Expand "How mitmproxy works". Clean up some un-needed sections.

1 files changed, 57 insertions, 40 deletions

diff --git a/doc-src/howmitmproxy.html b/doc-src/howmitmproxy.html
index 94c895d7..a3335094 100644
--- a/doc-src/howmitmproxy.html
+++ b/doc-src/howmitmproxy.html
@@ -1,15 +1,11 @@
 
-TODO:
-
-- Clarify terminology: SSL vs TLS
-
-
 Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
-process works will help you deploy it more creatively, and let you understand
+process works will help you deploy it creatively, and allow you to understand
 its fundamental assumptions and how to work around them. This document explains
-mitmproxy's proxy mechanism by example, starting with the simplest explicit
-proxy configuration, and working up to the most complicated interaction -
-transparent proxying of SSL-protected traffic in the presence of SNI.
+mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
+explicit proxying, and working up to the most complicated interaction -
+transparent proxying of SSL-protected traffic[^ssl] in the presence of
+[SNI](http://en.wikipedia.org/wiki/Server_Name_Indication).
 
 
 <div class="page-header">
@@ -75,9 +71,11 @@ This is where mitmproxy's fundamental trick comes into play. The MITM in its
 name stands for Man-In-The-Middle - a reference to the process we use to
 intercept and interfere with these theoretially opaque data streams. The basic
 idea is to pretend to be the server to the client, and pretend to be the client
-to the server. The tricky part is that the Certificate Authority system is
+to the server, while we sit in the middle decoding traffic from both sides. The
+tricky part is that the [Certificate
+Authority](http://en.wikipedia.org/wiki/Certificate_authority) system is
 designed to prevent exactly this attack, by allowing a trusted third-party to
-cryptographically sign a server's SSL certificates to verify that the certs are
+cryptographically sign a server's SSL certificates to verify that they are
 legit. If this signature is from a non-trusted party, a secure client will
 simply drop the connection and refuse to proceed. Despite the many shortcomings
 of the CA system as it exists today, this is usually fatal to attempts to MITM
@@ -86,7 +84,8 @@ an SSL connection for analysis.
 Our answer to this conundrum is to become a trusted Certificate Authority
 ourselves. Mitmproxy includes a full CA implementation that generates
 interception certificates on the fly. To get the client to trust these
-certificates, we register mitmproxy as a CA with the device manually.
+certificates, we [register mitmproxy as a trusted CA with the device
+manually](@!urlTo("ssl.html")!@).
 
 ## Complication 1: What's the remote hostname? 
 
@@ -103,25 +102,27 @@ Using the IP address is perfectly legitimate because it gives us enough
 information to initiate the pipe, even though it doesn't reveal the remote
 hostname. 
 
-Mitmproxy has a cunning mechanism that smooths this over - upstream certificate
-sniffing. As soon as we see the CONNECT request, we pause the client part of
-the conversation, and initiate a simultaneous connection to the server. We
-complete the SSL handshake with the server, and inspect the certificates it
-used. Now, we use the Common Name in the upstream SSL certificates to generate
-the dummy certificate for the client. Voila, we have the correct hostname to
-present to the client, even if it was never specified.
+Mitmproxy has a cunning mechanism that smooths this over - [upstream
+certificate sniffing](@!urlTo("features/upstreamcerts.html")!@). As soon as we
+see the CONNECT request, we pause the client part of the conversation, and
+initiate a simultaneous connection to the server. We complete the SSL handshake
+with the server, and inspect the certificates it used. Now, we use the Common
+Name in the upstream SSL certificates to generate the dummy certificate for the
+client. Voila, we have the correct hostname to present to the client, even if
+it was never specified.
 
 
-## Complication 2: Subject Alternate Name
+## Complication 2: Subject Alternative Name
 
 Enter the next complication. Sometimes, the certificate Common Name is not, in
 fact, the hostname that the client is connecting to. This is because of the
-optional Subject Alternate Name field in the SSL certificate that allows an
-arbitrary number of alternate domains to be specified. If the expected domain
-matches any of these, the client wil proceed, even though the domain doesn't
-match the certificate Common Name. The answer here is simple: when extract the
-CN from the upstream cert, we also extract the SANs, and add them to the
-generated dummy certificate.
+optional [Subject Alternative
+Name](http://en.wikipedia.org/wiki/SubjectAltName) field in the SSL certificate
+that allows an arbitrary number of alternative domains to be specified. If the
+expected domain matches any of these, the client wil proceed, even though the
+domain doesn't match the certificate Common Name. The answer here is simple:
+when extract the CN from the upstream cert, we also extract the SANs, and add
+them to the generated dummy certificate.
 
 
 ## Complication 3: Server Name Indication
@@ -130,9 +131,10 @@ One of the big limitations of conventional SSL is that each certificate
 requires its own IP address. This means that you couldn't do virtual hosting
 where multiple domains with independent certificates share the same IP address.
 In a world with a rapidly shrinking IPv4 address pool this is a problem, and we
-have a solution in the form of the Server Name Indication extension to the SSL
-and TLS protocols. This lets the client specify the remote server name at the
-start of the SSL handshake, which then lets the server select the right
+have a solution in the form of the [Server Name
+Indication](http://en.wikipedia.org/wiki/Server_Name_Indication) extension to
+the SSL and TLS protocols. This lets the client specify the remote server name
+at the start of the SSL handshake, which then lets the server select the right
 certificate to complete the process.
 
 SNI breaks our upstream certificate sniffing process, because when we connect
@@ -144,6 +146,15 @@ passed to us. Now we can pause the conversation, and initiate an upstream
 connection using the correct SNI value, which then serves us the correct
 upstream certificate, from which we can extract the expected CN and SANs.
 
+There's another wrinkle here. Due to a limitation of the SSL library mitmproxy
+uses, we can't detect that a connection _hasn't_ sent an SNI request until it's
+too late for upstream certificate sniffing. In practice, we therefore make a
+vanilla SSL connection upstream to sniff non-SNI certificates, and then discard
+the connection if the client sends an SNI notification. If you're watching your
+traffic with a packet sniffer, you'll see two connections to the server when an
+SNI request is made, the first of which is immediately closed after the SSL
+handshake. Luckily, this is almost never an issue in practice.
+
 
 ## Putting it all together
 
@@ -218,22 +229,28 @@ This makes transparent proxying ideal for those situations where you can't
 change client behaviour - proxy-oblivious Android applications being a common
 example.
 
-To achieve this, we need to introduce two extra components. The first new
-component is a router that transparently redirects the TCP connection to the
-proxy. Once the client has initiated the connection, it makes a vanilla HTTP
-request, which might look something like this:
+To achieve this, we need to introduce two extra components. The first is a
+redirection mechanism that transparently reroutes a TCP connection destined for
+a server on the Internet to a listening proxy server. This usually takes the
+form of a firewall on the same host as the proxy server -
+[iptables](http://www.netfilter.org/) on Linux or
+[pf](http://en.wikipedia.org/wiki/PF_(firewall)) on OSX. Once the client has
+initiated the connection, it makes a vanilla HTTP request, which might look
+something like this:
 
 <pre>GET /index.html HTTP/1.1</pre> 
 
 Note that this request differs from the explicit proxy variation, in that it
 omits the scheme and hostname. How, then, do we know which upstream host to
 forward the request to? The routing mechanism that has performed the
-redirection keeps track of the original destination. Each different routing
-mechanism has its own ideosyncratic way of exposing this data, so this
-introduces the second component required for working transparent proxying: a
-host module that knows how to retrieve the original destination address from
-the router. Once we have this information, the process is fairly
-straight-forward.
+redirection keeps track of the original destination for us.  Each routing
+mechanism has a different way of exposing this data, so this introduces the
+second component required for working transparent proxying: a host module that
+knows how to retrieve the original destination address from the router. In
+mitmproxy, this takes the form of a built-in set of
+[modules](https://github.com/cortesi/mitmproxy/tree/master/libmproxy/platform)
+that know how to talk to each platform's redirection mechanism.  Once we have
+this information, the process is fairly straight-forward.
 
 <img src="transparent.png"/>
 
@@ -338,4 +355,4 @@ and cope with SNI.
 </table>
 
 
-
+[^ssl]: I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified.