Merge pull request #2890 from mitmproxy/newdocs

All new documentation

1 files changed, 252 insertions, 0 deletions

diff --git a/docs/src/content/concepts-howmitmproxyworks.md b/docs/src/content/concepts-howmitmproxyworks.md
new file mode 100644
index 00000000..d747d71c
--- /dev/null
+++ b/docs/src/content/concepts-howmitmproxyworks.md
@@ -0,0 +1,252 @@
+---
+title: "How mitmproxy works"
+menu:
+    concepts:
+        weight: 1
+---
+
+# How mitmproxy works
+
+Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying
+process works will help you deploy it creatively, and take into account its
+fundamental assumptions and how to work around them. This document explains
+mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted
+explicit proxying, and working up to the most complicated interaction
+-transparent proxying of TLS-protected traffic[^1] in the presence of [Server
+Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication).
+
+## Explicit HTTP
+
+Configuring the client to use mitmproxy as an explicit proxy is the
+simplest and most reliable way to intercept traffic. The proxy protocol
+is codified in the [HTTP RFC](https://tools.ietf.org/html/rfc7230), so
+the behaviour of both the client and the server is well defined, and
+usually reliable. In the simplest possible interaction with mitmproxy, a
+client connects directly to the proxy, and makes a request that looks
+like this:
+
+{{< highlight http  >}}
+GET http://example.com/index.html HTTP/1.1
+{{< / highlight >}}
+
+This is a proxy GET request - an extended form of the vanilla HTTP GET
+request that includes a schema and host specification, and it includes
+all the information mitmproxy needs to proceed.
+
+
+{{< figure src="/schematics/how-mitmproxy-works-explicit.png" title="Explicit" >}}
+
+
+1. The client connects to the proxy and makes a request.
+2. Mitmproxy connects to the upstream server and simply forwards the request on.
+
+
+## Explicit HTTPS
+
+The process for an explicitly proxied HTTPS connection is quite
+different. The client connects to the proxy and makes a request that
+looks like this:
+
+{{< highlight http  >}}
+CONNECT example.com:443 HTTP/1.1
+{{< / highlight >}}
+
+A conventional proxy can neither view nor manipulate a TLS-encrypted
+data stream, so a CONNECT request simply asks the proxy to open a pipe
+between the client and server. The proxy here is just a facilitator - it
+blindly forwards data in both directions without knowing anything about
+the contents. The negotiation of the TLS connection happens over this
+pipe, and the subsequent flow of requests and responses are completely
+opaque to the proxy.
+
+### The MITM in mitmproxy
+
+This is where mitmproxy's fundamental trick comes into play. The MITM
+in its name stands for Man-In-The-Middle - a reference to the process we
+use to intercept and interfere with these theoretically opaque data
+streams. The basic idea is to pretend to be the server to the client,
+and pretend to be the client to the server, while we sit in the middle
+decoding traffic from both sides. The tricky part is that the
+[Certificate
+Authority](https://en.wikipedia.org/wiki/Certificate_authority) system
+is designed to prevent exactly this attack, by allowing a trusted
+third-party to cryptographically sign a server's certificates to verify
+that they are legit. If this signature doesn't match or is from a
+non-trusted party, a secure client will simply drop the connection and
+refuse to proceed. Despite the many shortcomings of the CA system as it
+exists today, this is usually fatal to attempts to MITM a TLS connection
+for analysis. Our answer to this conundrum is to become a trusted
+Certificate Authority ourselves. Mitmproxy includes a full CA
+implementation that generates interception certificates on the fly. To
+get the client to trust these certificates, we
+[register mitmproxy as a trusted CA with the device
+manually]({{< relref concepts-certificates >}}).
+
+### Complication 1: What's the remote hostname?
+
+To proceed with this plan, we need to know the domain name to use in the
+interception certificate - the client will verify that the certificate is for
+the domain it's connecting to, and abort if this is not the case. At first
+blush, it seems that the CONNECT request above gives us all we need - in this
+example, both of these values are "example.com". But what if the client had
+initiated the connection as follows:
+
+{{< highlight http  >}}
+CONNECT 10.1.1.1:443 HTTP/1.1
+{{< / highlight >}}
+
+Using the IP address is perfectly legitimate because it gives us enough
+information to initiate the pipe, even though it doesn't reveal the
+remote hostname.
+
+Mitmproxy has a cunning mechanism that smooths this over - [upstream certificate
+sniffing]({{< relref "overview-features#upstream-certificates" >}}). As soon as
+we see the CONNECT request, we pause the client part of the conversation, and
+initiate a simultaneous connection to the server. We complete the TLS handshake
+with the server, and inspect the certificates it used. Now, we use the Common
+Name in the upstream certificates to generate the dummy certificate for the
+client. Voila, we have the correct hostname to present to the client, even if it
+was never specified.
+
+### Complication 2: Subject Alternative Name
+
+Enter the next complication. Sometimes, the certificate Common Name is
+not, in fact, the hostname that the client is connecting to. This is
+because of the optional [Subject Alternative
+Name](https://en.wikipedia.org/wiki/SubjectAltName) field in the
+certificate that allows an arbitrary number of alternative domains to be
+specified. If the expected domain matches any of these, the client will
+proceed, even though the domain doesn't match the certificate CN. The
+answer here is simple: when we extract the CN from the upstream cert, we
+also extract the SANs, and add them to the generated dummy certificate.
+
+### Complication 3: Server Name Indication
+
+One of the big limitations of vanilla TLS is that each certificate
+requires its own IP address. This means that you couldn't do virtual
+hosting where multiple domains with independent certificates share the
+same IP address. In a world with a rapidly shrinking IPv4 address pool
+this is a problem, and we have a solution in the form of the [Server
+Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication)
+extension to the TLS protocols. This lets the client specify the remote
+server name at the start of the TLS handshake, which then lets the
+server select the right certificate to complete the process.
+
+SNI breaks our upstream certificate sniffing process, because when we
+connect without using SNI, we get served a default certificate that may
+have nothing to do with the certificate expected by the client. The
+solution is another tricky complication to the client connection
+process. After the client connects, we allow the TLS handshake to
+continue until just **after** the SNI value has been passed to us. Now
+we can pause the conversation, and initiate an upstream connection using
+the correct SNI value, which then serves us the correct upstream
+certificate, from which we can extract the expected CN and SANs.
+
+### Putting it all together
+
+Lets put all of this together into the complete explicitly proxied HTTPS
+flow.
+
+{{< figure src="/schematics/how-mitmproxy-works-explicit-https.png" title="Explicit HTTPS" >}}
+
+1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT
+    request.
+2. Mitmproxy responds with a `200 Connection Established`, as if it has set up
+    the CONNECT pipe.
+3. The client believes it's talking to the remote server, and initiates the TLS
+    connection. It uses SNI to indicate the hostname it is connecting to.
+4. Mitmproxy connects to the server, and establishes a TLS connection using the
+    SNI hostname indicated by the client.
+5. The server responds with the matching certificate, which contains the CN and
+    SAN values needed to generate the interception certificate.
+6. Mitmproxy generates the interception cert, and continues the client TLS
+    handshake paused in step 3.
+7. The client sends the request over the established TLS connection.
+8. Mitmproxy passes the request on to the server over the TLS connection
+    initiated in step 4.
+
+## Transparent HTTP
+
+When a transparent proxy is used, the connection is redirected into a
+proxy at the network layer, without any client configuration being
+required. This makes transparent proxying ideal for those situations
+where you can't change client behaviour - proxy-oblivious Android
+applications being a common example.
+
+To achieve this, we need to introduce two extra components. The first is
+a redirection mechanism that transparently reroutes a TCP connection
+destined for a server on the Internet to a listening proxy server. This
+usually takes the form of a firewall on the same host as the proxy
+server - [iptables](http://www.netfilter.org/) on Linux or
+[pf](https://en.wikipedia.org/wiki/PF_\(firewall\)) on OSX. Once the
+client has initiated the connection, it makes a vanilla HTTP request,
+which might look something like this:
+
+{{< highlight http  >}}
+GET /index.html HTTP/1.1
+{{< / highlight >}}
+
+Note that this request differs from the explicit proxy variation, in
+that it omits the scheme and hostname. How, then, do we know which
+upstream host to forward the request to? The routing mechanism that has
+performed the redirection keeps track of the original destination for
+us. Each routing mechanism has a different way of exposing this data, so
+this introduces the second component required for working transparent
+proxying: a host module that knows how to retrieve the original
+destination address from the router. In mitmproxy, this takes the form
+of a built-in set of
+[modules](https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/platform)
+that know how to talk to each platform's redirection mechanism. Once we
+have this information, the process is fairly straight-forward.
+
+{{< figure src="/schematics/how-mitmproxy-works-transparent.png" title="Transparent" >}}
+
+1. The client makes a connection to the server.
+2. The router redirects the connection to mitmproxy, which is typically
+    listening on a local port of the same host. Mitmproxy then consults
+    the routing mechanism to establish what the original destination
+    was.
+3. Now, we simply read the client's request...
+4. ... and forward it upstream.
+
+## Transparent HTTPS
+
+The first step is to determine whether we should treat an incoming
+connection as HTTPS. The mechanism for doing this is simple - we use the
+routing mechanism to find out what the original destination port is. All
+incoming connections pass through different layers which can determine
+the actual protocol to use. Automatic TLS detection works for SSLv3, TLS
+1.0, TLS 1.1, and TLS 1.2 by looking for a *ClientHello* message at the
+beginning of each connection. This works independently of the used TCP
+port.
+
+From here, the process is a merger of the methods we've described for
+transparently proxying HTTP, and explicitly proxying HTTPS. We use the
+routing mechanism to establish the upstream server address, and then
+proceed as for explicit HTTPS connections to establish the CN and SANs,
+and cope with SNI.
+
+{{< figure src="/schematics/how-mitmproxy-works-transparent-https.png" title="Transparent HTTPS" >}}
+
+1. The client makes a connection to the server.
+2. The router redirects the connection to mitmproxy, which is typically
+    listening on a local port of the same host. Mitmproxy then consults the
+    routing mechanism to establish what the original destination was.
+3. The client believes it's talking to the remote server, and
+    initiates the TLS connection. It uses SNI to indicate the hostname
+    it is connecting to.
+4. Mitmproxy connects to the server, and establishes a TLS connection
+    using the SNI hostname indicated by the client.
+5. The server responds with the matching certificate, which contains
+    the CN and SAN values needed to generate the interception
+    certificate.
+6. Mitmproxy generates the interception cert, and continues the client
+    TLS handshake paused in step 3.
+7. The client sends the request over the established TLS connection.
+8. Mitmproxy passes the request on to the server over the TLS
+    connection initiated in step 4.
+
+### Footnotes
+
+[^1]: The use of "TLS" refers to both SSL (outdated and insecure) and TLS (1.0
+    and up) in the generic sense, unless otherwise specified.