docs :tada:

10 files changed, 350 insertions, 2 deletions

diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst
new file mode 100644
index 00000000..83374955
--- /dev/null
+++ b/docs/features/passthrough.rst
@@ -0,0 +1,94 @@
+.. _passthrough:
+
+Ignore Domains
+==============
+
+There are two main reasons why you may want to exempt some traffic from mitmproxy's interception mechanism:
+
+- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
+  mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
+  the Apple App Store fail to work if mitmproxy is active.
+- **Convenience:** You really don't care about some parts of the traffic and just want them to go away.
+
+If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy` feature.
+If you want to ignore traffic from mitmproxy's processing because of large response bodies,
+take a look at the :ref:`responsestreaming` feature.
+
+How it works
+------------
+
+================== =============================
+command-line       :option:`--ignore regex`
+mitmproxy shortcut :kbd:`o` then :kbd:`I`
+================== =============================
+
+
+mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
+(e.g. "example.com:443") to determine hosts that should be excluded.
+
+There are two important quirks to consider:
+
+- **In transparent mode, the ignore pattern is matched against the IP.** While we usually infer the
+  hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not
+  have access to this information before the SSL handshake.
+- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is
+  applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
+
+Tutorial
+--------
+
+If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
+
+1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port``
+   information in the serverconnect messages. mitmproxy will filter on these.
+2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
+   and use this as your ignore pattern:
+
+.. code-block:: none
+    :emphasize-lines: 6,7,9
+
+    >>> mitmdump -v
+    127.0.0.1:50588: clientconnect
+    127.0.0.1:50588: request
+      -> CONNECT example.com:443 HTTP/1.1
+    127.0.0.1:50588: Set new server address: example.com:443
+    127.0.0.1:50588: serverconnect
+      -> example.com:443
+    ^C
+    >>> mitmproxy --ignore ^example\.com:443$
+
+
+Here are some other examples for ignore patterns:
+
+.. code-block:: none
+
+    # Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
+    --ignore apple.com:443
+    # "Correct" version without false-positives:
+    --ignore '^(.+\.)?apple\.com:443$'
+
+    # Ignore example.com, but not its subdomains:
+    --ignore '^example.com:'
+
+    # Ignore everything but example.com and mitmproxy.org:
+    --ignore '^(?!example\.com)(?!mitmproxy\.org)'
+
+    # Transparent mode:
+    --ignore 17\.178\.96\.59:443
+    # IP address range:
+    --ignore 17\.178\.\d+\.\d+:443
+
+
+.. seealso::
+
+    - :ref:`tcpproxy`
+    - :ref:`responsestreaming`
+
+.. rubric:: Footnotes
+
+.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
+    A single connection can be re-used for multiple target domains - a
+    ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
+    same connection. If we start to ignore the connection after the first request,
+    we would miss the relevant second one.
+.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning
\ No newline at end of file
diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst
new file mode 100644
index 00000000..edc428a7
--- /dev/null
+++ b/docs/features/proxyauth.rst
@@ -0,0 +1,17 @@
+.. _proxyauth:
+
+Proxy Authentication
+====================
+
+
+Asks the user for authentication before they are permitted to use the proxy.
+Authentication headers are stripped from the flows, so they are not passed to
+upstream servers. For now, only HTTP Basic authentication is supported. The
+proxy auth options are not compatible with the transparent, socks or reverse proxy
+mode.
+
+================== =============================
+command-line       :option:`--nonanonymous`,
+                   :option:`--singleuser USER`,
+                   :option:`--htpasswd PATH`
+================== =============================
\ No newline at end of file
diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst
new file mode 100644
index 00000000..50fd0614
--- /dev/null
+++ b/docs/features/responsestreaming.rst
@@ -0,0 +1,68 @@
+.. _responsestreaming:
+
+Response Streaming
+==================
+
+By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
+before they have been fully received by the proxy. This is especially useful for large binary files
+such as videos, where buffering the whole file slows down the client's browser.
+
+By default, mitmproxy will read the entire response, perform any indicated
+manipulations on it and then send the (possibly modified) response to
+the client. In some cases this is undesirable and you may wish to "stream"
+the reponse back to the client. When streaming is enabled, the response is
+not buffered on the proxy but directly sent back to the client instead.
+
+On the command-line
+-------------------
+
+Streaming can be enabled on the command line for all response bodies exceeding a certain size.
+The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
+
+================== =============================
+command-line       :option:`--stream SIZE`
+================== =============================
+
+.. warning::
+
+    When response streaming is enabled, **streamed response contents will not be
+    recorded or preserved in any way.**
+
+.. note::
+
+    When response streaming is enabled, the response body cannot be modified by the usual means.
+
+Customizing Response Streaming
+------------------------------
+
+You can also use an :ref:`inlinescripts` to customize exactly
+which responses are streamed.
+
+Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``:
+
+.. literalinclude:: ../../examples/stream.py
+   :caption: examples/stream.py
+   :language: python
+
+Implementation Details
+----------------------
+
+When response streaming is enabled, portions of the code which would have otherwise performed changes
+on the response body will see an empty response body instead (:py:data:`netlib.http.CONTENT_MISSING`).
+Any modifications will be ignored.
+
+Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a
+``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
+
+Modifying streamed data
+-----------------------
+
+If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all chunks.
+
+.. literalinclude:: ../../examples/stream_modify.py
+   :caption: examples/stream_modify.py
+   :language: python
+
+.. seealso::
+
+    - :ref:`passthrough`
\ No newline at end of file
diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst
new file mode 100644
index 00000000..87a598ff
--- /dev/null
+++ b/docs/features/reverseproxy.rst
@@ -0,0 +1,56 @@
+.. _reverseproxy:
+
+Reverse Proxy
+=============
+
+In reverse proxy mode, mitmproxy accepts standard HTTP requests and forwards
+them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
+mitmproxy forwards HTTP proxy requests to an upstream proxy server.
+
+================== =====================================
+command-line       :option:`-R http[s]://hostname[:port]`
+================== =====================================
+
+Here, **scheme** signifies if the proxy should use TLS to connect to the server.
+mitmproxy always accepts both encrypted and unencrypted requests and transforms
+them to what the server expects.
+
+.. code-block:: none
+
+    >>> mitmdump -R https://httpbin.org -p 80
+    >>> curl http://localhost/
+    # requests will be transparently upgraded to TLS by mitmproxy
+
+    >>> mitmdump -R https://httpbin.org -p 443
+    >>> curl https://localhost/
+    # mitmproxy will use TLS on both ends.
+
+
+Host Header
+-----------
+
+In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
+may lead to issues with public web servers. For example, consider the following scenario:
+
+.. code-block:: none
+    :emphasize-lines: 5
+
+    >>> mitmdump -d -R http://example.com/
+    >>> curl http://localhost:8080/
+
+    >> GET https://example.com/
+        Host: localhost:8080
+        User-Agent: curl/7.35.0
+        [...]
+
+    << 404 Not Found 345B
+
+Since the Host header doesn't match "example.com", an error is returned.
+There are two ways to solve this:
+
+1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP.
+   Then, access example.com directly. Make sure that your proxy can still resolve the original IP
+   or specify an IP in mitmproxy.
+2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header: ``--setheader :~q:Host:example.com``.
+   However, keep in mind that absolute URLs within the returned document or HTTP redirects will
+   cause the client application to bypass the proxy.
diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst
index 0a6c2296..f118e6f8 100644
--- a/docs/features/setheaders.rst
+++ b/docs/features/setheaders.rst
@@ -7,7 +7,8 @@ This feature lets you specify a set of headers to be added to requests or
 responses, based on a filter pattern. You can specify these either on the
 command-line, or through an interactive editor in mitmproxy.
 
-Example:
+Example: Set the **Host** header to "example.com" for all requests.
+
 .. code-block:: none
 
     mitmdump -R http://example.com --setheader :~q:Host:example.com
diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst
new file mode 100644
index 00000000..fb9117f2
--- /dev/null
+++ b/docs/features/socksproxy.rst
@@ -0,0 +1,10 @@
+.. _socksproxy:
+
+SOCKS Mode
+==========
+
+In this mode, mitmproxy acts as a SOCKS5 proxy server.
+
+================== =================
+command-line       :option:`--socks`
+================== =================
\ No newline at end of file
diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst
new file mode 100644
index 00000000..e155fb9b
--- /dev/null
+++ b/docs/features/sticky.rst
@@ -0,0 +1,41 @@
+.. _sticky:
+
+Sticky cookies and auth
+=======================
+
+Sticky cookies
+--------------
+
+When the sticky cookie option is set, __mitmproxy__ will add the cookie most
+recently set by the server to any cookie-less request. Consider a service that
+sets a cookie to track the session after authentication. Using sticky cookies,
+you can fire up mitmproxy, and authenticate to a service as you usually would
+using a browser. After authentication, you can request authenticated resources
+through mitmproxy as if they were unauthenticated, because mitmproxy will
+automatically add the session tracking cookie to requests. Among other things,
+this lets you script interactions with authenticated resources (using tools
+like wget or curl) without having to worry about authentication.
+
+Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
+record the authentication process once, and simply replay it on startup every time you need
+to interact with the secured resources.
+
+================== ======================
+command-line       :option:`-t FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`t`
+================== ======================
+
+
+Sticky auth
+-----------
+
+The sticky auth option is analogous to the sticky cookie option, in that HTTP
+**Authorization** headers are simply replayed to the server once they have been
+seen. This is enough to allow you to access a server resource using HTTP Basic
+authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
+replay of HTTP Digest authentication.
+
+================== ======================
+command-line       :option:`-u FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`A`
+================== ======================
\ No newline at end of file
diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst
new file mode 100644
index 00000000..53df8ed6
--- /dev/null
+++ b/docs/features/tcpproxy.rst
@@ -0,0 +1,30 @@
+.. _tcpproxy:
+
+TCP Proxy
+=========
+
+WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt
+hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
+This feature is closely related to the :ref:`passthrough` functionality,
+but differs in two important aspects:
+
+- The raw TCP messages are printed to the event log.
+- SSL connections will be intercepted.
+
+Please note that message interception or modification are not possible yet.
+If you are not interested in the raw TCP messages, you should use the ignore domains feature.
+
+How it works
+------------
+
+================== ======================
+command-line       :option:`--tcp HOST`
+mitmproxy shortcut :kbd:`o` then :kbd:`T`
+================== ======================
+
+For a detailed description how the hostname pattern works, please look at the :ref:`passthrough` feature.
+
+.. seealso::
+
+    - :ref:`passthrough`
+    - :ref:`responsestreaming`
diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst
index a287daef..84cfb84e 100644
--- a/docs/features/upstreamcerts.rst
+++ b/docs/features/upstreamcerts.rst
@@ -1,4 +1,23 @@
 .. _upstreamcerts:
 
 Upstream Certificates
-=====================
\ No newline at end of file
+=====================
+
+When mitmproxy receives a connection destined for an SSL-protected service, it
+freezes the connection before reading its request data, and makes a connection
+to the upstream server to "sniff" the contents of its SSL certificate. The
+information gained - the **Common Name** and **Subject Alternative Names** - is
+then used to generate the interception certificate, which is sent to the client
+so the connection can continue.
+
+This rather intricate little dance lets us seamlessly generate correct
+certificates even if the client has specifed only an IP address rather than the
+hostname. It also means that we don't need to sniff additional data to generate
+certs in transparent mode.
+
+Upstream cert sniffing is on by default, and can optionally be turned off.
+
+================== =============================
+command-line       :option:`--no-upstream-cert`
+mitmproxy shortcut :kbd:`o` then :kbd:`U`
+================== =============================
\ No newline at end of file
diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst
new file mode 100644
index 00000000..e06833c2
--- /dev/null
+++ b/docs/features/upstreamproxy.rst
@@ -0,0 +1,12 @@
+.. _upstreamproxy:
+
+Upstream proxy mode
+===================
+
+In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
+requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
+in which mitmproxy forwards ordinary HTTP requests to an upstream server.
+
+================== ===================================
+command-line       :option:`-U http://hostname[:port]`
+================== ===================================