Docs and examples to top level

15 files changed, 559 insertions, 0 deletions

diff --git a/docs/features/anticache.rst b/docs/features/anticache.rst
new file mode 100644
index 00000000..65d22bab
--- /dev/null
+++ b/docs/features/anticache.rst
@@ -0,0 +1,15 @@
+.. _anticache:
+
+Anticache
+=========
+When the :option:`--anticache` option is passed to mitmproxy, it removes headers
+(``if-none-match`` and ``if-modified-since``) that might elicit a
+``304 not modified`` response from the server. This is useful when you want to make
+sure you capture an HTTP exchange in its totality. It's also often used during
+:ref:`clientreplay`, when you want to make sure the server responds with complete data.
+
+
+================== ======================
+command-line       :option:`--anticache`
+mitmproxy shortcut :kbd:`o` then :kbd:`a`
+================== ======================
diff --git a/docs/features/clientreplay.rst b/docs/features/clientreplay.rst
new file mode 100644
index 00000000..b0eb6792
--- /dev/null
+++ b/docs/features/clientreplay.rst
@@ -0,0 +1,18 @@
+.. _clientreplay:
+
+Client-side replay
+==================
+
+Client-side replay does what it says on the tin: you provide a previously saved
+HTTP conversation, and mitmproxy replays the client requests one by one. Note
+that mitmproxy serializes the requests, waiting for a response from the server
+before starting the next request. This might differ from the recorded
+conversation, where requests may have been made concurrently.
+
+You may want to use client-side replay in conjunction with the
+:ref:`anticache` option, to make sure the server responds with complete data.
+
+================== =================
+command-line       :option:`-c path`
+mitmproxy shortcut :kbd:`c`
+================== =================
diff --git a/docs/features/filters.rst b/docs/features/filters.rst
new file mode 100644
index 00000000..509b5d6b
--- /dev/null
+++ b/docs/features/filters.rst
@@ -0,0 +1,39 @@
+.. _filters:
+
+Filter expressions
+==================
+
+Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression.
+Filter expressions consist of the following operators:
+
+.. documentedlist::
+    :header: "Expression" "Description"
+    :listobject: mitmproxy.filt.help
+
+- Regexes are Python-style
+- Regexes can be specified as quoted strings
+- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value".
+- Strings with no operators are matched against the request URL.
+- The default binary operator is &.
+
+Examples
+--------
+
+URL containing "google.com":
+
+.. code-block:: none
+
+    google\.com
+
+Requests whose body contains the string "test":
+
+.. code-block:: none
+
+    ~q ~b test
+
+Anything but requests with a text/html content type:
+
+.. code-block:: none
+
+    !(~q & ~t "text/html")
+
diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst
new file mode 100644
index 00000000..b7b5df84
--- /dev/null
+++ b/docs/features/passthrough.rst
@@ -0,0 +1,97 @@
+.. _passthrough:
+
+Ignore Domains
+==============
+
+There are two main reasons why you may want to exempt some traffic from mitmproxy's interception
+mechanism:
+
+- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and
+  mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or
+  the Apple App Store fail to work if mitmproxy is active.
+- **Convenience:** You really don't care about some parts of the traffic and just want them to go
+  away.
+
+If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy`
+feature.
+If you want to ignore traffic from mitmproxy's processing because of large response bodies,
+take a look at the :ref:`responsestreaming` feature.
+
+How it works
+------------
+
+================== =============================
+command-line       :option:`--ignore regex`
+mitmproxy shortcut :kbd:`o` then :kbd:`I`
+================== =============================
+
+
+mitmproxy allows you to specify a regex which is matched against a ``host:port`` string
+(e.g. "example.com:443") to determine hosts that should be excluded.
+
+There are two important quirks to consider:
+
+- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the
+  hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not
+  have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target.
+- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is
+  applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections.
+
+Tutorial
+--------
+
+If you just want to ignore one specific domain, there's usually a bulletproof method to do so:
+
+1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port``
+   information in the serverconnect messages. mitmproxy will filter on these.
+2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.)
+   and use this as your ignore pattern:
+
+.. code-block:: none
+    :emphasize-lines: 6,7,9
+
+    >>> mitmdump -v
+    127.0.0.1:50588: clientconnect
+    127.0.0.1:50588: request
+      -> CONNECT example.com:443 HTTP/1.1
+    127.0.0.1:50588: Set new server address: example.com:443
+    127.0.0.1:50588: serverconnect
+      -> example.com:443
+    ^C
+    >>> mitmproxy --ignore ^example\.com:443$
+
+
+Here are some other examples for ignore patterns:
+
+.. code-block:: none
+
+    # Exempt traffic from the iOS App Store (the regex is lax, but usually just works):
+    --ignore apple.com:443
+    # "Correct" version without false-positives:
+    --ignore '^(.+\.)?apple\.com:443$'
+
+    # Ignore example.com, but not its subdomains:
+    --ignore '^example.com:'
+
+    # Ignore everything but example.com and mitmproxy.org:
+    --ignore '^(?!example\.com)(?!mitmproxy\.org)'
+
+    # Transparent mode:
+    --ignore 17\.178\.96\.59:443
+    # IP address range:
+    --ignore 17\.178\.\d+\.\d+:443
+
+
+.. seealso::
+
+    - :ref:`tcpproxy`
+    - :ref:`responsestreaming`
+
+.. rubric:: Footnotes
+
+.. [#explicithttp] This stems from an limitation of explicit HTTP proxying:
+    A single connection can be re-used for multiple target domains - a
+    ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the
+    same connection. If we start to ignore the connection after the first request,
+    we would miss the relevant second one.
+.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning
diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst
new file mode 100644
index 00000000..bfd32fbd
--- /dev/null
+++ b/docs/features/proxyauth.rst
@@ -0,0 +1,17 @@
+.. _proxyauth:
+
+Proxy Authentication
+====================
+
+
+Asks the user for authentication before they are permitted to use the proxy.
+Authentication headers are stripped from the flows, so they are not passed to
+upstream servers. For now, only HTTP Basic authentication is supported. The
+proxy auth options are not compatible with the transparent, socks or reverse proxy
+mode.
+
+================== =============================
+command-line       :option:`--nonanonymous`,
+                   :option:`--singleuser USER`,
+                   :option:`--htpasswd PATH`
+================== =============================
diff --git a/docs/features/replacements.rst b/docs/features/replacements.rst
new file mode 100644
index 00000000..8f760866
--- /dev/null
+++ b/docs/features/replacements.rst
@@ -0,0 +1,72 @@
+.. _replacements:
+
+Replacements
+============
+
+Mitmproxy lets you specify an arbitrary number of patterns that define text
+replacements within flows. Each pattern has 3 components: a filter that defines
+which flows a replacement applies to, a regular expression that defines what
+gets replaced, and a target value that defines what is substituted in.
+
+Replace hooks fire when either a client request or a server response is
+received. Only the matching flow component is affected: so, for example, if a
+replace hook is triggered on server response, the replacement is only run on
+the Response object leaving the Request intact. You control whether the hook
+triggers on the request, response or both using the filter pattern. If you need
+finer-grained control than this, it's simple to create a script using the
+replacement API on Flow components.
+
+Replacement hooks are extremely handy in interactive testing of applications.
+For instance you can use a replace hook to replace the text "XSS" with a
+complicated XSS exploit, and then "inject" the exploit simply by interacting
+with the application through the browser. When used with tools like Firebug and
+mitmproxy's own interception abilities, replacement hooks can be an amazingly
+flexible and powerful feature.
+
+
+On the command-line
+-------------------
+
+The replacement hook command-line options use a compact syntax to make it easy
+to specify all three components at once. The general form is as follows:
+
+.. code-block:: none
+
+    /patt/regex/replacement
+
+Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python
+regular expression, and **replacement** is a string literal. The first
+character in the expression (``/`` in this case) defines what the separation
+character is. Here's an example of a valid expression that replaces "foo" with
+"bar" in all requests:
+
+.. code-block:: none
+
+    :~q:foo:bar
+
+In practice, it's pretty common for the replacement literal to be long and
+complex. For instance, it might be an XSS exploit that weighs in at hundreds or
+thousands of characters. To cope with this, there's a variation of the
+replacement hook specifier that lets you load the replacement text from a file.
+So, you might start **mitmdump** as follows:
+
+>>> mitmdump --replace-from-file :~q:foo:~/xss-exploit
+
+This will load the replacement text from the file ``~/xss-exploit``.
+
+Both the :option:`--replace` and :option:`--replace-from-file` flags can be passed multiple
+times.
+
+
+Interactively
+-------------
+
+The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`o`) lets you add and edit
+replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has
+complete usage information.
+
+================== =============================
+command-line       :option:`--replace`,
+                   :option:`--replace-from-file`
+mitmproxy shortcut :kbd:`o` then :kbd:`R`
+================== =============================
diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst
new file mode 100644
index 00000000..8975c1f8
--- /dev/null
+++ b/docs/features/responsestreaming.rst
@@ -0,0 +1,69 @@
+.. _responsestreaming:
+
+Response Streaming
+==================
+
+By using mitmproxy's streaming feature, response contents can be passed to the client incrementally
+before they have been fully received by the proxy. This is especially useful for large binary files
+such as videos, where buffering the whole file slows down the client's browser.
+
+By default, mitmproxy will read the entire response, perform any indicated
+manipulations on it and then send the (possibly modified) response to
+the client. In some cases this is undesirable and you may wish to "stream"
+the response back to the client. When streaming is enabled, the response is
+not buffered on the proxy but directly sent back to the client instead.
+
+On the command-line
+-------------------
+
+Streaming can be enabled on the command line for all response bodies exceeding a certain size.
+The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes.
+
+================== =============================
+command-line       :option:`--stream SIZE`
+================== =============================
+
+.. warning::
+
+    When response streaming is enabled, **streamed response contents will not be
+    recorded or preserved in any way.**
+
+.. note::
+
+    When response streaming is enabled, the response body cannot be modified by the usual means.
+
+Customizing Response Streaming
+------------------------------
+
+You can also use an :ref:`inlinescripts` to customize exactly
+which responses are streamed.
+
+Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``:
+
+.. literalinclude:: ../../examples/stream.py
+   :caption: examples/stream.py
+   :language: python
+
+Implementation Details
+----------------------
+
+When response streaming is enabled, portions of the code which would have otherwise performed
+changes on the response body will see an empty response body instead
+(:py:data:`netlib.http.CONTENT_MISSING`). Any modifications will be ignored.
+
+Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a
+``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time.
+
+Modifying streamed data
+-----------------------
+
+If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all
+chunks.
+
+.. literalinclude:: ../../examples/stream_modify.py
+   :caption: examples/stream_modify.py
+   :language: python
+
+.. seealso::
+
+    - :ref:`passthrough`
diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst
new file mode 100644
index 00000000..87065e73
--- /dev/null
+++ b/docs/features/reverseproxy.rst
@@ -0,0 +1,57 @@
+.. _reverseproxy:
+
+Reverse Proxy
+=============
+
+In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards
+them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which
+mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server.
+
+================== =====================================
+command-line       :option:`-R http[s]://hostname[:port]`
+================== =====================================
+
+Here, **http[s]** signifies if the proxy should use TLS to connect to the server.
+mitmproxy always accepts both encrypted and unencrypted requests and transforms
+them to what the server expects.
+
+.. code-block:: none
+
+    >>> mitmdump -R https://httpbin.org -p 80
+    >>> curl http://localhost/
+    # requests will be transparently upgraded to TLS by mitmproxy
+
+    >>> mitmdump -R https://httpbin.org -p 443
+    >>> curl https://localhost/
+    # mitmproxy will use TLS on both ends.
+
+
+Host Header
+-----------
+
+In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this
+may lead to issues with public web servers. For example, consider the following scenario:
+
+.. code-block:: none
+    :emphasize-lines: 5
+
+    >>> mitmdump -d -R http://example.com/
+    >>> curl http://localhost:8080/
+
+    >> GET https://example.com/
+        Host: localhost:8080
+        User-Agent: curl/7.35.0
+        [...]
+
+    << 404 Not Found 345B
+
+Since the Host header doesn't match "example.com", an error is returned.
+There are two ways to solve this:
+
+1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP.
+   Then, access example.com directly. Make sure that your proxy can still resolve the original IP
+   or specify an IP in mitmproxy.
+2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header:
+   ``--setheader :~q:Host:example.com``.
+   However, keep in mind that absolute URLs within the returned document or HTTP redirects will
+   cause the client application to bypass the proxy.
diff --git a/docs/features/serverreplay.rst b/docs/features/serverreplay.rst
new file mode 100644
index 00000000..261a1bd6
--- /dev/null
+++ b/docs/features/serverreplay.rst
@@ -0,0 +1,39 @@
+.. _serverreplay:
+
+Server-side replay
+==================
+
+Server-side replay lets us replay server responses from a saved HTTP
+conversation.
+
+Matching requests with responses
+--------------------------------
+
+By default, :program:`mitmproxy` excludes request headers when matching incoming
+requests with responses from the replay file. This works in most circumstances,
+and makes it possible to replay server responses in situations where request
+headers would naturally vary, e.g. using a different user agent.
+The :option:`--rheader headername` command-line option allows you to override
+this behaviour by specifying individual headers that should be included in matching.
+
+
+Response refreshing
+-------------------
+
+Simply replaying server responses without modification will often result in
+unexpected behaviour. For example cookie timeouts that were in the future at
+the time a conversation was recorded might be in the past at the time it is
+replayed. By default, :program:`mitmproxy` refreshes server responses before sending
+them to the client. The **date**, **expires** and **last-modified** headers are
+all updated to have the same relative time offset as they had at the time of
+recording. So, if they were in the past at the time of recording, they will be
+in the past at the time of replay, and vice versa. Cookie expiry times are
+updated in a similar way.
+
+You can turn off response refreshing using the :option:`--norefresh` argument, or using
+the :kbd:`o` options shortcut within :program:`mitmproxy`.
+
+================== =================
+command-line       :option:`-S path`
+mitmproxy shortcut :kbd:`S`
+================== =================
diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst
new file mode 100644
index 00000000..cbc8b6a5
--- /dev/null
+++ b/docs/features/setheaders.rst
@@ -0,0 +1,19 @@
+.. _setheaders:
+
+Set Headers
+===========
+
+This feature lets you specify a set of headers to be added to requests or
+responses, based on a filter pattern. You can specify these either on the
+command-line, or through an interactive editor in mitmproxy.
+
+Example: Set the **Host** header to "example.com" for all requests.
+
+.. code-block:: none
+
+    mitmdump -R http://example.com --setheader :~q:Host:example.com
+
+================== =============================
+command-line       :option:`--setheader PATTERN`
+mitmproxy shortcut :kbd:`o` then :kbd:`H`
+================== =============================
diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst
new file mode 100644
index 00000000..76d4cda9
--- /dev/null
+++ b/docs/features/socksproxy.rst
@@ -0,0 +1,10 @@
+.. _socksproxy:
+
+SOCKS Mode
+==========
+
+In this mode, mitmproxy acts as a SOCKS5 proxy server.
+
+================== =================
+command-line       :option:`--socks`
+================== =================
diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst
new file mode 100644
index 00000000..a79cbe8d
--- /dev/null
+++ b/docs/features/sticky.rst
@@ -0,0 +1,41 @@
+.. _sticky:
+
+Sticky cookies and auth
+=======================
+
+Sticky cookies
+--------------
+
+When the sticky cookie option is set, __mitmproxy__ will add the cookie most
+recently set by the server to any cookie-less request. Consider a service that
+sets a cookie to track the session after authentication. Using sticky cookies,
+you can fire up mitmproxy, and authenticate to a service as you usually would
+using a browser. After authentication, you can request authenticated resources
+through mitmproxy as if they were unauthenticated, because mitmproxy will
+automatically add the session tracking cookie to requests. Among other things,
+this lets you script interactions with authenticated resources (using tools
+like wget or curl) without having to worry about authentication.
+
+Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can
+record the authentication process once, and simply replay it on startup every time you need
+to interact with the secured resources.
+
+================== ======================
+command-line       :option:`-t FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`t`
+================== ======================
+
+
+Sticky auth
+-----------
+
+The sticky auth option is analogous to the sticky cookie option, in that HTTP
+**Authorization** headers are simply replayed to the server once they have been
+seen. This is enough to allow you to access a server resource using HTTP Basic
+authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support
+replay of HTTP Digest authentication.
+
+================== ======================
+command-line       :option:`-u FILTER`
+mitmproxy shortcut :kbd:`o` then :kbd:`A`
+================== ======================
diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst
new file mode 100644
index 00000000..fd0746a2
--- /dev/null
+++ b/docs/features/tcpproxy.rst
@@ -0,0 +1,31 @@
+.. _tcpproxy:
+
+TCP Proxy
+=========
+
+WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt
+hostnames from processing, so that mitmproxy acts as a generic TCP forwarder.
+This feature is closely related to the :ref:`passthrough` functionality,
+but differs in two important aspects:
+
+- The raw TCP messages are printed to the event log.
+- SSL connections will be intercepted.
+
+Please note that message interception or modification are not possible yet.
+If you are not interested in the raw TCP messages, you should use the ignore domains feature.
+
+How it works
+------------
+
+================== ======================
+command-line       :option:`--tcp HOST`
+mitmproxy shortcut :kbd:`o` then :kbd:`T`
+================== ======================
+
+For a detailed description how the hostname pattern works, please look at the :ref:`passthrough`
+feature.
+
+.. seealso::
+
+    - :ref:`passthrough`
+    - :ref:`responsestreaming`
diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst
new file mode 100644
index 00000000..af2e2226
--- /dev/null
+++ b/docs/features/upstreamcerts.rst
@@ -0,0 +1,23 @@
+.. _upstreamcerts:
+
+Upstream Certificates
+=====================
+
+When mitmproxy receives a connection destined for an SSL-protected service, it
+freezes the connection before reading its request data, and makes a connection
+to the upstream server to "sniff" the contents of its SSL certificate. The
+information gained - the **Common Name** and **Subject Alternative Names** - is
+then used to generate the interception certificate, which is sent to the client
+so the connection can continue.
+
+This rather intricate little dance lets us seamlessly generate correct
+certificates even if the client has specified only an IP address rather than the
+hostname. It also means that we don't need to sniff additional data to generate
+certs in transparent mode.
+
+Upstream cert sniffing is on by default, and can optionally be turned off.
+
+================== =============================
+command-line       :option:`--no-upstream-cert`
+mitmproxy shortcut :kbd:`o` then :kbd:`U`
+================== =============================
diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst
new file mode 100644
index 00000000..e06833c2
--- /dev/null
+++ b/docs/features/upstreamproxy.rst
@@ -0,0 +1,12 @@
+.. _upstreamproxy:
+
+Upstream proxy mode
+===================
+
+In this mode, mitmproxy accepts proxy requests and unconditionally forwards all
+requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`,
+in which mitmproxy forwards ordinary HTTP requests to an upstream server.
+
+================== ===================================
+command-line       :option:`-U http://hostname[:port]`
+================== ===================================