From 92597f82ea8e4747ce1836ecd5eb2479486e8647 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Thu, 18 Feb 2016 09:19:05 +1300 Subject: Docs and examples to top level --- docs/features/anticache.rst | 15 ++++++ docs/features/clientreplay.rst | 18 +++++++ docs/features/filters.rst | 39 +++++++++++++++ docs/features/passthrough.rst | 97 +++++++++++++++++++++++++++++++++++++ docs/features/proxyauth.rst | 17 +++++++ docs/features/replacements.rst | 72 +++++++++++++++++++++++++++ docs/features/responsestreaming.rst | 69 ++++++++++++++++++++++++++ docs/features/reverseproxy.rst | 57 ++++++++++++++++++++++ docs/features/serverreplay.rst | 39 +++++++++++++++ docs/features/setheaders.rst | 19 ++++++++ docs/features/socksproxy.rst | 10 ++++ docs/features/sticky.rst | 41 ++++++++++++++++ docs/features/tcpproxy.rst | 31 ++++++++++++ docs/features/upstreamcerts.rst | 23 +++++++++ docs/features/upstreamproxy.rst | 12 +++++ 15 files changed, 559 insertions(+) create mode 100644 docs/features/anticache.rst create mode 100644 docs/features/clientreplay.rst create mode 100644 docs/features/filters.rst create mode 100644 docs/features/passthrough.rst create mode 100644 docs/features/proxyauth.rst create mode 100644 docs/features/replacements.rst create mode 100644 docs/features/responsestreaming.rst create mode 100644 docs/features/reverseproxy.rst create mode 100644 docs/features/serverreplay.rst create mode 100644 docs/features/setheaders.rst create mode 100644 docs/features/socksproxy.rst create mode 100644 docs/features/sticky.rst create mode 100644 docs/features/tcpproxy.rst create mode 100644 docs/features/upstreamcerts.rst create mode 100644 docs/features/upstreamproxy.rst (limited to 'docs/features') diff --git a/docs/features/anticache.rst b/docs/features/anticache.rst new file mode 100644 index 00000000..65d22bab --- /dev/null +++ b/docs/features/anticache.rst @@ -0,0 +1,15 @@ +.. _anticache: + +Anticache +========= +When the :option:`--anticache` option is passed to mitmproxy, it removes headers +(``if-none-match`` and ``if-modified-since``) that might elicit a +``304 not modified`` response from the server. This is useful when you want to make +sure you capture an HTTP exchange in its totality. It's also often used during +:ref:`clientreplay`, when you want to make sure the server responds with complete data. + + +================== ====================== +command-line :option:`--anticache` +mitmproxy shortcut :kbd:`o` then :kbd:`a` +================== ====================== diff --git a/docs/features/clientreplay.rst b/docs/features/clientreplay.rst new file mode 100644 index 00000000..b0eb6792 --- /dev/null +++ b/docs/features/clientreplay.rst @@ -0,0 +1,18 @@ +.. _clientreplay: + +Client-side replay +================== + +Client-side replay does what it says on the tin: you provide a previously saved +HTTP conversation, and mitmproxy replays the client requests one by one. Note +that mitmproxy serializes the requests, waiting for a response from the server +before starting the next request. This might differ from the recorded +conversation, where requests may have been made concurrently. + +You may want to use client-side replay in conjunction with the +:ref:`anticache` option, to make sure the server responds with complete data. + +================== ================= +command-line :option:`-c path` +mitmproxy shortcut :kbd:`c` +================== ================= diff --git a/docs/features/filters.rst b/docs/features/filters.rst new file mode 100644 index 00000000..509b5d6b --- /dev/null +++ b/docs/features/filters.rst @@ -0,0 +1,39 @@ +.. _filters: + +Filter expressions +================== + +Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression. +Filter expressions consist of the following operators: + +.. documentedlist:: + :header: "Expression" "Description" + :listobject: mitmproxy.filt.help + +- Regexes are Python-style +- Regexes can be specified as quoted strings +- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". +- Strings with no operators are matched against the request URL. +- The default binary operator is &. + +Examples +-------- + +URL containing "google.com": + +.. code-block:: none + + google\.com + +Requests whose body contains the string "test": + +.. code-block:: none + + ~q ~b test + +Anything but requests with a text/html content type: + +.. code-block:: none + + !(~q & ~t "text/html") + diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst new file mode 100644 index 00000000..b7b5df84 --- /dev/null +++ b/docs/features/passthrough.rst @@ -0,0 +1,97 @@ +.. _passthrough: + +Ignore Domains +============== + +There are two main reasons why you may want to exempt some traffic from mitmproxy's interception +mechanism: + +- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and + mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or + the Apple App Store fail to work if mitmproxy is active. +- **Convenience:** You really don't care about some parts of the traffic and just want them to go + away. + +If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcpproxy` +feature. +If you want to ignore traffic from mitmproxy's processing because of large response bodies, +take a look at the :ref:`responsestreaming` feature. + +How it works +------------ + +================== ============================= +command-line :option:`--ignore regex` +mitmproxy shortcut :kbd:`o` then :kbd:`I` +================== ============================= + + +mitmproxy allows you to specify a regex which is matched against a ``host:port`` string +(e.g. "example.com:443") to determine hosts that should be excluded. + +There are two important quirks to consider: + +- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the + hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not + have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target. +- In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is + applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections. + +Tutorial +-------- + +If you just want to ignore one specific domain, there's usually a bulletproof method to do so: + +1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port`` + information in the serverconnect messages. mitmproxy will filter on these. +2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.) + and use this as your ignore pattern: + +.. code-block:: none + :emphasize-lines: 6,7,9 + + >>> mitmdump -v + 127.0.0.1:50588: clientconnect + 127.0.0.1:50588: request + -> CONNECT example.com:443 HTTP/1.1 + 127.0.0.1:50588: Set new server address: example.com:443 + 127.0.0.1:50588: serverconnect + -> example.com:443 + ^C + >>> mitmproxy --ignore ^example\.com:443$ + + +Here are some other examples for ignore patterns: + +.. code-block:: none + + # Exempt traffic from the iOS App Store (the regex is lax, but usually just works): + --ignore apple.com:443 + # "Correct" version without false-positives: + --ignore '^(.+\.)?apple\.com:443$' + + # Ignore example.com, but not its subdomains: + --ignore '^example.com:' + + # Ignore everything but example.com and mitmproxy.org: + --ignore '^(?!example\.com)(?!mitmproxy\.org)' + + # Transparent mode: + --ignore 17\.178\.96\.59:443 + # IP address range: + --ignore 17\.178\.\d+\.\d+:443 + + +.. seealso:: + + - :ref:`tcpproxy` + - :ref:`responsestreaming` + +.. rubric:: Footnotes + +.. [#explicithttp] This stems from an limitation of explicit HTTP proxying: + A single connection can be re-used for multiple target domains - a + ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the + same connection. If we start to ignore the connection after the first request, + we would miss the relevant second one. +.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst new file mode 100644 index 00000000..bfd32fbd --- /dev/null +++ b/docs/features/proxyauth.rst @@ -0,0 +1,17 @@ +.. _proxyauth: + +Proxy Authentication +==================== + + +Asks the user for authentication before they are permitted to use the proxy. +Authentication headers are stripped from the flows, so they are not passed to +upstream servers. For now, only HTTP Basic authentication is supported. The +proxy auth options are not compatible with the transparent, socks or reverse proxy +mode. + +================== ============================= +command-line :option:`--nonanonymous`, + :option:`--singleuser USER`, + :option:`--htpasswd PATH` +================== ============================= diff --git a/docs/features/replacements.rst b/docs/features/replacements.rst new file mode 100644 index 00000000..8f760866 --- /dev/null +++ b/docs/features/replacements.rst @@ -0,0 +1,72 @@ +.. _replacements: + +Replacements +============ + +Mitmproxy lets you specify an arbitrary number of patterns that define text +replacements within flows. Each pattern has 3 components: a filter that defines +which flows a replacement applies to, a regular expression that defines what +gets replaced, and a target value that defines what is substituted in. + +Replace hooks fire when either a client request or a server response is +received. Only the matching flow component is affected: so, for example, if a +replace hook is triggered on server response, the replacement is only run on +the Response object leaving the Request intact. You control whether the hook +triggers on the request, response or both using the filter pattern. If you need +finer-grained control than this, it's simple to create a script using the +replacement API on Flow components. + +Replacement hooks are extremely handy in interactive testing of applications. +For instance you can use a replace hook to replace the text "XSS" with a +complicated XSS exploit, and then "inject" the exploit simply by interacting +with the application through the browser. When used with tools like Firebug and +mitmproxy's own interception abilities, replacement hooks can be an amazingly +flexible and powerful feature. + + +On the command-line +------------------- + +The replacement hook command-line options use a compact syntax to make it easy +to specify all three components at once. The general form is as follows: + +.. code-block:: none + + /patt/regex/replacement + +Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python +regular expression, and **replacement** is a string literal. The first +character in the expression (``/`` in this case) defines what the separation +character is. Here's an example of a valid expression that replaces "foo" with +"bar" in all requests: + +.. code-block:: none + + :~q:foo:bar + +In practice, it's pretty common for the replacement literal to be long and +complex. For instance, it might be an XSS exploit that weighs in at hundreds or +thousands of characters. To cope with this, there's a variation of the +replacement hook specifier that lets you load the replacement text from a file. +So, you might start **mitmdump** as follows: + +>>> mitmdump --replace-from-file :~q:foo:~/xss-exploit + +This will load the replacement text from the file ``~/xss-exploit``. + +Both the :option:`--replace` and :option:`--replace-from-file` flags can be passed multiple +times. + + +Interactively +------------- + +The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`o`) lets you add and edit +replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has +complete usage information. + +================== ============================= +command-line :option:`--replace`, + :option:`--replace-from-file` +mitmproxy shortcut :kbd:`o` then :kbd:`R` +================== ============================= diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst new file mode 100644 index 00000000..8975c1f8 --- /dev/null +++ b/docs/features/responsestreaming.rst @@ -0,0 +1,69 @@ +.. _responsestreaming: + +Response Streaming +================== + +By using mitmproxy's streaming feature, response contents can be passed to the client incrementally +before they have been fully received by the proxy. This is especially useful for large binary files +such as videos, where buffering the whole file slows down the client's browser. + +By default, mitmproxy will read the entire response, perform any indicated +manipulations on it and then send the (possibly modified) response to +the client. In some cases this is undesirable and you may wish to "stream" +the response back to the client. When streaming is enabled, the response is +not buffered on the proxy but directly sent back to the client instead. + +On the command-line +------------------- + +Streaming can be enabled on the command line for all response bodies exceeding a certain size. +The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. + +================== ============================= +command-line :option:`--stream SIZE` +================== ============================= + +.. warning:: + + When response streaming is enabled, **streamed response contents will not be + recorded or preserved in any way.** + +.. note:: + + When response streaming is enabled, the response body cannot be modified by the usual means. + +Customizing Response Streaming +------------------------------ + +You can also use an :ref:`inlinescripts` to customize exactly +which responses are streamed. + +Responses that should be tagged for streaming by setting their ``.stream`` attribute to ``True``: + +.. literalinclude:: ../../examples/stream.py + :caption: examples/stream.py + :language: python + +Implementation Details +---------------------- + +When response streaming is enabled, portions of the code which would have otherwise performed +changes on the response body will see an empty response body instead +(:py:data:`netlib.http.CONTENT_MISSING`). Any modifications will be ignored. + +Streamed responses are usually sent in chunks of 4096 bytes. If the response is sent with a +``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. + +Modifying streamed data +----------------------- + +If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all +chunks. + +.. literalinclude:: ../../examples/stream_modify.py + :caption: examples/stream_modify.py + :language: python + +.. seealso:: + + - :ref:`passthrough` diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst new file mode 100644 index 00000000..87065e73 --- /dev/null +++ b/docs/features/reverseproxy.rst @@ -0,0 +1,57 @@ +.. _reverseproxy: + +Reverse Proxy +============= + +In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards +them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which +mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server. + +================== ===================================== +command-line :option:`-R http[s]://hostname[:port]` +================== ===================================== + +Here, **http[s]** signifies if the proxy should use TLS to connect to the server. +mitmproxy always accepts both encrypted and unencrypted requests and transforms +them to what the server expects. + +.. code-block:: none + + >>> mitmdump -R https://httpbin.org -p 80 + >>> curl http://localhost/ + # requests will be transparently upgraded to TLS by mitmproxy + + >>> mitmdump -R https://httpbin.org -p 443 + >>> curl https://localhost/ + # mitmproxy will use TLS on both ends. + + +Host Header +----------- + +In reverse proxy mode, mitmproxy does not rewrite the host header. While often useful, this +may lead to issues with public web servers. For example, consider the following scenario: + +.. code-block:: none + :emphasize-lines: 5 + + >>> mitmdump -d -R http://example.com/ + >>> curl http://localhost:8080/ + + >> GET https://example.com/ + Host: localhost:8080 + User-Agent: curl/7.35.0 + [...] + + << 404 Not Found 345B + +Since the Host header doesn't match "example.com", an error is returned. +There are two ways to solve this: + +1. Modify the hosts file of your OS so that "example.com" resolves to your proxy's IP. + Then, access example.com directly. Make sure that your proxy can still resolve the original IP + or specify an IP in mitmproxy. +2. Use mitmproxy's :ref:`setheaders` feature to rewrite the host header: + ``--setheader :~q:Host:example.com``. + However, keep in mind that absolute URLs within the returned document or HTTP redirects will + cause the client application to bypass the proxy. diff --git a/docs/features/serverreplay.rst b/docs/features/serverreplay.rst new file mode 100644 index 00000000..261a1bd6 --- /dev/null +++ b/docs/features/serverreplay.rst @@ -0,0 +1,39 @@ +.. _serverreplay: + +Server-side replay +================== + +Server-side replay lets us replay server responses from a saved HTTP +conversation. + +Matching requests with responses +-------------------------------- + +By default, :program:`mitmproxy` excludes request headers when matching incoming +requests with responses from the replay file. This works in most circumstances, +and makes it possible to replay server responses in situations where request +headers would naturally vary, e.g. using a different user agent. +The :option:`--rheader headername` command-line option allows you to override +this behaviour by specifying individual headers that should be included in matching. + + +Response refreshing +------------------- + +Simply replaying server responses without modification will often result in +unexpected behaviour. For example cookie timeouts that were in the future at +the time a conversation was recorded might be in the past at the time it is +replayed. By default, :program:`mitmproxy` refreshes server responses before sending +them to the client. The **date**, **expires** and **last-modified** headers are +all updated to have the same relative time offset as they had at the time of +recording. So, if they were in the past at the time of recording, they will be +in the past at the time of replay, and vice versa. Cookie expiry times are +updated in a similar way. + +You can turn off response refreshing using the :option:`--norefresh` argument, or using +the :kbd:`o` options shortcut within :program:`mitmproxy`. + +================== ================= +command-line :option:`-S path` +mitmproxy shortcut :kbd:`S` +================== ================= diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst new file mode 100644 index 00000000..cbc8b6a5 --- /dev/null +++ b/docs/features/setheaders.rst @@ -0,0 +1,19 @@ +.. _setheaders: + +Set Headers +=========== + +This feature lets you specify a set of headers to be added to requests or +responses, based on a filter pattern. You can specify these either on the +command-line, or through an interactive editor in mitmproxy. + +Example: Set the **Host** header to "example.com" for all requests. + +.. code-block:: none + + mitmdump -R http://example.com --setheader :~q:Host:example.com + +================== ============================= +command-line :option:`--setheader PATTERN` +mitmproxy shortcut :kbd:`o` then :kbd:`H` +================== ============================= diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst new file mode 100644 index 00000000..76d4cda9 --- /dev/null +++ b/docs/features/socksproxy.rst @@ -0,0 +1,10 @@ +.. _socksproxy: + +SOCKS Mode +========== + +In this mode, mitmproxy acts as a SOCKS5 proxy server. + +================== ================= +command-line :option:`--socks` +================== ================= diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst new file mode 100644 index 00000000..a79cbe8d --- /dev/null +++ b/docs/features/sticky.rst @@ -0,0 +1,41 @@ +.. _sticky: + +Sticky cookies and auth +======================= + +Sticky cookies +-------------- + +When the sticky cookie option is set, __mitmproxy__ will add the cookie most +recently set by the server to any cookie-less request. Consider a service that +sets a cookie to track the session after authentication. Using sticky cookies, +you can fire up mitmproxy, and authenticate to a service as you usually would +using a browser. After authentication, you can request authenticated resources +through mitmproxy as if they were unauthenticated, because mitmproxy will +automatically add the session tracking cookie to requests. Among other things, +this lets you script interactions with authenticated resources (using tools +like wget or curl) without having to worry about authentication. + +Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can +record the authentication process once, and simply replay it on startup every time you need +to interact with the secured resources. + +================== ====================== +command-line :option:`-t FILTER` +mitmproxy shortcut :kbd:`o` then :kbd:`t` +================== ====================== + + +Sticky auth +----------- + +The sticky auth option is analogous to the sticky cookie option, in that HTTP +**Authorization** headers are simply replayed to the server once they have been +seen. This is enough to allow you to access a server resource using HTTP Basic +authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support +replay of HTTP Digest authentication. + +================== ====================== +command-line :option:`-u FILTER` +mitmproxy shortcut :kbd:`o` then :kbd:`A` +================== ====================== diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst new file mode 100644 index 00000000..fd0746a2 --- /dev/null +++ b/docs/features/tcpproxy.rst @@ -0,0 +1,31 @@ +.. _tcpproxy: + +TCP Proxy +========= + +WebSockets or other non-HTTP protocols are not supported by mitmproxy yet. However, you can exempt +hostnames from processing, so that mitmproxy acts as a generic TCP forwarder. +This feature is closely related to the :ref:`passthrough` functionality, +but differs in two important aspects: + +- The raw TCP messages are printed to the event log. +- SSL connections will be intercepted. + +Please note that message interception or modification are not possible yet. +If you are not interested in the raw TCP messages, you should use the ignore domains feature. + +How it works +------------ + +================== ====================== +command-line :option:`--tcp HOST` +mitmproxy shortcut :kbd:`o` then :kbd:`T` +================== ====================== + +For a detailed description how the hostname pattern works, please look at the :ref:`passthrough` +feature. + +.. seealso:: + + - :ref:`passthrough` + - :ref:`responsestreaming` diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst new file mode 100644 index 00000000..af2e2226 --- /dev/null +++ b/docs/features/upstreamcerts.rst @@ -0,0 +1,23 @@ +.. _upstreamcerts: + +Upstream Certificates +===================== + +When mitmproxy receives a connection destined for an SSL-protected service, it +freezes the connection before reading its request data, and makes a connection +to the upstream server to "sniff" the contents of its SSL certificate. The +information gained - the **Common Name** and **Subject Alternative Names** - is +then used to generate the interception certificate, which is sent to the client +so the connection can continue. + +This rather intricate little dance lets us seamlessly generate correct +certificates even if the client has specified only an IP address rather than the +hostname. It also means that we don't need to sniff additional data to generate +certs in transparent mode. + +Upstream cert sniffing is on by default, and can optionally be turned off. + +================== ============================= +command-line :option:`--no-upstream-cert` +mitmproxy shortcut :kbd:`o` then :kbd:`U` +================== ============================= diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst new file mode 100644 index 00000000..e06833c2 --- /dev/null +++ b/docs/features/upstreamproxy.rst @@ -0,0 +1,12 @@ +.. _upstreamproxy: + +Upstream proxy mode +=================== + +In this mode, mitmproxy accepts proxy requests and unconditionally forwards all +requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`, +in which mitmproxy forwards ordinary HTTP requests to an upstream server. + +================== =================================== +command-line :option:`-U http://hostname[:port]` +================== =================================== -- cgit v1.2.3