From 982508d30f887b4fe8b2a855792ae1e33f378222 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Thu, 22 Feb 2018 17:21:34 +1300 Subject: All new documentation This patch does a lot. - Ditch sphinx in favor of hugo. This gives us complete control of the layout and presentation of our docs. Henceforth, docs will be hosted on our website rather than ReadTheDocs. - Create a simple, clean doc layout and theme. - Remove large parts of the documentaion. I've ditched anything that was a) woefully out of date, b) too detailed, or c) too hard to maintain in the long term. - Huge updates to the docs themselves: completely rewrite addons documentation, add docs for core concepts like commands and options, and revise and tweak a lot of the existing docs. With this patch, we're also changing the way we publish and maintain the docs. From now on, we don't publish docs for every release. Instead, the website will contain ONE set of docs for each major release. The online docs will be updated if needed as minor releases are made. Docs are free to improve during minor releases, but anything that changes behaviour sufficiently to require a doc change warrants a new major release. This also leaves us free to progressively update and improve docs out of step with our release cadence. With this new scheme, I feel CI over the docs is less important. I've removed it for now, but won't object if someone wants to add it back in. --- docs/features/anticache.rst | 15 ------ docs/features/clientreplay.rst | 18 ------- docs/features/filters.rst | 38 --------------- docs/features/passthrough.rst | 102 ---------------------------------------- docs/features/proxyauth.rst | 17 ------- docs/features/replacements.rst | 71 ---------------------------- docs/features/reverseproxy.rst | 43 ----------------- docs/features/serverreplay.rst | 52 -------------------- docs/features/setheaders.rst | 19 -------- docs/features/socksproxy.rst | 10 ---- docs/features/sticky.rst | 41 ---------------- docs/features/streaming.rst | 102 ---------------------------------------- docs/features/upstreamcerts.rst | 23 --------- docs/features/upstreamproxy.rst | 12 ----- 14 files changed, 563 deletions(-) delete mode 100644 docs/features/anticache.rst delete mode 100644 docs/features/clientreplay.rst delete mode 100644 docs/features/filters.rst delete mode 100644 docs/features/passthrough.rst delete mode 100644 docs/features/proxyauth.rst delete mode 100644 docs/features/replacements.rst delete mode 100644 docs/features/reverseproxy.rst delete mode 100644 docs/features/serverreplay.rst delete mode 100644 docs/features/setheaders.rst delete mode 100644 docs/features/socksproxy.rst delete mode 100644 docs/features/sticky.rst delete mode 100644 docs/features/streaming.rst delete mode 100644 docs/features/upstreamcerts.rst delete mode 100644 docs/features/upstreamproxy.rst (limited to 'docs/features') diff --git a/docs/features/anticache.rst b/docs/features/anticache.rst deleted file mode 100644 index a0c3187a..00000000 --- a/docs/features/anticache.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _anticache: - -Anticache -========= -When the ``--anticache`` option is passed to mitmproxy, it removes headers -(``if-none-match`` and ``if-modified-since``) that might elicit a -``304 not modified`` response from the server. This is useful when you want to make -sure you capture an HTTP exchange in its totality. It's also often used during -:ref:`clientreplay`, when you want to make sure the server responds with complete data. - - -================== ====================== -command-line ``--anticache`` -mitmproxy shortcut :kbd:`O` then :kbd:`a` -================== ====================== diff --git a/docs/features/clientreplay.rst b/docs/features/clientreplay.rst deleted file mode 100644 index ebe40b5f..00000000 --- a/docs/features/clientreplay.rst +++ /dev/null @@ -1,18 +0,0 @@ -.. _clientreplay: - -Client-side replay -================== - -Client-side replay does what it says on the tin: you provide a previously saved -HTTP conversation, and mitmproxy replays the client requests one by one. Note -that mitmproxy serializes the requests, waiting for a response from the server -before starting the next request. This might differ from the recorded -conversation, where requests may have been made concurrently. - -You may want to use client-side replay in conjunction with the -:ref:`anticache` option, to make sure the server responds with complete data. - -================== =========== -command-line ``-c path`` -mitmproxy shortcut :kbd:`R` then :kbd:`c` -================== =========== diff --git a/docs/features/filters.rst b/docs/features/filters.rst deleted file mode 100644 index e531f734..00000000 --- a/docs/features/filters.rst +++ /dev/null @@ -1,38 +0,0 @@ -.. _filters: - -Filter expressions -================== - -Many commands in :program:`mitmproxy` and :program:`mitmdump` take a filter expression. -Filter expressions consist of the following operators: - -.. documentedlist:: - :header: "Expression" "Description" - :listobject: mitmproxy.flowfilter.help - -- Regexes are Python-style -- Regexes can be specified as quoted strings -- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". -- Strings with no operators are matched against the request URL. -- The default binary operator is &. - -Examples --------- - -URL containing "google.com": - -.. code-block:: none - - google\.com - -Requests whose body contains the string "test": - -.. code-block:: none - - ~q ~b test - -Anything but requests with a text/html content type: - -.. code-block:: none - - !(~q & ~t "text/html") diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst deleted file mode 100644 index 91fcb9b6..00000000 --- a/docs/features/passthrough.rst +++ /dev/null @@ -1,102 +0,0 @@ -.. _passthrough: - -Ignore Domains -============== - -There are two main reasons why you may want to exempt some traffic from mitmproxy's interception -mechanism: - -- **Certificate pinning:** Some traffic is is protected using `Certificate Pinning`_ and - mitmproxy's interception leads to errors. For example, the Twitter app, Windows Update or - the Apple App Store fail to work if mitmproxy is active. -- **Convenience:** You really don't care about some parts of the traffic and just want them to go - away. Note that mitmproxy's "Limit" option is often the better alternative here, as it is - not affected by the limitations listed below. - -If you want to peek into (SSL-protected) non-HTTP connections, check out the :ref:`tcp_proxy` -feature. -If you want to ignore traffic from mitmproxy's processing because of large response bodies, -take a look at the :ref:`streaming` feature. - -How it works ------------- - -================== ====================== -command-line ``--ignore regex`` -mitmproxy shortcut :kbd:`O` then :kbd:`I` -================== ====================== - - -mitmproxy allows you to specify a regex which is matched against a ``host:port`` string -(e.g. "example.com:443") to determine hosts that should be excluded. - -Limitations ------------ - -There are two important quirks to consider: - -- **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the - hostname from the Host header if the ``--host`` argument is passed to mitmproxy, we do not - have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target. -- **In regular and upstream proxy mode, explicit HTTP requests are never ignored.** [#explicithttp]_ The ignore pattern is - applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections. - -Tutorial --------- - -If you just want to ignore one specific domain, there's usually a bulletproof method to do so: - -1. Run mitmproxy or mitmdump in verbose mode (``-v``) and observe the ``host:port`` - information in the serverconnect messages. mitmproxy will filter on these. -2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.) - and use this as your ignore pattern: - -.. code-block:: none - :emphasize-lines: 6,7,9 - - >>> mitmdump -v - 127.0.0.1:50588: clientconnect - 127.0.0.1:50588: request - -> CONNECT example.com:443 HTTP/1.1 - 127.0.0.1:50588: Set new server address: example.com:443 - 127.0.0.1:50588: serverconnect - -> example.com:443 - ^C - >>> mitmproxy --ignore ^example\.com:443$ - - -Here are some other examples for ignore patterns: - -.. code-block:: none - - # Exempt traffic from the iOS App Store (the regex is lax, but usually just works): - --ignore apple.com:443 - # "Correct" version without false-positives: - --ignore '^(.+\.)?apple\.com:443$' - - # Ignore example.com, but not its subdomains: - --ignore '^example.com:' - - # Ignore everything but example.com and mitmproxy.org: - --ignore '^(?!example\.com)(?!mitmproxy\.org)' - - # Transparent mode: - --ignore 17\.178\.96\.59:443 - # IP address range: - --ignore 17\.178\.\d+\.\d+:443 - - -.. seealso:: - - - :ref:`tcp_proxy` - - :ref:`streaming` - - mitmproxy's "Limit" feature - -.. rubric:: Footnotes - -.. [#explicithttp] This stems from an limitation of explicit HTTP proxying: - A single connection can be re-used for multiple target domains - a - ``GET http://example.com/`` request may be followed by a ``GET http://evil.com/`` request on the - same connection. If we start to ignore the connection after the first request, - we would miss the relevant second one. -.. _Certificate Pinning: https://security.stackexchange.com/questions/29988/what-is-certificate-pinning diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst deleted file mode 100644 index afdbb639..00000000 --- a/docs/features/proxyauth.rst +++ /dev/null @@ -1,17 +0,0 @@ -.. _proxyauth: - -Proxy Authentication -==================== - - -Asks the user for authentication before they are permitted to use the proxy. -Authentication headers are stripped from the flows, so they are not passed to -upstream servers. For now, only HTTP Basic authentication is supported. The -proxy auth options are not compatible with the transparent, socks or reverse proxy -mode. - -================== ====================== -command-line ``--nonanonymous``, - ``--singleuser USER``, - ``--htpasswd PATH`` -================== ====================== diff --git a/docs/features/replacements.rst b/docs/features/replacements.rst deleted file mode 100644 index 39dccca2..00000000 --- a/docs/features/replacements.rst +++ /dev/null @@ -1,71 +0,0 @@ -.. _replacements: - -Replacements -============ - -Mitmproxy lets you specify an arbitrary number of patterns that define text -replacements within flows. Each pattern has 3 components: a filter that defines -which flows a replacement applies to, a regular expression that defines what -gets replaced, and a target value that defines what is substituted in. - -Replace hooks fire when either a client request or a server response is -received. Only the matching flow component is affected: so, for example, if a -replace hook is triggered on server response, the replacement is only run on -the Response object leaving the Request intact. You control whether the hook -triggers on the request, response or both using the filter pattern. If you need -finer-grained control than this, it's simple to create a script using the -replacement API on Flow components. - -Replacement hooks are extremely handy in interactive testing of applications. -For instance you can use a replace hook to replace the text "XSS" with a -complicated XSS exploit, and then "inject" the exploit simply by interacting -with the application through the browser. When used with tools like Firebug and -mitmproxy's own interception abilities, replacement hooks can be an amazingly -flexible and powerful feature. - - -On the command-line -------------------- - -The replacement hook command-line options use a compact syntax to make it easy -to specify all three components at once. The general form is as follows: - -.. code-block:: none - - /patt/regex/replacement - -Here, **patt** is a mitmproxy filter expression, **regex** is a valid Python -regular expression, and **replacement** is a string literal. The first -character in the expression (``/`` in this case) defines what the separation -character is. Here's an example of a valid expression that replaces "foo" with -"bar" in all requests: - -.. code-block:: none - - :~q:foo:bar - -In practice, it's pretty common for the replacement literal to be long and -complex. For instance, it might be an XSS exploit that weighs in at hundreds or -thousands of characters. To cope with this, there's a variation of the -replacement hook specifier that lets you load the replacement text from a file. -To specify a file as replacement, prefix the file path with ``@``. -You might start **mitmdump** as follows: - ->>> mitmdump --replacements :~q:foo:@~/xss-exploit - -This will load the replacement text from the file ``~/xss-exploit``. - -The ``--replacements`` flag can be passed multiple times. - - -Interactively -------------- - -The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`O`) lets you add and edit -replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has -complete usage information. - -================== ======================= -command-line ``--replacements`` -mitmproxy shortcut :kbd:`O` then :kbd:`R` -================== ======================= diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst deleted file mode 100644 index 57b353ae..00000000 --- a/docs/features/reverseproxy.rst +++ /dev/null @@ -1,43 +0,0 @@ -.. _reverseproxy: - -Reverse Proxy -============= - -In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards -them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which -mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server. - -================== ================================ -command-line ``-R http[s]://hostname[:port]`` -================== ================================ - -Here, **http[s]** signifies if the proxy should use TLS to connect to the server. -mitmproxy always accepts both encrypted and unencrypted requests and transforms -them to what the server expects. - -.. code-block:: none - - >>> mitmdump -R https://httpbin.org -p 80 - >>> curl http://localhost/ - # requests will be transparently upgraded to TLS by mitmproxy - - >>> mitmdump -R https://httpbin.org -p 443 - >>> curl https://localhost/ - # mitmproxy will use TLS on both ends. - - -Host Header ------------ - -In reverse proxy mode, mitmproxy automatically rewrites the Host header to match the -upstream server. This allows mitmproxy to easily connect to existing endpoints on the -open web (e.g. ``mitmproxy -R https://example.com``). You can disable this behaviour -by passing ``--keep-host-header`` on the console. - -However, keep in mind that absolute URLs within the returned document or HTTP redirects will -NOT be rewritten by mitmproxy. This means that if you click on a link for "http://example.com" -in the returned web page, you will be taken directly to that URL, bypassing mitmproxy. - -One possible way to address this is to modify the hosts file of your OS so that "example.com" -resolves to your proxy's IP, and then access the proxy by going directly to example.com. -Make sure that your proxy can still resolve the original IP, or specify an IP in mitmproxy. diff --git a/docs/features/serverreplay.rst b/docs/features/serverreplay.rst deleted file mode 100644 index aef0296e..00000000 --- a/docs/features/serverreplay.rst +++ /dev/null @@ -1,52 +0,0 @@ -.. _serverreplay: - -Server-side replay -================== - -Server-side replay lets us replay server responses from a saved HTTP -conversation. - -Matching requests with responses --------------------------------- - -By default, :program:`mitmproxy` excludes request headers when matching incoming -requests with responses from the replay file. This works in most circumstances, -and makes it possible to replay server responses in situations where request -headers would naturally vary, e.g. using a different user agent. -The ``--rheader headername`` command-line option allows you to override -this behaviour by specifying individual headers that should be included in matching. - - -Response refreshing -------------------- - -Simply replaying server responses without modification will often result in -unexpected behaviour. For example cookie timeouts that were in the future at -the time a conversation was recorded might be in the past at the time it is -replayed. By default, :program:`mitmproxy` refreshes server responses before sending -them to the client. The **date**, **expires** and **last-modified** headers are -all updated to have the same relative time offset as they had at the time of -recording. So, if they were in the past at the time of recording, they will be -in the past at the time of replay, and vice versa. Cookie expiry times are -updated in a similar way. - -You can turn off response refreshing using the ``--norefresh`` argument, or using -the :kbd:`O` options shortcut within :program:`mitmproxy`. - - -Replaying a session recorded in Reverse-proxy Mode --------------------------------------------------- - -If you have captured the session in reverse proxy mode, in order to replay it you -still have to specify the server URL, otherwise you may get the error: -'HTTP protocol error in client request: Invalid HTTP request form (expected authority or absolute...)'. - -During replay, when the client's requests match previously recorded requests, then the -respective recorded responses are simply replayed by mitmproxy. -Otherwise, the unmatched requests is forwarded to the upstream server. -If forwarding is not desired, you can use the --kill (-k) switch to prevent that. - -================== =========== -command-line ``-S path`` -mitmproxy shortcut :kbd:`R` then :kbd:`s` -================== =========== diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst deleted file mode 100644 index 486f8c76..00000000 --- a/docs/features/setheaders.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _setheaders: - -Set Headers -=========== - -This feature lets you specify a set of headers to be added to requests or -responses, based on a filter pattern. You can specify these either on the -command-line, or through an interactive editor in mitmproxy. - -Example: Set the **Host** header to "example.com" for all requests. - -.. code-block:: none - - mitmdump -R http://example.com --setheader :~q:Host:example.com - -================== ======================= -command-line ``--setheader PATTERN`` -mitmproxy shortcut :kbd:`O` then :kbd:`H` -================== ======================= diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst deleted file mode 100644 index e1686f45..00000000 --- a/docs/features/socksproxy.rst +++ /dev/null @@ -1,10 +0,0 @@ -.. _socksproxy: - -SOCKS Mode -========== - -In this mode, mitmproxy acts as a SOCKS5 proxy server. - -================== =========== -command-line ``--socks`` -================== =========== diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst deleted file mode 100644 index 5cf32299..00000000 --- a/docs/features/sticky.rst +++ /dev/null @@ -1,41 +0,0 @@ -.. _sticky: - -Sticky cookies and auth -======================= - -Sticky cookies --------------- - -When the sticky cookie option is set, __mitmproxy__ will add the cookie most -recently set by the server to any cookie-less request. Consider a service that -sets a cookie to track the session after authentication. Using sticky cookies, -you can fire up mitmproxy, and authenticate to a service as you usually would -using a browser. After authentication, you can request authenticated resources -through mitmproxy as if they were unauthenticated, because mitmproxy will -automatically add the session tracking cookie to requests. Among other things, -this lets you script interactions with authenticated resources (using tools -like wget or curl) without having to worry about authentication. - -Sticky cookies are especially powerful when used in conjunction with :ref:`clientreplay` - you can -record the authentication process once, and simply replay it on startup every time you need -to interact with the secured resources. - -================== ====================== -command-line ``-t FILTER`` -mitmproxy shortcut :kbd:`O` then :kbd:`t` -================== ====================== - - -Sticky auth ------------ - -The sticky auth option is analogous to the sticky cookie option, in that HTTP -**Authorization** headers are simply replayed to the server once they have been -seen. This is enough to allow you to access a server resource using HTTP Basic -authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) support -replay of HTTP Digest authentication. - -================== ====================== -command-line ``-u FILTER`` -mitmproxy shortcut :kbd:`O` then :kbd:`A` -================== ====================== diff --git a/docs/features/streaming.rst b/docs/features/streaming.rst deleted file mode 100644 index 82510843..00000000 --- a/docs/features/streaming.rst +++ /dev/null @@ -1,102 +0,0 @@ -.. _streaming: - -HTTP Streaming -============== - -By default, mitmproxy will read the entire request/response, perform any indicated -manipulations on it and then send the (possibly modified) message to -the other party. In some cases this is undesirable and you may wish to "stream" -the request/response. When streaming is enabled, the request/response is -not buffered on the proxy but directly sent to the server/client instead. -HTTP headers are still fully buffered before being sent. - -Request Streaming ------------------ - -Request streaming can be used to incrementally stream a request body to the server -before it has been fully received by the proxy. This is useful for large file uploads. - -Response Streaming ------------------- - -By using mitmproxy's streaming feature, response contents can be passed to the client incrementally -before they have been fully received by the proxy. This is especially useful for large binary files -such as videos, where buffering the whole file slows down the client's browser. - -On the command-line -------------------- - -Streaming can be enabled on the command line for all request and response bodies exceeding a certain size. -The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. - -================== ================= -command-line ``--set stream_large_bodies=SIZE`` -================== ================= - -.. warning:: - - When streaming is enabled, **streamed request/response contents will not be - recorded or preserved in any way.** - -.. note:: - - When streaming is enabled, the request/response body cannot be modified by the usual means. - -Customizing Streaming ---------------------- - -You can also use a script to customize exactly which requests or responses are streamed. - -Requests/Responses that should be tagged for streaming by setting their ``.stream`` -attribute to ``True``: - -.. literalinclude:: ../../examples/complex/stream.py - :caption: examples/complex/stream.py - :language: python - -Implementation Details ----------------------- - -When response streaming is enabled, portions of the code which would have otherwise performed -changes on the request/response body will see an empty body. Any modifications will be ignored. - -Streamed bodies are usually sent in chunks of 4096 bytes. If the response is sent with a -``Transfer-Encoding: chunked`` header, the response will be streamed one chunk at a time. - -Modifying streamed data ------------------------ - -If the ``.stream`` attribute is callable, ``.stream`` will wrap the generator that yields all -chunks. - -.. literalinclude:: ../../examples/complex/stream_modify.py - :caption: examples/complex/stream_modify.py - :language: python - -WebSocket Streaming -=================== - -The WebSocket streaming feature can be used to send the frames as soon as they arrive. This can be useful for large binary file transfers. - -On the command-line -------------------- - -Streaming can be enabled on the command line for all WebSocket frames - -================== ================= -command-line ``--set stream_websockets=true`` -================== ================= - -.. note:: - - When Web Socket streaming is enabled, the message payload cannot be modified. - -Implementation Details ----------------------- -When WebSocket streaming is enabled, portions of the code which may perform changes to the WebSocket message payloads will not have -any effect on the actual payload sent to the server as the frames are immediately forwarded to the server. -In contrast to HTTP streaming, where the body is not stored, the message payload will still be stored in the WebSocket Flow. - -.. seealso:: - - - :ref:`passthrough` diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst deleted file mode 100644 index 4ef79e1b..00000000 --- a/docs/features/upstreamcerts.rst +++ /dev/null @@ -1,23 +0,0 @@ -.. _upstreamcerts: - -Upstream Certificates -===================== - -When mitmproxy receives a connection destined for an SSL-protected service, it -freezes the connection before reading its request data, and makes a connection -to the upstream server to "sniff" the contents of its SSL certificate. The -information gained - the **Common Name** and **Subject Alternative Names** - is -then used to generate the interception certificate, which is sent to the client -so the connection can continue. - -This rather intricate little dance lets us seamlessly generate correct -certificates even if the client has specified only an IP address rather than the -hostname. It also means that we don't need to sniff additional data to generate -certs in transparent mode. - -Upstream cert sniffing is on by default, and can optionally be turned off. - -================== ====================== -command-line ``--no-upstream-cert`` -mitmproxy shortcut :kbd:`O` then :kbd:`U` -================== ====================== diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst deleted file mode 100644 index a4ccf57f..00000000 --- a/docs/features/upstreamproxy.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. _upstreamproxy: - -Upstream proxy mode -=================== - -In this mode, mitmproxy accepts proxy requests and unconditionally forwards all -requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`, -in which mitmproxy forwards ordinary HTTP requests to an upstream server. - -================== ============================= -command-line ``-U http://hostname[:port]`` -================== ============================= -- cgit v1.2.3