diff options
Diffstat (limited to 'docs')
| -rw-r--r-- | docs/_static/theme_overrides.css | 11 | ||||
| -rw-r--r-- | docs/conf.py | 7 | ||||
| -rw-r--r-- | docs/config.rst | 4 | ||||
| -rw-r--r-- | docs/dev/models.rst | 14 | ||||
| -rw-r--r-- | docs/features/anticache.rst | 4 | ||||
| -rw-r--r-- | docs/features/clientreplay.rst | 6 | ||||
| -rw-r--r-- | docs/features/passthrough.rst | 10 | ||||
| -rw-r--r-- | docs/features/proxyauth.rst | 10 | ||||
| -rw-r--r-- | docs/features/replacements.rst | 10 | ||||
| -rw-r--r-- | docs/features/responsestreaming.rst | 6 | ||||
| -rw-r--r-- | docs/features/reverseproxy.rst | 6 | ||||
| -rw-r--r-- | docs/features/serverreplay.rst | 10 | ||||
| -rw-r--r-- | docs/features/setheaders.rst | 6 | ||||
| -rw-r--r-- | docs/features/socksproxy.rst | 6 | ||||
| -rw-r--r-- | docs/features/sticky.rst | 4 | ||||
| -rw-r--r-- | docs/features/tcpproxy.rst | 2 | ||||
| -rw-r--r-- | docs/features/upstreamcerts.rst | 6 | ||||
| -rw-r--r-- | docs/features/upstreamproxy.rst | 6 | ||||
| -rw-r--r-- | docs/index.rst | 12 | ||||
| -rw-r--r-- | docs/mitmdump.rst | 8 | ||||
| -rw-r--r-- | docs/pathod/intro.rst | 307 | ||||
| -rw-r--r-- | docs/pathod/language.rst | 257 | ||||
| -rw-r--r-- | docs/pathod/library.rst | 14 | ||||
| -rw-r--r-- | docs/pathod/test.rst | 35 | ||||
| -rw-r--r-- | docs/transparent/linux.rst | 2 | ||||
| -rw-r--r-- | docs/transparent/osx.rst | 2 |
26 files changed, 704 insertions, 61 deletions
diff --git a/docs/_static/theme_overrides.css b/docs/_static/theme_overrides.css new file mode 100644 index 00000000..63c7cc78 --- /dev/null +++ b/docs/_static/theme_overrides.css @@ -0,0 +1,11 @@ + +/* override table width restrictions */ +.wy-table-responsive table td, .wy-table-responsive table th { + white-space: normal; +} + +.wy-table-responsive { + margin-bottom: 24px; + max-width: 100%; + overflow: visible; +} diff --git a/docs/conf.py b/docs/conf.py index 01bcce1b..3ef0c63d 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -153,7 +153,7 @@ html_favicon = "favicon.ico" # Add any paths that contain custom static files (such as style sheets) here, # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". -# html_static_path = ['_static'] +html_static_path = ['_static'] # Add any extra paths that contain custom files (such as robots.txt or # .htaccess) here, relative to this directory. These files are copied @@ -216,4 +216,7 @@ html_favicon = "favicon.ico" #html_search_scorer = 'scorer.js' # Output file base name for HTML help builder. -htmlhelp_basename = 'mitmproxydoc'
\ No newline at end of file +htmlhelp_basename = 'mitmproxydoc' + +def setup(app): + app.add_stylesheet('theme_overrides.css') diff --git a/docs/config.rst b/docs/config.rst index 634b8703..ad55baed 100644 --- a/docs/config.rst +++ b/docs/config.rst @@ -62,7 +62,7 @@ Options The options available in the config files are precisely those available as command-line flags, with the key being the option's long name. To get a -complete list of these, use the :option:`--help` option on each of the tools. Be +complete list of these, use the ``--help`` option on each of the tools. Be careful to only specify common options in the **common.conf** file - unsupported options in this file will be detected as an error on startup. @@ -72,7 +72,7 @@ Examples common.conf ^^^^^^^^^^^ -Note that :option:`--port` is an option supported by all tools. +Note that ``--port`` is an option supported by all tools. .. code-block:: none diff --git a/docs/dev/models.rst b/docs/dev/models.rst index f2ddf242..02f36f58 100644 --- a/docs/dev/models.rst +++ b/docs/dev/models.rst @@ -56,6 +56,10 @@ Datastructures :special-members: :no-undoc-members: + .. autoclass:: decoded + +.. automodule:: netlib.multidict + .. autoclass:: MultiDictView .. automethod:: get_all @@ -67,9 +71,11 @@ Datastructures .. automethod:: items .. automethod:: to_dict - .. autoclass:: decoded +.. autoclass:: mitmproxy.models.Error + :show-inheritance: -.. automodule:: mitmproxy.models +.. autoclass:: mitmproxy.models.ServerConnection :show-inheritance: - :noindex: - :members: Error, ServerConnection, ClientConnection
\ No newline at end of file + +.. autoclass:: mitmproxy.models.ClientConnection + :show-inheritance:
\ No newline at end of file diff --git a/docs/features/anticache.rst b/docs/features/anticache.rst index 65d22bab..411b284b 100644 --- a/docs/features/anticache.rst +++ b/docs/features/anticache.rst @@ -2,7 +2,7 @@ Anticache ========= -When the :option:`--anticache` option is passed to mitmproxy, it removes headers +When the ``--anticache`` option is passed to mitmproxy, it removes headers (``if-none-match`` and ``if-modified-since``) that might elicit a ``304 not modified`` response from the server. This is useful when you want to make sure you capture an HTTP exchange in its totality. It's also often used during @@ -10,6 +10,6 @@ sure you capture an HTTP exchange in its totality. It's also often used during ================== ====================== -command-line :option:`--anticache` +command-line ``--anticache`` mitmproxy shortcut :kbd:`o` then :kbd:`a` ================== ====================== diff --git a/docs/features/clientreplay.rst b/docs/features/clientreplay.rst index b0eb6792..50740bcf 100644 --- a/docs/features/clientreplay.rst +++ b/docs/features/clientreplay.rst @@ -12,7 +12,7 @@ conversation, where requests may have been made concurrently. You may want to use client-side replay in conjunction with the :ref:`anticache` option, to make sure the server responds with complete data. -================== ================= -command-line :option:`-c path` +================== =========== +command-line ``-c path`` mitmproxy shortcut :kbd:`c` -================== ================= +================== =========== diff --git a/docs/features/passthrough.rst b/docs/features/passthrough.rst index b7b5df84..ea628dad 100644 --- a/docs/features/passthrough.rst +++ b/docs/features/passthrough.rst @@ -20,10 +20,10 @@ take a look at the :ref:`responsestreaming` feature. How it works ------------ -================== ============================= -command-line :option:`--ignore regex` +================== ====================== +command-line ``--ignore regex`` mitmproxy shortcut :kbd:`o` then :kbd:`I` -================== ============================= +================== ====================== mitmproxy allows you to specify a regex which is matched against a ``host:port`` string @@ -32,7 +32,7 @@ mitmproxy allows you to specify a regex which is matched against a ``host:port`` There are two important quirks to consider: - **In transparent mode, the ignore pattern is matched against the IP and ClientHello SNI host.** While we usually infer the - hostname from the Host header if the :option:`--host` argument is passed to mitmproxy, we do not + hostname from the Host header if the ``--host`` argument is passed to mitmproxy, we do not have access to this information before the SSL handshake. If the client uses SNI however, then we treat the SNI host as an ignore target. - In regular mode, explicit HTTP requests are never ignored. [#explicithttp]_ The ignore pattern is applied on CONNECT requests, which initiate HTTPS or clear-text WebSocket connections. @@ -42,7 +42,7 @@ Tutorial If you just want to ignore one specific domain, there's usually a bulletproof method to do so: -1. Run mitmproxy or mitmdump in verbose mode (:option:`-v`) and observe the ``host:port`` +1. Run mitmproxy or mitmdump in verbose mode (``-v``) and observe the ``host:port`` information in the serverconnect messages. mitmproxy will filter on these. 2. Take the ``host:port`` string, surround it with ^ and $, escape all dots (. becomes \\.) and use this as your ignore pattern: diff --git a/docs/features/proxyauth.rst b/docs/features/proxyauth.rst index bfd32fbd..afdbb639 100644 --- a/docs/features/proxyauth.rst +++ b/docs/features/proxyauth.rst @@ -10,8 +10,8 @@ upstream servers. For now, only HTTP Basic authentication is supported. The proxy auth options are not compatible with the transparent, socks or reverse proxy mode. -================== ============================= -command-line :option:`--nonanonymous`, - :option:`--singleuser USER`, - :option:`--htpasswd PATH` -================== ============================= +================== ====================== +command-line ``--nonanonymous``, + ``--singleuser USER``, + ``--htpasswd PATH`` +================== ====================== diff --git a/docs/features/replacements.rst b/docs/features/replacements.rst index 8f760866..b4643dd2 100644 --- a/docs/features/replacements.rst +++ b/docs/features/replacements.rst @@ -54,7 +54,7 @@ So, you might start **mitmdump** as follows: This will load the replacement text from the file ``~/xss-exploit``. -Both the :option:`--replace` and :option:`--replace-from-file` flags can be passed multiple +Both the ``--replace`` and ``--replace-from-file`` flags can be passed multiple times. @@ -65,8 +65,8 @@ The :kbd:`R` shortcut key in the mitmproxy options menu (:kbd:`o`) lets you add replacement hooks using a built-in editor. The context-sensitive help (:kbd:`?`) has complete usage information. -================== ============================= -command-line :option:`--replace`, - :option:`--replace-from-file` +================== ======================= +command-line ``--replace``, + ``--replace-from-file`` mitmproxy shortcut :kbd:`o` then :kbd:`R` -================== ============================= +================== ======================= diff --git a/docs/features/responsestreaming.rst b/docs/features/responsestreaming.rst index 9dc27bf4..66b5cae0 100644 --- a/docs/features/responsestreaming.rst +++ b/docs/features/responsestreaming.rst @@ -19,9 +19,9 @@ On the command-line Streaming can be enabled on the command line for all response bodies exceeding a certain size. The SIZE argument understands k/m/g suffixes, e.g. 3m for 3 megabytes. -================== ============================= -command-line :option:`--stream SIZE` -================== ============================= +================== ================= +command-line ``--stream SIZE`` +================== ================= .. warning:: diff --git a/docs/features/reverseproxy.rst b/docs/features/reverseproxy.rst index 87065e73..86a5b2e4 100644 --- a/docs/features/reverseproxy.rst +++ b/docs/features/reverseproxy.rst @@ -7,9 +7,9 @@ In reverse proxy mode, mitmproxy accepts standard HTTP(S) requests and forwards them to the specified upstream server. This is in contrast to :ref:`upstreamproxy`, in which mitmproxy forwards HTTP(S) proxy requests to an upstream proxy server. -================== ===================================== -command-line :option:`-R http[s]://hostname[:port]` -================== ===================================== +================== ================================ +command-line ``-R http[s]://hostname[:port]`` +================== ================================ Here, **http[s]** signifies if the proxy should use TLS to connect to the server. mitmproxy always accepts both encrypted and unencrypted requests and transforms diff --git a/docs/features/serverreplay.rst b/docs/features/serverreplay.rst index 261a1bd6..d70b6514 100644 --- a/docs/features/serverreplay.rst +++ b/docs/features/serverreplay.rst @@ -13,7 +13,7 @@ By default, :program:`mitmproxy` excludes request headers when matching incoming requests with responses from the replay file. This works in most circumstances, and makes it possible to replay server responses in situations where request headers would naturally vary, e.g. using a different user agent. -The :option:`--rheader headername` command-line option allows you to override +The ``--rheader headername`` command-line option allows you to override this behaviour by specifying individual headers that should be included in matching. @@ -30,10 +30,10 @@ recording. So, if they were in the past at the time of recording, they will be in the past at the time of replay, and vice versa. Cookie expiry times are updated in a similar way. -You can turn off response refreshing using the :option:`--norefresh` argument, or using +You can turn off response refreshing using the ``--norefresh`` argument, or using the :kbd:`o` options shortcut within :program:`mitmproxy`. -================== ================= -command-line :option:`-S path` +================== =========== +command-line ``-S path`` mitmproxy shortcut :kbd:`S` -================== ================= +================== =========== diff --git a/docs/features/setheaders.rst b/docs/features/setheaders.rst index cbc8b6a5..aa991bf4 100644 --- a/docs/features/setheaders.rst +++ b/docs/features/setheaders.rst @@ -13,7 +13,7 @@ Example: Set the **Host** header to "example.com" for all requests. mitmdump -R http://example.com --setheader :~q:Host:example.com -================== ============================= -command-line :option:`--setheader PATTERN` +================== ======================= +command-line ``--setheader PATTERN`` mitmproxy shortcut :kbd:`o` then :kbd:`H` -================== ============================= +================== ======================= diff --git a/docs/features/socksproxy.rst b/docs/features/socksproxy.rst index 76d4cda9..e1686f45 100644 --- a/docs/features/socksproxy.rst +++ b/docs/features/socksproxy.rst @@ -5,6 +5,6 @@ SOCKS Mode In this mode, mitmproxy acts as a SOCKS5 proxy server. -================== ================= -command-line :option:`--socks` -================== ================= +================== =========== +command-line ``--socks`` +================== =========== diff --git a/docs/features/sticky.rst b/docs/features/sticky.rst index a79cbe8d..7be5f842 100644 --- a/docs/features/sticky.rst +++ b/docs/features/sticky.rst @@ -21,7 +21,7 @@ record the authentication process once, and simply replay it on startup every ti to interact with the secured resources. ================== ====================== -command-line :option:`-t FILTER` +command-line ``-t FILTER`` mitmproxy shortcut :kbd:`o` then :kbd:`t` ================== ====================== @@ -36,6 +36,6 @@ authentication through the proxy. Note that :program:`mitmproxy` doesn't (yet) s replay of HTTP Digest authentication. ================== ====================== -command-line :option:`-u FILTER` +command-line ``-u FILTER`` mitmproxy shortcut :kbd:`o` then :kbd:`A` ================== ====================== diff --git a/docs/features/tcpproxy.rst b/docs/features/tcpproxy.rst index fd0746a2..1d6fbd12 100644 --- a/docs/features/tcpproxy.rst +++ b/docs/features/tcpproxy.rst @@ -18,7 +18,7 @@ How it works ------------ ================== ====================== -command-line :option:`--tcp HOST` +command-line ``--tcp HOST`` mitmproxy shortcut :kbd:`o` then :kbd:`T` ================== ====================== diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst index af2e2226..d2a82ca2 100644 --- a/docs/features/upstreamcerts.rst +++ b/docs/features/upstreamcerts.rst @@ -17,7 +17,7 @@ certs in transparent mode. Upstream cert sniffing is on by default, and can optionally be turned off. -================== ============================= -command-line :option:`--no-upstream-cert` +================== ====================== +command-line ``--no-upstream-cert`` mitmproxy shortcut :kbd:`o` then :kbd:`U` -================== ============================= +================== ====================== diff --git a/docs/features/upstreamproxy.rst b/docs/features/upstreamproxy.rst index e06833c2..a4ccf57f 100644 --- a/docs/features/upstreamproxy.rst +++ b/docs/features/upstreamproxy.rst @@ -7,6 +7,6 @@ In this mode, mitmproxy accepts proxy requests and unconditionally forwards all requests to a specified upstream proxy server. This is in contrast to :ref:`reverseproxy`, in which mitmproxy forwards ordinary HTTP requests to an upstream server. -================== =================================== -command-line :option:`-U http://hostname[:port]` -================== =================================== +================== ============================= +command-line ``-U http://hostname[:port]`` +================== ============================= diff --git a/docs/index.rst b/docs/index.rst index d3b6f434..28c0c66f 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -64,6 +64,17 @@ tutorials/gamecenter tutorials/transparent-dhcp + +.. toctree:: + :hidden: + :caption: Pathod & Pathoc + + pathod/intro + pathod/language + pathod/library + pathod/test + + .. toctree:: :hidden: :caption: Hacking @@ -80,4 +91,3 @@ * :ref:`genindex` * :ref:`modindex` - diff --git a/docs/mitmdump.rst b/docs/mitmdump.rst index d9b4a26b..369bf0eb 100644 --- a/docs/mitmdump.rst +++ b/docs/mitmdump.rst @@ -7,7 +7,7 @@ mitmdump **mitmdump** is the command-line companion to mitmproxy. It provides tcpdump-like functionality to let you view, record, and programmatically -transform HTTP traffic. See the :option:`--help` flag output for complete +transform HTTP traffic. See the ``--help`` flag output for complete documentation. @@ -28,7 +28,7 @@ Filtering saved traffic >>> mitmdump -nr infile -w outfile "~m post" -Start mitmdump without binding to the proxy port (:option:`-n`), read all flows from +Start mitmdump without binding to the proxy port (``-n``), read all flows from infile, apply the specified filter expression (only match POSTs), and write to outfile. @@ -38,8 +38,8 @@ Client replay >>> mitmdump -nc outfile -Start mitmdump without binding to the proxy port (:option:`-n`), then replay all -requests from outfile (:option:`-c filename`). Flags combine in the obvious way, so +Start mitmdump without binding to the proxy port (``-n``), then replay all +requests from outfile (``-c filename``). Flags combine in the obvious way, so you can replay requests from one file, and write the resulting flows to another: diff --git a/docs/pathod/intro.rst b/docs/pathod/intro.rst new file mode 100644 index 00000000..bf0c531f --- /dev/null +++ b/docs/pathod/intro.rst @@ -0,0 +1,307 @@ +.. _intro: + +Pathology 101 +============= + + +pathod +------ + +Pathod is a pathological HTTP daemon designed to let you craft almost any +conceivable HTTP response, including ones that creatively violate the +standards. HTTP responses are specified using a :ref:`small, terse language +<language>` which pathod shares with its evil twin :ref:`pathoc`. To start +playing with pathod, fire up the daemon: + +>>> pathod + +By default, the service listens on port 9999 of localhost, and the default +crafting anchor point is the path **/p/**. Anything after this URL prefix is +treated as a response specifier. So, hitting the following URL will generate an +HTTP 200 response with 100 bytes of random data: + + http://localhost:9999/p/200:b@100 + +See the :ref:`language documentation <language>` to get (much) fancier. The +pathod daemon also takes a range of configuration options. To view those, use +the command-line help: + +>>> pathod --help + +Mimicing a proxy +^^^^^^^^^^^^^^^^ + +Pathod automatically responds to both straight HTTP and proxy requests. For +proxy requests, the upstream host is ignored, and the path portion of the URL +is used to match anchors. This lets you test software that supports a proxy +configuration by spoofing responses from upstream servers. + +By default, we treat all proxy CONNECT requests as HTTPS traffic, serving the +response using either pathod's built-in certificates, or the cert/key pair +specified by the user. You can over-ride this behaviour if you're testing a +client that makes a non-SSL CONNECT request using the **-C** command-line +option. + +Anchors +^^^^^^^ + +Anchors provide an alternative to specifying the response in the URL. Instead, +you attach a response to a pre-configured anchor point, specified with a regex. +When a URL matching the regex is requested, the specified response is served. + +>>> pathod -a "/foo=200" + +Here, "/foo" is the regex specifying the anchor path, and the part after the "=" +is a response specifier. + + +File Access +^^^^^^^^^^^ + +There are two operators in the :ref:`language <language>` that load contents +from file - the **+** operator to load an entire request specification from +file, and the **>** value specifier. In pathod, both of these operators are +restricted to a directory specified at startup, or disabled if no directory is +specified: + +>>> pathod -d ~/staticdir" + + +Internal Error Responses +^^^^^^^^^^^^^^^^^^^^^^^^ + +Pathod uses the non-standard 800 response code to indicate internal errors, to +distinguish them from crafted responses. For example, a request to: + + http://localhost:9999/p/foo + +... will return an 800 response because "foo" is not a valid page specifier. + + + + + +.. _pathoc: + + +pathoc +------ + +Pathoc is a perverse HTTP daemon designed to let you craft almost any +conceivable HTTP request, including ones that creatively violate the standards. +HTTP requests are specified using a :ref:`small, terse language <language>`, +which pathod shares with its server-side twin pathod. To view pathoc's complete +range of options, use the command-line help: + +>>> pathoc --help + + +Getting Started +^^^^^^^^^^^^^^^ + +The basic pattern for pathoc commands is as follows: + + pathoc hostname request [request ...] + +That is, we specify the hostname to connect to, followed by one or more +requests. Lets start with a simple example:: + + > pathoc google.com get:/ + 07-06-16 12:13:43: >> 'GET':/ + << 302 Found: 261 bytes + +Here, we make a GET request to the path / on port 80 of google.com. Pathoc's +output tells us that the server responded with a 302 redirection. We can tell +pathoc to connect using SSL, in which case the default port is changed to 443 +(you can over-ride the default port with the **-p** command-line option):: + + > pathoc -s www.google.com get:/ + 07-06-16 12:14:56: >> 'GET':/ + << 302 Found: 262 bytes + + +Multiple Requests +^^^^^^^^^^^^^^^^^ + +There are two ways to tell pathoc to issue multiple requests. The first is to specify +them on the command-line, like so:: + + > pathoc google.com get:/ get:/ + 07-06-16 12:21:04: >> 'GET':/ + << 302 Found: 261 bytes + 07-06-16 12:21:04: >> 'GET':/ + << 302 Found: 261 bytes + +In this case, pathoc issues the specified requests over the same TCP connection - +so in the above example only one connection is made to google.com + +The other way to issue multiple requests is to use the **-n** flag:: + + > pathoc -n 2 google.com get:/ + 07-06-16 12:21:04: >> 'GET':/ + << 302 Found: 261 bytes + 07-06-16 12:21:04: >> 'GET':/ + << 302 Found: 261 bytes + +The output is identical, but two separate TCP connections are made to the +upstream server. These two specification styles can be combined:: + + pathoc -n 2 google.com get:/ get:/ + + +Here, two distinct TCP connections are made, with two requests issued over +each. + + + +Basic Fuzzing +^^^^^^^^^^^^^ + +The combination of pathoc's powerful request specification language and a few +of its command-line options makes for quite a powerful basic fuzzer. Here's an +example:: + + pathoc -e -I 200 -t 2 -n 1000 localhost get:/:b@10:ir,@1 + +The request specified here is a valid GET with a body consisting of 10 random bytes, +but with 1 random byte inserted in a random place. This could be in the headers, +in the initial request line, or in the body itself. There are a few things +to note here: + +- Corrupting the request in this way will often make the server enter a state where + it's awaiting more input from the client. This is where the + **-t** option comes in, which sets a timeout that causes pathoc to + disconnect after two seconds. +- The **-n** option tells pathoc to repeat the request 1000 times. +- The **-I** option tells pathoc to ignore HTTP 200 response codes. + You can use this to fine-tune what pathoc considers to be an exceptional + condition, and therefore log-worthy. +- The **-e** option tells pathoc to print an explanation of each logged + request, in the form of an expanded pathoc specification with all random + portions and automatic header additions resolved. This lets you precisely + replay a request that triggered an error. + + +Interacting with Proxies +^^^^^^^^^^^^^^^^^^^^^^^^ + +Pathoc has a reasonably sophisticated suite of features for interacting with +proxies. The proxy request syntax very closely mirrors that of straight HTTP, +which means that it is possible to make proxy-style requests using pathoc +without any additional syntax, by simply specifying a full URL instead of a +simple path: + +>>> pathoc -p 8080 localhost "get:'http://google.com'" + +Another common use case is to use an HTTP CONNECT request to probe remote +servers via a proxy. This is done with the **-c** command-line option, which +allows you to specify a remote host and port pair: + +>>> pathoc -c google.com:80 -p 8080 localhost get:/ + +Note that pathoc does **not** negotiate SSL without being explictly instructed +to do so. If you're making a CONNECT request to an SSL-protected resource, you +must also pass the **-s** flag: + +>>> pathoc -sc google.com:443 -p 8080 localhost get:/ + + + +Embedded response specification +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One interesting feature of the Request specification language is that you can +embed a response specification in it, which is then added to the request path. +Here's an example: + +>>> pathoc localhost:9999 "get:/p/:s'401:ir,@1'" + +This crafts a request that connects to the pathod server, and which then crafts +a response that generates a 401, with one random byte embedded at a random +point. The response specification is parsed and expanded by pathoc, so you see +syntax errors immediately. This really becomes handy when combined with the +**-e** flag to show the expanded request:: + + 07-06-16 12:32:01: >> 'GET':/p/:s'401:i35,\x27\\x1b\x27:h\x27Content-Length\x27=\x270\x27:h\x27Content-Length\x27=\x270\x27':h'Host'='localhost' + << 401 Unauthorized: 0 bytes + +Note that the embedded response has been resolved *before* being sent to +the server, so that "ir,@1" (embed a random byte at a random location) has +become "i15,\'o\'" (embed the character "o" at offset 15). You now have a +pathoc request specification that is precisely reproducible, even with random +components. This feature comes in terribly handy when testing a proxy, since +you can now drive the server response completely from the client, and have a +complete log of reproducible requests to analyze afterwards. + + +Request Examples +---------------- + +.. list-table:: + :widths: 50 50 + :header-rows: 0 + + * - get:/ + - Get path / + + * - get:/:b@100 + - 100 random bytes as the body + + * - get:/:h"Etag"="&;drop table browsers;" + - Add a header + + * - get:/:u"&;drop table browsers;" + - Add a User-Agent header + + * - get:/:b@100:dr + - Drop the connection randomly + + * - get:/:b@100,ascii:ir,@1 + - 100 ASCII bytes as the body, and randomly inject a random byte + + * - ws:/ + - Initiate a websocket handshake. + + +Response Examples +----------------- + +.. list-table:: + :widths: 50 50 + :header-rows: 0 + + + * - 200 + - A basic HTTP 200 response. + + * - 200:r + - A basic HTTP 200 response with no Content-Length header. This will hang. + + * - 200:da + - Server-side disconnect after all content has been sent. + + * - 200:b\@100 + - 100 random bytes as the body. A Content-Length header is added, so the disconnect + is no longer needed. + + * - 200:b\@100:h"Etag"="';drop table servers;" + - Add a Server header + + * - 200:b\@100:dr + - Drop the connection randomly + + * - 200:b\@100,ascii:ir,@1 + - 100 ASCII bytes as the body, and randomly inject a random byte + + * - 200:b\@1k:c"text/json" + - 1k of random bytes, with a text/json content type + + * - 200:b\@1k:p50,120 + - 1k of random bytes, pause for 120 seconds after 50 bytes + + * - 200:b\@1k:pr,f + - 1k of random bytes, but hang forever at a random location + + * - 200:b\@100:h\@1k,ascii_letters='foo' + - 100 ASCII bytes as the body, randomly generated 100k header name, with the value + 'foo'. diff --git a/docs/pathod/language.rst b/docs/pathod/language.rst new file mode 100644 index 00000000..672e21b6 --- /dev/null +++ b/docs/pathod/language.rst @@ -0,0 +1,257 @@ +.. _language: + +language spec +============= + +************ +HTTP Request +************ + + **method:path:[colon-separated list of features]** + +.. list-table:: + :widths: 20 80 + :header-rows: 0 + + * - method + - A :ref:`VALUE` specifying the HTTP method to + use. Standard methods do not need to be enclosed in quotes, while + non-standard methods can be specified as quoted strings. + + The special method **ws** creates a valid websocket upgrade + GET request, and signals to pathoc to switch to websocket recieve + mode if the server responds correctly. Apart from that, websocket + requests are just like any other, and all aspects of the request + can be over-ridden. + * - h\:\ :ref:`VALUE`\ =\ :ref:`VALUE`\ + - Set a header. + * - r + - Set the **raw** flag on this response. Pathod will not calculate a + *Content-Length* header if a body is set. + * - c\ :ref:`VALUE` + - A shortcut for setting the Content-Type header. Equivalent to + ``h"Content-Type"=VALUE`` + * - u\ :ref:`VALUE` + uSHORTCUT + - Set a User-Agent header on this request. You can specify either a + complete :ref:`VALUE`, or a User-Agent shortcut: **android**, + **blackberry**, **bingbot**, **chrome**, **firefox**, **googlebot**, + **ie9**, **ipad**, **iphone**, **safari**. + * - b\ :ref:`VALUE` + - Set the body. The appropriate Content-Length header is added + automatically unless the **r** flag is set. + * - s\ :ref:`VALUE` + - An embedded Response specification, appended to the path of the request. + * - x\ :ref:`INTEGER` + - Repeat this message N times. + * - d\ :ref:`OFFSET` + - Disconnect after OFFSET bytes (HTTP/1 only). + * - i\ :ref:`OFFSET`,\ :ref:`VALUE` + - Inject the specified value at the offset (HTTP/1 only) + * - p\ :ref:`OFFSET`,SECONDS + - Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer + or "f" to pause forever (HTTP/1 only) + + +************* +HTTP Response +************* + + **code:[colon-separated list of features]** + +.. list-table:: + :widths: 20 80 + :header-rows: 0 + + * - code + - An integer specifying the HTTP response code. + + The special method **ws** creates a valid websocket upgrade + response (code 101), and moves pathod to websocket mode. Apart + from that, websocket responses are just like any other, and all + aspects of the response can be over-ridden. + * - m\ :ref:`VALUE` + - HTTP Reason message. Automatically chosen according to the response + code if not specified. (HTTP/1 only) + * - h\:\ :ref:`VALUE`\ =\ :ref:`VALUE`\ + - Set a header. + * - r + - Set the **raw** flag on this response. Pathod will not calculate a + *Content-Length* header if a body is set. + * - l\ :ref:`VALUE` + - A shortcut for setting the Location header. Equivalent to + ``h"Location"=VALUE`` + * - c\ :ref:`VALUE` + - A shortcut for setting the Content-Type header. Equivalent to + ``h"Content-Type"=VALUE`` + * - b\ :ref:`VALUE` + - Set the body. The appropriate Content-Length header is added + automatically unless the **r** flag is set. + * - d\ :ref:`OFFSET` + - Disconnect after OFFSET bytes (HTTP/1 only). + * - i\ :ref:`OFFSET`,\ :ref:`VALUE` + - Inject the specified value at the offset (HTTP/1 only) + * - p\ :ref:`OFFSET`,SECONDS + - Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer + or "f" to pause forever (HTTP/1 only) + +*************** +Websocket Frame +*************** + + **wf:[colon-separated list of features]** + +.. list-table:: + :widths: 20 80 + :header-rows: 0 + + * - b\ :ref:`VALUE` + - Set the frame payload. If a masking key is present, the value is + encoded automatically. + * - c\ :ref:`INTEGER` + - Set the op code. This can either be an integer from 0-15, or be one of + the following opcode names: **text** (the default), **continue**, + **binary**, **close**, **ping**, **pong**. + * - d\ :ref:`OFFSET` + - Disconnect after OFFSET bytes + * - i\ :ref:`OFFSET`,\ :ref:`VALUE` + - Inject the specified value at the offset + * - p\ :ref:`OFFSET`,SECONDS + - Pause for SECONDS seconds after OFFSET bytes. SECONDS can be an integer + or "f" to pause forever + * - x\ :ref:`INTEGER` + - Repeat this message N times. + * - [-]fin + - Set or un-set the **fin** bit. + * - k\ :ref:`VALUE` + - Set the masking key. The resulting value must be exactly 4 bytes long. + The special form **knone** specifies that no key should be set, even if + the mask bit is on. + * - l\ :ref:`INTEGER` + - Set the payload length in the frame header, regardless of the actual + body length. + * - [-]mask + - Set or un-set the <b>mask</b> bit. + * - r\ :ref:`VALUE` + - Set the raw frame payload. This disables masking, even if the key is present. + * - [-]rsv1 + - Set or un-set the **rsv1** bit. + * - [-]rsv2 + - Set or un-set the **rsv2** bit. + * - [-]rsv2 + - Set or un-set the **rsv2** bit. + + + +********** +Data types +********** + +.. _INTEGER: + +INTEGER +^^^^^^^ + +.. _OFFSET: + +OFFSET +^^^^^^ + +Offsets are calculated relative to the base message, before any injections or +other transforms are applied. They have 3 flavors: + +======= ========================== +integer An integer byte offset +**r** A random location +**a** The end of the message +======= ========================== + + +.. _VALUE: + +VALUE +^^^^^ + +Literals +"""""""" + +Literal values are specified as a quoted strings:: + + "foo" + +Either single or double quotes are accepted, and quotes can be escaped with +backslashes within the string:: + + 'fo\'o' + +Literal values can contain Python-style backslash escape sequences:: + + 'foo\r\nbar' + + + +Generated +""""""""" + +An @-symbol lead-in specifies that generated data should be used. There are two +components to a generator specification - a size, and a data type. By default +pathod assumes a data type of "bytes". + +Here's a value specifier for generating 100 bytes:: + + @100 + +You can use standard suffixes to indicate larger values. Here, for instance, is +a specifier for generating 100 megabytes: + + @100m + +Data is generated and served efficiently - if you really want to send a +terabyte of data to a client, pathod can do it. The supported suffixes are: + +========== ==================== +b 1024**0 (bytes) +k 1024**1 (kilobytes) +m 1024**2 (megabytes) +g 1024**3 (gigabytes) +t 1024**4 (terabytes) +========== ==================== + +Data types are separated from the size specification by a comma. This specification +generates 100mb of ASCII:: + + @100m,ascii + +Supported data types are: + +================= ============================================== +ascii All ASCII characters +ascii_letters A-Za-z +ascii_lowercase a-z +ascii_uppercase A-Z +bytes All 256 byte values +digits 0-9 +hexdigits 0-f +octdigits 0-7 +punctuation !"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~ and space +whitespace \\t \\n \\x0b \\x0c \\r and space +================= ============================================== + + + +Files +""""" + +You can load a value from a specified file path. To do so, you have to specify a +_staticdir_ option to pathod on the command-line, like so: + +>>> pathod -d ~/myassets + +All paths are relative paths under this directory. File loads are indicated by +starting the value specifier with the left angle bracket:: + + <my/path + +The path value can also be a quoted string, with the same syntax as literals:: + + <"my/path" diff --git a/docs/pathod/library.rst b/docs/pathod/library.rst new file mode 100644 index 00000000..b055d089 --- /dev/null +++ b/docs/pathod/library.rst @@ -0,0 +1,14 @@ +.. _library: + +pathod library +============== + +Behind the pathod and pathoc command-line tools lurks the **pathod** library, a +powerful way to manipulate and serve HTTP requests and responses from code. The +canonical documentation for the library is in the code, and can be accessed +using pydoc. + + +.. literalinclude:: ../../examples/pathod/libpathod_pathoc.py + :caption: examples/pathod/libpathod_pathoc.py + :language: python diff --git a/docs/pathod/test.rst b/docs/pathod/test.rst new file mode 100644 index 00000000..cd6e8a29 --- /dev/null +++ b/docs/pathod/test.rst @@ -0,0 +1,35 @@ +.. _test: + +pathod.test +=========== + +The **pathod.test** module is a light, flexible testing layer for HTTP clients. +It works by firing up a Pathod instance in a separate thread, letting you use +Pathod's full abilities to generate responses, and then query Pathod's internal +logs to establish what happened. All the mechanics of startup, shutdown, finding +free ports and so forth are taken care of for you. + +The canonical docs can be accessed using pydoc: + +>>> pydoc pathod.test + +The remainder of this page demonstrates some common interaction patterns using +<a href="http://nose.readthedocs.org/en/latest/">nose</a>. These examples are +also applicable with only minor modification to most commonly used Python testing +engines. + + +Context Manager +--------------- + +.. literalinclude:: ../../examples/pathod/test_context.py + :caption: examples/pathod/test_context.py + :language: python + + +One instance per test +--------------------- + +.. literalinclude:: ../../examples/pathod/test_setup.py + :caption: examples/pathod/test_setup.py + :language: python diff --git a/docs/transparent/linux.rst b/docs/transparent/linux.rst index ce79128c..1878008c 100644 --- a/docs/transparent/linux.rst +++ b/docs/transparent/linux.rst @@ -35,7 +35,7 @@ achieve transparent mode. >>> mitmproxy -T --host - The :option:`-T` flag turns on transparent mode, and the :option:`--host` + The ``-T`` flag turns on transparent mode, and the ``--host`` argument tells mitmproxy to use the value of the Host header for URL display. 6. Finally, configure your test device to use the host on which mitmproxy is diff --git a/docs/transparent/osx.rst b/docs/transparent/osx.rst index 1791105f..46f0e2df 100644 --- a/docs/transparent/osx.rst +++ b/docs/transparent/osx.rst @@ -50,7 +50,7 @@ Note that this means we don't support transparent mode for earlier versions of O >>> mitmproxy -T --host - The :option:`-T` flag turns on transparent mode, and the :option:`--host` + The ``-T`` flag turns on transparent mode, and the ``--host`` argument tells mitmproxy to use the value of the Host header for URL display. 8. Finally, configure your test device to use the host on which mitmproxy is |