From 101f92b2566adea605b9db69dba29e575414cbba Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Thu, 3 Jan 2013 08:25:24 +1300 Subject: Docs: move features into their own directory --- doc-src/anticache.html | 18 --------- doc-src/clientreplay.html | 22 ----------- doc-src/features/anticache.html | 18 +++++++++ doc-src/features/clientreplay.html | 22 +++++++++++ doc-src/features/filters.html | 36 ++++++++++++++++++ doc-src/features/index.py | 14 +++++++ doc-src/features/proxyauth.html | 25 +++++++++++++ doc-src/features/replacements.html | 74 +++++++++++++++++++++++++++++++++++++ doc-src/features/reverseproxy.html | 17 +++++++++ doc-src/features/serverreplay.html | 35 ++++++++++++++++++ doc-src/features/setheaders.html | 18 +++++++++ doc-src/features/sticky.html | 62 +++++++++++++++++++++++++++++++ doc-src/features/upstreamcerts.html | 21 +++++++++++ doc-src/filters.html | 36 ------------------ doc-src/index.py | 11 ------ doc-src/proxyauth.html | 25 ------------- doc-src/replacements.html | 74 ------------------------------------- doc-src/reverseproxy.html | 17 --------- doc-src/serverreplay.html | 35 ------------------ doc-src/setheaders.html | 18 --------- doc-src/sticky.html | 62 ------------------------------- doc-src/upstreamcerts.html | 21 ----------- 22 files changed, 342 insertions(+), 339 deletions(-) delete mode 100644 doc-src/anticache.html delete mode 100644 doc-src/clientreplay.html create mode 100644 doc-src/features/anticache.html create mode 100644 doc-src/features/clientreplay.html create mode 100644 doc-src/features/filters.html create mode 100644 doc-src/features/index.py create mode 100644 doc-src/features/proxyauth.html create mode 100644 doc-src/features/replacements.html create mode 100644 doc-src/features/reverseproxy.html create mode 100644 doc-src/features/serverreplay.html create mode 100644 doc-src/features/setheaders.html create mode 100644 doc-src/features/sticky.html create mode 100644 doc-src/features/upstreamcerts.html delete mode 100644 doc-src/filters.html delete mode 100644 doc-src/proxyauth.html delete mode 100644 doc-src/replacements.html delete mode 100644 doc-src/reverseproxy.html delete mode 100644 doc-src/serverreplay.html delete mode 100644 doc-src/setheaders.html delete mode 100644 doc-src/sticky.html delete mode 100644 doc-src/upstreamcerts.html (limited to 'doc-src') diff --git a/doc-src/anticache.html b/doc-src/anticache.html deleted file mode 100644 index f42903e8..00000000 --- a/doc-src/anticache.html +++ /dev/null @@ -1,18 +0,0 @@ - -When the __anticache__ option is passed to mitmproxy, it removes headers -(__if-none-match__ and __if-modified-since__) that might elicit a -304-not-modified response from the server. This is useful when you want to make -sure you capture an HTTP exchange in its totality. It's also often used during -[client replay](@!urlTo("clientreplay.html")!@), when you want to make sure the -server responds with complete data. - - - - - - - - - - -
command-line --anticache
mitmproxy shortcut o then a
diff --git a/doc-src/clientreplay.html b/doc-src/clientreplay.html deleted file mode 100644 index 6638d078..00000000 --- a/doc-src/clientreplay.html +++ /dev/null @@ -1,22 +0,0 @@ - -Client-side replay does what it says on the tin: you provide a previously saved -HTTP conversation, and mitmproxy replays the client requests one by one. Note -that mitmproxy serializes the requests, waiting for a response from the server -before starting the next request. This might differ from the recorded -conversation, where requests may have been made concurrently. - -You may want to use client-side replay in conjunction with the -[anticache](@!urlTo("anticache.html")!@) option, to make sure the server -responds with complete data. - - - - - - - - - - - -
command-line -c path
mitmproxy shortcut c
diff --git a/doc-src/features/anticache.html b/doc-src/features/anticache.html new file mode 100644 index 00000000..f42903e8 --- /dev/null +++ b/doc-src/features/anticache.html @@ -0,0 +1,18 @@ + +When the __anticache__ option is passed to mitmproxy, it removes headers +(__if-none-match__ and __if-modified-since__) that might elicit a +304-not-modified response from the server. This is useful when you want to make +sure you capture an HTTP exchange in its totality. It's also often used during +[client replay](@!urlTo("clientreplay.html")!@), when you want to make sure the +server responds with complete data. + + + + + + + + + + +
command-line --anticache
mitmproxy shortcut o then a
diff --git a/doc-src/features/clientreplay.html b/doc-src/features/clientreplay.html new file mode 100644 index 00000000..6638d078 --- /dev/null +++ b/doc-src/features/clientreplay.html @@ -0,0 +1,22 @@ + +Client-side replay does what it says on the tin: you provide a previously saved +HTTP conversation, and mitmproxy replays the client requests one by one. Note +that mitmproxy serializes the requests, waiting for a response from the server +before starting the next request. This might differ from the recorded +conversation, where requests may have been made concurrently. + +You may want to use client-side replay in conjunction with the +[anticache](@!urlTo("anticache.html")!@) option, to make sure the server +responds with complete data. + + + + + + + + + + + +
command-line -c path
mitmproxy shortcut c
diff --git a/doc-src/features/filters.html b/doc-src/features/filters.html new file mode 100644 index 00000000..c7f0f78b --- /dev/null +++ b/doc-src/features/filters.html @@ -0,0 +1,36 @@ + +Many commands in __mitmproxy__ and __mitmdump__ take a filter expression. +Filter expressions consist of the following operators: + + + + + + + + + + +
@!i[0]!@@!i[1]!@
+ +- Regexes are Python-style +- Regexes can be specified as quoted strings +- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". +- Strings with no operators are matched against the request URL. +- The default binary operator is &. + +Examples +======== + +URL containing "google.com": + + google\.com + +Requests whose body contains the string "test": + + ~q ~b test + +Anything but requests with a text/html content type: + + !(~q & ~t \"text/html\") + diff --git a/doc-src/features/index.py b/doc-src/features/index.py new file mode 100644 index 00000000..f7efc23a --- /dev/null +++ b/doc-src/features/index.py @@ -0,0 +1,14 @@ +from countershape import Page + +pages = [ + Page("anticache.html", "Anticache"), + Page("clientreplay.html", "Client-side replay"), + Page("filters.html", "Filter expressions"), + Page("setheaders.html", "Set Headers"), + Page("serverreplay.html", "Server-side replay"), + Page("sticky.html", "Sticky cookies and auth"), + Page("proxyauth.html", "Proxy Authentication"), + Page("replacements.html", "Replacements"), + Page("reverseproxy.html", "Reverse proxy mode"), + Page("upstreamcerts.html", "Upstream Certs"), +] diff --git a/doc-src/features/proxyauth.html b/doc-src/features/proxyauth.html new file mode 100644 index 00000000..0618ee68 --- /dev/null +++ b/doc-src/features/proxyauth.html @@ -0,0 +1,25 @@ + + +Asks the user for authentication before they are permitted to use the proxy. +Authentication headers are stripped from the flows, so they are not passed to +upstream servers. For now, only HTTP Basic authentication is supported. + + + + + + + + +
command-line +
    +
  • --nonanonymous
    • + +
  • --singleuser USER
    • + +
  • --htpasswd PATH
    • +
+
+ + + diff --git a/doc-src/features/replacements.html b/doc-src/features/replacements.html new file mode 100644 index 00000000..c10fe2c3 --- /dev/null +++ b/doc-src/features/replacements.html @@ -0,0 +1,74 @@ +Mitmproxy lets you specify an arbitrary number of patterns that define text +replacements within flows. Each pattern has 3 components: a filter that defines +which flows a replacement applies to, a regular expression that defines what +gets replaced, and a target value that defines what is substituted in. + +Replace hooks fire when either a client request or a server response is +received. Only the matching flow component is affected: so, for example, if a +replace hook is triggered on server response, the replacement is only run on +the Response object leaving the Request intact. You control whether the hook +triggers on the request, response or both using the filter pattern. If you need +finer-grained control than this, it's simple to create a script using the +replacement API on Flow components. + +Replacement hooks are extremely handy in interactive testing of applications. +For instance you can use a replace hook to replace the text "XSS" with a +complicated XSS exploit, and then "inject" the exploit simply by interacting +with the application through the browser. When used with tools like Firebug and +mitmproxy's own interception abilities, replacement hooks can be an amazingly +flexible and powerful feature. + + +## On the command-line + +The replacement hook command-line options use a compact syntax to make it easy +to specify all three components at once. The general form is as follows: + + /patt/regex/replacement + +Here, __patt__ is a mitmproxy filter expression, __regex__ is a valid Python +regular expression, and __replacement__ is a string literal. The first +character in the expression (__/__ in this case) defines what the separation +character is. Here's an example of a valid expression that replaces "foo" with +"bar" in all requests: + + :~q:foo:bar + +In practice, it's pretty common for the replacement literal to be long and +complex. For instance, it might be an XSS exploit that weighs in at hundreds or +thousands of characters. To cope with this, there's a variation of the +replacement hook specifier that lets you load the replacement text from a file. +So, you might start __mitmdump__ as follows: + +
+mitmdump --replace-from-file :~q:foo:~/xss-exploit
+
+ +This will load the replacement text from the file __~/xss-exploit__. + +Both the _--replace_ and _--replace-from-file_ flags can be passed multiple +times. + + +## Interactively + +The _R_ shortcut key in mitmproxy lets you add and edit replacement hooks using +a built-in editor. The context-sensitive help (_h_) has complete usage +information. + + + + + + + + + + + +
command-line +
    +
  • --replace
    • +
  • --replace-from-file
    • +
+
mitmproxy shortcut R
diff --git a/doc-src/features/reverseproxy.html b/doc-src/features/reverseproxy.html new file mode 100644 index 00000000..524df083 --- /dev/null +++ b/doc-src/features/reverseproxy.html @@ -0,0 +1,17 @@ + +In reverse proxy mode, mitmproxy acts as a standard HTTP server and forwards +all requests to the specified upstream server. Note that the displayed URL for +flows in this mode will use the value of the __Host__ header field from the +request, not the reverse proxy server. + + + + + + + + + + + +
command-line -P http[s]://hostname[:port]
mitmproxy shortcut P
diff --git a/doc-src/features/serverreplay.html b/doc-src/features/serverreplay.html new file mode 100644 index 00000000..1282be06 --- /dev/null +++ b/doc-src/features/serverreplay.html @@ -0,0 +1,35 @@ + +- command-line: _-S path_ +- mitmproxy shortcut: _S_ + +Server-side replay lets us replay server responses from a saved HTTP +conversation. + +Matching requests with responses +-------------------------------- + +By default, __mitmproxy__ excludes request headers when matching incoming +requests with responses from the replay file. This works in most circumstances, +and makes it possible to replay server responses in situations where request +headers would naturally vary, e.g. using a different user agent. The _--rheader +headername_ command-line option allows you to override this behaviour by +specifying individual headers that should be included in matching. + + +Response refreshing +------------------- + +Simply replaying server responses without modification will often result in +unexpected behaviour. For example cookie timeouts that were in the future at +the time a conversation was recorded might be in the past at the time it is +replayed. By default, __mitmproxy__ refreshes server responses before sending +them to the client. The __date__, __expires__ and __last-modified__ headers are +all updated to have the same relative time offset as they had at the time of +recording. So, if they were in the past at the time of recording, they will be +in the past at the time of replay, and vice versa. Cookie expiry times are +updated in a similar way. + +You can turn off response refreshing using the _--norefresh_ argument, or using +the _o_ options shortcut within __mitmproxy__. + + diff --git a/doc-src/features/setheaders.html b/doc-src/features/setheaders.html new file mode 100644 index 00000000..b74525df --- /dev/null +++ b/doc-src/features/setheaders.html @@ -0,0 +1,18 @@ + +This feature lets you specify a set of headers to be added to requests or +responses, based on a filter pattern. You can specify these either on the +command-line, or through an interactive editor in mitmproxy. + + + + + + + + + + + +
command-line + --setheader PATTERN +
mitmproxy shortcut H
diff --git a/doc-src/features/sticky.html b/doc-src/features/sticky.html new file mode 100644 index 00000000..1b79f922 --- /dev/null +++ b/doc-src/features/sticky.html @@ -0,0 +1,62 @@ + +## Sticky cookies + +When the sticky cookie option is set, __mitmproxy__ will add the cookie most +recently set by the server to any cookie-less request. Consider a service that +sets a cookie to track the session after authentication. Using sticky cookies, +you can fire up mitmproxy, and authenticate to a service as you usually would +using a browser. After authentication, you can request authenticated resources +through mitmproxy as if they were unauthenticated, because mitmproxy will +automatically add the session tracking cookie to requests. Among other things, +this lets you script interactions with authenticated resources (using tools +like wget or curl) without having to worry about authentication. + +Sticky cookies are especially powerful when used in conjunction with [client +replay](@!urlTo("clientreplay.html")!@) - you can record the authentication +process once, and simply replay it on startup every time you need to interact +with the secured resources. + + + + + + + + + + + +
command-line +
    +
  • -t (sticky cookies on all requests)
    • +
  • -T FILTER (sticky cookies on requests matching filter
    • +
+
mitmproxy shortcut t
+ + +## Sticky auth + +The sticky auth option is analogous to the sticky cookie option, in that HTTP +__Authorization__ headers are simply replayed to the server once they have been +seen. This is enough to allow you to access a server resource using HTTP Basic +authentication through the proxy. Note that __mitmproxy__ doesn't (yet) support +replay of HTTP Digest authentication. + + + + + + + + + + + +
command-line +
    +
  • -u (sticky auth on all requests)
    • +
  • -U FILTER (sticky auth on requests matching filter
    • +
+
mitmproxy shortcut u
+ + diff --git a/doc-src/features/upstreamcerts.html b/doc-src/features/upstreamcerts.html new file mode 100644 index 00000000..8de75ee3 --- /dev/null +++ b/doc-src/features/upstreamcerts.html @@ -0,0 +1,21 @@ +When mitmproxy receives a connection destined for an SSL-protected service, it +freezes the connection before reading its request data, and makes a connection +to the upstream server to "sniff" the contents of its SSL certificate. The +information gained - the __Common Name__ and __Subject Alternative Names__ - is +then used to generate the interception certificate, which is sent to the client +so the connection can continue. + +This rather intricate little dance lets us seamlessly generate correct +certificates even if the client has specifed only an IP address rather than the +hostname. It also means that we don't need to sniff additional data to generate +certs in transparent mode. + +Upstream cert sniffing is on by default, and can optionally be turned off. + + + + + + + +
command-line --no-upstream-cert
diff --git a/doc-src/filters.html b/doc-src/filters.html deleted file mode 100644 index c7f0f78b..00000000 --- a/doc-src/filters.html +++ /dev/null @@ -1,36 +0,0 @@ - -Many commands in __mitmproxy__ and __mitmdump__ take a filter expression. -Filter expressions consist of the following operators: - - - - - - - - - - -
@!i[0]!@@!i[1]!@
- -- Regexes are Python-style -- Regexes can be specified as quoted strings -- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". -- Strings with no operators are matched against the request URL. -- The default binary operator is &. - -Examples -======== - -URL containing "google.com": - - google\.com - -Requests whose body contains the string "test": - - ~q ~b test - -Anything but requests with a text/html content type: - - !(~q & ~t \"text/html\") - diff --git a/doc-src/index.py b/doc-src/index.py index f2a07db8..01b9363d 100644 --- a/doc-src/index.py +++ b/doc-src/index.py @@ -66,17 +66,6 @@ pages = [ Page("mitmproxy.html", "mitmproxy"), Page("mitmdump.html", "mitmdump"), - Page("anticache.html", "Anticache"), - Page("clientreplay.html", "Client-side replay"), - Page("filters.html", "Filter expressions"), - Page("setheaders.html", "Set Headers"), - Page("serverreplay.html", "Server-side replay"), - Page("sticky.html", "Sticky cookies and auth"), - Page("proxyauth.html", "Proxy Authentication"), - Page("replacements.html", "Replacements"), - Page("reverseproxy.html", "Reverse proxy mode"), - Page("upstreamcerts.html", "Upstream Certs"), - Page("ssl.html", "Overview"), Page("transparent.html", "Overview"), Directory("certinstall"), diff --git a/doc-src/proxyauth.html b/doc-src/proxyauth.html deleted file mode 100644 index 0618ee68..00000000 --- a/doc-src/proxyauth.html +++ /dev/null @@ -1,25 +0,0 @@ - - -Asks the user for authentication before they are permitted to use the proxy. -Authentication headers are stripped from the flows, so they are not passed to -upstream servers. For now, only HTTP Basic authentication is supported. - - - - - - - - -
command-line -
    -
  • --nonanonymous
    • - -
  • --singleuser USER
    • - -
  • --htpasswd PATH
    • -
-
- - - diff --git a/doc-src/replacements.html b/doc-src/replacements.html deleted file mode 100644 index c10fe2c3..00000000 --- a/doc-src/replacements.html +++ /dev/null @@ -1,74 +0,0 @@ -Mitmproxy lets you specify an arbitrary number of patterns that define text -replacements within flows. Each pattern has 3 components: a filter that defines -which flows a replacement applies to, a regular expression that defines what -gets replaced, and a target value that defines what is substituted in. - -Replace hooks fire when either a client request or a server response is -received. Only the matching flow component is affected: so, for example, if a -replace hook is triggered on server response, the replacement is only run on -the Response object leaving the Request intact. You control whether the hook -triggers on the request, response or both using the filter pattern. If you need -finer-grained control than this, it's simple to create a script using the -replacement API on Flow components. - -Replacement hooks are extremely handy in interactive testing of applications. -For instance you can use a replace hook to replace the text "XSS" with a -complicated XSS exploit, and then "inject" the exploit simply by interacting -with the application through the browser. When used with tools like Firebug and -mitmproxy's own interception abilities, replacement hooks can be an amazingly -flexible and powerful feature. - - -## On the command-line - -The replacement hook command-line options use a compact syntax to make it easy -to specify all three components at once. The general form is as follows: - - /patt/regex/replacement - -Here, __patt__ is a mitmproxy filter expression, __regex__ is a valid Python -regular expression, and __replacement__ is a string literal. The first -character in the expression (__/__ in this case) defines what the separation -character is. Here's an example of a valid expression that replaces "foo" with -"bar" in all requests: - - :~q:foo:bar - -In practice, it's pretty common for the replacement literal to be long and -complex. For instance, it might be an XSS exploit that weighs in at hundreds or -thousands of characters. To cope with this, there's a variation of the -replacement hook specifier that lets you load the replacement text from a file. -So, you might start __mitmdump__ as follows: - -
-mitmdump --replace-from-file :~q:foo:~/xss-exploit
-
- -This will load the replacement text from the file __~/xss-exploit__. - -Both the _--replace_ and _--replace-from-file_ flags can be passed multiple -times. - - -## Interactively - -The _R_ shortcut key in mitmproxy lets you add and edit replacement hooks using -a built-in editor. The context-sensitive help (_h_) has complete usage -information. - - - - - - - - - - - -
command-line -
    -
  • --replace
    • -
  • --replace-from-file
    • -
-
mitmproxy shortcut R
diff --git a/doc-src/reverseproxy.html b/doc-src/reverseproxy.html deleted file mode 100644 index 524df083..00000000 --- a/doc-src/reverseproxy.html +++ /dev/null @@ -1,17 +0,0 @@ - -In reverse proxy mode, mitmproxy acts as a standard HTTP server and forwards -all requests to the specified upstream server. Note that the displayed URL for -flows in this mode will use the value of the __Host__ header field from the -request, not the reverse proxy server. - - - - - - - - - - - -
command-line -P http[s]://hostname[:port]
mitmproxy shortcut P
diff --git a/doc-src/serverreplay.html b/doc-src/serverreplay.html deleted file mode 100644 index 1282be06..00000000 --- a/doc-src/serverreplay.html +++ /dev/null @@ -1,35 +0,0 @@ - -- command-line: _-S path_ -- mitmproxy shortcut: _S_ - -Server-side replay lets us replay server responses from a saved HTTP -conversation. - -Matching requests with responses --------------------------------- - -By default, __mitmproxy__ excludes request headers when matching incoming -requests with responses from the replay file. This works in most circumstances, -and makes it possible to replay server responses in situations where request -headers would naturally vary, e.g. using a different user agent. The _--rheader -headername_ command-line option allows you to override this behaviour by -specifying individual headers that should be included in matching. - - -Response refreshing -------------------- - -Simply replaying server responses without modification will often result in -unexpected behaviour. For example cookie timeouts that were in the future at -the time a conversation was recorded might be in the past at the time it is -replayed. By default, __mitmproxy__ refreshes server responses before sending -them to the client. The __date__, __expires__ and __last-modified__ headers are -all updated to have the same relative time offset as they had at the time of -recording. So, if they were in the past at the time of recording, they will be -in the past at the time of replay, and vice versa. Cookie expiry times are -updated in a similar way. - -You can turn off response refreshing using the _--norefresh_ argument, or using -the _o_ options shortcut within __mitmproxy__. - - diff --git a/doc-src/setheaders.html b/doc-src/setheaders.html deleted file mode 100644 index b74525df..00000000 --- a/doc-src/setheaders.html +++ /dev/null @@ -1,18 +0,0 @@ - -This feature lets you specify a set of headers to be added to requests or -responses, based on a filter pattern. You can specify these either on the -command-line, or through an interactive editor in mitmproxy. - - - - - - - - - - - -
command-line - --setheader PATTERN -
mitmproxy shortcut H
diff --git a/doc-src/sticky.html b/doc-src/sticky.html deleted file mode 100644 index 1b79f922..00000000 --- a/doc-src/sticky.html +++ /dev/null @@ -1,62 +0,0 @@ - -## Sticky cookies - -When the sticky cookie option is set, __mitmproxy__ will add the cookie most -recently set by the server to any cookie-less request. Consider a service that -sets a cookie to track the session after authentication. Using sticky cookies, -you can fire up mitmproxy, and authenticate to a service as you usually would -using a browser. After authentication, you can request authenticated resources -through mitmproxy as if they were unauthenticated, because mitmproxy will -automatically add the session tracking cookie to requests. Among other things, -this lets you script interactions with authenticated resources (using tools -like wget or curl) without having to worry about authentication. - -Sticky cookies are especially powerful when used in conjunction with [client -replay](@!urlTo("clientreplay.html")!@) - you can record the authentication -process once, and simply replay it on startup every time you need to interact -with the secured resources. - - - - - - - - - - - -
command-line -
    -
  • -t (sticky cookies on all requests)
    • -
  • -T FILTER (sticky cookies on requests matching filter
    • -
-
mitmproxy shortcut t
- - -## Sticky auth - -The sticky auth option is analogous to the sticky cookie option, in that HTTP -__Authorization__ headers are simply replayed to the server once they have been -seen. This is enough to allow you to access a server resource using HTTP Basic -authentication through the proxy. Note that __mitmproxy__ doesn't (yet) support -replay of HTTP Digest authentication. - - - - - - - - - - - -
command-line -
    -
  • -u (sticky auth on all requests)
    • -
  • -U FILTER (sticky auth on requests matching filter
    • -
-
mitmproxy shortcut u
- - diff --git a/doc-src/upstreamcerts.html b/doc-src/upstreamcerts.html deleted file mode 100644 index 8de75ee3..00000000 --- a/doc-src/upstreamcerts.html +++ /dev/null @@ -1,21 +0,0 @@ -When mitmproxy receives a connection destined for an SSL-protected service, it -freezes the connection before reading its request data, and makes a connection -to the upstream server to "sniff" the contents of its SSL certificate. The -information gained - the __Common Name__ and __Subject Alternative Names__ - is -then used to generate the interception certificate, which is sent to the client -so the connection can continue. - -This rather intricate little dance lets us seamlessly generate correct -certificates even if the client has specifed only an IP address rather than the -hostname. It also means that we don't need to sniff additional data to generate -certs in transparent mode. - -Upstream cert sniffing is on by default, and can optionally be turned off. - - - - - - - -
command-line --no-upstream-cert
-- cgit v1.2.3