diff options
Diffstat (limited to 'docs/src/content')
20 files changed, 2350 insertions, 0 deletions
diff --git a/docs/src/content/_index.md b/docs/src/content/_index.md new file mode 100644 index 00000000..a977e2db --- /dev/null +++ b/docs/src/content/_index.md @@ -0,0 +1,32 @@ +--- +title: "Introduction" +menu: + overview: + weight: 1 +--- + +# Introduction + +**mitmproxy** is an interactive man-in-the-middle proxy for HTTP and HTTPS +with a console interface. + +**mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP. + +**mitmweb** is a web-based interface for mitmproxy. + +Documentation, tutorials and distribution packages can be found on the +[mitmproxy website](https://mitmproxy.org). + + +## Features + +- Intercept HTTP & HTTPS requests and responses and modify them on the fly +- Save complete HTTP conversations for later replay and analysis +- Replay the client-side of an HTTP conversations +- Replay HTTP responses of a previously recorded server +- Reverse proxy mode to forward traffic to a specified server +- Transparent proxy mode on OSX and Linux +- Make scripted changes to HTTP traffic using Python +- SSL/TLS certificates for interception are generated on the fly +- And much, much more... + diff --git a/docs/src/content/addons-commands.md b/docs/src/content/addons-commands.md new file mode 100644 index 00000000..f689d763 --- /dev/null +++ b/docs/src/content/addons-commands.md @@ -0,0 +1,134 @@ +--- +title: "Commands" +menu: + addons: + weight: 4 +--- + +# Commands + +Commands allow users to actively interact with addons - querying their state, +commanding them to perform actions, and having them transform data. Like +[options]({{< relref addons-options >}}), commands are typed, and both +invocations and data returned from commands are checked at runtime. Commands are +a very powerful construct - for instance, all user interaction in mitmproxy +console are built by binding commands to keys. + + +## Simple example + +Let's begin with a simple example. + +{{< example src="examples/addons/commands-simple.py" lang="py" >}} + +To see this example in action, start mitmproxy console with the addon loaded: + +{{< highlight bash >}} +> mitmproxy -s ./examples/addons/commands-simple.py +{{< /highlight >}} + +Now, make sure the event log is showing, and then execute the command at the +prompt (started by typing ":"): + +{{< highlight none>}} +:myaddon.inc +{{< /highlight >}} + +Notice that tab completion works - our addon command has complete parity with +builtin commands. There are a few things to note about this example: + +- Commands are declared through the `command.command` decorator. Each command + has a unique name - by convention, we use period-separated names, with the + name of the addon as a prefix. +- Annotating commands with types is mandatory, including the return type (in + this case `None`). This allows mitmproxy to support addon commands throughout + its toolset - runtime invocations are type checked, addon commands are + included in the built-in help, the command editor in mitmproxy console can + perform sophisticated completion and error checking, and so forth. + + +## Working with flows + +Since command arguments are typed, we can provide special conveniences for +working with certain important data types. The most useful of these are the +`Flows` classes that represent mitmproxy traffic. + +Consider the following addon: + +{{< example src="examples/addons/commands-flows.py" lang="py" >}} + +The `myaddon.addheader` command is quite simple: it takes a sequence of flows, +and adds a header to every request. The really interesting aspect of this +example is how users specify flows. Because mitmproxy can inspect the type +signature, it can expand a text flow selector into a sequence of flows for us +transparently. This means that the user has the full flexibility of [flow +filters]({{< relref addons-options >}}) available. Let's try it out. + +Start by loading the addon into mitmproxy and sending some traffic through so we +have flows to work with: + +{{< highlight bash >}} +> mitmproxy -s ./examples/addons/commands-flows.py +{{< /highlight >}} + +We can now invoke our toy command in various ways. Let's begin by running it +just on the currently focused flow: + +{{< highlight none >}} +:myaddon.addheader @focus +{{< /highlight >}} + +We can also invoke it on all flows: + +{{< highlight none >}} +:myaddon.addheader @all +{{< /highlight >}} + +Or only flows from **google.com**: + +{{< highlight none >}} +:myaddon.addheader ~d google.com +{{< /highlight >}} + +What's more, we can trivially bind these commands to keyboard shortcuts within +mitmproxy if we plan to use them frequently. Flow selectors combined with +commands are amazingly powerful, and lets us build and expose re-usable functions +for operating on flows. + + +## Paths + +Commands can take an arbitrary number of arguments. Let's build on the previous +example to illustrate this, and also demonstrate another special type: paths. + +{{< example src="examples/addons/commands-paths.py" lang="py" >}} + +Our command calculates a histogram of the domains in the specified set of flows, +and writes it to a path which is specified as the second argument to the +command. Try invoking it like this: + +{{< highlight none >}} +:myaddon.histogram @all /tmp/xxx +{{< /highlight >}} + +Notice that mitmproxy provides tab completion both for the flow specification +and the path. + + + +## Supported Types + +The following types are supported for options. If you need to use a type not +listed here, please send us a pull request. + +- Primitive types: `str`, `int`, `bool` +- Sequences: `typing.Sequence[str]` +- Flows and flow sequences: `flow.Flow` and `typing.Sequence[flow.Flow]` +- Multiple choice strings: `types.Choice` +- Meta-types: `types.Command` and `types.Arg`. These are for constructing + commands that invoke other commands. This is most commonly useful in + keybinding - see the built-in mitmproxy console keybindings for a rich suite + of examples. +- Data types: `types.CutSpec` and `types.Data`. The cuts mechanism is in alpha + at the moment, and provides a convenient way to snip up flow data. +- Path: `types.Path` diff --git a/docs/src/content/addons-events.md b/docs/src/content/addons-events.md new file mode 100644 index 00000000..489ea342 --- /dev/null +++ b/docs/src/content/addons-events.md @@ -0,0 +1,24 @@ +--- +title: "Events" +menu: + addons: + weight: 2 +--- + +# Events + +Addons hook into mitmproxy's internal mechanisms through events. These are +implemented on addons as methods with a set of well-known names. Many events +receive `Flow` objects as arguments - by modifying these objects, addons can +change traffic on the fly. For instance, here is an addon that adds a response +header with a count of the number of responses seen: + +{{< example src="examples/addons/addheader.py" lang="py" >}} + + +## Supported Events + +Below is an addon class that implements stubs for all events. We've added +annotations to illustrate the argument types for the various events. + +{{< example src="examples/addons/events.py" lang="py" >}}
\ No newline at end of file diff --git a/docs/src/content/addons-options.md b/docs/src/content/addons-options.md new file mode 100644 index 00000000..930847d0 --- /dev/null +++ b/docs/src/content/addons-options.md @@ -0,0 +1,101 @@ +--- +title: "Options" +menu: + addons: + weight: 3 +--- + +# Options + +At the heart of mitmproxy is a global options store, containing the settings +that determine the behaviour of both mitmproxy and its addons. Options can be +read from a configuration file, set on the command-line and changed +interactively by users on the fly. + +All options are annotated with one of a set of supported types. Mitmproxy knows +how to serialise and deserialise these types, and has standard ways of +presenting typed values for editing in interactive programs. Attempting to set a +value with the wrong type will result in an error. This means that addon options +get full support throughout mitmproxy's toolchain simply by declaring a type. + + +## Simple example + +{{< example src="examples/addons/options-simple.py" lang="py" >}} + +The `load` event receives an instance of `mitmproxy.addonmanager.Loader`, which +allows addons declare options and commands. In this case, the addon adds a +single `addheader` option with type `bool`. Let's try this out by running the +script in mitmproxy console: + +{{< highlight bash >}} +> mitmproxy -s ./examples/addons/options-simple.py +{{< /highlight >}} + +You can now use CURL to make a request through the proxy like this: + +{{< highlight bash >}} +> env http_proxy=http://localhost:8080 curl -I http://google.com +{{< /highlight >}} + +If you run this request immediately, you'll notice that no count header is +added. This is because our default value for the option was `false`. Press `O` +to enter the options editor, and find the `addheader` option. You'll notice that +mitmproxy knows this is a boolean, and lets you toggle the value between true +and false. Set the value to `true`, and you should see a result something like +this: + +{{< highlight bash >}} +> env http_proxy=http://localhost:8080 curl -I http://google.com +HTTP/1.1 301 Moved Permanently +Location: http://www.google.com/ +Content-Length: 219 +count: 1 +{{< /highlight >}} + +When this addon is loaded, the `addheader` setting is available in the +persistent [YAML configuration file]({{< relref "concepts-options" >}}). You can +also over-ride the value directly from the command-line for any of the tools +using the `--set` flag: + +{{< highlight bash >}} +mitmproxy -s ./examples/addons/options-simple.py --set addheader=true +{{< /highlight >}} + + +## Handling configuration updates + +Sometimes, simply testing the value of an option from an event is not +sufficient. Instead, we want to act immediately when an option is changed by the +user. This is what the `configure` event is for - when it is triggered, it +receives a set of changed options. An addon can check if an option is in this +set, and then read the value from the options object on the context. + +One common use for this function is to check that an option is valid, and give +the user feedback if it's not. If an `exceptions.OptionsError` exception is +raised during configure, all the changes in the update are automatically rolled +back, and an error is displayed to the user. Let's see an example. + +{{< example src="examples/addons/options-configure.py" lang="py" >}} + +There are a few things to note here. First, the option we add uses +`typing.Optional`. This signals to mitmproxy that `None` is a valid value for +this option - that is, it can be unset. Second, the `configure` method is first +called with our default value (`None`), and then later with an updated value if +the option is changed. If we try to load the script with an incorrect value, we +now see an error: + +{{< highlight none >}} +> mitmdump -s ./examples/addons/options-configure.py --set addheader=1000 +Loading script: ./examples/addons/options-configure.py +/Users/cortesi/mitmproxy/mitmproxy/venv/bin/mitmdump: addheader must be <= 100 +{{< /highlight >}} + + +## Supported Types + +The following types are supported for options. + +- Primitive types - `str`, `int`, `float`, `bool`. +- Optional values, annotated using `typing.Optional`. +- Sequences of values, annotated using `typing.Sequence`. diff --git a/docs/src/content/addons-overview.md b/docs/src/content/addons-overview.md new file mode 100644 index 00000000..6aa1207e --- /dev/null +++ b/docs/src/content/addons-overview.md @@ -0,0 +1,74 @@ +--- +title: "Addons" +menu: + addons: + weight: 1 +--- + +# Addons + +Mitmproxy's addon mechanism consists of a set of APIs that support components of +any complexity. Addons interact with mitmproxy by responding to **events**, +which allow them to hook into and change mitmproxy's behaviour. They are +configured through **[options]({{< relref concepts-options >}})**, which can be +set in mitmproxy's config file, changed interactively by users, or passed on the +command-line. Finally, they can expose **commands**, which allows users to +invoke their actions either directly or by binding them to keys in the +interactive tools. + +Addons are an exceptionally powerful part of mitmproxy. In fact, much of +mitmproxy's own functionality is defined in [a suite of built-in +addons](https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/addons), +implementing everything from functionality like [anticaching]({{< relref +"overview-features#anticache" >}}) and [sticky cookies]({{< relref +"overview-features#sticky-cookies" >}}) to our onboarding webapp. The built-in +addons make for instructive reading, and you will quickly see that quite complex +functionality can often boil down to a very small, completely self-contained +modules. Mitmproxy provides the exact same set of facilities it uses for its own +functionality to third-party scripters and extenders. + +This document will show you how to build addons using **events**, **options** +and **commands**. However, this is not an API manual, and the mitmproxy source +code remains the canonical reference. One easy way to explore the API from the +command-line is to use [pydoc](https://docs.python.org/3/library/pydoc.html). +Here, for example, is a command that shows the API documentation for the +mitmproxy's HTTP flow classes: + +{{< highlight bash >}} +pydoc mimtproxy.http +{{< /highlight >}} + +You will be referring to the mitmproxy API documentation frequently, so keep +**pydoc** or an equivalent handy. + +# Anatomy of an addon + +{{< example src="examples/addons/anatomy.py" lang="py" >}} + +Above is a simple addon that keeps track of the number of flows (or more +specifically HTTP requests) we've seen. Every time it sees a new flow, it uses +mitmproxy's internal logging mechanism to announce its tally. The output can be +found in the event log in the interactive tools, or on the console in mitmdump. + +Take it for a spin and make sure that it does what it's supposed to, by loading +it into your mitmproxy tool of choice. We'll use mitmpdump in these examples, +but the flag is identical for all tools: + +{{< highlight bash >}} +> mitmdump -s ./anatomy.py +{{< /highlight >}} + +Here are a few things to note about the code above: + +- Mitmproxy picks up the contents of the `addons` global list and loads what it + finds into the addons mechanism. +- Addons are just objects - in this case our addon is an instance of `Counter`. +- The `request` method is an example of an **event**. Addons simply implement a + method for each event they wan to handle. Each event has a signature + consisting of arguments that are passed to the method. For `request`, this is + an instance of `mitmproxy.http.HTTPFlow`. +- Finally, the `ctx` module is a holdall module that exposes a set of standard + objects that are commonly used in addons. We could pass a `ctx` object as the + first parameter to every event, but we've found it neater to just expose it as + an importable global. In this case, we're using the `ctx.log` object to do our + logging.
\ No newline at end of file diff --git a/docs/src/content/addons-scripting.md b/docs/src/content/addons-scripting.md new file mode 100644 index 00000000..e31d291a --- /dev/null +++ b/docs/src/content/addons-scripting.md @@ -0,0 +1,17 @@ +--- +title: "Scripting" +menu: + addons: + weight: 5 +--- + +# Scripting + +Sometimes, we would like to write a quick script without going through the +trouble of creating a class. The addons mechanism has a shorthand that allows a +module as a whole to be treated as an addon object. This lets us place event +handler functions in the module scope. For instance, here is a complete script +that adds a header to every request. + + +{{< example src="examples/addons/scripting.py" lang="py" >}}
\ No newline at end of file diff --git a/docs/src/content/concepts-certificates.md b/docs/src/content/concepts-certificates.md new file mode 100644 index 00000000..6956ff3f --- /dev/null +++ b/docs/src/content/concepts-certificates.md @@ -0,0 +1,164 @@ +--- +title: "Certificates" +menu: + concepts: + weight: 3 +--- + +# About Certificates + +Mitmproxy can decrypt encrypted traffic on the fly, as long as the client trusts +its built-in certificate authority. Usually this means that the mitmproxy CA +certificates have to be installed on the client device. + +## Quick Setup + +By far the easiest way to install the mitmproxy certificates is to use the +built-in certificate installation app. To do this, just start mitmproxy and +configure your target device with the correct proxy settings. Now start a +browser on the device, and visit the magic domain **mitm.it**. You should see +something like this: + +{{< figure src="/certinstall-webapp.png" >}} + +Click on the relevant icon, follow the setup instructions for the platform +you're on and you are good to go. + +## Installing the mitmproxy CA certificate manually + +Sometimes using the quick install app is not an option - Java or the iOS +Simulator spring to mind - or you just need to do it manually for some other +reason. Below is a list of pointers to manual certificate installation +documentation for some common platforms. The mitmproxy CA cert is located in +`~/.mitmproxy` after it has been generated at the first start of mitmproxy. + +- [IOS](http://jasdev.me/intercepting-ios-traffic) On + iOS 10.3 and onwards, you also need to enable full trust for the mitmproxy + root certificate: + 1. Go to Settings > General > About > Certificate Trust Settings. + 2. Under "Enable full trust for root certificates", turn on trust for + the mitmproxy certificate. +- [IOS Simulator](https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore) +- [Java](https://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html) +- [Android/Android Simulator](http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets) +- [Windows](https://web.archive.org/web/20160612045445/http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7) +- [Windows (automated)](https://technet.microsoft.com/en-us/library/cc732443.aspx) + +{{< highlight bash >}} +certutil.exe -importpfx Root mitmproxy-ca-cert.p12 +{{< / highlight >}} + +- [Mac OS X](https://support.apple.com/kb/PH7297?locale=en_US) +- [Ubuntu/Debian]( https://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861) +- [Mozilla Firefox](https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox) +- [Chrome on Linux](https://stackoverflow.com/a/15076602/198996) + +## The mitmproxy certificate authority + +The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate +Authority (CA) is created in the config directory (`~/.mitmproxy` by default). +This CA is used for on-the-fly generation of dummy certificates for each of the +SSL sites that your client visits. Since your browser won't trust the mitmproxy +CA out of the box, you will see an SSL certificate warning every time you visit +a new SSL domain through mitmproxy. When you are testing a single site through a +browser, just accepting the bogus SSL cert manually is not too much trouble, but +there are a many circumstances where you will want to configure your testing +system or browser to trust the mitmproxy CA as a signing root authority. For +security reasons, the mitmproxy CA is generated uniquely on the first start and +is not shared between mitmproxy installations on different devices. + +### Certificate Pinning + +Some applications employ [Certificate +Pinning](https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning) to prevent +man-in-the-middle attacks. This means that **mitmproxy** and **mitmdump's** +certificates will not be accepted by these applications without modifying them. +It is recommended to use the passthrough feature in order to prevent +**mitmproxy** and **mitmdump** from intercepting traffic to these specific +domains. If you want to intercept the pinned connections, you need to patch the +application manually. For Android and (jailbroken) iOS devices, various tools +exist to accomplish this. + +## CA and cert files + +The files created by mitmproxy in the .mitmproxy directory are as follows: + +| | | +| --------------------- | ------------------------------------------------------------------------------------ | +| mitmproxy-ca.pem | The certificate **and the private key** in PEM format. | +| mitmproxy-ca-cert.pem | The certificate in PEM format. Use this to distribute on most non-Windows platforms. | +| mitmproxy-ca-cert.p12 | The certificate in PKCS12 format. For use on Windows. | +| mitmproxy-ca-cert.cer | Same file as .pem, but with an extension expected by some Android devices. | + +## Using a custom certificate + +You can use your own (leaf) certificate by passing the `--cert +[domain=]path_to_certificate` option to mitmproxy. Mitmproxy then uses the +provided certificate for interception of the specified domain instead of +generating a certificate signed by its own CA. + +The certificate file is expected to be in the PEM format. You can include +intermediary certificates right below your leaf certificate, so that your PEM +file roughly looks like this: + + -----BEGIN PRIVATE KEY----- + <private key> + -----END PRIVATE KEY----- + -----BEGIN CERTIFICATE----- + <cert> + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + <intermediary cert (optional)> + -----END CERTIFICATE----- + +For example, you can generate a certificate in this format using these +instructions: + +{{< highlight bash >}} +openssl genrsa -out cert.key 2048 +# (Specify the mitm domain as Common Name, e.g. \*.google.com) +openssl req -new -x509 -key cert.key -out cert.crt +cat cert.key cert.crt \> cert.pem +{{< / highlight >}} + +Now, you can run mitmproxy with the generated certificate: + +**For all domain names** + +{{< highlight bash >}} +mitmproxy --cert *=cert.pem +{{< / highlight >}} + +**For specific domain names** + +{{< highlight bash >}} +mitmproxy --cert *.example.com=cert.pem +{{< / highlight >}} + +**Note:** `*.example.com` is for all the subdomains. You can also use +`www.example.com` for a particular subdomain. + +## Using a custom certificate authority + +By default, mitmproxy will use `~/.mitmproxy/mitmproxy-ca.pem` as the +certificate authority to generate certificates for all domains for which +no custom certificate is provided (see above). You can use your own +certificate authority by passing the `--cadir DIRECTORY` option to +mitmproxy. Mitmproxy will then look for `mitmproxy-ca.pem` in the +specified directory. If no such file exists, it will be generated +automatically. + +## Using a client side certificate + +You can use a client certificate by passing the `--client-certs DIRECTORY|FILE` +option to mitmproxy. Using a directory allows certs to be selected based on +hostname, while using a filename allows a single specific certificate to be used +for all SSL connections. Certificate files must be in the PEM format and should +contain both the unencrypted private key and the certificate. + +### Multiple certs by Hostname + +You can specify a directory to `--client-certs`, in which case the matching +certificate is looked up by filename. So, if you visit example.org, mitmproxy +looks for a file named `example.org.pem` in the specified directory and uses +this as the client cert. diff --git a/docs/src/content/concepts-commands.md b/docs/src/content/concepts-commands.md new file mode 100644 index 00000000..72f17252 --- /dev/null +++ b/docs/src/content/concepts-commands.md @@ -0,0 +1,72 @@ +--- +title: "Commands" +menu: + concepts: + weight: 6 +--- + +# Commands + +Commands are the mechanism that allows users to actively interact with addons. +Perhaps the most prominent example of this is the mitmproxy console user +interface - every interaction in this tool consists of commands bound to keys. +Commands also form a flexible and very powerful way to interact with mitmproxy +from the command prompt. In mitmproxy console you can enter the command prompt +with the `:` key. The prompt has intelligent tab completion for command names +and many of the built-in argument types - give it a try. + +The canonical reference for commands is the `--commands` flag, which is exposed +by each of the mitmproxy tools. Passing this flag will dump an annotated list of +all registered commands, their arguments and their return values to screen. In +mimtproxy console you can also view a palette of all commands in the command +browser (by default accessible with the `C` key binding). + + +# Working with flows + +Many of mitmproxy's commands take flows as arguments. For instance, the +signature for the client replay commands looks like this: + +{{< highlight none >}} +replay.client [flow] +{{< /highlight >}} + + +That means that it expects a sequence of one or more flows. This is where [flow +specifications]({{< relref concepts-filters >}}) come in - mitmproxy will +intelligently expand a flexible flow selection language to a list of flows when +invoking commands. + +Fire up mitmproxy console, and intercept some traffic so we have flows to work +with. Now type the following command: + +{{< highlight none >}} +:replay.client @focus +{{< /highlight >}} + +Make sure you try using tab completion for the command name and the flow +specification. The `@focus` specifiers expands to the currently focused flow, so +you should see this flow replay. However, replay can take any number of flows. +Try the following command: + +{{< highlight none >}} +:replay.client @all +{{< /highlight >}} + +Now you should see all flows replay one by one. We have the full power of the +mitmproxy filter language at our disposal here, so we could also, for example, +just replay flows for a specific domain: + +{{< highlight none >}} +:replay.client "~d google.com" +{{< /highlight >}} + + + + + + + + + + diff --git a/docs/src/content/concepts-filters.md b/docs/src/content/concepts-filters.md new file mode 100644 index 00000000..ada24e32 --- /dev/null +++ b/docs/src/content/concepts-filters.md @@ -0,0 +1,81 @@ +--- +title: "Filter expressions" +menu: + concepts: + weight: 4 +--- + +# Filter expressions + +Many commands in the mitmproxy tool make use of filter expressions. Filter +expressions consist of the following operators: + + +<table class="table filtertable"><tbody> +<tr><th>~a</th><td>Match asset in response: CSS, Javascript, Flash, images.</td></tr> +<tr><th>~b regex</th><td>Body</td></tr> +<tr><th>~bq regex</th><td>Request body</td></tr> +<tr><th>~bs regex</th><td>Response body</td></tr> +<tr><th>~c int</th><td>HTTP response code</td></tr> +<tr><th>~d regex</th><td>Domain</td></tr> +<tr><th>~dst regex</th><td>Match destination address</td></tr> +<tr><th>~e</th><td>Match error</td></tr> +<tr><th>~h regex</th><td>Header</td></tr> +<tr><th>~hq regex</th><td>Request header</td></tr> +<tr><th>~hs regex</th><td>Response header</td></tr> +<tr><th>~http</th><td>Match HTTP flows</td></tr> +<tr><th>~m regex</th><td>Method</td></tr> +<tr><th>~marked</th><td>Match marked flows</td></tr> +<tr><th>~q</th><td>Match request with no response</td></tr> +<tr><th>~s</th><td>Match response</td></tr> +<tr><th>~src regex</th><td>Match source address</td></tr> +<tr><th>~t regex</th><td>Content-type header</td></tr> +<tr><th>~tcp</th><td>Match TCP flows</td></tr> +<tr><th>~tq regex</th><td>Request Content-Type header</td></tr> +<tr><th>~ts regex</th><td>Response Content-Type header</td></tr> +<tr><th>~u regex</th><td>URL</td></tr> +<tr><th>~websocket</th><td>Match WebSocket flows</td></tr> +<tr><th>!</th><td>unary not</td></tr> +<tr><th>&</th><td>and</td></tr> +<tr><th>|</th><td>or</td></tr> +<tr><th>(...)</th><td>grouping</td></tr> +</tbody></table> + + +- Regexes are Python-style +- Regexes can be specified as quoted strings +- Header matching (~h, ~hq, ~hs) is against a string of the form "name: value". +- Strings with no operators are matched against the request URL. +- The default binary operator is &. + + +## View flow selectors + +In interactive contexts, mitmproxy has a set of convenient flow selectors that +operate on the current view: + +<table class="table filtertable"><tbody> +<tr><th>@all</th><td>All flows</td></tr> +<tr><th>@focus</th><td>The currently focused flow</td></tr> +<tr><th>@shown</th><td>All flows currently shown</td></tr> +<tr><th>@hidden</th><td>All flows currently hidden</td></tr> +<tr><th>@marked</th><td>All marked flows</td></tr> +<tr><th>@unmarked</th><td>All unmarked flows</td></tr> +</tbody></table> + +These are frequently used in commands and key bindings. + + +## Examples + +URL containing "google.com": + + google\.com + +Requests whose body contains the string "test": + + ~q ~b test + +Anything but requests with a text/html content type: + + !(~q & ~t "text/html") diff --git a/docs/src/content/concepts-howmitmproxyworks.md b/docs/src/content/concepts-howmitmproxyworks.md new file mode 100644 index 00000000..d747d71c --- /dev/null +++ b/docs/src/content/concepts-howmitmproxyworks.md @@ -0,0 +1,252 @@ +--- +title: "How mitmproxy works" +menu: + concepts: + weight: 1 +--- + +# How mitmproxy works + +Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying +process works will help you deploy it creatively, and take into account its +fundamental assumptions and how to work around them. This document explains +mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted +explicit proxying, and working up to the most complicated interaction +-transparent proxying of TLS-protected traffic[^1] in the presence of [Server +Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication). + +## Explicit HTTP + +Configuring the client to use mitmproxy as an explicit proxy is the +simplest and most reliable way to intercept traffic. The proxy protocol +is codified in the [HTTP RFC](https://tools.ietf.org/html/rfc7230), so +the behaviour of both the client and the server is well defined, and +usually reliable. In the simplest possible interaction with mitmproxy, a +client connects directly to the proxy, and makes a request that looks +like this: + +{{< highlight http >}} +GET http://example.com/index.html HTTP/1.1 +{{< / highlight >}} + +This is a proxy GET request - an extended form of the vanilla HTTP GET +request that includes a schema and host specification, and it includes +all the information mitmproxy needs to proceed. + + +{{< figure src="/schematics/how-mitmproxy-works-explicit.png" title="Explicit" >}} + + +1. The client connects to the proxy and makes a request. +2. Mitmproxy connects to the upstream server and simply forwards the request on. + + +## Explicit HTTPS + +The process for an explicitly proxied HTTPS connection is quite +different. The client connects to the proxy and makes a request that +looks like this: + +{{< highlight http >}} +CONNECT example.com:443 HTTP/1.1 +{{< / highlight >}} + +A conventional proxy can neither view nor manipulate a TLS-encrypted +data stream, so a CONNECT request simply asks the proxy to open a pipe +between the client and server. The proxy here is just a facilitator - it +blindly forwards data in both directions without knowing anything about +the contents. The negotiation of the TLS connection happens over this +pipe, and the subsequent flow of requests and responses are completely +opaque to the proxy. + +### The MITM in mitmproxy + +This is where mitmproxy's fundamental trick comes into play. The MITM +in its name stands for Man-In-The-Middle - a reference to the process we +use to intercept and interfere with these theoretically opaque data +streams. The basic idea is to pretend to be the server to the client, +and pretend to be the client to the server, while we sit in the middle +decoding traffic from both sides. The tricky part is that the +[Certificate +Authority](https://en.wikipedia.org/wiki/Certificate_authority) system +is designed to prevent exactly this attack, by allowing a trusted +third-party to cryptographically sign a server's certificates to verify +that they are legit. If this signature doesn't match or is from a +non-trusted party, a secure client will simply drop the connection and +refuse to proceed. Despite the many shortcomings of the CA system as it +exists today, this is usually fatal to attempts to MITM a TLS connection +for analysis. Our answer to this conundrum is to become a trusted +Certificate Authority ourselves. Mitmproxy includes a full CA +implementation that generates interception certificates on the fly. To +get the client to trust these certificates, we +[register mitmproxy as a trusted CA with the device +manually]({{< relref concepts-certificates >}}). + +### Complication 1: What's the remote hostname? + +To proceed with this plan, we need to know the domain name to use in the +interception certificate - the client will verify that the certificate is for +the domain it's connecting to, and abort if this is not the case. At first +blush, it seems that the CONNECT request above gives us all we need - in this +example, both of these values are "example.com". But what if the client had +initiated the connection as follows: + +{{< highlight http >}} +CONNECT 10.1.1.1:443 HTTP/1.1 +{{< / highlight >}} + +Using the IP address is perfectly legitimate because it gives us enough +information to initiate the pipe, even though it doesn't reveal the +remote hostname. + +Mitmproxy has a cunning mechanism that smooths this over - [upstream certificate +sniffing]({{< relref "overview-features#upstream-certificates" >}}). As soon as +we see the CONNECT request, we pause the client part of the conversation, and +initiate a simultaneous connection to the server. We complete the TLS handshake +with the server, and inspect the certificates it used. Now, we use the Common +Name in the upstream certificates to generate the dummy certificate for the +client. Voila, we have the correct hostname to present to the client, even if it +was never specified. + +### Complication 2: Subject Alternative Name + +Enter the next complication. Sometimes, the certificate Common Name is +not, in fact, the hostname that the client is connecting to. This is +because of the optional [Subject Alternative +Name](https://en.wikipedia.org/wiki/SubjectAltName) field in the +certificate that allows an arbitrary number of alternative domains to be +specified. If the expected domain matches any of these, the client will +proceed, even though the domain doesn't match the certificate CN. The +answer here is simple: when we extract the CN from the upstream cert, we +also extract the SANs, and add them to the generated dummy certificate. + +### Complication 3: Server Name Indication + +One of the big limitations of vanilla TLS is that each certificate +requires its own IP address. This means that you couldn't do virtual +hosting where multiple domains with independent certificates share the +same IP address. In a world with a rapidly shrinking IPv4 address pool +this is a problem, and we have a solution in the form of the [Server +Name Indication](https://en.wikipedia.org/wiki/Server_Name_Indication) +extension to the TLS protocols. This lets the client specify the remote +server name at the start of the TLS handshake, which then lets the +server select the right certificate to complete the process. + +SNI breaks our upstream certificate sniffing process, because when we +connect without using SNI, we get served a default certificate that may +have nothing to do with the certificate expected by the client. The +solution is another tricky complication to the client connection +process. After the client connects, we allow the TLS handshake to +continue until just **after** the SNI value has been passed to us. Now +we can pause the conversation, and initiate an upstream connection using +the correct SNI value, which then serves us the correct upstream +certificate, from which we can extract the expected CN and SANs. + +### Putting it all together + +Lets put all of this together into the complete explicitly proxied HTTPS +flow. + +{{< figure src="/schematics/how-mitmproxy-works-explicit-https.png" title="Explicit HTTPS" >}} + +1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT + request. +2. Mitmproxy responds with a `200 Connection Established`, as if it has set up + the CONNECT pipe. +3. The client believes it's talking to the remote server, and initiates the TLS + connection. It uses SNI to indicate the hostname it is connecting to. +4. Mitmproxy connects to the server, and establishes a TLS connection using the + SNI hostname indicated by the client. +5. The server responds with the matching certificate, which contains the CN and + SAN values needed to generate the interception certificate. +6. Mitmproxy generates the interception cert, and continues the client TLS + handshake paused in step 3. +7. The client sends the request over the established TLS connection. +8. Mitmproxy passes the request on to the server over the TLS connection + initiated in step 4. + +## Transparent HTTP + +When a transparent proxy is used, the connection is redirected into a +proxy at the network layer, without any client configuration being +required. This makes transparent proxying ideal for those situations +where you can't change client behaviour - proxy-oblivious Android +applications being a common example. + +To achieve this, we need to introduce two extra components. The first is +a redirection mechanism that transparently reroutes a TCP connection +destined for a server on the Internet to a listening proxy server. This +usually takes the form of a firewall on the same host as the proxy +server - [iptables](http://www.netfilter.org/) on Linux or +[pf](https://en.wikipedia.org/wiki/PF_\(firewall\)) on OSX. Once the +client has initiated the connection, it makes a vanilla HTTP request, +which might look something like this: + +{{< highlight http >}} +GET /index.html HTTP/1.1 +{{< / highlight >}} + +Note that this request differs from the explicit proxy variation, in +that it omits the scheme and hostname. How, then, do we know which +upstream host to forward the request to? The routing mechanism that has +performed the redirection keeps track of the original destination for +us. Each routing mechanism has a different way of exposing this data, so +this introduces the second component required for working transparent +proxying: a host module that knows how to retrieve the original +destination address from the router. In mitmproxy, this takes the form +of a built-in set of +[modules](https://github.com/mitmproxy/mitmproxy/tree/master/mitmproxy/platform) +that know how to talk to each platform's redirection mechanism. Once we +have this information, the process is fairly straight-forward. + +{{< figure src="/schematics/how-mitmproxy-works-transparent.png" title="Transparent" >}} + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically + listening on a local port of the same host. Mitmproxy then consults + the routing mechanism to establish what the original destination + was. +3. Now, we simply read the client's request... +4. ... and forward it upstream. + +## Transparent HTTPS + +The first step is to determine whether we should treat an incoming +connection as HTTPS. The mechanism for doing this is simple - we use the +routing mechanism to find out what the original destination port is. All +incoming connections pass through different layers which can determine +the actual protocol to use. Automatic TLS detection works for SSLv3, TLS +1.0, TLS 1.1, and TLS 1.2 by looking for a *ClientHello* message at the +beginning of each connection. This works independently of the used TCP +port. + +From here, the process is a merger of the methods we've described for +transparently proxying HTTP, and explicitly proxying HTTPS. We use the +routing mechanism to establish the upstream server address, and then +proceed as for explicit HTTPS connections to establish the CN and SANs, +and cope with SNI. + +{{< figure src="/schematics/how-mitmproxy-works-transparent-https.png" title="Transparent HTTPS" >}} + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically + listening on a local port of the same host. Mitmproxy then consults the + routing mechanism to establish what the original destination was. +3. The client believes it's talking to the remote server, and + initiates the TLS connection. It uses SNI to indicate the hostname + it is connecting to. +4. Mitmproxy connects to the server, and establishes a TLS connection + using the SNI hostname indicated by the client. +5. The server responds with the matching certificate, which contains + the CN and SAN values needed to generate the interception + certificate. +6. Mitmproxy generates the interception cert, and continues the client + TLS handshake paused in step 3. +7. The client sends the request over the established TLS connection. +8. Mitmproxy passes the request on to the server over the TLS + connection initiated in step 4. + +### Footnotes + +[^1]: The use of "TLS" refers to both SSL (outdated and insecure) and TLS (1.0 + and up) in the generic sense, unless otherwise specified. diff --git a/docs/src/content/concepts-modes.md b/docs/src/content/concepts-modes.md new file mode 100644 index 00000000..86bb7b0f --- /dev/null +++ b/docs/src/content/concepts-modes.md @@ -0,0 +1,215 @@ +--- +title: "Modes of operation" +menu: + concepts: + weight: 2 +--- + +# Modes of Operation + +- [Regular](#regular-proxy) (the default) +- [Transparent](#transparent-proxy) +- [Reverse Proxy](#reverse-proxy) +- [Upstream Proxy](#upstream-proxy) +- [SOCKS Proxy](#socks-proxy) + +Now, which one should you pick? Use this flow chart: + +{{< figure src="/schematics/proxy-modes-flowchart.png" >}} + + +## Regular Proxy + +Mitmproxy's regular mode is the simplest and the easiest to set up. + +1. Start mitmproxy. +2. Configure your client to use mitmproxy by explicitly setting an HTTP + proxy. +3. Quick Check: You should already be able to visit an unencrypted HTTP + site through the proxy. +4. Open the magic domain **mitm.it** and install the certificate for your + device. + +{{< note >}} +Unfortunately, some applications bypass the system HTTP proxy settings - +Android applications are a common example. In these cases, you need to +use mitmproxy's transparent mode. +{{< /note >}} + +If you are proxying an external device, your network will probably look +like this: + +{{< figure src="/schematics/proxy-modes-regular.png" >}} + +The square brackets signify the source and destination IP addresses. +Your client explicitly connects to mitmproxy and mitmproxy explicitly +connects to the target server. + +## Transparent Proxy + +In transparent mode, traffic is directed into a proxy at the network +layer, without any client configuration required. This makes transparent +proxying ideal for situations where you can't change client behaviour. +In the graphic below, a machine running mitmproxy has been inserted +between the router and the internet: + +{{< figure src="/schematics/proxy-modes-transparent-1.png" >}} + +The square brackets signify the source and destination IP addresses. +Round brackets mark the next hop on the *Ethernet/data link* layer. This +distinction is important: when the packet arrives at the mitmproxy +machine, it must still be addressed to the target server. This means +that Network Address Translation should not be applied before the +traffic reaches mitmproxy, since this would remove the target +information, leaving mitmproxy unable to determine the real destination. + +{{< figure src="/schematics/proxy-modes-transparent-wrong.png" title="Modes Transparent Wrong" >}} + +### Common Configurations + +There are many ways to configure your network for transparent proxying. +We'll look at two common scenarios: + +1. Configuring the client to use a custom gateway/router/"next hop" +2. Implementing custom routing on the router + +In most cases, the first option is recommended due to its ease of use. + +#### (a) Custom Gateway + +One simple way to get traffic to the mitmproxy machine with the +destination IP intact, is to simply configure the client with the +mitmproxy box as the default gateway. + +{{< figure src="/schematics/proxy-modes-transparent-2.png" >}} + +In this scenario, we would: + +1. Configure the proxy machine for transparent mode. You can find instructions + in the [transparent]({{< relref "howto-transparent" + >}}) section. +2. Configure the client to use the proxy machine's IP as the default gateway. +3. Quick Check: At this point, you should already be able to visit an + unencrypted HTTP site over the proxy. +4. Open the magic domain **mitm.it** and install the certificate for your + device. + +Setting the custom gateway on clients can be automated by serving the +settings out to clients over DHCP. This lets set up an interception +network where all clients are proxied automatically, which can save time +and effort. + +{{% note %}} + +### Troubleshooting Transparent Mode + +Incorrect transparent mode configurations are a frequent source of +error. If it doesn't work for you, try the following things: + +- Open mitmproxy's event log - do you see clientconnect messages? If not, the + packets are not arriving at the proxy. One common cause is the occurrence of + ICMP redirects, which means that your machine is telling the client that + there's a faster way to the internet by contacting your router directly (see + the [transparent]({{< relref "howto-transparent" + >}}) section on how to disable them). If in doubt, + [Wireshark](https://wireshark.org/) may help you to see whether something + arrives at your machine or not. +- Make sure you have not explicitly configured an HTTP proxy on the client. This + is not needed in transparent mode. +- Re-check the instructions in the [transparent]({{< relref "howto-transparent" + >}}) section. Anything you missed? + +If you encounter any other pitfalls that should be listed here, please +let us know! +{{% /note %}} + +#### (b) Custom Routing + +In some cases, you may need more fine-grained control of which traffic +reaches the mitmproxy instance, and which doesn't. You may, for +instance, choose only to divert traffic to some hosts into the +transparent proxy. There are a huge number of ways to accomplish this, +and much will depend on the router or packet filter you're using. In +most cases, the configuration will look like this: + +{{< figure src="/schematics/proxy-modes-transparent-3.png" >}} + +## Reverse Proxy + +mitmproxy is usually used with a client that uses the proxy to access +the Internet. Using reverse proxy mode, you can use mitmproxy to act +like a normal HTTP server: + +{{< figure src="/schematics/proxy-modes-reverse.png" >}} + +There are various use-cases: + +- Say you have an internal API running at <http://example.local/>. You could now + set up mitmproxy in reverse proxy mode at <http://debug.example.local/> and + dynamically point clients to this new API endpoint, which provides them with + the same data and you with debug information. Similarly, you could move your + real server to a different IP/port and set up mitmproxy in the original + place to debug and or redirect all sessions. +- Say you're a web developer working on <http://example.com/> (with a + development version running on <http://localhost:8000/>). You can modify + your hosts file so that example.com points to 127.0.0.1 and then run + mitmproxy in reverse proxy mode on port 80. You can test your app on the + example.com domain and get all requests recorded in mitmproxy. +- Say you have some toy project that should get SSL support. Simply set up + mitmproxy as a reverse proxy on port 443 and you're done (`mitmdump -p 443 + -R http://localhost:80/`). Mitmproxy auto-detects TLS traffic and intercepts + it dynamically. There are better tools for this specific task, but mitmproxy + is very quick and simple way to set up an SSL-speaking server. +- Want to add a non-SSL-capable compression proxy in front of your server? You + could even spawn a mitmproxy instance that terminates SSL (`-R http://...`), + point it to the compression proxy and let the compression proxy point to a + SSL-initiating mitmproxy (`-R https://...`), which then points to the real + server. As you see, it's a fairly flexible thing. + +### Host Header + +In reverse proxy mode, mitmproxy automatically rewrites the Host header to match +the upstream server. This allows mitmproxy to easily connect to existing +endpoints on the open web (e.g. `mitmproxy -R https://example.com`). You can +disable this behaviour with the `keep_host_header` option. + +However, keep in mind that absolute URLs within the returned document or HTTP +redirects will NOT be rewritten by mitmproxy. This means that if you click on a +link for "<http://example.com>" in the returned web page, you will be taken +directly to that URL, bypassing mitmproxy. + +One possible way to address this is to modify the hosts file of your OS so that +"example.com" resolves to your proxy's IP, and then access the proxy by going +directly to example.com. Make sure that your proxy can still resolve the +original IP, or specify an IP in mitmproxy. + + +{{% note %}} + +### Caveat: Interactive Use + +Reverse Proxy mode is usually not sufficient to create a copy of an +interactive website at different URL. The HTML served to the client +remains unchanged - as soon as the user clicks on an non-relative URL +(or downloads a non-relative image resource), traffic no longer passes +through mitmproxy. +{{% /note %}} + + +## Upstream Proxy + +If you want to chain proxies by adding mitmproxy in front of a different +proxy appliance, you can use mitmproxy's upstream mode. In upstream +mode, all requests are unconditionally transferred to an upstream proxy +of your choice. + +{{< figure src="/schematics/proxy-modes-upstream.png" >}} + +mitmproxy supports both explicit HTTP and explicit HTTPS in upstream +proxy mode. You could in theory chain multiple mitmproxy instances in a +row, but that doesn't make any sense in practice (i.e. outside of our +tests). + + +## SOCKS Proxy + diff --git a/docs/src/content/concepts-options.md b/docs/src/content/concepts-options.md new file mode 100644 index 00000000..18776841 --- /dev/null +++ b/docs/src/content/concepts-options.md @@ -0,0 +1,66 @@ +--- +title: "Options" +menu: + concepts: + weight: 5 +--- + +# Options + +The mitmproxy tools share a common [YAML](http://yaml.org/) configuration file +located at `~/.mitmproxy/config.yaml`. This file controls **options** - typed +values that determine the behaviour of mitmproxy. The options mechanism is very +comprehensive - in fact, options control all of mitmproxy's runtime behaviour. +Most command-line flags are simply aliases for underlying options, and +interactive settings changes made in **mitmproxy** and **mitmweb** just change +values in our runtime options store. This means that almost any facet of +mitmproxy's behaviour can be controlled through options. + +The canonical reference for options is the `--options` flag, which is exposed by +each of the mitmproxy tools. Passing this flag will dump an annotated YAML +configuration to console, which includes all options and their default values. + +The options mechanism is extensible - third-party addons can define options that +are treated exactly like mitmproxy's own. This means that addons can also be +configured through the central configuration file, and their options will appear +in the options editors in interactive tools. + + +## Tools + +Both **mitmproxy** and **mitmweb** have built-in editors that let you view and +manipulate the complete configuration state of mitmproxy. Values you change +interactively have immediate effect in the running instance, and can be made +persistent by saving the settings out to a YAML configuration file (please see +the specific tool's interactive help for details on how to do this). + +For all tools, options can be set directly by name using the `--set` command-line +option. Please see the command-line help (`--help`) for usage. + + +## Example + +Here is an excerpt showing the first few lines of the complete `--options` +output: + +{{< highlight yaml >}} + +# Add all certificates of the upstream server to the certificate chain +# that will be served to the proxy client, as extras. Type bool. +add_upstream_certs_to_client_chain: false + +# Allow remote clients to connect to proxy. If set to false, client will +# not be able to connect to proxy unless it is on the same network or +# the proxyauth option is set Type bool. +allow_remote: false + +# Strip out request headers that might cause the server to return +# 304-not-modified. Type bool. +anticache: false + +# Try to convince servers to send us un-compressed data. Type bool. +anticomp: false + +{{< /highlight >}} + + diff --git a/docs/src/content/howto-ignoredomains.md b/docs/src/content/howto-ignoredomains.md new file mode 100644 index 00000000..b1b14831 --- /dev/null +++ b/docs/src/content/howto-ignoredomains.md @@ -0,0 +1,102 @@ +--- +title: "Ignoring Domains" +menu: + howto: + weight: 2 +--- + +# Ignoring Domains + +There are two main reasons why you may want to exempt some traffic from +mitmproxy's interception mechanism: + +- **Certificate pinning:** Some traffic is is protected using [Certificate + Pinning](https://security.stackexchange.com/questions/29988/what-is-certificate-pinning) + and mitmproxy's interception leads to errors. For example, the Twitter app, + Windows Update or the Apple App Store fail to work if mitmproxy is active. +- **Convenience:** You really don't care about some parts of the traffic and + just want them to go away. Note that mitmproxy's "Limit" option is often the + better alternative here, as it is not affected by the limitations listed + below. + +If you want to peek into (SSL-protected) non-HTTP connections, check out the +**tcp_proxy** feature. If you want to ignore traffic from mitmproxy's processing +because of large response bodies, take a look at the [streaming]({{< relref "overview-features#streaming" >}}) feature. + + +## ignore_hosts + +The `ignore_hosts` option allows you to specify a regex which is matched against +a `host:port` string (e.g. "example.com:443") of a connection. Matching hosts +are excluded from interception, and passed on unmodified. + +| | | +| ------------------ | ------------------------------------------------------------------ | +| command-line alias | `--ignore-hosts regex` | +| mitmproxy option | `ignore_hosts` | + + +## Limitations + +There are two important quirks to consider: + +- **In transparent mode, the ignore pattern is matched against the IP and + ClientHello SNI host.** While we usually infer the hostname from the Host + header if the `ignore_hosts` option is set, we do not have access to this + information before the SSL handshake. If the client uses SNI however, then we + treat the SNI host as an ignore target. +- **In regular and upstream proxy mode, explicit HTTP requests are never + ignored.**\[1\] The ignore pattern is applied on CONNECT requests, which + initiate HTTPS or clear-text WebSocket connections. + +## Tutorial + +If you just want to ignore one specific domain, there's usually a bulletproof +method to do so: + +1. Run mitmproxy or mitmdump in verbose mode (`-v`) and observe the `host:port` + information in the serverconnect messages. mitmproxy will filter on these. +2. Take the `host:port` string, surround it with ^ and $, escape all dots (. + becomes \\.) and use this as your ignore pattern: + + +{{< highlight none >}} +>>> mitmdump -v +127.0.0.1:50588: clientconnect +127.0.0.1:50588: request + -> CONNECT example.com:443 HTTP/1.1 +127.0.0.1:50588: Set new server address: example.com:443 +127.0.0.1:50588: serverconnect + -> example.com:443 +^C +>>> mitmproxy --ignore-hosts ^example\.com:443$ +{{< /highlight >}} + +Here are some other examples for ignore +patterns: + +{{< highlight none >}} +# Exempt traffic from the iOS App Store (the regex is lax, but usually just works): +--ignore-hosts apple.com:443 +# "Correct" version without false-positives: +--ignore-hosts '^(.+\.)?apple\.com:443$' + +# Ignore example.com, but not its subdomains: +--ignore-hosts '^example.com:' + +# Ignore everything but example.com and mitmproxy.org: +--ignore-hosts '^(?!example\.com)(?!mitmproxy\.org)' + +# Transparent mode: +--ignore-hosts 17\.178\.96\.59:443 +# IP address range: +--ignore-hosts 17\.178\.\d+\.\d+:443 +{{< / highlight >}} + +**Footnotes** + +1. This stems from an limitation of explicit HTTP proxying: A single connection + can be re-used for multiple target domains - a `GET http://example.com/` + request may be followed by a `GET http://evil.com/` request on the same + connection. If we start to ignore the connection after the first request, we + would miss the relevant second one. diff --git a/docs/src/content/howto-transparent-vms.md b/docs/src/content/howto-transparent-vms.md new file mode 100644 index 00000000..b186fd39 --- /dev/null +++ b/docs/src/content/howto-transparent-vms.md @@ -0,0 +1,114 @@ +--- +title: "Transparently Proxying VMs" +menu: + howto: + weight: 3 +--- + +# Transparently proxify virtual machines + +This walkthrough illustrates how to set up transparent proxying with +mitmproxy. We use VirtualBox VMs with an Ubuntu proxy machine in this +example, but the general *Internet \<--\> Proxy VM \<--\> (Virtual) +Internal Network* setup can be applied to other setups. + +## 1. Configure Proxy VM + +On the proxy machine, **eth0** is connected to the internet. **eth1** is +connected to the internal network that will be proxified and configured +to use a static ip (192.168.3.1). + +### VirtualBox configuration + + +{{< figure src="/transparent-vms/step1_vbox_eth0.png" >}} + +{{< figure src="/transparent-vms/step1_vbox_eth1.png" >}} + + +### VM Network Configuration + +{{< figure src="/transparent-vms/step1_proxy.png" >}} + +## 2. Configure DHCP and DNS + +We use dnsmasq to provide DHCP and DNS in our internal network. Dnsmasq is a +lightweight server designed to provide DNS (and optionally DHCP and TFTP) +services to a small-scale network. Before we get to that, we need to fix some +Ubuntu quirks: **Ubuntu \>12.04** runs an internal dnsmasq instance (listening +on loopback only) by default +[\[1\]](https://www.stgraber.org/2012/02/24/dns-in-ubuntu-12-04/). For our use +case, this needs to be disabled by changing `dns=dnsmasq` to `#dns=dnsmasq` in +**/etc/NetworkManager/NetworkManager.conf** and if on Ubuntu 16.04 or newer +running: + +{{< highlight bash >}} +sudo systemctl restart NetworkManager +{{< / highlight >}} + +If on Ubuntu 12.04 or 14.04 running: + +{{< highlight bash >}} +sudo restart network-manager +{{< / highlight >}} + +afterwards. + +Now, dnsmasq can be be installed and configured: + +{{< highlight bash >}} +sudo apt-get install dnsmasq +{{< / highlight >}} + +Replace **/etc/dnsmasq.conf** with the following configuration: + +{{< highlight none >}} +# Listen for DNS requests on the internal network +interface=eth1 +# Act as a DHCP server, assign IP addresses to clients +dhcp-range=192.168.3.10,192.168.3.100,96h +# Broadcast gateway and dns server information +dhcp-option=option:router,192.168.3.1 +dhcp-option=option:dns-server,192.168.3.1 +{{< / highlight >}} + +Apply changes: + +If on Ubuntu 16.04 or newer: + +{{< highlight bash >}} +sudo systemctl restart dnsmasq +{{< / highlight >}} + +If on Ubuntu 12.04 or 14.04: + +{{< highlight bash >}} +sudo service dnsmasq restart +{{< / highlight >}} + +Your **proxied machine** in the internal virtual network should now receive an +IP address via DHCP: + +{{< figure src="/transparent-vms/step2_proxied_vm.png" >}} + +## 3. Redirect traffic to mitmproxy + +To redirect traffic to mitmproxy, we need to add two iptables +rules: + +{{< highlight bash >}} +sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 80 -j REDIRECT --to-port 8080 +sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-port 8080 +{{< / highlight >}} + +## 4. Run mitmproxy + +Finally, we can run mitmproxy in transparent mode with + +{{< highlight bash >}} +mitmproxy -T +{{< / highlight >}} + +The proxied machine cannot to leak any data outside of HTTP or DNS requests. If +required, you can now [install the mitmproxy certificates on the proxied +machine]({{< relref "concepts-certificates" >}}). diff --git a/docs/src/content/howto-transparent.md b/docs/src/content/howto-transparent.md new file mode 100644 index 00000000..224cb5ee --- /dev/null +++ b/docs/src/content/howto-transparent.md @@ -0,0 +1,269 @@ +--- +title: "Transparent Proxying" +menu: + howto: + weight: 1 +--- + +# Transparent Proxying + +When a transparent proxy is used, traffic is redirected into a proxy at the +network layer, without any client configuration being required. This makes +transparent proxying ideal for those situations where you can't change client +behaviour - proxy-oblivious mobile applications being a common example. + +To set up transparent proxying, we need two new components. The first is a +redirection mechanism that transparently reroutes a TCP connection destined for +a server on the Internet to a listening proxy server. This usually takes the +form of a firewall on the same host as the proxy server - +[iptables](http://www.netfilter.org/) on Linux or +[pf](https://en.wikipedia.org/wiki/PF_(firewall)) on OSX. When the proxy +receives a redirected connection, it sees a vanilla HTTP request, without a host +specification. This is where the second new component comes in - a host module +that allows us to query the redirector for the original destination of the TCP +connection. + +At the moment, mitmproxy supports transparent proxying on OSX Lion and above, +and all current flavors of Linux. + + +## Linux fully transparent mode + +By default mitmproxy will use its own local IP address for its server-side +connections. In case this isn't desired, the --spoof-source-address argument can +be used to use the client's IP address for server-side connections. The +following config is required for this mode to work: + +{{< highlight bash >}} +CLIENT_NET=192.168.1.0/24 +TABLE_ID=100 +MARK=1 + +echo "$TABLE_ID mitmproxy" >> /etc/iproute2/rt_tables +iptables -t mangle -A PREROUTING -d $CLIENT_NET -j MARK --set-mark $MARK +iptables -t nat \ + -A PREROUTING -p tcp -s $CLIENT_NET \ + --match multiport --dports 80,443 -j \ + REDIRECT --to-port 8080 + +ip rule add fwmark $MARK lookup $TABLE_ID +ip route add local $CLIENT_NET dev lo table $TABLE_ID +{{< / highlight >}} + +This mode does require root privileges though. There's a wrapper in the examples +directory called 'mitmproxy_shim.c', which will enable you to use this mode with +dropped privileges. It can be used as follows: + +{{< highlight bash >}} +gcc examples/complex/full_transparency_shim.c -o mitmproxy_shim -lcap +sudo chown root:root mitmproxy_shim +sudo chmod u+s mitmproxy_shim +./mitmproxy_shim $(which mitmproxy) -T --spoof-source-address +{{< / highlight >}} + + + +## Linux + +On Linux, mitmproxy integrates with the iptables redirection mechanism to +achieve transparent mode. + +### 1. [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding: + +{{< highlight bash >}} +sysctl -w net.ipv4.ip_forward=1 +sysctl -w net.ipv6.conf.all.forwarding=1 +{{< / highlight >}} + +You may also want to consider enabling this permanently in `/etc/sysctl.conf` or +newly created `/etc/sysctl.d/mitmproxy.conf`, see +[here](https://superuser.com/a/625852). + +### 3. If your target machine is on the same physical network and you configured it to use a custom gateway, disable ICMP redirects: + +{{< highlight bash >}} +sysctl -w net.ipv4.conf.all.send_redirects=0 +{{< / highlight >}} + +You may also want to consider enabling this permanently in `/etc/sysctl.conf` or +a newly created `/etc/sysctl.d/mitmproxy.conf`, see +[here](https://superuser.com/a/625852). + +### 4. Create an iptables ruleset that redirects the desired traffic to the mitmproxy port + +Details will differ according to your setup, but the ruleset should look +something like this: + +{{< highlight bash >}} + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080 + ip6tables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 + ip6tables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080 +{{< / highlight >}} + + You may also want to consider enabling this permanently with the +`iptables-persistent` package, see +[here](http://www.microhowto.info/howto/make_the_configuration_of_iptables_persistent_on_debian.html). + +### 5. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells + mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. For a detailed walkthrough, have a look at the [tutorial for +transparently proxying VMs]({{< relref "howto-transparent-vms" >}}). + + +## OpenBSD + +### 1 [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding + +{{< highlight bash >}} +sudo sysctl -w net.inet.ip.forwarding=1 +{{< / highlight >}} + +### 3. Place the following two lines in **/etc/pf.conf** + +{{< highlight none >}} +mitm_if = "re2" +pass in quick proto tcp from $mitm_if to port { 80, 443 } divert-to 127.0.0.1 port 8080 +{{< / highlight >}} + +These rules tell pf to divert all traffic from `$mitm_if` destined for port 80 +or 443 to the local mitmproxy instance running on port 8080. You should replace +`$mitm_if` value with the interface on which your test device will appear. + +### 4. Enable the pf ruleset and enable it + +{{< highlight bash >}} +doas pfctl -f /etc/pf.conf +{{< / highlight >}} + +And now enable it: + +{{< highlight bash >}} +doas pfctl -e +{{< / highlight >}} + +### 5. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells +mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. + + +{{% note %}} +Note that the **divert-to** rules in the pf.conf given above only apply +to inbound traffic. **This means that they will NOT redirect traffic +coming from the box running pf itself.** We can't distinguish between an +outbound connection from a non-mitmproxy app, and an outbound connection +from mitmproxy itself - if you want to intercept your traffic, you +should use an external host to run mitmproxy. Nonetheless, pf is +flexible to cater for a range of creative possibilities, like +intercepting traffic emanating from VMs. See the **pf.conf** man page +for more. +{{% /note %}} + + +## macOS + +OSX Lion integrated the [pf](https://en.wikipedia.org/wiki/PF_(firewall)) +packet filter from the OpenBSD project, which mitmproxy uses to implement +transparent mode on OSX. Note that this means we don't support transparent mode +for earlier versions of OSX. + +### 1. [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding + +{{< highlight bash >}} +sudo sysctl -w net.inet.ip.forwarding=1 +{{< / highlight >}} + +### 3. Place the following two lines in a file called, say, **pf.conf** + + +{{< highlight none >}} +rdr on en0 inet proto tcp to any port {80, 443} -> 127.0.0.1 port 8080 +{{< / highlight >}} + +These rules tell pf to redirect all traffic destined for port 80 or 443 +to the local mitmproxy instance running on port 8080. You should replace +`en2` with the interface on which your test device will appear. + +### 4. Configure pf with the rules + +{{< highlight bash >}} +sudo pfctl -f pf.conf +{{< / highlight >}} + +### 5. And now enable it + +{{< highlight bash >}} +sudo pfctl -e +{{< / highlight >}} + +### 6. Configure sudoers to allow mitmproxy to access pfctl + +Edit the file **/etc/sudoers** on your system as root. Add the following line to +the end of the file: + +{{< highlight none >}} +ALL ALL=NOPASSWD: /sbin/pfctl -s state +{{< / highlight >}} + +Note that this allows any user on the system to run the command `/sbin/pfctl -s +state` as root without a password. This only allows inspection of the state +table, so should not be an undue security risk. If you're special feel free to +tighten the restriction up to the user running mitmproxy. + +### 7. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells +mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. + +{{% note %}} +Note that the **rdr** rules in the pf.conf given above only apply to +inbound traffic. **This means that they will NOT redirect traffic coming +from the box running pf itself.** We can't distinguish between an +outbound connection from a non-mitmproxy app, and an outbound connection +from mitmproxy itself - if you want to intercept your OSX traffic, you +should use an external host to run mitmproxy. Nonetheless, pf is +flexible to cater for a range of creative possibilities, like +intercepting traffic emanating from VMs. See the **pf.conf** man page +for more. +{{% /note %}} diff --git a/docs/src/content/overview-features.md b/docs/src/content/overview-features.md new file mode 100644 index 00000000..9a4c5468 --- /dev/null +++ b/docs/src/content/overview-features.md @@ -0,0 +1,219 @@ +--- +title: "Features" +menu: "overview" +menu: + overview: + weight: 4 +--- + +# Mitmproxy Core Features + + +- [Anticache](#anticache) +- [Client-side replay](#client-side-replay) +- [Proxy Authentication](#proxy-authentication) +- [Replacements](#replacements) +- [Server-side replay](#server-side-replay) +- [Set Headers](#set-headers) +- [Sticky Auth](#sticky-auth) +- [Sticky Cookies](#sticky-cookies) +- [Streaming](#streaming) +- [Upstream Certificates](#upstream-certificates) + + +## Anticache + +When the `anticache` option is set, it removes headers (`if-none-match` and +`if-modified-since`) that might elicit a `304 not modified` response from the +server. This is useful when you want to make sure you capture an HTTP exchange +in its totality. It's also often used during client-side replay, when you want +to make sure the server responds with complete data. + + +## Client-side replay + +Client-side replay does what it says on the tin: you provide a previously saved +HTTP conversation, and mitmproxy replays the client requests one by one. Note +that mitmproxy serialises the requests, waiting for a response from the server +before starting the next request. This might differ from the recorded +conversation, where requests may have been made concurrently. + +You may want to use client-side replay in conjunction with the `anticache` +option, to make sure the server responds with complete data. + +## Proxy Authentication + +Asks the user for authentication before they are permitted to use the proxy. +Authentication headers are stripped from the flows, so they are not passed to +upstream servers. For now, only HTTP Basic authentication is supported. The +proxy auth options are not compatible with the transparent, socks or reverse +proxy mode. + + +## Replacements + +The `replacements` option lets you specify an arbitrary number of patterns that +define text replacements within flows. A replacement pattern looks like this: + +{{< highlight none >}} +/patt/regex/replacement +{{< / highlight >}} + +Here, **patt** is a mitmproxy filter expression that defines which flows a +replacement applies to, **regex** is a valid Python regular expression that +defines what gets replaced, and **replacement** is a string literal that is +substituted in. The separator is arbitrary, and defined by the first character. +If the replacement string literal starts with `@`, it is treated as a file path +from which the replacement is read. + +Replace hooks fire when either a client request or a server response is +received. Only the matching flow component is affected: so, for example, +if a replace hook is triggered on server response, the replacement is +only run on the Response object leaving the Request intact. You control +whether the hook triggers on the request, response or both using the +filter pattern. If you need finer-grained control than this, it's simple +to create a script using the replacement API on Flow components. + +### Examples + +Replace `foo` with `bar` in requests: + +{{< highlight none >}} +:~q:foo:bar +{{< / highlight >}} + +Replace `foo` with with the data read from `~/xss-exploit`: + +{{< highlight bash >}} +mitmdump --replacements :~q:foo:@~/xss-exploit +{{< / highlight >}} + + +## Server-side replay + +The `server_replay` option lets us replay server responses from saved HTTP +conversations. To do this, we use a set of heuristics to match incoming requests +with saved responses. By default, we exclude request headers when matching +incoming requests with responses from the replay file, and use only the URL and +request method for matching. This works in most circumstances, and makes it +possible to replay server responses in situations where request headers would +naturally vary, e.g. using a different user agent. + +There is a slew of ways to customise the matching heuristic, including +specifying headers to include, request parameters to exclude, etc. These options +are collected under the `server_replay` prefix - please see the built-in +documentation for details. + +### Response refreshing + +Simply replaying server responses without modification will often result in +unexpected behaviour. For example cookie timeouts that were in the future at the +time a conversation was recorded might be in the past at the time it is +replayed. By default, mitmproxy refreshes server responses before sending them +to the client. The **date**, **expires** and **last-modified** headers are all +updated to have the same relative time offset as they had at the time of +recording. So, if they were in the past at the time of recording, they will be +in the past at the time of replay, and vice versa. Cookie expiry times are +updated in a similar way. + +You can turn off this behaviour by setting the `server_replay_refresh` option to +`false`. + +### Replaying a session recorded in Reverse-proxy Mode + +If you have captured the session in reverse proxy mode, in order to replay it +you still have to specify the server URL, otherwise you may get the error: 'HTTP +protocol error in client request: Invalid HTTP request form (expected authority +or absolute...)'. + +During replay, when the client's requests match previously recorded requests, +then the respective recorded responses are simply replayed by mitmproxy. +Otherwise, the unmatched requests is forwarded to the upstream server. If +forwarding is not desired, you can use the --kill (-k) switch to prevent that. + +## Set Headers + +The `setheaders` option lets you specify a set of headers to be added to +requests or responses, based on a filter pattern. A `setheaders` expression +looks like this: + +{{< highlight none >}} +/patt/name/value +{{< / highlight >}} + +Here, **patt** is a mitmproxy filter expression that defines which flows to set +headers on, and **name** and **value** are the header name and the value to set +respectively. + +## Sticky auth + +The `stickyauth` option is analogous to the sticky cookie option, in that HTTP +**Authorization** headers are simply replayed to the server once they have been +seen. This is enough to allow you to access a server resource using HTTP Basic +authentication through the proxy. Note that <span +data-role="program">mitmproxy</span> doesn't (yet) support replay of HTTP Digest +authentication. + +## Sticky cookies + +When the `stickycookie` option is set, **mitmproxy** will add the cookie most +recently set by the server to any cookie-less request. Consider a service that +sets a cookie to track the session after authentication. Using sticky cookies, +you can fire up mitmproxy, and authenticate to a service as you usually would +using a browser. After authentication, you can request authenticated resources +through mitmproxy as if they were unauthenticated, because mitmproxy will +automatically add the session tracking cookie to requests. Among other things, +this lets you script interactions with authenticated resources (using tools like +wget or curl) without having to worry about authentication. + +Sticky cookies are especially powerful when used in conjunction with [client +replay]({{< relref "#client-side-replay" >}}) - you can record the +authentication process once, and simply replay it on startup every time you need +to interact with the secured resources. + +## Streaming + +By default, mitmproxy will read an entire request/response, perform any +indicated manipulations on it, and then send the message on to the other party. +This can be problematic when downloading or uploading large files. When +streaming is enabled, message bodies are not buffered on the proxy but instead +sent directly to the server/client. HTTP headers are still fully buffered before +being sent. + +Request/response streaming is enabled by specifying a size cutoff in the +`stream_large_bodies` option. + +### Customizing Streaming + +You can also use a script to customise exactly which requests or responses are +streamed. Requests/Responses that should be tagged for streaming by setting +their ``.stream`` attribute to ``True``: + +{{< example src="examples/complex/stream.py" lang="py" >}} + + +### Websockets + +The `stream_websockets` option enables an analogous behaviour for websockets. +When WebSocket streaming is enabled, portions of the code which may perform +changes to the WebSocket message payloads will not have any effect on the actual +payload sent to the server as the frames are immediately forwarded to the +server. In contrast to HTTP streaming, where the body is not stored, the message +payload will still be stored in the WebSocket flow. + +## Upstream Certificates + +When mitmproxy receives a connection destined for an SSL-protected service, it +freezes the connection before reading its request data, and makes a connection +to the upstream server to "sniff" the contents of its SSL certificate. The +information gained - the **Common Name** and **Subject Alternative Names** - is +then used to generate the interception certificate, which is sent to the client +so the connection can continue. + +This rather intricate little dance lets us seamlessly generate correct +certificates even if the client has specified only an IP address rather than the +hostname. It also means that we don't need to sniff additional data to generate +certs in transparent mode. + +Upstream cert sniffing is on by default, and can optionally be turned off with +the `upstream_cert` option.
\ No newline at end of file diff --git a/docs/src/content/overview-installation.md b/docs/src/content/overview-installation.md new file mode 100644 index 00000000..00941cb8 --- /dev/null +++ b/docs/src/content/overview-installation.md @@ -0,0 +1,117 @@ +--- +title: "Installation" +menu: "overview" +menu: + overview: + weight: 2 +--- + +# Installation + +Please follow the steps for your operating system. + +## macOS + +The recommended way to install mitmproxy on macOS is to use +[Homebrew](https://brew.sh/): + +{{< highlight bash >}} +brew install mitmproxy +{{< / highlight >}} + +Alternatively you can download the binary-packages from our +[releases](https://github.com/mitmproxy/mitmproxy/releases/latest) page. + +## Linux + +The recommended way to install mitmproxy on Linux is to download the +binary-packages from our +[releases](https://github.com/mitmproxy/mitmproxy/releases/latest) page. + +Some Linux distributions provide community-supported mitmproxy packages through +their native package repositories (e.g., Arch Linux, Debian, Ubuntu, Kali Linux, +OpenSUSE, etc.). We are not involved in the maintenance of downstream packaging +efforts, and they often lag behind the current mitmproxy release. Please contact +the repository maintainers directly for issues with native packages. + +## Windows + +The recommended way to install mitmproxy on Windows is to download our +binary-packages from our +[releases](https://github.com/mitmproxy/mitmproxy/releases/latest) page. + +FIXME: WSL + +After installation, you'll find shortcuts for mitmweb and mitmdump in the start +menu. Both executables are added to your PATH and can be invoked from the +command line. The console interface is not supported on native Windows. + + +# Advanced Installation + +## Self-contained Pre-built Binary Packages + +For some platforms we provide pre-built binary packages containing +ready-to-run executables. This includes a self-contained Python 3 +environment, a recent OpenSSL that support ALPN and HTTP/2, and other +dependencies that would otherwise we cumbersome to compile and install. + +Please be advised that we do not update these binaries after the initial +release. This means we do not include security-related updates of our +dependencies in already released mitmproxy versions. If there is a +severe issue, we might consider releasing a bugfix release of mitmproxy +and corresponding binary packages. + +We only support the latest version of mitmproxy with bugfix and security +updates through regular minor releases. + +## Docker Images + +You can use the official mitmproxy images from +[DockerHub](https://hub.docker.com/r/mitmproxy/mitmproxy/). The same +security considerations apply as for our binary packages. + +## Installation on Linux via pip3 + +Please make sure to install Python 3.5 (or higher) and pip3 for your +distribution. If your distribution does not provide a suitable Python +version, you can use [pyenv](https://github.com/yyuu/pyenv) to get a +recent Python environment. + +{{< highlight bash >}} +sudo apt install python3-pip # Debian 8 or higher, Ubuntu 16.04 or higher +sudo dnf install python3-pip # Fedora 24 or higher +sudo pacman -S python-pip # Arch Linux +{{< / highlight >}} + +Please make sure to upgrade pip3 itself: + +{{< highlight bash >}} +sudo pip3 install -U pip +{{< / highlight >}} + +Now you can install mitmproxy via pip3: + +{{< highlight bash >}} +sudo pip3 install mitmproxy +{{< / highlight >}} + +## Installation on Windows via pip3 + +First, install the latest version of Python 3.5 or higher from the +[Python website](https://www.python.org/downloads/windows/). During +installation, make sure to select Add Python to PATH. There are no other +dependencies on Windows. + +Now you can install mitmproxy via pip3: + +{{< highlight bash >}} +pip3 install mitmproxy +{{< / highlight >}} + +## Installation from Source + +If you would like to install mitmproxy directly from source code or the +GitHub master branch, please see the our +[README](https://github.com/mitmproxy/mitmproxy/blob/master/README.rst) +on GitHub. diff --git a/docs/src/content/overview-tools.md b/docs/src/content/overview-tools.md new file mode 100644 index 00000000..7612383a --- /dev/null +++ b/docs/src/content/overview-tools.md @@ -0,0 +1,105 @@ +--- +title: "Tools" +menu: "overview" +menu: + overview: + weight: 3 +--- + +# Overview + +You should thin of the mitmproxy project's tools as a set of front-ends that +expose the same underlying functionality. We aim to have feature parity across +all of our tooling, and all tools share a common configuration mechanism and +most command-line options. + +## mitmproxy + +{{< figure src="/screenshots/mitmproxy.png" >}} + +**mitmproxy** is a console tool that allows interactive examination and +modification of HTTP traffic. It differs from mitmdump in that all flows are +kept in memory, which means that it's intended for taking and manipulating +small-ish samples. Use the `?` shortcut key to view, context-sensitive +documentation from any **mitmproxy** screen. + + +## mitmweb + +{{< figure src="/screenshots/mitmweb.png" >}} + +**mitmweb** is mitmproxy's web-based user interface that allows +interactive examination and modification of HTTP traffic. Like +mitmproxy, it differs from mitmdump in that all flows are kept in +memory, which means that it's intended for taking and manipulating +small-ish samples. + +{{% note %}} +Mitmweb is currently in beta. We consider it stable for all features +currently exposed in the UI, but it still misses a lot of mitmproxy's +features. +{{% /note %}} + + +## mitmdump + +**mitmdump** is the command-line companion to mitmproxy. It provides +tcpdump-like functionality to let you view, record, and programmatically +transform HTTP traffic. See the `--help` flag output for complete +documentation. + + +### Example: Saving traffic + +{{< highlight bash >}} +mitmdump -w outfile +{{< / highlight >}} + +Start up mitmdump in proxy mode, and write all traffic to **outfile**. + +### Filtering saved traffic + +{{< highlight bash >}} +mitmdump -nr infile -w outfile "~m post" +{{< / highlight >}} + +Start mitmdump without binding to the proxy port (`-n`), read all flows +from infile, apply the specified filter expression (only match POSTs), +and write to outfile. + +### Client replay + +{{< highlight bash >}} +mitmdump -nc outfile +{{< / highlight >}} + +Start mitmdump without binding to the proxy port (`-n`), then replay all +requests from outfile (`-c filename`). Flags combine in the obvious way, +so you can replay requests from one file, and write the resulting flows +to another: + +{{< highlight bash >}} +mitmdump -nc srcfile -w dstfile +{{< / highlight >}} + +See the [client-side replay]({{< relref "overview-features#client-side-replay" +>}}) section for more information. + +### Running a script + +{{< highlight bash >}} +mitmdump -s examples/add_header.py +{{< / highlight >}} + +This runs the **add_header.py** example script, which simply adds a new +header to all responses. + +### Scripted data transformation + +{{< highlight bash >}} +mitmdump -ns examples/add_header.py -r srcfile -w dstfile +{{< / highlight >}} + +This command loads flows from **srcfile**, transforms it according to +the specified script, then writes it back to **dstfile**. + diff --git a/docs/src/content/tute-clientreplay.md b/docs/src/content/tute-clientreplay.md new file mode 100644 index 00000000..1bf69031 --- /dev/null +++ b/docs/src/content/tute-clientreplay.md @@ -0,0 +1,69 @@ +--- +title: "Client replay" +menu: + tutes: + weight: 1 +--- + +# Client playback: a 30 second example + +My local cafe is serviced by a rickety and unreliable wireless network, +generously sponsored with ratepayers' money by our city council. After +connecting, you are redirected to an SSL-protected page that prompts you for a +username and password. Once you've entered your details, you are free to enjoy +the intermittent dropouts, treacle-like speeds and incorrectly configured +transparent proxy. + +I tend to automate this kind of thing at the first opportunity, on the theory +that time spent now will be more than made up in the long run. In this case, I +might use [Firebug](https://getfirebug.com/) to ferret out the form post +parameters and target URL, then fire up an editor to write a little script using +Python's [urllib](https://docs.python.org/library/urllib.html) to simulate a +submission. That's a lot of futzing about. With mitmproxy we can do the job in +literally 30 seconds, without having to worry about any of the details. Here's +how. + +## 1. Run mitmdump to record our HTTP conversation to a file. + +{{< highlight bash >}} +mitmdump -w wireless-login +{{< / highlight >}} + +## 2. Point your browser at the mitmdump instance. + +I use a tiny Firefox addon called [Toggle +Proxy](https://addons.mozilla.org/en-us/firefox/addon/toggle-proxy-51740/) to +switch quickly to and from mitmproxy. I'm assuming you've already [configured +your browser with mitmproxy's SSL certificate authority]({{< relref +"concepts-certificates" >}}). + +## 3. Log in as usual + +And that's it\! You now have a serialised version of the login process +in the file wireless-login, and you can replay it at any time like this: + +{{< highlight bash >}} +mitmdump -c wireless-login +{{< / highlight >}} + +## Embellishments + +We're really done at this point, but there are a couple of embellishments we +could make if we wanted. I use [wicd](https://launchpad.net/wicd) to +automatically join wireless networks I frequent, and it lets me specify a +command to run after connecting. I used the client replay command above and +voila\! - totally hands-free wireless network startup. + +We might also want to prune requests that download CSS, JS, images and so forth. +These add only a few moments to the time it takes to replay, but they're not +really needed and I somehow feel compelled to trim them anyway. So, we fire up +the mitmproxy console tool on our serialised conversation, like so: + +{{< highlight bash >}} +mitmproxy -r wireless-login +{{< / highlight >}} + +We can now go through and manually delete (using the <span +data-role="kbd">d</span> keyboard shortcut) everything we want to trim. When +we're done, we use <span data-role="kbd">w</span> to save the conversation back +to the file. diff --git a/docs/src/content/tute-highscores.md b/docs/src/content/tute-highscores.md new file mode 100644 index 00000000..f5cbd7bc --- /dev/null +++ b/docs/src/content/tute-highscores.md @@ -0,0 +1,123 @@ +--- +title: "Setting highscores on Apple GameCenter" +menu: + tutes: + weight: 2 +--- + +# Setting highscores on Apple's GameCenter + +## The setup + +In this tutorial, I'm going to show you how simple it is to creatively interfere +with Apple Game Center traffic using mitmproxy. To set things up, [install the +mitmproxy root certificate]({{< relref concepts-certificates >}}). Then start +mitmproxy on your desktop, and configure the iPhone to use it as a proxy. + +## Taking a look at the Game Center traffic + +Lets take a first look at the Game Center traffic. The game I'll use in this +tutorial is [Super Mega +Worm](https://itunes.apple.com/us/app/super-mega-worm/id388541990?mt=8) - a +great little retro-apocalyptic sidescroller for the iPhone: + +{{< figure src="/tute-highscores/supermega.png" >}} + +After finishing a game (take your time), watch the traffic flowing through +mitmproxy: + +{{< figure src="/tute-highscores/one.png" >}} + +We see a bunch of things we might expect - initialisation, the retrieval +of leaderboards and so forth. Then, right at the end, there's a POST to +this tantalising +URL: + +{{< highlight none >}} +https://service.gc.apple.com/WebObjects/GKGameStatsService.woa/wa/submitScore +{{< / highlight >}} + +The contents of the submission are particularly interesting: + +{{< highlight xml >}} +<plist version="1.0"> + <dict> + <key>scores</key> + <array> + <dict> + <key>category</key> + <string>SMW_Adv_USA1</string> + <key>context</key> + <integer>0</integer> + <key>score-value</key> + <integer>55</integer> + <key>timestamp</key> + <integer>1363515361321</integer> + </dict> + </array> + </dict> +</plist> +{{< / highlight >}} + +This is a [property list](https://en.wikipedia.org/wiki/Property_list), +containing an identifier for the game, a score (55, in this case), and a +timestamp. Looks pretty simple to mess with. + +## Modifying and replaying the score submission + +Lets edit the score submission. First, select it in mitmproxy, then +press <span data-role="kbd">enter</span> to view it. Make sure you're +viewing the request, not the response -you can use +<span data-role="kbd">tab</span> to flick between the two. Now press +<span data-role="kbd">e</span> for edit. You'll be prompted for the part +of the request you want to change - press <span data-role="kbd">r</span> +for raw body. Your preferred editor (taken from the EDITOR environment +variable) will now fire up. Lets bump the score up to something a bit +more ambitious: + +{{< highlight xml >}} +<plist version="1.0"> + <dict> + <key>scores</key> + <array> + <dict> + <key>category</key> + <string>SMW_Adv_USA1</string> + <key>context</key> + <integer>0</integer> + <key>score-value</key> + <integer>2200272667</integer> + <key>timestamp</key> + <integer>1363515361321</integer> + </dict> + </array> + </dict> +</plist> +{{< / highlight >}} + +Save the file and exit your editor. + +The final step is to replay this modified request. Simply press +<span data-role="kbd">r</span> for replay. + +## The glorious result and some intrigue + +{{< figure src="/tute-highscores/leaderboard.png" >}} + +And that's it - according to the records, I am the greatest Super Mega Worm +player of all time. + +There's a curious addendum to this tale. When I first wrote this tutorial, all +the top competitors' scores were the same: 2,147,483,647 (this is no longer the +case, because there are now so many fellow cheaters using this tutorial). If you +think that number seems familiar, you're right: it's 2^31-1, the maximum value +you can fit into a signed 32-bit int. Now let me tell you another peculiar thing +about Super Mega Worm - at the end of every game, it submits your highest +previous score to the Game Center, not your current score. This means that it +stores your highscore somewhere, and I'm guessing that it reads that stored +score back into a signed integer. So, if you **were** to cheat by the relatively +pedestrian means of modifying the saved score on your jailbroken phone, then +2^31-1 might well be the maximum score you could get. Then again, if the game +itself stores its score in a signed 32-bit int, you could get the same score +through perfect play, effectively beating the game. So, which is it in this +case? I'll leave that for you to decide. |