Merge branch 'master' of github.com:mitmproxy/mitmproxy

Conflicts:
	doc-src/modes.html

12 files changed, 361 insertions, 225 deletions

diff --git a/CHANGELOG b/CHANGELOG
index 69e7339b..c78fdcce 100644
--- a/CHANGELOG
+++ b/CHANGELOG
@@ -1,3 +1,46 @@
+
+23 October 2014: mitmproxy 0.11:
+
+    * SOCKS5 proxy mode allows mitmproxy to act as a SOCKS5 proxy server
+
+    * Data streaming for response bodies exceeding a threshold
+      (bradpeabody@gmail.com)
+
+    * Ignore hosts or IP addresses, forwarding both HTTP and HTTPS traffic
+      untouched
+
+    * Finer-grained control of traffic replay, including options to ignore
+      contents or parameters when matching flows (marcelo.glezer@gmail.com)
+
+    * Pass arguments to inline scripts 
+
+    * Configurable size limit on HTTP request and response bodies
+
+    * Per-domain specification of interception certificates and keys (see
+      --cert option)
+
+    * Certificate forwarding, relaying upstream SSL certificates verbatim (see
+      --cert-forward)
+
+    * Search and highlighting for HTTP request and response bodies in
+      mitmproxy console (pedro@worcel.com)
+
+    * Transparent proxy support on Windows
+
+    * Improved error messages and logging
+
+    * Support for FreeBSD in transparent mode, using pf (zbrdge@gmail.com)
+
+    * Content view mode for WBXML (davidshaw835@air-watch.com)
+
+    * Better documentation, with a new section on proxy modes
+
+    * Generic TCP proxy mode
+
+    * Countless bugfixes and other small improvements
+
+
+
 28 January 2014: mitmproxy 0.10:
 	
     * Support for multiple scripts and multiple script arguments
diff --git a/CONTRIBUTORS b/CONTRIBUTORS
index bed636fa..a9688d92 100644
--- a/CONTRIBUTORS
+++ b/CONTRIBUTORS
@@ -1,51 +1,65 @@
-   854	Aldo Cortesi
-    64	Maximilian Hils
+   902	Aldo Cortesi
+   323	Maximilian Hils
     18	Henrik Nordstrom
     13	Thomas Roth
+    12	Pedro Worcel
     11	Stephen Altamirano
     10	András Veres-Szentkirályi
-     8	Jason A. Novak
      8	Rouli
+     8	Jason A. Novak
      7	Alexis Hildebrandt
-     6	Pedro Worcel
      5	Tomaz Muraus
+     5	Brad Peabody
      5	Matthias Urlichs
      4	root
-     4	Bryan Bishop
      4	Marc Liyanage
      4	Valtteri Virtanen
-     3	Kyle Manna
+     4	Bryan Bishop
      3	Chris Neasbitt
-     2	alts
-     2	Heikki Hannikainen
-     2	Jim Lloyd
+     3	Zack B
+     3	Eli Shvartsman
+     3	Kyle Manna
      2	Michael Frister
+     2	Bennett Blodinger
+     2	Jim Lloyd
      2	Rob Wills
-     2	Jaime Soriano Pastor
      2	israel
+     2	Jaime Soriano Pastor
+     2	Heikki Hannikainen
      2	Mark E. Haase
+     2	alts
+     1	davidpshaw
+     1	deployable
+     1	joebowbeer
+     1	meeee
+     1	phil plante
+     1	Michael Bisbjerg
+     1	Andy Smith
+     1	Dan Wilbraham
+     1	David Shaw
+     1	Eric Entzel
+     1	Felix Wolfsteller
+     1	Henrik Nordström
+     1	Ivaylo Popov
+     1	JC
+     1	Jakub Nawalaniec
+     1	James Billingham
+     1	Jean Regisser
+     1	Kit Randel
+     1	Marcelo Glezer
+     1	Mathieu Mitchell
+     1	Mikhail Korobov
+     1	Nicolas Esteves
+     1	Oleksandr Sheremet
      1	Paul
      1	Rich Somerfield
      1	Rory McCann
-     1	Felix Wolfsteller
      1	Rune Halvorsen
      1	Sahn Lam
-     1	Eric Entzel
-     1	Dan Wilbraham
+     1	Seppo Yli-Olli
+     1	Sergey Chipiga
+     1	Steven Van Acker
      1	Ulrich Petri
-     1	Andy Smith
+     1	Vyacheslav Bakhmutov
      1	Yuangxuan Wang
      1	capt8bit
-     1	joebowbeer
-     1	meeee
-     1	James Billingham
-     1	Jakub Nawalaniec
-     1	JC
-     1	Kit Randel
-     1	phil plante
-     1	Mathieu Mitchell
-     1	Ivaylo Popov
-     1	Henrik Nordström
-     1	Michael Bisbjerg
-     1	Nicolas Esteves
-     1	Oleksandr Sheremet
diff --git a/README.mkd b/README.mkd
index 49582612..ccc09138 100644
--- a/README.mkd
+++ b/README.mkd
@@ -13,6 +13,9 @@ mitmproxy.org website:
 [mitmproxy.org](http://mitmproxy.org).
 
 
+You can find complete directions for installing mitmproxy [here](http://mitmproxy.org/doc/install.html). 
+
+
 Features
 --------
 
@@ -26,17 +29,17 @@ Features
 - SSL certificates for interception are generated on the fly.
 - And much, much more.
 
+__mitmproxy__ is tested and developed on OSX, Linux and OpenBSD. On Windows,
+only mitmdump is supported, which does not have a graphical user interface.
 
-Installation
-------------
 
 
-The recommended way to install mitmproxy is running <code>pip install mitmproxy</code>.  
-For convenience, we provide binary packages on [mitmproxy.org](http://mitmproxy.org/).
+Hacking
+-------
+
 
+### Requirements
 
-Requirements
-------------
 
 * [Python](http://www.python.org) 2.7.x.
 * [netlib](http://pypi.python.org/pypi/netlib), version matching mitmproxy.
@@ -49,28 +52,41 @@ Optional packages for extended content decoding:
 * [cssutils](http://cthedot.de/cssutils/) version 1.0 or newer.
 
 For convenience, all optional dependencies can be installed with 
-`pip install mitmproxy[contenviews]`
 
-__mitmproxy__ is tested and developed on OSX, Linux and OpenBSD. On Windows,
-only mitmdump is supported, which does not have a graphical user interface.
+`pip install "mitmproxy[contentviews]"`
 
+### Setting up a dev environment
 
-Hacking
--------
+The following procedure is recommended to set up your dev environment:
 
-The following components are needed if you plan to hack on mitmproxy:
-
-* The test suite requires the `dev` extra requirements listed in [setup.py](https://github.com/mitmproxy/mitmproxy/blob/master/setup.py) and [pathod](http://pathod.net), version matching mitmproxy.
-* Rendering the documentation requires [countershape](http://github.com/cortesi/countershape).
-
-For convenience, the following procedure is recommended to set up your environment:
 ```
 $ git clone https://github.com/mitmproxy/mitmproxy.git
 $ cd mitmproxy
 $ pip install --src . -r requirements.txt
 ```
+
 This installs the latest GitHub versions of mitmproxy, netlib and pathod into `mitmproxy/`. All other development dependencies save countershape are installed into their usual locations. 
 
+
+### Testing
+
+The test suite requires the `dev` extra requirements listed in [setup.py](https://github.com/mitmproxy/mitmproxy/blob/master/setup.py) and [pathod](http://pathod.net), version matching mitmproxy. Install these with:
+
+`
+pip install "mitmproxy[dev]""`
+
+
 Please ensure that all patches are accompanied by matching changes in the test
 suite. The project maintains 100% test coverage.
 
+
+### Docs
+
+Rendering the documentation requires [countershape](http://github.com/cortesi/countershape). After installation, you can render the documentation to the doc like this:
+
+`cshape doc-src doc`
+
+
+
+
+
diff --git a/doc-src/_nav.html b/doc-src/_nav.html
index 8bd03db2..0ae0fa67 100644
--- a/doc-src/_nav.html
+++ b/doc-src/_nav.html
@@ -17,13 +17,14 @@
         $!nav("serverreplay.html", this, state)!$
         $!nav("setheaders.html", this, state)!$
         $!nav("passthrough.html", this, state)!$
-        $!nav("tcpproxy.html", this, state)!$
-        $!nav("sticky.html", this, state)!$
+        $!nav("proxyauth.html", this, state)!$
         $!nav("reverseproxy.html", this, state)!$
+        $!nav("responsestreaming.html", this, state)!$
+        $!nav("socksproxy.html", this, state)!$
+        $!nav("sticky.html", this, state)!$
+        $!nav("tcpproxy.html", this, state)!$
         $!nav("upstreamproxy.html", this, state)!$
         $!nav("upstreamcerts.html", this, state)!$
-        $!nav("proxyauth.html", this, state)!$
-        $!nav("responsestreaming.html", this, state)!$
 
 
     <li class="nav-header">Installing Certificates</li>
diff --git a/doc-src/features/index.py b/doc-src/features/index.py
index 40a2669c..693b4439 100644
--- a/doc-src/features/index.py
+++ b/doc-src/features/index.py
@@ -9,6 +9,7 @@ pages = [
     Page("replacements.html", "Replacements"),
     Page("responsestreaming.html", "Response Streaming"),
     Page("reverseproxy.html", "Reverse proxy mode"),
+    Page("socksproxy.html", "SOCKS Mode"),
     Page("setheaders.html", "Set Headers"),
     Page("serverreplay.html", "Server-side replay"),
     Page("sticky.html", "Sticky cookies and auth"),
diff --git a/doc-src/features/reverseproxy.html b/doc-src/features/reverseproxy.html
index e6de4f33..1c57f0b2 100644
--- a/doc-src/features/reverseproxy.html
+++ b/doc-src/features/reverseproxy.html
@@ -7,10 +7,22 @@ mitmproxy forwards HTTP proxy requests to an upstream proxy server.
 <table class="table">
     <tbody>
         <tr>
-            <th width="20%">command-line</th> <td>-R http[s]://hostname[:port]</td>
-        </tr>
-        <tr>
-            <th>mitmproxy shortcut</th> <td><b>P</b></td>
+            <th width="20%">command-line</th> <td>-R <i>schema</i>://hostname[:port]</td>
         </tr>
     </tbody>
 </table>
+
+Here, **schema** is one of http, https, http2https or https2http. The latter
+two extended schema specifications control the use of HTTP and HTTPS on
+mitmproxy and the upstream server. You can indicate that mitmproxy should use
+HTTP, and the upstream server uses HTTPS like this:
+
+	http2https://hostname:port
+
+And you can indicate that mitmproxy should use HTTPS while the upstream
+service uses HTTP like this:
+
+	https2http://hostname:port
+
+
+
diff --git a/doc-src/features/socksproxy.html b/doc-src/features/socksproxy.html
new file mode 100644
index 00000000..f436cbf5
--- /dev/null
+++ b/doc-src/features/socksproxy.html
@@ -0,0 +1,10 @@
+
+In this mode, mitmproxy acts as a SOCKS5 proxy server.
+
+<table class="table">
+    <tbody>
+        <tr>
+            <th width="20%">command-line</th> <td>--socks</td>
+        </tr>
+    </tbody>
+</table>
diff --git a/doc-src/features/upstreamproxy.html b/doc-src/features/upstreamproxy.html
index 6039f4df..47bc115d 100644
--- a/doc-src/features/upstreamproxy.html
+++ b/doc-src/features/upstreamproxy.html
@@ -9,8 +9,19 @@ mitmproxy forwards ordinary HTTP requests to an upstream server.
         <tr>
             <th width="20%">command-line</th> <td>-U http://hostname[:port]</td>
         </tr>
-        <tr>
-            <th>mitmproxy shortcut</th> <td><b>U</b></td>
-        </tr>
     </tbody>
 </table>
+
+Here, **schema** is one of http, https, http2https or https2http. The latter
+two extended schema specifications control the use of HTTP and HTTPS on
+mitmproxy and the upstream server. You can indicate that mitmproxy should use
+HTTP, and the upstream server uses HTTPS like this:
+
+	http2https://hostname:port
+
+And you can indicate that mitmproxy should use HTTPS while the upstream
+service uses HTTP like this:
+
+	https2http://hostname:port
+
+
diff --git a/doc-src/index.html b/doc-src/index.html
index 79687ec6..23da7223 100644
--- a/doc-src/index.html
+++ b/doc-src/index.html
@@ -1,4 +1,27 @@
 
-@!index_contents!@
+__mitmproxy__ is an interactive, SSL-capable man-in-the-middle proxy for HTTP
+with a console interface.
 
+__mitmdump__ is the command-line version of mitmproxy. Think tcpdump for HTTP.
+
+__libmproxy__ is the library that mitmproxy and mitmdump are built on.
+
+Documentation, tutorials and distribution packages can be found on the
+mitmproxy.org website:
+
+[mitmproxy.org](http://mitmproxy.org).
+
+
+Features
+--------
+
+- Intercept HTTP requests and responses and modify them on the fly.
+- Save complete HTTP conversations for later replay and analysis.
+- Replay the client-side of an HTTP conversations.
+- Replay HTTP responses of a previously recorded server.
+- Reverse proxy mode to forward traffic to a specified server.
+- Transparent proxy mode on OSX and Linux.
+- Make scripted changes to HTTP traffic using Python.
+- SSL certificates for interception are generated on the fly.
+- And much, much more.
 
diff --git a/doc-src/index.py b/doc-src/index.py
index b7ab9995..e6064e3a 100644
--- a/doc-src/index.py
+++ b/doc-src/index.py
@@ -1,6 +1,8 @@
-import os, sys, datetime
+import os
+import sys
+import datetime
 import countershape
-from countershape import Page, Directory, PythonModule, markup, model
+from countershape import Page, Directory, markup, model
 import countershape.template
 sys.path.insert(0, "..")
 from libmproxy import filt, version
@@ -23,18 +25,18 @@ ns.docMaintainer = "Aldo Cortesi"
 ns.docMaintainerEmail = "aldo@corte.si"
 ns.copyright = u"\u00a9 mitmproxy project, %s" % datetime.date.today().year
 
+
 def mpath(p):
     p = os.path.join(MITMPROXY_SRC, p)
     return os.path.expanduser(p)
 
-with open(mpath("README.mkd")) as f:
-        readme = f.read()
-        ns.index_contents = readme.split("\n", 1)[1] #remove first line (contains build status)
 
 def example(s):
     d = file(mpath(s)).read().rstrip()
     extemp = """<div class="example">%s<div class="example_legend">(%s)</div></div>"""
     return extemp%(countershape.template.Syntax("py")(d), s)
+
+
 ns.example = example
 
 
@@ -73,6 +75,7 @@ def nav(page, current, state):
 ns.nav = nav
 ns.navbar = countershape.template.File(None, "_nav.html")
 
+
 pages = [
     Page("index.html", "Introduction"),
     Page("install.html", "Installation"),
diff --git a/doc-src/install.html b/doc-src/install.html
index 5d412459..8a76e3ae 100644
--- a/doc-src/install.html
+++ b/doc-src/install.html
@@ -1,40 +1,35 @@
 
+
+
+## Installing from source
+
 The preferred way to install mitmproxy - whether you're installing the latest
 release or from source - is to use [pip](http://www.pip-installer.org/). If you
 don't already have pip on your system, you can find installation instructions
 [here](http://www.pip-installer.org/en/latest/installing.html).
 
-
-## Installing the latest release
-
-A single command will download and install the latest release of mitmproxy,
-along with all its dependencies:
-
 <pre class="terminal">
 pip install mitmproxy
 </pre>
 
+If you also want to install the optional packages AMF, protobuf and CSS
+content views, do this:
 
-## Installing from source
-
-When installing from source, the easiest method is still to use pip. In this
-case run:
-    
 <pre class="terminal">
-pip install /path/to/source
+pip install "mitmproxy[contentviews]"
 </pre>
 
-Note that if you're installing current git master, you will also have to
-install the current git master of [netlib](http://github.com/mitmproxy/netlib) by
-hand.
 
 ## OSX
 
+The easiest way to get up and running on OSX is to download the pre-built
+binary packages from [mitmproxy.org](http://mitmproxy.org). If you still want
+to install using pip, there are a few things to keep in mind:
+
 - If you're running a Python interpreter installed with homebrew (or similar),
 you may have to install some dependencies by hand. 
 - Make sure that XCode is installed from the App Store, and that the
 command-line tools have been downloaded (XCode/Preferences/Downloads).
-- Now use __pip__ to do the installation, as above.
 
 There are a few bits of customization you might want to do to make mitmproxy
 comfortable to use on OSX. The default color scheme is optimized for a dark
@@ -64,8 +59,3 @@ from source:
 - libxslt1-dev
 
 
-
-
-
-
-
diff --git a/doc-src/modes.html b/doc-src/modes.html
index ddbeae0c..8870009d 100644
--- a/doc-src/modes.html
+++ b/doc-src/modes.html
@@ -1,210 +1,222 @@
-Mitmproxy comes with several modes of operation, which allow you to use mitmproxy in a variety of scenarios.
-This documents briefly explains each mode and possible setups.
-<hr>
-Mitmproxy has four modes of operation:
-<ul>
-    <li>Regular Mode (this is what you get by default)</li>
-    <li>Transparent Mode</li>
-    <li>Reverse Proxy Mode</li>
-    <li>Upstream Proxy Mode</li>
-</ul>
-
-<p>Now, which one should you pick? Use this flow chart:
-</p>
-
-<img src="@!urlTo('schematics/proxy-modes-flowchart.png')!@"><br><br>
+
+Mitmproxy has four modes of operation that allow you to use mitmproxy in a
+variety of scenarios:
+
+- **Regular** (the default)
+- **Transparent**
+- **Reverse Proxy**
+- **Upstream Proxy**
+
+Now, which one should you pick? Use this flow chart:
+
+<img src="@!urlTo('schematics/proxy-modes-flowchart.png')!@"/>
 
 <div class="page-header">
     <h1>Regular Proxy</h1>
 </div>
 
-Mitmproxy's regular mode it the most simple one and the easiest to set up.
+Mitmproxy's regular mode is the simplest and the easiest to set up.
 
-<ol>
-    <li>Start mitmproxy.</li>
-    <li>Configure your client to use mitmproxy. This means that you either adjust the proxy setting of your local browser
-    or point an external device to your proxy (which should look like
-        <a href="@!urlTo('screenshots/ios-manual.png')!@">this</a>).</li>
-    <li>Quick Check: You can already visit an unencrypted HTTP site over the proxy.</li>
-    <li>Open the magic domain <strong>mitm.it</strong> and install the certificate for your device.</li>
-</ol>
+1. Start mitmproxy.
+2. Configure your client to use mitmproxy. For instance on IOS, the settings might look like <a href="@!urlTo('screenshots/ios-manual.png')!@">this</a>.
+3. Quick Check: You should already be able to visit an unencrypted HTTP site
+through the proxy.
+4. Open the magic domain <strong>mitm.it</strong> and install the certificate for your device.
 
 <div class="well">
-    <strong>Heads Up:</strong> Unfortunately, some applications prefer to bypass the HTTP proxy settings of the system -
-    Android applications are a common example. In these cases, you need to use mitmproxy's transparent mode.
+<strong>Heads Up:</strong> Unfortunately, some applications bypass the
+system HTTP proxy settings - Android applications are a common example. In
+these cases, you need to use mitmproxy's transparent mode.
 </div>
 
-<p>If you are proxying an external device, your network will probably look like this:</p>
+If you are proxying an external device, your network will probably look like this:
+
 <img src="@!urlTo('schematics/proxy-modes-regular.png')!@">
-<br><br>
-<p>The square brackets signify the source and destination IP addresses. Your client explicitly connects
-    to mitmproxy and mitmproxy explicitly connects to the target server.
-</p>
+
+The square brackets signify the source and destination IP addresses. Your
+client explicitly connects to mitmproxy and mitmproxy explicitly connects
+to the target server.
 
 <div class="page-header">
     <h1>Transparent Proxy</h1>
 </div>
 
-When a transparent proxy is used, traffic is redirected into a proxy at the network layer, without any client
-configuration being required. This makes transparent proxying ideal for those situations where you can't change client
-behaviour. The basic principle is that mitmproxy sits somewhere on the line from the client to the internet and
-transparently intercepts the request. In the graphic below, a machine running mitmproxy has been inserted between
-the router and the internet:
+In transparent mode, traffic is directed into a proxy at the network layer,
+without any client configuration required. This makes transparent proxying
+ideal for situations where you can't change client behaviour. In the graphic
+below, a machine running mitmproxy has been inserted between the router and
+the internet:
 
 <a href="@!urlTo('schematics/proxy-modes-transparent-1.png')!@">
-    <img src="@!urlTo('schematics/proxy-modes-transparent-1.png')!@"></a>
-<p>The square brackets signify the source and destination IP addresses. Round brackets mark the next
-    hop on the <strong>Ethernet</strong>/data link layer. This distinction is important to make: When the packet arrives
-    at the mitmproxy machine, it must still be addressed to the target server. In other words: A simple IP redirect on
-    the router does not work - this would remove the target information, leaving mitmproxy unable to
-    determine the real destination.
-</p>
+    <img src="@!urlTo('schematics/proxy-modes-transparent-1.png')!@">
+</a>
+
+The square brackets signify the source and destination IP addresses. Round
+brackets mark the next hop on the *Ethernet/data link* layer. This distinction
+is important: when the packet arrives at the mitmproxy machine, it must still
+be addressed to the target server. This means that Network Address Translation
+should not be applied before the traffic reaches mitmproxy, since this would
+remove the target information, leaving mitmproxy unable to determine the real
+destination.
+
 <a href="@!urlTo('schematics/proxy-modes-transparent-wrong.png')!@">
     <img src="@!urlTo('schematics/proxy-modes-transparent-wrong.png')!@"></a>
 
 <h2>Common Configurations</h2>
 
-The first graphic is a little bit idealistic: Usually, you'll have your local wireless lan network and no
-machines between your router and the internet. Fortunately, there are other ways to configure your network:
-(a) Configuring the client to use a custom gateway/router/"next hop", (b) Implementing custom routing on the router
-or (c) setting up a separate wireless network router which gets proxied.
-There are of course other options, but we'll look at these three. In most cases, setting (a) is recommended due to its
-ease of use.
+There are many ways to configure your network for transparent proxying. We'll
+look at three common scenarios:
+
+1. Configuring the client to use a custom gateway/router/"next hop"
+2. Implementing custom routing on the router
+
+In most cases, the first option is recommended due to its ease of use.
 
 <h3>(a) Custom Gateway</h3>
 
-<p>Looking at your local home network, it's clear what happens if you enter "example.com" into your address bar: After you
-press enter, your OS sends a packet to your router, which then sends this to your ISP, which then sends it to some
-Tier-1 carrier, which then sends it... I think you get the idea. The important part for us is the first step here:
-Your machine is configured to use your router as the next hop. Your router certainly doesn't host example.com, but your
-machine knows that your router will forward it upstream. On the technical level, your router probably provides a DHCP
-server, which instructs all clients to use his address as the <em>Default Gateway</em> for connections that leave the
-current subnet (your local network).</p>
-<p>
-How does this help us? Here comes our trick: By configuring the client to use our machine as its Gateway, all traffic
-will be sent to our machine, which then forwards it to the router. This provides us with the scenario we'd like to have,
-namely packets on our doorstep that are addressed for someone else:
-</p>
+One simple way to get traffic to the mitmproxy machine with the destination IP
+intact, is to simply configure the client with the mitmproxy box as the
+default gateway.
+
 <a href="@!urlTo('schematics/proxy-modes-transparent-2.png')!@">
     <img src="@!urlTo('schematics/proxy-modes-transparent-2.png')!@"></a>
 
-Given this concept, we can set up mitmproxy:
-<ol>
-    <li>Configure your proxy machine for transparent mode.<br>You can find instructions
-    in the <em>Transparent Proxying</em> section of the mitmproxy docs.</li>
-    <li>Configure your client to use your proxy machine's IP as the default gateway. This setting is usually called
-        <em>Standard Gateway, Router</em> or something along these lines
-        (<a href="@!urlTo('screenshots/ios-gateway.png')!@">iOS screenshot</a>).</li>
-    <li>Quick Check: You can already visit an unencrypted HTTP site over the proxy.</li>
-    <li>Open the magic domain <strong>mitm.it</strong> and install the certificate for your device.</li>
-</ol>
+In this scenario, we would:
+
+- Configure the proxy machine for transparent mode. You can find instructions
+in the <em>Transparent Proxying</em> section of the mitmproxy docs.
+
+- Configure the client to use the proxy machine's IP as the default gateway.
+<a href="@!urlTo('screenshots/ios-gateway.png')!@">Here</a> is what this would
+look like on IOS.
+
+- Quick Check: At this point, you should already be able to visit an
+unencrypted HTTP site over the proxy.
+
+- Open the magic domain <strong>mitm.it</strong> and install the certificate
+for your device.
+
+Setting the custom gateway on clients can be automated by serving the settings
+out to clients over DHCP. This lets set up an interception network where all
+clients are proxied automatically, which can save time and effort. 
+
 
 <div class="well">
     <strong style="text-align: center; display: block">Troubleshooting Transparent Mode</strong>
-    <p>Wrong transparent mode configurations are a frequent source of
+
+    <p>Incorrect transparent mode configurations are a frequent source of
     error. If it doesn't work for you, try the following things:</p>
+
     <ul>
-        <li>Open mitmproxy's event log (press `e`) - can you spot clientconnect messages?
-        If not, the packets are not arriving at the proxy. A common source is the occurence of ICMP redirects,
-            which means that your machine is telling the client that there's a faster way to the internet by contacting
-            your router directly (see the <em>Transparent Proxying</em> section on how to disable them). If in doubt,
-            <a href="https://wireshark.org/">Wireshark</a> may help you to see whether something arrives at your machine
-            or not.
+        <li>
+            Open mitmproxy's event log (press `e`) - do you see clientconnect
+            messages? If not, the packets are not arriving at the proxy. One common
+            cause is the occurrence of ICMP redirects, which means that your
+            machine is telling the client that there's a faster way to the
+            internet by contacting your router directly (see the
+            <em>Transparent Proxying</em> section on how to disable them). If in
+            doubt, <a href="https://wireshark.org/">Wireshark</a> may help you
+            to see whether something arrives at your machine or not.
         </li>
         <li>
-            Have you explicitly configured an HTTP proxy on your device? You do not need mitmproxy's transparent mode
-            then, just start mitmproxy normally. Explicitly setting a proxy and transparent mode contradict each other,
-            settle for one. Do not explicitly redirect traffic to mitmproxy anywhere except for the Gateway setting.
+            Make sure you have not explicitly configured an HTTP proxy on the
+            client. This is not needed in transparent mode.
         </li>
         <li>
             Re-check the instructions in the <em>Transparent Proxying</em> section. Anything you missed?
         </li>
     </ul>
+
     If you encounter any other pitfalls that should be listed here, please let us know!
+
 </div>
 
 <h3>(b) Custom Routing</h3>
 
-Custom routing is a fairly advanced setup which we'll only document briefly here.
-First and foremost, it usually requires root on your router. The basic idea is to teach your router a custom routing
-table that says "for requests from ip X, the proxy machine is the next gateway".
+In some cases, you may need more fine-grained control of which traffic reaches
+the mitmproxy instance, and which doesn't. You may, for instance, choose only
+to divert traffic to some hosts into the transparent proxy. There are a huge
+number of ways to accomplish this, and much will depend on the router or
+packet filter you're using. In most cases, the configuration will look like
+this:
 
 <a href="@!urlTo('schematics/proxy-modes-transparent-3.png')!@">
-    <img src="@!urlTo('schematics/proxy-modes-transparent-3.png')!@"></a>
-
-For this setup, we expect you to have a basic understanding of networking in general. In short, you should get started
-with <a href="@!urlTo('custom-routing.txt')!@">these routing commands</a>. The Troubleshooting part directly above this
-section might be helpful for you as well.
-
-<h3>(c) Separate Network</h3>
-
-Setting up a separate network using a cheap router might be a viable option, too. Such a configuration mostly resembles
-the idealistic graphic from the beginning (Variant 1). Take a look at the
-<a href="@!urlTo('tutorials/transparent-dhcp.html')!@">Transparently proxify virtual machines</a> tutorial to see how
-such a network could be implemented. The troubleshooting section for custom gateways may be helpful for you, too.
+    <img src="@!urlTo('schematics/proxy-modes-transparent-3.png')!@">
+</a>
 
 
 <div class="page-header">
     <h1>Reverse Proxy</h1>
 </div>
 
-Mitmproxy is usually used with a client that uses the proxy to access the Internet. Using reverse proxy mode, you can
-use mitmproxy to represent a server:
+Mitmproxy is usually used with a client that uses the proxy to access the
+Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal
+HTTP server:
 
 <a href="@!urlTo('schematics/proxy-modes-reverse.png')!@">
-    <img src="@!urlTo('schematics/proxy-modes-reverse.png')!@"></a>
+    <img src="@!urlTo('schematics/proxy-modes-reverse.png')!@">
+</a>
 
 There are various use-cases:
-<ul>
-<li>
-    Say you have an internal API running at http://example.local/. You could now setup mitmproxy in
-    reverse proxy mode at http://debug.example.local/ and dynamically point clients to this new API endpoint,
-    which provides clients with the same data and you with debug information. Similarly, you could move your real server
-    to a different ip/port and setup mitmproxy at the original place to debug all sessions.
-</li>
-<li>
-    Say you're a web developer working on example.com (with a development version running on localhost:8000).
-    You can modify your hosts file so that example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy
-    mode on port 80. You can test your app on the example.com domain and get all requests recorded in mitmproxy.
-</li>
-<li>
-    Say you have some toy project that should get SSL support. Simply setup mitmproxy with SSL termination and you're
-    done (<code>mitmdump -p 443 -R https2http://localhost:80/</code>). There are better tools for this specific task (we don't
-    have C performance obviously), but it's definitely a nice and very quick way to setup an SSL-speaking server.
-</li>
-<li>
-    Want to add a non-SSL-capable compression proxy in front of your server? You could even spawn a mitmproxy instance
-    that terminates SSL (https2http://...), point it to the compression proxy and let the compression proxy point
-    to a SSL-initiating mitmproxy (http2https://...), which then points to the real server. As you see, it's a fairly
-    flexible thing.
-</li>
-</ul>
-
-<p>
-Please note that cloning Google by using <code>mitmproxy -R http://google.com/</code> does <em>not</em> really work
-(as in <a href="@!urlTo('screenshots/ios-reverse.png')!@">this screenshot</a>).
-This may work for the first request, but the HTML remains unchanged: As soon as the user clicks on an non-relative URL
-(or downloads a non-relative image resource), they speak with Google directly again.
-</p>
-<p>
-    On another note, mitmproxy either supports an HTTP or an HTTPS upstream server, not both at the same time. You can
-    simply work around this by spawning a second mitmproxy instance. Each instance listens to one port and talks to one
-    port.
-</p>
+
+- Say you have an internal API running at http://example.local/. You could now
+set up mitmproxy in reverse proxy mode at http://debug.example.local/ and
+dynamically point clients to this new API endpoint, which provides clients
+with the same data and you with debug information. Similarly, you could move
+your real server to a different IP/port and set up mitmproxy at the original
+place to debug all sessions.
+
+- Say you're a web developer working on example.com (with a development
+version running on localhost:8000). You can modify your hosts file so that
+example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode
+on port 80. You can test your app on the example.com domain and get all
+requests recorded in mitmproxy.
+
+- Say you have some toy project that should get SSL support. Simply set up
+mitmproxy with SSL termination and you're done (<code>mitmdump -p 443 -R
+https2http://localhost:80/</code>). There are better tools for this specific
+task, but mitmproxy is very quick and simple way to set up an SSL-speaking
+server.
+
+- Want to add a non-SSL-capable compression proxy in front of your server? You
+could even spawn a mitmproxy instance that terminates SSL (https2http://...),
+point it to the compression proxy and let the compression proxy point to a
+SSL-initiating mitmproxy (http2https://...), which then points to the real
+server. As you see, it's a fairly flexible thing.
+
+Note that mitmproxy supports either an HTTP or an HTTPS upstream server, not
+both at the same time. You can work around this by spawning a second mitmproxy
+instance.
+
+<div class="well">
+    <strong style="text-align: center; display: block">Caveat: Interactive Use</strong>
+
+
+One caveat is that reverse proxy mode is often not sufficient for interactive
+browsing. Consider trying to clone Google by using:
+
+<code>mitmproxy -R http://google.com/</code>
+
+This works for the initial request, but the HTML served to the client remains
+unchanged. As soon as the user clicks on an non-relative URL (or downloads a
+non-relative image resource), traffic no longer passes through mitmproxy, and
+the client connects to Google directly again.
+
+</div>
+
+
 
 <div class="page-header">
     <h1>Upstream Proxy</h1>
 </div>
 
-<p>
-If you want to add mitmproxy in front of a different proxy appliance, you can use mitmproxy's upstream mode.
-In upstream mode, all requests are unconditionally transferred to an upstream proxy of your choice.
-</p>
+If you want to chain proxies by adding mitmproxy in front of a different proxy
+appliance, you can use mitmproxy's upstream mode. In upstream mode, all
+requests are unconditionally transferred to an upstream proxy of your choice.
 
 <a href="@!urlTo('schematics/proxy-modes-upstream.png')!@">
     <img src="@!urlTo('schematics/proxy-modes-upstream.png')!@"></a>
 
-<p>
-mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy mode. You could in theory chain multiple
-mitmproxy instances in a row, but that doesn't make any sense in practice (i.e. outside of our tests).
-</p>
\ No newline at end of file
+mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy
+mode. You could in theory chain multiple mitmproxy instances in a row, but
+that doesn't make any sense in practice (i.e. outside of our tests).