From 982508d30f887b4fe8b2a855792ae1e33f378222 Mon Sep 17 00:00:00 2001 From: Aldo Cortesi Date: Thu, 22 Feb 2018 17:21:34 +1300 Subject: All new documentation This patch does a lot. - Ditch sphinx in favor of hugo. This gives us complete control of the layout and presentation of our docs. Henceforth, docs will be hosted on our website rather than ReadTheDocs. - Create a simple, clean doc layout and theme. - Remove large parts of the documentaion. I've ditched anything that was a) woefully out of date, b) too detailed, or c) too hard to maintain in the long term. - Huge updates to the docs themselves: completely rewrite addons documentation, add docs for core concepts like commands and options, and revise and tweak a lot of the existing docs. With this patch, we're also changing the way we publish and maintain the docs. From now on, we don't publish docs for every release. Instead, the website will contain ONE set of docs for each major release. The online docs will be updated if needed as minor releases are made. Docs are free to improve during minor releases, but anything that changes behaviour sufficiently to require a doc change warrants a new major release. This also leaves us free to progressively update and improve docs out of step with our release cadence. With this new scheme, I feel CI over the docs is less important. I've removed it for now, but won't object if someone wants to add it back in. --- docs/src/content/howto-transparent.md | 269 ++++++++++++++++++++++++++++++++++ 1 file changed, 269 insertions(+) create mode 100644 docs/src/content/howto-transparent.md (limited to 'docs/src/content/howto-transparent.md') diff --git a/docs/src/content/howto-transparent.md b/docs/src/content/howto-transparent.md new file mode 100644 index 00000000..224cb5ee --- /dev/null +++ b/docs/src/content/howto-transparent.md @@ -0,0 +1,269 @@ +--- +title: "Transparent Proxying" +menu: + howto: + weight: 1 +--- + +# Transparent Proxying + +When a transparent proxy is used, traffic is redirected into a proxy at the +network layer, without any client configuration being required. This makes +transparent proxying ideal for those situations where you can't change client +behaviour - proxy-oblivious mobile applications being a common example. + +To set up transparent proxying, we need two new components. The first is a +redirection mechanism that transparently reroutes a TCP connection destined for +a server on the Internet to a listening proxy server. This usually takes the +form of a firewall on the same host as the proxy server - +[iptables](http://www.netfilter.org/) on Linux or +[pf](https://en.wikipedia.org/wiki/PF_(firewall)) on OSX. When the proxy +receives a redirected connection, it sees a vanilla HTTP request, without a host +specification. This is where the second new component comes in - a host module +that allows us to query the redirector for the original destination of the TCP +connection. + +At the moment, mitmproxy supports transparent proxying on OSX Lion and above, +and all current flavors of Linux. + + +## Linux fully transparent mode + +By default mitmproxy will use its own local IP address for its server-side +connections. In case this isn't desired, the --spoof-source-address argument can +be used to use the client's IP address for server-side connections. The +following config is required for this mode to work: + +{{< highlight bash >}} +CLIENT_NET=192.168.1.0/24 +TABLE_ID=100 +MARK=1 + +echo "$TABLE_ID mitmproxy" >> /etc/iproute2/rt_tables +iptables -t mangle -A PREROUTING -d $CLIENT_NET -j MARK --set-mark $MARK +iptables -t nat \ + -A PREROUTING -p tcp -s $CLIENT_NET \ + --match multiport --dports 80,443 -j \ + REDIRECT --to-port 8080 + +ip rule add fwmark $MARK lookup $TABLE_ID +ip route add local $CLIENT_NET dev lo table $TABLE_ID +{{< / highlight >}} + +This mode does require root privileges though. There's a wrapper in the examples +directory called 'mitmproxy_shim.c', which will enable you to use this mode with +dropped privileges. It can be used as follows: + +{{< highlight bash >}} +gcc examples/complex/full_transparency_shim.c -o mitmproxy_shim -lcap +sudo chown root:root mitmproxy_shim +sudo chmod u+s mitmproxy_shim +./mitmproxy_shim $(which mitmproxy) -T --spoof-source-address +{{< / highlight >}} + + + +## Linux + +On Linux, mitmproxy integrates with the iptables redirection mechanism to +achieve transparent mode. + +### 1. [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding: + +{{< highlight bash >}} +sysctl -w net.ipv4.ip_forward=1 +sysctl -w net.ipv6.conf.all.forwarding=1 +{{< / highlight >}} + +You may also want to consider enabling this permanently in `/etc/sysctl.conf` or +newly created `/etc/sysctl.d/mitmproxy.conf`, see +[here](https://superuser.com/a/625852). + +### 3. If your target machine is on the same physical network and you configured it to use a custom gateway, disable ICMP redirects: + +{{< highlight bash >}} +sysctl -w net.ipv4.conf.all.send_redirects=0 +{{< / highlight >}} + +You may also want to consider enabling this permanently in `/etc/sysctl.conf` or +a newly created `/etc/sysctl.d/mitmproxy.conf`, see +[here](https://superuser.com/a/625852). + +### 4. Create an iptables ruleset that redirects the desired traffic to the mitmproxy port + +Details will differ according to your setup, but the ruleset should look +something like this: + +{{< highlight bash >}} + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 + iptables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080 + ip6tables -t nat -A PREROUTING -i eth0 -p tcp --dport 80 -j REDIRECT --to-port 8080 + ip6tables -t nat -A PREROUTING -i eth0 -p tcp --dport 443 -j REDIRECT --to-port 8080 +{{< / highlight >}} + +   You may also want to consider enabling this permanently with the +`iptables-persistent` package, see +[here](http://www.microhowto.info/howto/make_the_configuration_of_iptables_persistent_on_debian.html). + +### 5. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells + mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. For a detailed walkthrough, have a look at the [tutorial for +transparently proxying VMs]({{< relref "howto-transparent-vms" >}}). + + +## OpenBSD + +### 1 [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding + +{{< highlight bash >}} +sudo sysctl -w net.inet.ip.forwarding=1 +{{< / highlight >}} + +### 3. Place the following two lines in **/etc/pf.conf** + +{{< highlight none >}} +mitm_if = "re2" +pass in quick proto tcp from $mitm_if to port { 80, 443 } divert-to 127.0.0.1 port 8080 +{{< / highlight >}} + +These rules tell pf to divert all traffic from `$mitm_if` destined for port 80 +or 443 to the local mitmproxy instance running on port 8080. You should replace +`$mitm_if` value with the interface on which your test device will appear. + +### 4. Enable the pf ruleset and enable it + +{{< highlight bash >}} +doas pfctl -f /etc/pf.conf +{{< / highlight >}} + +And now enable it: + +{{< highlight bash >}} +doas pfctl -e +{{< / highlight >}} + +### 5. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells +mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. + + +{{% note %}} +Note that the **divert-to** rules in the pf.conf given above only apply +to inbound traffic. **This means that they will NOT redirect traffic +coming from the box running pf itself.** We can't distinguish between an +outbound connection from a non-mitmproxy app, and an outbound connection +from mitmproxy itself - if you want to intercept your traffic, you +should use an external host to run mitmproxy. Nonetheless, pf is +flexible to cater for a range of creative possibilities, like +intercepting traffic emanating from VMs. See the **pf.conf** man page +for more. +{{% /note %}} + + +## macOS + +OSX Lion integrated the [pf](https://en.wikipedia.org/wiki/PF_(firewall)) +packet filter from the OpenBSD project, which mitmproxy uses to implement +transparent mode on OSX. Note that this means we don't support transparent mode +for earlier versions of OSX. + +### 1. [Install the mitmproxy certificate on the test device]({{< relref "concepts-certificates" >}}) + +### 2. Enable IP forwarding + +{{< highlight bash >}} +sudo sysctl -w net.inet.ip.forwarding=1 +{{< / highlight >}} + +### 3. Place the following two lines in a file called, say, **pf.conf** + + +{{< highlight none >}} +rdr on en0 inet proto tcp to any port {80, 443} -> 127.0.0.1 port 8080 +{{< / highlight >}} + +These rules tell pf to redirect all traffic destined for port 80 or 443 +to the local mitmproxy instance running on port 8080. You should replace +`en2` with the interface on which your test device will appear. + +### 4. Configure pf with the rules + +{{< highlight bash >}} +sudo pfctl -f pf.conf +{{< / highlight >}} + +### 5. And now enable it + +{{< highlight bash >}} +sudo pfctl -e +{{< / highlight >}} + +### 6. Configure sudoers to allow mitmproxy to access pfctl + +Edit the file **/etc/sudoers** on your system as root. Add the following line to +the end of the file: + +{{< highlight none >}} +ALL ALL=NOPASSWD: /sbin/pfctl -s state +{{< / highlight >}} + +Note that this allows any user on the system to run the command `/sbin/pfctl -s +state` as root without a password. This only allows inspection of the state +table, so should not be an undue security risk. If you're special feel free to +tighten the restriction up to the user running mitmproxy. + +### 7. Fire up mitmproxy + +You probably want a command like this: + +{{< highlight bash >}} +mitmproxy -T --host +{{< / highlight >}} + +The `-T` flag turns on transparent mode, and the `--host` argument tells +mitmproxy to use the value of the Host header for URL display. + +### 6. Finally, configure your test device + +Set the test device up to use the host on which mitmproxy is running as the +default gateway. + +{{% note %}} +Note that the **rdr** rules in the pf.conf given above only apply to +inbound traffic. **This means that they will NOT redirect traffic coming +from the box running pf itself.** We can't distinguish between an +outbound connection from a non-mitmproxy app, and an outbound connection +from mitmproxy itself - if you want to intercept your OSX traffic, you +should use an external host to run mitmproxy. Nonetheless, pf is +flexible to cater for a range of creative possibilities, like +intercepting traffic emanating from VMs. See the **pf.conf** man page +for more. +{{% /note %}} -- cgit v1.2.3