organize examples

This commit is largely based on work by Thiago Arrais (@thiagoarrais)
and Shane Bradfield (@l33tLumberjack). I wasn't really able to get their
PR reasonably merged onto the latest master, so I reapplied their changes
manually here and did some further improvements on that.

40 files changed, 159 insertions, 236 deletions

diff --git a/examples/README b/examples/README
deleted file mode 100644
index 90edf468..00000000
--- a/examples/README
+++ /dev/null
@@ -1,31 +0,0 @@
-Some inline scripts may require additional dependencies, which can be installed using
-`pip install mitmproxy[examples]`.
-
-
-# inline script examples
-add_header.py             Simple script that just adds a header to every request.
-change_upstream_proxy.py  Dynamically change the upstream proxy
-dns_spoofing.py           Use mitmproxy in a DNS spoofing scenario.
-dup_and_replay.py         Duplicates each request, changes it, and then replays the modified request.
-fail_with_500.py          Turn every response into an Internal Server Error.
-filt.py                   Use mitmproxy's filter expressions in your script.
-flowwriter.py             Only write selected flows into a mitmproxy dumpfile.
-iframe_injector.py        Inject configurable iframe into pages.
-modify_form.py            Modify all form submissions to add a parameter.
-modify_querystring.py     Modify all query strings to add a parameters.
-modify_response_body.py   Replace arbitrary strings in all responses
-nonblocking.py            Demonstrate parallel processing with a blocking script.
-proxapp.py                How to embed a WSGI app in a mitmproxy server
-redirect_requests.py      Redirect requests or directly reply to them.
-stub.py                   Script stub with a method definition for every event.
-upsidedownternet.py       Rewrites traffic to turn images upside down.
-
-
-# mitmproxy examples
-flowbasic                 Basic use of mitmproxy as a library.
-stickycookies             An example of writing a custom proxy with mitmproxy.
-
-
-# misc
-read_dumpfile             Read a dumpfile generated by mitmproxy.
-mitmproxywrapper.py       Bracket mitmproxy run with proxy enable/disable on OS X
diff --git a/examples/README.md b/examples/README.md
new file mode 100644
index 00000000..f46f322d
--- /dev/null
+++ b/examples/README.md
@@ -0,0 +1,15 @@
+# Mitmproxy Scripting API
+
+Mitmproxy has a powerful scripting API that allows you to control almost any aspect of traffic being 
+proxied. In fact, much of mitmproxy’s own core functionality is implemented using the exact same API 
+exposed to scripters (see [mitmproxy/addons](../mitmproxy/addons)).
+
+This directory contains some examples of the scripting API. We recommend to start with the
+ones in [simple/](./simple).
+
+|  :warning: | If you are browsing this on GitHub, make sure to select the git tag matching your mitmproxy version. |
+|------------|------------------------------------------------------------------------------------------------------|
+
+
+Some inline scripts may require additional dependencies, which can be installed using
+`pip install mitmproxy[examples]`.
\ No newline at end of file
diff --git a/examples/complex/README.md b/examples/complex/README.md
new file mode 100644
index 00000000..d3b2e77a
--- /dev/null
+++ b/examples/complex/README.md
@@ -0,0 +1,19 @@
+## Complex Examples
+
+| Filename                 | Description                                                                                   |
+|:-------------------------|:----------------------------------------------------------------------------------------------|
+| change_upstream_proxy.py | Dynamically change the upstream proxy.                                                        |
+| dns_spoofing.py          | Use mitmproxy in a DNS spoofing scenario.                                                     |
+| dup_and_replay.py        | Duplicates each request, changes it, and then replays the modified request.                   |
+| flowbasic.py             | Basic use of mitmproxy's FlowMaster directly.                                                 |
+| full_transparency_shim.c | Setuid wrapper that can be used to run mitmproxy in full transparency mode, as a normal user. |
+| har_dump.py              | Dump flows as HAR files.                                                                      |
+| mitmproxywrapper.py      | Bracket mitmproxy run with proxy enable/disable on OS X                                       |
+| nonblocking.py           | Demonstrate parallel processing with a blocking script                                        |
+| remote_debug.py          | This script enables remote debugging of the mitmproxy _UI_ with PyCharm.                      |
+| sslstrip.py              | sslstrip-like funtionality implemented with mitmproxy                                         |
+| stickycookies            | An advanced example of using mitmproxy's FlowMaster directly.                                 |
+| stream                   | Enable streaming for all responses.                                                           |
+| stream_modify.py         | Modify a streamed response body.                                                              |
+| tcp_message.py           | Modify a raw TCP connection                                                                   |
+| tls_passthrough.py       | Use conditional TLS interception based on a user-defined strategy.                            |
\ No newline at end of file
diff --git a/examples/change_upstream_proxy.py b/examples/complex/change_upstream_proxy.py
index 49d5379f..49d5379f 100644
--- a/examples/change_upstream_proxy.py
+++ b/examples/complex/change_upstream_proxy.py
diff --git a/examples/dns_spoofing.py b/examples/complex/dns_spoofing.py
index c020047f..c020047f 100644
--- a/examples/dns_spoofing.py
+++ b/examples/complex/dns_spoofing.py
diff --git a/examples/dup_and_replay.py b/examples/complex/dup_and_replay.py
index bf7c2a4e..bf7c2a4e 100644
--- a/examples/dup_and_replay.py
+++ b/examples/complex/dup_and_replay.py
diff --git a/examples/flowbasic b/examples/complex/flowbasic.py
index cb1e4ea4..25b0b1a9 100755..100644
--- a/examples/flowbasic
+++ b/examples/complex/flowbasic.py
@@ -35,6 +35,7 @@ class MyMaster(master.Master):
     def log(self, l):
         print("log", l.msg)
 
+
 opts = options.Options(cadir="~/.mitmproxy/")
 config = ProxyConfig(opts)
 server = ProxyServer(config)
diff --git a/examples/full_transparency_shim.c b/examples/complex/full_transparency_shim.c
index 923eea76..923eea76 100644
--- a/examples/full_transparency_shim.c
+++ b/examples/complex/full_transparency_shim.c
diff --git a/examples/har_dump.py b/examples/complex/har_dump.py
index aeb154d2..aeb154d2 100644
--- a/examples/har_dump.py
+++ b/examples/complex/har_dump.py
diff --git a/examples/mitmproxywrapper.py b/examples/complex/mitmproxywrapper.py
index eade0fe2..eade0fe2 100644
--- a/examples/mitmproxywrapper.py
+++ b/examples/complex/mitmproxywrapper.py
diff --git a/examples/nonblocking.py b/examples/complex/nonblocking.py
index 264a1fdb..264a1fdb 100644
--- a/examples/nonblocking.py
+++ b/examples/complex/nonblocking.py
diff --git a/examples/remote_debug.py b/examples/complex/remote_debug.py
index fb864f78..fb864f78 100644
--- a/examples/remote_debug.py
+++ b/examples/complex/remote_debug.py
diff --git a/examples/sslstrip.py b/examples/complex/sslstrip.py
index 9a090c0c..2f60c8b9 100644
--- a/examples/sslstrip.py
+++ b/examples/complex/sslstrip.py
@@ -1,3 +1,7 @@
+"""
+This script implements an sslstrip-like attack based on mitmproxy.
+https://moxie.org/software/sslstrip/
+"""
 import re
 import urllib
 
@@ -28,7 +32,7 @@ def response(flow):
     flow.response.headers.pop('Public-Key-Pins', None)
 
     # strip links in response body
-    flow.response.content = flow.response.content.replace('https://', 'http://')
+    flow.response.content = flow.response.content.replace(b'https://', b'http://')
 
     # strip meta tag upgrade-insecure-requests in response body
     csp_meta_tag_pattern = b'<meta.*http-equiv=["\']Content-Security-Policy[\'"].*upgrade-insecure-requests.*?>'
diff --git a/examples/stickycookies b/examples/complex/stickycookies
index a0ee90ff..4631fa73 100755..100644
--- a/examples/stickycookies
+++ b/examples/complex/stickycookies
@@ -6,7 +6,7 @@ implement functionality similar to the "sticky cookies" option.
 Heads Up: In the majority of cases, you want to use inline scripts.
 """
 import os
-from mitmproxy import controller, proxy
+from mitmproxy import controller, proxy, master
 from mitmproxy.proxy.server import ProxyServer
 
 
diff --git a/examples/stream.py b/examples/complex/stream.py
index 8598f329..1993cf7f 100644
--- a/examples/stream.py
+++ b/examples/complex/stream.py
@@ -1,5 +1,6 @@
 def responseheaders(flow):
     """
     Enables streaming for all responses.
+    This is equivalent to passing `--stream 0` to mitmproxy.
     """
     flow.response.stream = True
diff --git a/examples/stream_modify.py b/examples/complex/stream_modify.py
index 5e5da95b..5e5da95b 100644
--- a/examples/stream_modify.py
+++ b/examples/complex/stream_modify.py
diff --git a/examples/tcp_message.py b/examples/complex/tcp_message.py
index d7c9c42e..d7c9c42e 100644
--- a/examples/tcp_message.py
+++ b/examples/complex/tcp_message.py
diff --git a/examples/tls_passthrough.py b/examples/complex/tls_passthrough.py
index 40c1051d..40c1051d 100644
--- a/examples/tls_passthrough.py
+++ b/examples/complex/tls_passthrough.py
diff --git a/examples/context_logging.py b/examples/context_logging.py
deleted file mode 100644
index dccfd8b2..00000000
--- a/examples/context_logging.py
+++ /dev/null
@@ -1,6 +0,0 @@
-from mitmproxy import ctx
-
-
-def start():
-    ctx.log.info("This is some informative text.")
-    ctx.log.error("This is an error.")
diff --git a/examples/custom_contentviews.py b/examples/custom_contentviews.py
deleted file mode 100644
index 3558eaca..00000000
--- a/examples/custom_contentviews.py
+++ /dev/null
@@ -1,70 +0,0 @@
-import string
-import lxml.html
-import lxml.etree
-from mitmproxy import contentviews
-from mitmproxy.utils import strutils
-
-
-class ViewPigLatin(contentviews.View):
-    name = "pig_latin_HTML"
-    prompt = ("pig latin HTML", "l")
-    content_types = ["text/html"]
-
-    def __call__(self, data, **metadata):
-        if strutils.is_xml(data):
-            parser = lxml.etree.HTMLParser(
-                strip_cdata=True,
-                remove_blank_text=True
-            )
-            d = lxml.html.fromstring(data, parser=parser)
-            docinfo = d.getroottree().docinfo
-
-            def piglify(src):
-                words = src.split()
-                ret = ''
-                for word in words:
-                    idx = -1
-                    while word[idx] in string.punctuation and (idx * -1) != len(word):
-                        idx -= 1
-                    if word[0].lower() in 'aeiou':
-                        if idx == -1:
-                            ret += word[0:] + "hay"
-                        else:
-                            ret += word[0:len(word) + idx + 1] + "hay" + word[idx + 1:]
-                    else:
-                        if idx == -1:
-                            ret += word[1:] + word[0] + "ay"
-                        else:
-                            ret += word[1:len(word) + idx + 1] + word[0] + "ay" + word[idx + 1:]
-                    ret += ' '
-                return ret.strip()
-
-            def recurse(root):
-                if hasattr(root, 'text') and root.text:
-                    root.text = piglify(root.text)
-                if hasattr(root, 'tail') and root.tail:
-                    root.tail = piglify(root.tail)
-
-                if len(root):
-                    for child in root:
-                        recurse(child)
-
-            recurse(d)
-
-            s = lxml.etree.tostring(
-                d,
-                pretty_print=True,
-                doctype=docinfo.doctype
-            )
-            return "HTML", contentviews.format_text(s)
-
-
-pig_view = ViewPigLatin()
-
-
-def start():
-    contentviews.add(pig_view)
-
-
-def done():
-    contentviews.remove(pig_view)
diff --git a/examples/fail_with_500.py b/examples/fail_with_500.py
deleted file mode 100644
index 9710f74a..00000000
--- a/examples/fail_with_500.py
+++ /dev/null
@@ -1,3 +0,0 @@
-def response(flow):
-    flow.response.status_code = 500
-    flow.response.content = b""
diff --git a/examples/redirect_requests.py b/examples/redirect_requests.py
deleted file mode 100644
index c28042db..00000000
--- a/examples/redirect_requests.py
+++ /dev/null
@@ -1,18 +0,0 @@
-"""
-This example shows two ways to redirect flows to other destinations.
-"""
-from mitmproxy import http
-
-
-def request(flow):
-    # pretty_host takes the "Host" header of the request into account,
-    # which is useful in transparent mode where we usually only have the IP
-    # otherwise.
-
-    # Method 1: Answer with a locally generated response
-    if flow.request.pretty_host.endswith("example.com"):
-        flow.response = http.HTTPResponse.make(200, b"Hello World", {"Content-Type": "text/html"})
-
-    # Method 2: Redirect the request to a different server
-    if flow.request.pretty_host.endswith("example.org"):
-        flow.request.host = "mitmproxy.org"
diff --git a/examples/simple/README.md b/examples/simple/README.md
new file mode 100644
index 00000000..52f15627
--- /dev/null
+++ b/examples/simple/README.md
@@ -0,0 +1,18 @@
+## Simple Examples
+
+| Filename                     | Description                                                                |
+|:-----------------------------|:---------------------------------------------------------------------------|
+| add_header.py                | Simple script that just adds a header to every request.                    |
+| custom_contentview.py        | Add a custom content view to the mitmproxy UI.                             |
+| filter_flows.py              | This script demonstrates how to use mitmproxy's filter pattern in scripts. |
+| io_read_dumpfile.py          | Read a dumpfile generated by mitmproxy.                                    |
+| io_write_dumpfile.py         | Only write selected flows into a mitmproxy dumpfile.                       |
+| logging.py                   | Use mitmproxy's logging API.                                               |
+| modify_body_inject_iframe.py | Inject configurable iframe into pages.                                     |
+| modify_form.py               | Modify HTTP form submissions.                                              |
+| modify_querystring.py        | Modify HTTP query strings.                                                 |
+| redirect_requests.py         | Redirect a request to a different server.                                  |
+| script_arguments.py          | Add arguments to a script.                                                 |
+| send_reply_from_proxy.py     | Send a HTTP response directly from the proxy.                              |
+| upsidedownternet.py          | Turn all images upside down.                                               |
+| wsgi_flask_app.py            | Embed a WSGI app into mitmproxy.                                           |
\ No newline at end of file
diff --git a/examples/add_header.py b/examples/simple/add_header.py
index 3e0b5f1e..3e0b5f1e 100644
--- a/examples/add_header.py
+++ b/examples/simple/add_header.py
diff --git a/examples/classes.py b/examples/simple/add_header_class.py
index 6443798a..6443798a 100644
--- a/examples/classes.py
+++ b/examples/simple/add_header_class.py
diff --git a/examples/simple/custom_contentview.py b/examples/simple/custom_contentview.py
new file mode 100644
index 00000000..35216397
--- /dev/null
+++ b/examples/simple/custom_contentview.py
@@ -0,0 +1,28 @@
+"""
+This example shows how one can add a custom contentview to mitmproxy.
+The content view API is explained in the mitmproxy.contentviews module.
+"""
+from mitmproxy import contentviews
+
+
+class ViewSwapCase(contentviews.View):
+    name = "swapcase"
+
+    # We don't have a good solution for the keyboard shortcut yet -
+    # you manually need to find a free letter. Contributions welcome :)
+    prompt = ("swap case text", "p")
+    content_types = ["text/plain"]
+
+    def __call__(self, data: bytes, **metadata):
+        return "case-swapped text", contentviews.format_text(data.swapcase())
+
+
+view = ViewSwapCase()
+
+
+def start():
+    contentviews.add(view)
+
+
+def done():
+    contentviews.remove(view)
diff --git a/examples/flowfilter.py b/examples/simple/filter_flows.py
index 34d97275..29d0a9b8 100644
--- a/examples/flowfilter.py
+++ b/examples/simple/filter_flows.py
@@ -1,6 +1,8 @@
-# This scripts demonstrates how to use mitmproxy's filter pattern in scripts.
-# Usage: mitmdump -s "flowfilter.py FILTER"
-
+"""
+This scripts demonstrates how to use mitmproxy's filter pattern in scripts.
+Usage:
+    mitmdump -s "flowfilter.py FILTER"
+"""
 import sys
 from mitmproxy import flowfilter
 
diff --git a/examples/read_dumpfile b/examples/simple/io_read_dumpfile.py
index e0e9064a..edbbe2dd 100644
--- a/examples/read_dumpfile
+++ b/examples/simple/io_read_dumpfile.py
@@ -3,7 +3,7 @@
 # Simple script showing how to read a mitmproxy dump file
 #
 
-from mitmproxy import flow
+from mitmproxy import io
 from mitmproxy.exceptions import FlowReadException
 import pprint
 import sys
diff --git a/examples/flowwriter.py b/examples/simple/io_write_dumpfile.py
index a9768542..ff1fd0f4 100644
--- a/examples/flowwriter.py
+++ b/examples/simple/io_write_dumpfile.py
@@ -1,3 +1,10 @@
+"""
+This script how to generate a mitmproxy dump file,
+as it would also be generated by passing `-w` to mitmproxy.
+In contrast to `-w`, this gives you full control over which
+flows should be saved and also allows you to rotate files or log
+to multiple files in parallel.
+"""
 import random
 import sys
 from mitmproxy import io
diff --git a/examples/simple/logging.py b/examples/simple/logging.py
new file mode 100644
index 00000000..ab1baf75
--- /dev/null
+++ b/examples/simple/logging.py
@@ -0,0 +1,12 @@
+"""
+It is recommended to use `ctx.log` for logging within a script.
+This goes to the event log in mitmproxy and to stdout in mitmdump.
+
+If you want to help us out: https://github.com/mitmproxy/mitmproxy/issues/1530 :-)
+"""
+from mitmproxy import ctx
+
+
+def start():
+    ctx.log.info("This is some informative text.")
+    ctx.log.error("This is an error.")
diff --git a/examples/iframe_injector.py b/examples/simple/modify_body_inject_iframe.py
index 33d18bbd..33d18bbd 100644
--- a/examples/iframe_injector.py
+++ b/examples/simple/modify_body_inject_iframe.py
diff --git a/examples/modify_form.py b/examples/simple/modify_form.py
index b63a1586..b425efb0 100644
--- a/examples/modify_form.py
+++ b/examples/simple/modify_form.py
@@ -1,7 +1,9 @@
 def request(flow):
     if flow.request.urlencoded_form:
+        # If there's already a form, one can just add items to the dict:
         flow.request.urlencoded_form["mitmproxy"] = "rocks"
     else:
+        # One can also just pass new form data.
         # This sets the proper content type and overrides the body.
         flow.request.urlencoded_form = [
             ("foo", "bar")
diff --git a/examples/modify_querystring.py b/examples/simple/modify_querystring.py
index ee8a89ad..ee8a89ad 100644
--- a/examples/modify_querystring.py
+++ b/examples/simple/modify_querystring.py
diff --git a/examples/simple/redirect_requests.py b/examples/simple/redirect_requests.py
new file mode 100644
index 00000000..51876df7
--- /dev/null
+++ b/examples/simple/redirect_requests.py
@@ -0,0 +1,11 @@
+"""
+This example shows two ways to redirect flows to another server.
+"""
+
+
+def request(flow):
+    # pretty_host takes the "Host" header of the request into account,
+    # which is useful in transparent mode where we usually only have the IP
+    # otherwise.
+    if flow.request.pretty_host == "example.org":
+        flow.request.host = "mitmproxy.org"
diff --git a/examples/arguments.py b/examples/simple/script_arguments.py
index 70851192..70851192 100644
--- a/examples/arguments.py
+++ b/examples/simple/script_arguments.py
diff --git a/examples/simple/send_reply_from_proxy.py b/examples/simple/send_reply_from_proxy.py
new file mode 100644
index 00000000..bef2e7e7
--- /dev/null
+++ b/examples/simple/send_reply_from_proxy.py
@@ -0,0 +1,17 @@
+"""
+This example shows how to send a reply from the proxy immediately
+without sending any data to the remote server.
+"""
+from mitmproxy import http
+
+
+def request(flow):
+    # pretty_url takes the "Host" header of the request into account, which
+    # is useful in transparent mode where we usually only have the IP otherwise.
+
+    if flow.request.pretty_url == "http://example.com/path":
+        flow.response = http.HTTPResponse.make(
+            200,  # (optional) status code
+            b"Hello World",  # (optional) content
+            {"Content-Type": "text/html"}  # (optional) headers
+        )
diff --git a/examples/simple/upsidedownternet.py b/examples/simple/upsidedownternet.py
new file mode 100644
index 00000000..8ba450ab
--- /dev/null
+++ b/examples/simple/upsidedownternet.py
@@ -0,0 +1,16 @@
+"""
+This script rotates all images passing through the proxy by 180 degrees.
+"""
+import io
+
+from PIL import Image
+
+
+def response(flow):
+    if flow.response.headers.get("content-type", "").startswith("image"):
+        s = io.BytesIO(flow.response.content)
+        img = Image.open(s).rotate(180)
+        s2 = io.BytesIO()
+        img.save(s2, "png")
+        flow.response.content = s2.getvalue()
+        flow.response.headers["content-type"] = "image/png"
diff --git a/examples/proxapp.py b/examples/simple/wsgi_flask_app.py
index f95c41e5..f95c41e5 100644
--- a/examples/proxapp.py
+++ b/examples/simple/wsgi_flask_app.py
diff --git a/examples/stub.py b/examples/stub.py
deleted file mode 100644
index 4f5061e2..00000000
--- a/examples/stub.py
+++ /dev/null
@@ -1,87 +0,0 @@
-import mitmproxy
-"""
-    This is a script stub, with definitions for all events.
-"""
-
-
-def start():
-    """
-        Called once on script startup before any other events
-    """
-    mitmproxy.ctx.log("start")
-
-
-def configure(options, updated):
-    """
-        Called once on script startup before any other events, and whenever options changes.
-    """
-    mitmproxy.ctx.log("configure")
-
-
-def clientconnect(root_layer):
-    """
-        Called when a client initiates a connection to the proxy. Note that a
-        connection can correspond to multiple HTTP requests
-    """
-    mitmproxy.ctx.log("clientconnect")
-
-
-def request(flow):
-    """
-        Called when a client request has been received.
-    """
-    mitmproxy.ctx.log("request")
-
-
-def serverconnect(server_conn):
-    """
-        Called when the proxy initiates a connection to the target server. Note that a
-        connection can correspond to multiple HTTP requests
-    """
-    mitmproxy.ctx.log("serverconnect")
-
-
-def responseheaders(flow):
-    """
-        Called when the response headers for a server response have been received,
-        but the response body has not been processed yet. Can be used to tell mitmproxy
-        to stream the response.
-    """
-    mitmproxy.ctx.log("responseheaders")
-
-
-def response(flow):
-    """
-       Called when a server response has been received.
-    """
-    mitmproxy.ctx.log("response")
-
-
-def error(flow):
-    """
-        Called when a flow error has occured, e.g. invalid server responses, or
-        interrupted connections. This is distinct from a valid server HTTP error
-        response, which is simply a response with an HTTP error code.
-    """
-    mitmproxy.ctx.log("error")
-
-
-def serverdisconnect(server_conn):
-    """
-        Called when the proxy closes the connection to the target server.
-    """
-    mitmproxy.ctx.log("serverdisconnect")
-
-
-def clientdisconnect(root_layer):
-    """
-        Called when a client disconnects from the proxy.
-    """
-    mitmproxy.ctx.log("clientdisconnect")
-
-
-def done():
-    """
-        Called once on script shutdown, after any other events.
-    """
-    mitmproxy.ctx.log("done")
diff --git a/examples/upsidedownternet.py b/examples/upsidedownternet.py
deleted file mode 100644
index d4de7e25..00000000
--- a/examples/upsidedownternet.py
+++ /dev/null
@@ -1,15 +0,0 @@
-import io
-from PIL import Image
-
-
-def response(flow):
-    if flow.response.headers.get("content-type", "").startswith("image"):
-        try:
-            s = io.StringIO(flow.response.content)
-            img = Image.open(s).rotate(180)
-            s2 = io.StringIO()
-            img.save(s2, "png")
-            flow.response.content = s2.getvalue()
-            flow.response.headers["content-type"] = "image/png"
-        except:  # Unknown image types etc.
-            pass