From 74b3b842fefec6a05d17bbdf365cd92c82fd3503 Mon Sep 17 00:00:00 2001 From: Maximilian Hils Date: Fri, 4 Sep 2015 16:17:55 +0200 Subject: rewrite basic docs for readthedocs --- doc-src/dev-docs/.gitignore | 1 - doc-src/dev-docs/Makefile | 195 ----------------- doc-src/dev-docs/_static/favicon.ico | Bin 5430 -> 0 bytes doc-src/dev-docs/_static/mitmproxy-long.png | Bin 123829 -> 0 bytes doc-src/dev-docs/conf.py | 215 ------------------- doc-src/dev-docs/exceptions.rst | 9 - doc-src/dev-docs/index.rst | 24 --- doc-src/dev-docs/inlinescripts.rst | 211 ------------------ doc-src/dev-docs/models.rst | 19 -- doc-src/dev-docs/protocols.rst | 15 -- doc-src/dev-docs/proxy.rst | 12 -- docs/.gitignore | 1 + docs/Makefile | 195 +++++++++++++++++ docs/certinstall-webapp.png | Bin 0 -> 61683 bytes docs/certinstall.rst | 173 +++++++++++++++ docs/conf.py | 215 +++++++++++++++++++ docs/dev/exceptions.rst | 9 + docs/dev/models.rst | 19 ++ docs/dev/protocols.rst | 15 ++ docs/dev/proxy.rst | 12 ++ docs/favicon.ico | Bin 0 -> 5430 bytes docs/features/upstreamcerts.rst | 4 + docs/howmitmproxy.rst | 238 +++++++++++++++++++++ docs/index.rst | 43 ++++ docs/install.rst | 100 +++++++++ docs/introduction.rst | 26 +++ docs/mitmdump.rst | 4 + docs/mitmproxy-long.png | Bin 0 -> 123829 bytes docs/mitmproxy.rst | 4 + docs/modes.rst | 193 +++++++++++++++++ docs/schematics/architecture.pdf | Bin 0 -> 182446 bytes docs/schematics/architecture.png | Bin 0 -> 87365 bytes docs/schematics/architecture.vsdx | Bin 0 -> 60922 bytes .../how-mitmproxy-works-explicit-https.png | Bin 0 -> 78951 bytes docs/schematics/how-mitmproxy-works-explicit.png | Bin 0 -> 65305 bytes .../how-mitmproxy-works-transparent-https.png | Bin 0 -> 79758 bytes .../schematics/how-mitmproxy-works-transparent.png | Bin 0 -> 69375 bytes docs/schematics/proxy-modes-flowchart.png | Bin 0 -> 34635 bytes docs/schematics/proxy-modes-regular.png | Bin 0 -> 18283 bytes docs/schematics/proxy-modes-reverse.png | Bin 0 -> 16719 bytes docs/schematics/proxy-modes-transparent-1.png | Bin 0 -> 14558 bytes docs/schematics/proxy-modes-transparent-2.png | Bin 0 -> 23375 bytes docs/schematics/proxy-modes-transparent-3.png | Bin 0 -> 23855 bytes docs/schematics/proxy-modes-transparent-wrong.png | Bin 0 -> 14719 bytes docs/schematics/proxy-modes-upstream.png | Bin 0 -> 14781 bytes docs/schematics/proxy-modes.pdf | Bin 0 -> 335485 bytes docs/schematics/proxy-modes.vsdx | Bin 0 -> 190788 bytes docs/screenshots/firefox3-import.jpg | Bin 0 -> 55496 bytes docs/screenshots/firefox3-trust.jpg | Bin 0 -> 31495 bytes docs/screenshots/firefox3.jpg | Bin 0 -> 57366 bytes docs/screenshots/ios-gateway.png | Bin 0 -> 154469 bytes docs/screenshots/ios-installed.png | Bin 0 -> 80251 bytes docs/screenshots/ios-manual.png | Bin 0 -> 196431 bytes docs/screenshots/ios-profile.png | Bin 0 -> 83364 bytes docs/screenshots/ios-reverse.png | Bin 0 -> 66150 bytes docs/screenshots/ios-warning.png | Bin 0 -> 75604 bytes docs/screenshots/mitmproxy-flowview.png | Bin 0 -> 315864 bytes docs/screenshots/mitmproxy-intercept-filt.png | Bin 0 -> 18332 bytes docs/screenshots/mitmproxy-intercept-mid.png | Bin 0 -> 19841 bytes docs/screenshots/mitmproxy-intercept-options.png | Bin 0 -> 41281 bytes docs/screenshots/mitmproxy-intercept-result.png | Bin 0 -> 22855 bytes docs/screenshots/mitmproxy-kveditor-editmode.png | Bin 0 -> 44528 bytes docs/screenshots/mitmproxy-kveditor.png | Bin 0 -> 44852 bytes docs/screenshots/mitmproxy.png | Bin 0 -> 152596 bytes docs/screenshots/osx-addcert-alwaystrust.png | Bin 0 -> 47146 bytes docs/screenshots/win7-certstore-trustedroot.png | Bin 0 -> 39236 bytes docs/screenshots/win7-certstore.png | Bin 0 -> 37453 bytes docs/screenshots/win7-wizard.png | Bin 0 -> 66456 bytes docs/screenshots/winpythoninstaller.jpg | Bin 0 -> 46628 bytes docs/scripting/inlinescripts.rst | 216 +++++++++++++++++++ docs/scripting/libmproxy.rst | 27 +++ docs/transparent.rst | 6 + 72 files changed, 1500 insertions(+), 701 deletions(-) delete mode 100644 doc-src/dev-docs/.gitignore delete mode 100644 doc-src/dev-docs/Makefile delete mode 100644 doc-src/dev-docs/_static/favicon.ico delete mode 100644 doc-src/dev-docs/_static/mitmproxy-long.png delete mode 100644 doc-src/dev-docs/conf.py delete mode 100644 doc-src/dev-docs/exceptions.rst delete mode 100644 doc-src/dev-docs/index.rst delete mode 100644 doc-src/dev-docs/inlinescripts.rst delete mode 100644 doc-src/dev-docs/models.rst delete mode 100644 doc-src/dev-docs/protocols.rst delete mode 100644 doc-src/dev-docs/proxy.rst create mode 100644 docs/.gitignore create mode 100644 docs/Makefile create mode 100644 docs/certinstall-webapp.png create mode 100644 docs/certinstall.rst create mode 100644 docs/conf.py create mode 100644 docs/dev/exceptions.rst create mode 100644 docs/dev/models.rst create mode 100644 docs/dev/protocols.rst create mode 100644 docs/dev/proxy.rst create mode 100644 docs/favicon.ico create mode 100644 docs/features/upstreamcerts.rst create mode 100644 docs/howmitmproxy.rst create mode 100644 docs/index.rst create mode 100644 docs/install.rst create mode 100644 docs/introduction.rst create mode 100644 docs/mitmdump.rst create mode 100644 docs/mitmproxy-long.png create mode 100644 docs/mitmproxy.rst create mode 100644 docs/modes.rst create mode 100644 docs/schematics/architecture.pdf create mode 100644 docs/schematics/architecture.png create mode 100644 docs/schematics/architecture.vsdx create mode 100644 docs/schematics/how-mitmproxy-works-explicit-https.png create mode 100644 docs/schematics/how-mitmproxy-works-explicit.png create mode 100644 docs/schematics/how-mitmproxy-works-transparent-https.png create mode 100644 docs/schematics/how-mitmproxy-works-transparent.png create mode 100644 docs/schematics/proxy-modes-flowchart.png create mode 100644 docs/schematics/proxy-modes-regular.png create mode 100644 docs/schematics/proxy-modes-reverse.png create mode 100644 docs/schematics/proxy-modes-transparent-1.png create mode 100644 docs/schematics/proxy-modes-transparent-2.png create mode 100644 docs/schematics/proxy-modes-transparent-3.png create mode 100644 docs/schematics/proxy-modes-transparent-wrong.png create mode 100644 docs/schematics/proxy-modes-upstream.png create mode 100644 docs/schematics/proxy-modes.pdf create mode 100644 docs/schematics/proxy-modes.vsdx create mode 100644 docs/screenshots/firefox3-import.jpg create mode 100644 docs/screenshots/firefox3-trust.jpg create mode 100644 docs/screenshots/firefox3.jpg create mode 100644 docs/screenshots/ios-gateway.png create mode 100644 docs/screenshots/ios-installed.png create mode 100644 docs/screenshots/ios-manual.png create mode 100644 docs/screenshots/ios-profile.png create mode 100644 docs/screenshots/ios-reverse.png create mode 100644 docs/screenshots/ios-warning.png create mode 100644 docs/screenshots/mitmproxy-flowview.png create mode 100644 docs/screenshots/mitmproxy-intercept-filt.png create mode 100644 docs/screenshots/mitmproxy-intercept-mid.png create mode 100644 docs/screenshots/mitmproxy-intercept-options.png create mode 100644 docs/screenshots/mitmproxy-intercept-result.png create mode 100644 docs/screenshots/mitmproxy-kveditor-editmode.png create mode 100644 docs/screenshots/mitmproxy-kveditor.png create mode 100644 docs/screenshots/mitmproxy.png create mode 100644 docs/screenshots/osx-addcert-alwaystrust.png create mode 100644 docs/screenshots/win7-certstore-trustedroot.png create mode 100644 docs/screenshots/win7-certstore.png create mode 100644 docs/screenshots/win7-wizard.png create mode 100644 docs/screenshots/winpythoninstaller.jpg create mode 100644 docs/scripting/inlinescripts.rst create mode 100644 docs/scripting/libmproxy.rst create mode 100644 docs/transparent.rst diff --git a/doc-src/dev-docs/.gitignore b/doc-src/dev-docs/.gitignore deleted file mode 100644 index 69fa449d..00000000 --- a/doc-src/dev-docs/.gitignore +++ /dev/null @@ -1 +0,0 @@ -_build/ diff --git a/doc-src/dev-docs/Makefile b/doc-src/dev-docs/Makefile deleted file mode 100644 index 666a3a86..00000000 --- a/doc-src/dev-docs/Makefile +++ /dev/null @@ -1,195 +0,0 @@ -# Makefile for Sphinx documentation -# - -# You can set these variables from the command line. -SPHINXOPTS = -SPHINXBUILD = sphinx-build -PAPER = -BUILDDIR = _build - -# User-friendly check for sphinx-build -ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) -$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) -endif - -# Internal variables. -PAPEROPT_a4 = -D latex_paper_size=a4 -PAPEROPT_letter = -D latex_paper_size=letter -ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . -# the i18n builder cannot share the environment and doctrees with the others -I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . - -.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext - -help: - @echo "Please use \`make ' where is one of" - @echo " html to make standalone HTML files" - @echo " dirhtml to make HTML files named index.html in directories" - @echo " singlehtml to make a single large HTML file" - @echo " pickle to make pickle files" - @echo " json to make JSON files" - @echo " htmlhelp to make HTML files and a HTML help project" - @echo " qthelp to make HTML files and a qthelp project" - @echo " applehelp to make an Apple Help Book" - @echo " devhelp to make HTML files and a Devhelp project" - @echo " epub to make an epub" - @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" - @echo " latexpdf to make LaTeX files and run them through pdflatex" - @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" - @echo " text to make text files" - @echo " man to make manual pages" - @echo " texinfo to make Texinfo files" - @echo " info to make Texinfo files and run them through makeinfo" - @echo " gettext to make PO message catalogs" - @echo " changes to make an overview of all changed/added/deprecated items" - @echo " xml to make Docutils-native XML files" - @echo " pseudoxml to make pseudoxml-XML files for display purposes" - @echo " linkcheck to check all external links for integrity" - @echo " doctest to run all doctests embedded in the documentation (if enabled)" - @echo " coverage to run coverage check of the documentation (if enabled)" - -clean: - rm -rf $(BUILDDIR)/* - -html: - $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." - -dirhtml: - $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml - @echo - @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." - -singlehtml: - $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml - @echo - @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." - -pickle: - $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle - @echo - @echo "Build finished; now you can process the pickle files." - -json: - $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json - @echo - @echo "Build finished; now you can process the JSON files." - -htmlhelp: - $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp - @echo - @echo "Build finished; now you can run HTML Help Workshop with the" \ - ".hhp project file in $(BUILDDIR)/htmlhelp." - -qthelp: - $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp - @echo - @echo "Build finished; now you can run "qcollectiongenerator" with the" \ - ".qhcp project file in $(BUILDDIR)/qthelp, like this:" - @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/mitmproxy.qhcp" - @echo "To view the help file:" - @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/mitmproxy.qhc" - -applehelp: - $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp - @echo - @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." - @echo "N.B. You won't be able to view it unless you put it in" \ - "~/Library/Documentation/Help or install it in your application" \ - "bundle." - -devhelp: - $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp - @echo - @echo "Build finished." - @echo "To view the help file:" - @echo "# mkdir -p $$HOME/.local/share/devhelp/mitmproxy" - @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/mitmproxy" - @echo "# devhelp" - -epub: - $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub - @echo - @echo "Build finished. The epub file is in $(BUILDDIR)/epub." - -latex: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo - @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." - @echo "Run \`make' in that directory to run these through (pdf)latex" \ - "(use \`make latexpdf' here to do that automatically)." - -latexpdf: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through pdflatex..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -latexpdfja: - $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex - @echo "Running LaTeX files through platex and dvipdfmx..." - $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja - @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." - -text: - $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text - @echo - @echo "Build finished. The text files are in $(BUILDDIR)/text." - -man: - $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man - @echo - @echo "Build finished. The manual pages are in $(BUILDDIR)/man." - -texinfo: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo - @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." - @echo "Run \`make' in that directory to run these through makeinfo" \ - "(use \`make info' here to do that automatically)." - -info: - $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo - @echo "Running Texinfo files through makeinfo..." - make -C $(BUILDDIR)/texinfo info - @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." - -gettext: - $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale - @echo - @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." - -changes: - $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes - @echo - @echo "The overview file is in $(BUILDDIR)/changes." - -linkcheck: - $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck - @echo - @echo "Link check complete; look for any errors in the above output " \ - "or in $(BUILDDIR)/linkcheck/output.txt." - -doctest: - $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest - @echo "Testing of doctests in the sources finished, look at the " \ - "results in $(BUILDDIR)/doctest/output.txt." - -coverage: - $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage - @echo "Testing of coverage in the sources finished, look at the " \ - "results in $(BUILDDIR)/coverage/python.txt." - -xml: - $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml - @echo - @echo "Build finished. The XML files are in $(BUILDDIR)/xml." - -pseudoxml: - $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml - @echo - @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." - -livehtml: - sphinx-autobuild -b html -z '../../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html \ No newline at end of file diff --git a/doc-src/dev-docs/_static/favicon.ico b/doc-src/dev-docs/_static/favicon.ico deleted file mode 100644 index 3c3b891c..00000000 Binary files a/doc-src/dev-docs/_static/favicon.ico and /dev/null differ diff --git a/doc-src/dev-docs/_static/mitmproxy-long.png b/doc-src/dev-docs/_static/mitmproxy-long.png deleted file mode 100644 index f9397d1e..00000000 Binary files a/doc-src/dev-docs/_static/mitmproxy-long.png and /dev/null differ diff --git a/doc-src/dev-docs/conf.py b/doc-src/dev-docs/conf.py deleted file mode 100644 index 23db112c..00000000 --- a/doc-src/dev-docs/conf.py +++ /dev/null @@ -1,215 +0,0 @@ -# -*- coding: utf-8 -*- -# -# mitmproxy documentation build configuration file, created by -# sphinx-quickstart on Thu Sep 03 14:04:13 2015. -# -# This file is execfile()d with the current directory set to its -# containing dir. -# -# Note that not all possible configuration values are present in this -# autogenerated file. -# -# All configuration values have a default; values that are commented out -# serve to show the default. - -import sys -import os -import shlex - -# If extensions (or modules to document with autodoc) are in another directory, -# add these directories to sys.path here. If the directory is relative to the -# documentation root, use os.path.abspath to make it absolute, like shown here. -sys.path.insert(0, os.path.abspath('..')) - -import libmproxy.version - -# -- General configuration ------------------------------------------------ - -# If your documentation needs a minimal Sphinx version, state it here. -#needs_sphinx = '1.0' - -# Add any Sphinx extension module names here, as strings. They can be -# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom -# ones. - -extensions = [ - 'sphinx.ext.autodoc', - 'sphinx.ext.doctest', - 'sphinx.ext.viewcode', - 'sphinx.ext.napoleon' -] - -autodoc_member_order = "bysource" - -# Add any paths that contain templates here, relative to this directory. -#templates_path = ['_templates'] - -# The suffix(es) of source filenames. -# You can specify multiple suffix as a list of string: -# source_suffix = ['.rst', '.md'] -source_suffix = '.rst' - -# The encoding of source files. -#source_encoding = 'utf-8-sig' - -# The master toctree document. -master_doc = 'index' - -# General information about the project. -project = u'mitmproxy dev docs' -copyright = u'2015, the mitmproxy project' -author = u'The mitmproxy project' - -# The version info for the project you're documenting, acts as replacement for -# |version| and |release|, also used in various other places throughout the -# built documents. -# -# The short X.Y version. -version = libmproxy.version.VERSION -# The full version, including alpha/beta/rc tags. -release = libmproxy.version.VERSION - -# The language for content autogenerated by Sphinx. Refer to documentation -# for a list of supported languages. -# -# This is also used if you do content translation via gettext catalogs. -# Usually you set "language" from the command line for these cases. -language = None - -# There are two options for replacing |today|: either, you set today to some -# non-false value, then it is used: -#today = '' -# Else, today_fmt is used as the format for a strftime call. -#today_fmt = '%B %d, %Y' - -# List of patterns, relative to source directory, that match files and -# directories to ignore when looking for source files. -exclude_patterns = ['_build'] - -# The reST default role (used for this markup: `text`) to use for all -# documents. -#default_role = None - -# If true, '()' will be appended to :func: etc. cross-reference text. -#add_function_parentheses = True - -# If true, the current module name will be prepended to all description -# unit titles (such as .. function::). -#add_module_names = True - -# If true, sectionauthor and moduleauthor directives will be shown in the -# output. They are ignored by default. -#show_authors = False - -# The name of the Pygments (syntax highlighting) style to use. -pygments_style = 'sphinx' - -# A list of ignored prefixes for module index sorting. -modindex_common_prefix = ['libmproxy.'] - -# If true, keep warnings as "system message" paragraphs in the built documents. -#keep_warnings = False - -# If true, `todo` and `todoList` produce output, else they produce nothing. -todo_include_todos = False - - -# -- Options for HTML output ---------------------------------------------- - -# The theme to use for HTML and HTML Help pages. See the documentation for -# a list of builtin themes. -html_theme = 'sphinx_rtd_theme' - -# Theme options are theme-specific and customize the look and feel of a theme -# further. For a list of options available for each theme, see the -# documentation. -html_theme_options = { - # 'logo_only': True, -} - -# Add any paths that contain custom themes here, relative to this directory. -#html_theme_path = [] - -# The name for this set of Sphinx documents. If None, it defaults to -# " v documentation". -html_title = "mitmproxy %s documentation" % version - -# A shorter title for the navigation bar. Default is the same as html_title. -#html_short_title = None - -# The name of an image file (relative to this directory) to place at the top -# of the sidebar. -html_logo = "mitmproxy-long.png" - -# The name of an image file (within the static path) to use as favicon of the -# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 -# pixels large. -html_favicon = "favicon.ico" - -# Add any paths that contain custom static files (such as style sheets) here, -# relative to this directory. They are copied after the builtin static files, -# so a file named "default.css" will overwrite the builtin "default.css". -html_static_path = ['_static'] - -# Add any extra paths that contain custom files (such as robots.txt or -# .htaccess) here, relative to this directory. These files are copied -# directly to the root of the documentation. -#html_extra_path = [] - -# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, -# using the given strftime format. -#html_last_updated_fmt = '%b %d, %Y' - -# If true, SmartyPants will be used to convert quotes and dashes to -# typographically correct entities. -#html_use_smartypants = True - -# Custom sidebar templates, maps document names to template names. -#html_sidebars = {} - -# Additional templates that should be rendered to pages, maps page names to -# template names. -#html_additional_pages = {} - -# If false, no module index is generated. -#html_domain_indices = True - -# If false, no index is generated. -#html_use_index = True - -# If true, the index is split into individual pages for each letter. -#html_split_index = False - -# If true, links to the reST sources are added to the pages. -#html_show_sourcelink = True - -# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. -#html_show_sphinx = True - -# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. -#html_show_copyright = True - -# If true, an OpenSearch description file will be output, and all pages will -# contain a tag referring to it. The value of this option must be the -# base URL from which the finished HTML is served. -#html_use_opensearch = '' - -# This is the file name suffix for HTML files (e.g. ".xhtml"). -#html_file_suffix = None - -# Language to be used for generating the HTML full-text search index. -# Sphinx supports the following languages: -# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' -# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' -#html_search_language = 'en' - -# A dictionary with options for the search language support, empty by default. -# Now only 'ja' uses this config value -#html_search_options = {'type': 'default'} - -# The name of a javascript file (relative to the configuration directory) that -# implements a search results scorer. If empty, the default will be used. -#html_search_scorer = 'scorer.js' - -# Output file base name for HTML help builder. -htmlhelp_basename = 'mitmproxydoc' \ No newline at end of file diff --git a/doc-src/dev-docs/exceptions.rst b/doc-src/dev-docs/exceptions.rst deleted file mode 100644 index d1e4bfe5..00000000 --- a/doc-src/dev-docs/exceptions.rst +++ /dev/null @@ -1,9 +0,0 @@ -.. _exceptions: - -Exceptions -========== - -.. automodule:: libmproxy.exceptions - :show-inheritance: - :members: - :undoc-members: \ No newline at end of file diff --git a/doc-src/dev-docs/index.rst b/doc-src/dev-docs/index.rst deleted file mode 100644 index 15ec3617..00000000 --- a/doc-src/dev-docs/index.rst +++ /dev/null @@ -1,24 +0,0 @@ -Welcome to mitmproxy's dev documentation! -========================================= - - -Contents: - -.. toctree:: - :maxdepth: 2 - - End User Documentation - - inlinescripts - protocols - proxy - exceptions - models - - -Indices and tables -================== - -* :ref:`genindex` -* :ref:`modindex` - diff --git a/doc-src/dev-docs/inlinescripts.rst b/doc-src/dev-docs/inlinescripts.rst deleted file mode 100644 index 0c53b1a4..00000000 --- a/doc-src/dev-docs/inlinescripts.rst +++ /dev/null @@ -1,211 +0,0 @@ -Inline Scripts -============== - -**mitmproxy** has a powerful scripting API that allows you to modify flows -on-the-fly or rewrite previously saved flows locally. - -The mitmproxy scripting API is event driven - a script is simply a Python -module that exposes a set of event methods. Here's a complete mitmproxy script -that adds a new header to every HTTP response before it is returned to the -client: - -.. literalinclude:: ../../examples/add_header.py - :caption: examples/add_header.py - :language: python - -The first argument to each event method is an instance of -:py:class:`~libmproxy.script.ScriptContext` that lets the script interact with the global mitmproxy -state. The **response** event also gets an instance of :py:class:`~libmproxy.script.ScriptContext`, -which we can use to manipulate the response itself. - -We can now run this script using mitmdump or mitmproxy as follows: - ->>> mitmdump -s add_header.py - -The new header will be added to all responses passing through the proxy. - -Examples --------- - -mitmproxy comes with a variety of example inline scripts, which demonstrate many basic tasks. -We encourage you to either browse them locally or on `GitHub`_. - - -Events ------- - -.. TODO: Split this into Connection, HTTP and TCP events once we have TCP events. - -The ``context`` argument passed to each event method is always a -:py:class:`~libmproxy.script.ScriptContext` instance. It is guaranteed to be the same object -for the scripts lifetime and is not shared between multiple inline scripts. You can safely use it -to store any form of state you require. - -Events are listed in the order they usually occur. - -.. py:function:: start(context, argv) - - Called once on startup, before any other events. - - :param List[str] argv: The inline scripts' arguments. - For example, ``mitmproxy -s 'example.py --foo 42'`` sets argv to ``["--foo", "42"]``. - -.. py:function:: clientconnect(context, root_layer) - - Called when a client initiates a connection to the proxy. Note that - a connection can correspond to multiple HTTP requests. - - .. versionchanged:: 0.14 - :param Layer root_layer: The root layer (see :ref:`protocols` for an explanation what the root - layer is), which provides transparent access to all attributes of the - :py:class:`~libmproxy.proxy.RootContext`. For example, ``root_layer.client_conn.address`` - gives the remote address of the connecting client. - - -.. py:function:: request(context, flow) - - Called when a client request has been received. The ``flow`` object is - guaranteed to have a non-None ``request`` attribute. - - :param HTTPFlow flow: The flow containing the request which has been received. - The object is guaranteed to have a non-None ``request`` attribute. - -.. py:function:: serverconnect(context, server_conn) - - Called before the proxy initiates a connection to the target server. Note that - a connection can correspond to multiple HTTP requests. - - :param ServerConnection server_conn: The server connection object. It is guaranteed to have a - non-None ``address`` attribute. - -.. py:function:: responseheaders(context, flow) - - Called when the headers of a server response have been received. - This will always be called before the response hook. - - :param HTTPFlow flow: The flow containing the request and response. - The object is guaranteed to have non-None ``request`` and - ``response`` attributes. ``response.content`` will be ``None``, - as the response body has not been read yet. - -.. py:function:: response(context, flow) - - Called when a server response has been received. - - :param HTTPFlow flow: The flow containing the request and response. - The object is guaranteed to have non-None ``request`` and - ``response`` attributes. ``response.body`` will contain the raw response body, - unless response streaming has been enabled. - -.. py:function:: error(context, flow) - - Called when a flow error has occurred, e.g. invalid server responses, or - interrupted connections. This is distinct from a valid server HTTP error - response, which is simply a response with an HTTP error code. - - :param HTTPFlow flow: The flow containing the error. - It is guaranteed to have non-None ``error`` attribute. - -.. py:function:: serverdisconnect(context, server_conn) - - Called when the proxy has closed the server connection. - - .. versionadded:: 0.14 - :param ServerConnection server_conn: see :py:func:`serverconnect` - -.. py:function:: clientdisconnect(context, root_layer) - - Called when a client disconnects from the proxy. - - .. versionchanged:: 0.14 - :param Layer root_layer: see :py:func:`clientconnect` - -.. py:function:: done(context) - - Called once on script shutdown, after any other events. - - -API ---- - -The canonical API documentation is the code, which you can browse here, locally or on `GitHub`_. -*Use the Source, Luke!* - -The main classes you will deal with in writing mitmproxy scripts are: - -:py:class:`~libmproxy.script.ScriptContext` - - A handle for interacting with mitmproxy's Flow Master from within scripts. -:py:class:`~libmproxy.models.ClientConnection` - - Describes a client connection. -:py:class:`~libmproxy.models.ServerConnection` - - Describes a server connection. -:py:class:`~libmproxy.models.HTTPFlow` - - A collection of objects representing a single HTTP transaction. -:py:class:`~libmproxy.models.HTTPRequest` - - An HTTP request. -:py:class:`~libmproxy.models.HTTPResponse` - - An HTTP response. -:py:class:`~libmproxy.models.Error` - - A communications error. -:py:class:`netlib.odict.ODDict` - - A dictionary-like object for managing sets of key/value data. There - is also a variant called ODictCaseless that ignores key case for some - calls (used mainly for headers). -:py:class:`netlib.certutils.SSLCert` - - Exposes information SSL certificates. -:py:class:`libmproxy.flow.FlowMaster` - - The "heart" of libmproxy, usually subclassed as :py:class:`libmproxy.dump.DumpMaster` or - :py:class:`libmproxy.console.ConsoleMaster`. - -Script Context --------------- - -.. autoclass:: libmproxy.script.ScriptContext - :members: - :undoc-members: - -Running scripts in parallel ---------------------------- - -We have a single flow primitive, so when a script is blocking, other requests are not processed. -While that's usually a very desirable behaviour, blocking scripts can be run threaded by using the -:py:obj:`libmproxy.script.concurrent` decorator. -**If your script does not block, you should avoid the overhead of the decorator.** - -.. literalinclude:: ../../examples/nonblocking.py - :caption: examples/nonblocking.py - :language: python - -Make scripts configurable with arguments ----------------------------------------- - -Sometimes, you want to pass runtime arguments to the inline script. This can be simply done by -surrounding the script call with quotes, e.g. ```mitmdump -s 'script.py --foo 42'``. -The arguments are then exposed in the start event: - -.. literalinclude:: ../../examples/modify_response_body.py - :caption: examples/modify_response_body.py - :language: python - -Running scripts on saved flows ------------------------------- - -Sometimes, we want to run a script on :py:class:`~libmproxy.models.Flow` objects that are already -complete. This happens when you start a script, and then load a saved set of flows from a file -(see the "scripted data transformation" example `here `_). -It also happens when you run a one-shot script on a single flow through the ``|`` (pipe) shortcut -in mitmproxy. - -In this case, there are no client connections, and the events are run in the following order: -**start**, **request**, **responseheaders**, **response**, **error**, **done**. -If the flow doesn't have a **response** or **error** associated with it, the matching events will -be skipped. - -Spaces in the script path -------------------------- - -By default, spaces are interpreted as a separator between the inline script and its arguments -(e.g. ``-s 'foo.py 42'``). Consequently, the script path needs to be wrapped in a separate pair of -quotes if it contains spaces: ``-s '\'./foo bar/baz.py\' 42'``. - -.. _GitHub: https://github.com/mitmproxy/mitmproxy diff --git a/doc-src/dev-docs/models.rst b/doc-src/dev-docs/models.rst deleted file mode 100644 index 1ac7d042..00000000 --- a/doc-src/dev-docs/models.rst +++ /dev/null @@ -1,19 +0,0 @@ -.. _models: - -Models -====== - -.. warning:: - The documentation for models has not been converted to rst yet and **many attributes/features - are missing**. - Please read the source code instead. - -.. automodule:: libmproxy.models - :show-inheritance: - :members: - :undoc-members: - - -.. automodule:: netlib.http.semantics - :members: Request, Response - :undoc-members: \ No newline at end of file diff --git a/doc-src/dev-docs/protocols.rst b/doc-src/dev-docs/protocols.rst deleted file mode 100644 index 498f634d..00000000 --- a/doc-src/dev-docs/protocols.rst +++ /dev/null @@ -1,15 +0,0 @@ -.. _protocols: - -Protocols -========= - -.. automodule:: libmproxy.protocol - - .. autoclass:: Layer - :members: - :special-members: - - .. autoclass:: ServerConnectionMixin - :members: - - .. autoexception:: Kill \ No newline at end of file diff --git a/doc-src/dev-docs/proxy.rst b/doc-src/dev-docs/proxy.rst deleted file mode 100644 index e772e5b9..00000000 --- a/doc-src/dev-docs/proxy.rst +++ /dev/null @@ -1,12 +0,0 @@ -.. _exceptions: - -Proxy -===== - -.. automodule:: libmproxy.proxy - - .. autoclass:: ProxyServer - .. autoclass:: DummyServer - .. autoclass:: ProxyConfig - .. autoclass:: RootContext - :members: \ No newline at end of file diff --git a/docs/.gitignore b/docs/.gitignore new file mode 100644 index 00000000..69fa449d --- /dev/null +++ b/docs/.gitignore @@ -0,0 +1 @@ +_build/ diff --git a/docs/Makefile b/docs/Makefile new file mode 100644 index 00000000..a22bc8a2 --- /dev/null +++ b/docs/Makefile @@ -0,0 +1,195 @@ +# Makefile for Sphinx documentation +# + +# You can set these variables from the command line. +SPHINXOPTS = +SPHINXBUILD = sphinx-build +PAPER = +BUILDDIR = _build + +# User-friendly check for sphinx-build +ifeq ($(shell which $(SPHINXBUILD) >/dev/null 2>&1; echo $$?), 1) +$(error The '$(SPHINXBUILD)' command was not found. Make sure you have Sphinx installed, then set the SPHINXBUILD environment variable to point to the full path of the '$(SPHINXBUILD)' executable. Alternatively you can add the directory with the executable to your PATH. If you don't have Sphinx installed, grab it from http://sphinx-doc.org/) +endif + +# Internal variables. +PAPEROPT_a4 = -D latex_paper_size=a4 +PAPEROPT_letter = -D latex_paper_size=letter +ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . +# the i18n builder cannot share the environment and doctrees with the others +I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . + +.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest coverage gettext + +help: + @echo "Please use \`make ' where is one of" + @echo " html to make standalone HTML files" + @echo " dirhtml to make HTML files named index.html in directories" + @echo " singlehtml to make a single large HTML file" + @echo " pickle to make pickle files" + @echo " json to make JSON files" + @echo " htmlhelp to make HTML files and a HTML help project" + @echo " qthelp to make HTML files and a qthelp project" + @echo " applehelp to make an Apple Help Book" + @echo " devhelp to make HTML files and a Devhelp project" + @echo " epub to make an epub" + @echo " latex to make LaTeX files, you can set PAPER=a4 or PAPER=letter" + @echo " latexpdf to make LaTeX files and run them through pdflatex" + @echo " latexpdfja to make LaTeX files and run them through platex/dvipdfmx" + @echo " text to make text files" + @echo " man to make manual pages" + @echo " texinfo to make Texinfo files" + @echo " info to make Texinfo files and run them through makeinfo" + @echo " gettext to make PO message catalogs" + @echo " changes to make an overview of all changed/added/deprecated items" + @echo " xml to make Docutils-native XML files" + @echo " pseudoxml to make pseudoxml-XML files for display purposes" + @echo " linkcheck to check all external links for integrity" + @echo " doctest to run all doctests embedded in the documentation (if enabled)" + @echo " coverage to run coverage check of the documentation (if enabled)" + +clean: + rm -rf $(BUILDDIR)/* + +html: + $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/html." + +dirhtml: + $(SPHINXBUILD) -b dirhtml $(ALLSPHINXOPTS) $(BUILDDIR)/dirhtml + @echo + @echo "Build finished. The HTML pages are in $(BUILDDIR)/dirhtml." + +singlehtml: + $(SPHINXBUILD) -b singlehtml $(ALLSPHINXOPTS) $(BUILDDIR)/singlehtml + @echo + @echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml." + +pickle: + $(SPHINXBUILD) -b pickle $(ALLSPHINXOPTS) $(BUILDDIR)/pickle + @echo + @echo "Build finished; now you can process the pickle files." + +json: + $(SPHINXBUILD) -b json $(ALLSPHINXOPTS) $(BUILDDIR)/json + @echo + @echo "Build finished; now you can process the JSON files." + +htmlhelp: + $(SPHINXBUILD) -b htmlhelp $(ALLSPHINXOPTS) $(BUILDDIR)/htmlhelp + @echo + @echo "Build finished; now you can run HTML Help Workshop with the" \ + ".hhp project file in $(BUILDDIR)/htmlhelp." + +qthelp: + $(SPHINXBUILD) -b qthelp $(ALLSPHINXOPTS) $(BUILDDIR)/qthelp + @echo + @echo "Build finished; now you can run "qcollectiongenerator" with the" \ + ".qhcp project file in $(BUILDDIR)/qthelp, like this:" + @echo "# qcollectiongenerator $(BUILDDIR)/qthelp/mitmproxy.qhcp" + @echo "To view the help file:" + @echo "# assistant -collectionFile $(BUILDDIR)/qthelp/mitmproxy.qhc" + +applehelp: + $(SPHINXBUILD) -b applehelp $(ALLSPHINXOPTS) $(BUILDDIR)/applehelp + @echo + @echo "Build finished. The help book is in $(BUILDDIR)/applehelp." + @echo "N.B. You won't be able to view it unless you put it in" \ + "~/Library/Documentation/Help or install it in your application" \ + "bundle." + +devhelp: + $(SPHINXBUILD) -b devhelp $(ALLSPHINXOPTS) $(BUILDDIR)/devhelp + @echo + @echo "Build finished." + @echo "To view the help file:" + @echo "# mkdir -p $$HOME/.local/share/devhelp/mitmproxy" + @echo "# ln -s $(BUILDDIR)/devhelp $$HOME/.local/share/devhelp/mitmproxy" + @echo "# devhelp" + +epub: + $(SPHINXBUILD) -b epub $(ALLSPHINXOPTS) $(BUILDDIR)/epub + @echo + @echo "Build finished. The epub file is in $(BUILDDIR)/epub." + +latex: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo + @echo "Build finished; the LaTeX files are in $(BUILDDIR)/latex." + @echo "Run \`make' in that directory to run these through (pdf)latex" \ + "(use \`make latexpdf' here to do that automatically)." + +latexpdf: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through pdflatex..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +latexpdfja: + $(SPHINXBUILD) -b latex $(ALLSPHINXOPTS) $(BUILDDIR)/latex + @echo "Running LaTeX files through platex and dvipdfmx..." + $(MAKE) -C $(BUILDDIR)/latex all-pdf-ja + @echo "pdflatex finished; the PDF files are in $(BUILDDIR)/latex." + +text: + $(SPHINXBUILD) -b text $(ALLSPHINXOPTS) $(BUILDDIR)/text + @echo + @echo "Build finished. The text files are in $(BUILDDIR)/text." + +man: + $(SPHINXBUILD) -b man $(ALLSPHINXOPTS) $(BUILDDIR)/man + @echo + @echo "Build finished. The manual pages are in $(BUILDDIR)/man." + +texinfo: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo + @echo "Build finished. The Texinfo files are in $(BUILDDIR)/texinfo." + @echo "Run \`make' in that directory to run these through makeinfo" \ + "(use \`make info' here to do that automatically)." + +info: + $(SPHINXBUILD) -b texinfo $(ALLSPHINXOPTS) $(BUILDDIR)/texinfo + @echo "Running Texinfo files through makeinfo..." + make -C $(BUILDDIR)/texinfo info + @echo "makeinfo finished; the Info files are in $(BUILDDIR)/texinfo." + +gettext: + $(SPHINXBUILD) -b gettext $(I18NSPHINXOPTS) $(BUILDDIR)/locale + @echo + @echo "Build finished. The message catalogs are in $(BUILDDIR)/locale." + +changes: + $(SPHINXBUILD) -b changes $(ALLSPHINXOPTS) $(BUILDDIR)/changes + @echo + @echo "The overview file is in $(BUILDDIR)/changes." + +linkcheck: + $(SPHINXBUILD) -b linkcheck $(ALLSPHINXOPTS) $(BUILDDIR)/linkcheck + @echo + @echo "Link check complete; look for any errors in the above output " \ + "or in $(BUILDDIR)/linkcheck/output.txt." + +doctest: + $(SPHINXBUILD) -b doctest $(ALLSPHINXOPTS) $(BUILDDIR)/doctest + @echo "Testing of doctests in the sources finished, look at the " \ + "results in $(BUILDDIR)/doctest/output.txt." + +coverage: + $(SPHINXBUILD) -b coverage $(ALLSPHINXOPTS) $(BUILDDIR)/coverage + @echo "Testing of coverage in the sources finished, look at the " \ + "results in $(BUILDDIR)/coverage/python.txt." + +xml: + $(SPHINXBUILD) -b xml $(ALLSPHINXOPTS) $(BUILDDIR)/xml + @echo + @echo "Build finished. The XML files are in $(BUILDDIR)/xml." + +pseudoxml: + $(SPHINXBUILD) -b pseudoxml $(ALLSPHINXOPTS) $(BUILDDIR)/pseudoxml + @echo + @echo "Build finished. The pseudo-XML files are in $(BUILDDIR)/pseudoxml." + +livehtml: + sphinx-autobuild -b html -z '../libmproxy' -r '___jb_(old|bak)___$$' $(ALLSPHINXOPTS) $(BUILDDIR)/html \ No newline at end of file diff --git a/docs/certinstall-webapp.png b/docs/certinstall-webapp.png new file mode 100644 index 00000000..10e795cd Binary files /dev/null and b/docs/certinstall-webapp.png differ diff --git a/docs/certinstall.rst b/docs/certinstall.rst new file mode 100644 index 00000000..f0e71223 --- /dev/null +++ b/docs/certinstall.rst @@ -0,0 +1,173 @@ +.. _certinstall: + +About Certificates +================== + +Introduction +------------ + +Mitmproxy can decrypt encrypted traffic on the fly, as long as the client +trusts its built-in certificate authority. Usually this means that the +mitmproxy CA certificates have to be installed on the client device. + +Quick Setup +----------- + +By far the easiest way to install the mitmproxy certificates is to use the +built-in certificate installation app. To do this, just start mitmproxy and +configure your target device with the correct proxy settings. Now start a +browser on the device, and visit the magic domain **mitm.it**. You should see +something like this: + +.. image:: certinstall-webapp.png + +Click on the relevant icon, follow the setup instructions for the platform +you're on and you are good to go. + + +Installing the mitmproxy CA certificate manually +------------------------------------------------ + +Sometimes using the quick install app is not an option - Java or the iOS +Simulator spring to mind - or you just need to do it manually for some other +reason. Below is a list of pointers to manual certificate installation +documentation for some common platforms. + +The mitmproxy CA cert is located in ``~/.mitmproxy`` after it has been generated at the first +start of mitmproxy. + + +iOS +^^^ + +http://kb.mit.edu/confluence/pages/viewpage.action?pageId=152600377 + +iOS Simulator +^^^^^^^^^^^^^ + +See https://github.com/ADVTOOLS/ADVTrustStore#how-to-use-advtruststore + +Java +^^^^ + +See http://docs.oracle.com/cd/E19906-01/820-4916/geygn/index.html + +Android/Android Simulator +^^^^^^^^^^^^^^^^^^^^^^^^^ + +See http://wiki.cacert.org/FAQ/ImportRootCert#Android_Phones_.26_Tablets + +Windows +^^^^^^^ + +See http://windows.microsoft.com/en-ca/windows/import-export-certificates-private-keys#1TC=windows-7 + +Windows (automated) +^^^^^^^^^^^^^^^^^^^ + +>>> certutil.exe -importpfx mitmproxy-ca-cert.p12 + +See also: https://technet.microsoft.com/en-us/library/cc732443.aspx + +Mac OS X +^^^^^^^^ + +See https://support.apple.com/kb/PH7297?locale=en_US + +Ubuntu/Debian +^^^^^^^^^^^^^ + +See http://askubuntu.com/questions/73287/how-do-i-install-a-root-certificate/94861#94861 + +Mozilla Firefox +^^^^^^^^^^^^^^^ + +See https://wiki.mozilla.org/MozillaRootCertificate#Mozilla_Firefox + +Chrome on Linux +^^^^^^^^^^^^^^^ + +See https://code.google.com/p/chromium/wiki/LinuxCertManagement + + +More on mitmproxy certificates +------------------------------ + +The first time **mitmproxy** or **mitmdump** is run, the mitmproxy Certificate +Authority (CA) is created in the config directory (``~/.mitmproxy`` by default). +This CA is used for on-the-fly generation of dummy certificates for each of the +SSL sites that your client visits. Since your browser won't trust the +mitmproxy CA out of the box , you will see an SSL certificate warning every +time you visit a new SSL domain through mitmproxy. When you are testing a +single site through a browser, just accepting the bogus SSL cert manually is +not too much trouble, but there are a many circumstances where you will want to +configure your testing system or browser to trust the mitmproxy CA as a +signing root authority. For security reasons, the mitmproxy CA is generated uniquely on the first +start and is not shared between mitmproxy installations on different devices. + + +CA and cert files +----------------- + +The files created by mitmproxy in the .mitmproxy directory are as follows: + +===================== ==================================================================================== +mitmproxy-ca.pem The certificate **and the private key** in PEM format. +mitmproxy-ca-cert.pem The certificate in PEM format. Use this to distribute on most non-Windows platforms. +mitmproxy-ca-cert.p12 The certificate in PKCS12 format. For use on Windows. +mitmproxy-ca-cert.cer Same file as .pem, but with an extension expected by some Android devices. +===================== ==================================================================================== + +Using a custom certificate +-------------------------- + +You can use your own certificate by passing the ``--cert`` option to +mitmproxy. Mitmproxy then uses the provided certificate for interception of the +specified domains instead of generating a certificate signed by its own CA. + +The certificate file is expected to be in the PEM format. You can include +intermediary certificates right below your leaf certificate, so that you PEM +file roughly looks like this: + +.. code-block:: none + + -----BEGIN PRIVATE KEY----- + + -----END PRIVATE KEY----- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + -----BEGIN CERTIFICATE----- + + -----END CERTIFICATE----- + + +For example, you can generate a certificate in this format using these instructions: + + +>>> openssl genrsa -out cert.key 2048 +>>> openssl req -new -x509 -key cert.key -out cert.crt + (Specify the mitm domain as Common Name, e.g. *.google.com) +>>> cat cert.key cert.crt > cert.pem +>>> mitmproxy --cert=cert.pem + + +Using a custom certificate authority +------------------------------------ + +By default, mitmproxy will use ``~/.mitmproxy/mitmproxy-ca.pem`` as +the certificate authority to generate certificates for all domains for which no +custom certificate is provided (see above). You can use your own certificate +authority by passing the ``--confdir`` option to mitmproxy. Mitmproxy +will then look for ``mitmproxy-ca.pem`` in the specified directory. If +no such file exists, it will be generated automatically. + + +Using a client side certificate +------------------------------- + +You can use a client certificate by passing the ``--client-certs DIRECTORY`` +option to mitmproxy. If you visit example.org, mitmproxy looks for a file named ``example.org.pem`` +in the specified directory and uses this as the client cert. The certificate file needs to be in +the PEM format and should contain both the unencrypted private key and the certificate. + diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 00000000..65aa19dc --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,215 @@ +# -*- coding: utf-8 -*- +# +# mitmproxy documentation build configuration file, created by +# sphinx-quickstart on Thu Sep 03 14:04:13 2015. +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +import sys +import os +import shlex + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +sys.path.insert(0, os.path.abspath('..')) + +import libmproxy.version + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +#needs_sphinx = '1.0' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. + +extensions = [ + 'sphinx.ext.autodoc', + 'sphinx.ext.doctest', + 'sphinx.ext.viewcode', + 'sphinx.ext.napoleon' +] + +autodoc_member_order = "bysource" + +# Add any paths that contain templates here, relative to this directory. +#templates_path = ['_templates'] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The encoding of source files. +#source_encoding = 'utf-8-sig' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'mitmproxy dev docs' +copyright = u'2015, the mitmproxy project' +author = u'The mitmproxy project' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# The short X.Y version. +version = libmproxy.version.VERSION +# The full version, including alpha/beta/rc tags. +release = libmproxy.version.VERSION + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# There are two options for replacing |today|: either, you set today to some +# non-false value, then it is used: +#today = '' +# Else, today_fmt is used as the format for a strftime call. +#today_fmt = '%B %d, %Y' + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +exclude_patterns = ['_build'] + +# The reST default role (used for this markup: `text`) to use for all +# documents. +#default_role = None + +# If true, '()' will be appended to :func: etc. cross-reference text. +#add_function_parentheses = True + +# If true, the current module name will be prepended to all description +# unit titles (such as .. function::). +#add_module_names = True + +# If true, sectionauthor and moduleauthor directives will be shown in the +# output. They are ignored by default. +#show_authors = False + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# A list of ignored prefixes for module index sorting. +modindex_common_prefix = ['libmproxy.'] + +# If true, keep warnings as "system message" paragraphs in the built documents. +#keep_warnings = False + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +html_theme = 'sphinx_rtd_theme' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +html_theme_options = { + # 'logo_only': True, +} + +# Add any paths that contain custom themes here, relative to this directory. +#html_theme_path = [] + +# The name for this set of Sphinx documents. If None, it defaults to +# " v documentation". +html_title = "mitmproxy %s documentation" % version + +# A shorter title for the navigation bar. Default is the same as html_title. +#html_short_title = None + +# The name of an image file (relative to this directory) to place at the top +# of the sidebar. +html_logo = "mitmproxy-long.png" + +# The name of an image file (within the static path) to use as favicon of the +# docs. This file should be a Windows icon file (.ico) being 16x16 or 32x32 +# pixels large. +html_favicon = "favicon.ico" + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +# html_static_path = ['_static'] + +# Add any extra paths that contain custom files (such as robots.txt or +# .htaccess) here, relative to this directory. These files are copied +# directly to the root of the documentation. +#html_extra_path = [] + +# If not '', a 'Last updated on:' timestamp is inserted at every page bottom, +# using the given strftime format. +#html_last_updated_fmt = '%b %d, %Y' + +# If true, SmartyPants will be used to convert quotes and dashes to +# typographically correct entities. +#html_use_smartypants = True + +# Custom sidebar templates, maps document names to template names. +#html_sidebars = {} + +# Additional templates that should be rendered to pages, maps page names to +# template names. +#html_additional_pages = {} + +# If false, no module index is generated. +#html_domain_indices = True + +# If false, no index is generated. +#html_use_index = True + +# If true, the index is split into individual pages for each letter. +#html_split_index = False + +# If true, links to the reST sources are added to the pages. +#html_show_sourcelink = True + +# If true, "Created using Sphinx" is shown in the HTML footer. Default is True. +#html_show_sphinx = True + +# If true, "(C) Copyright ..." is shown in the HTML footer. Default is True. +#html_show_copyright = True + +# If true, an OpenSearch description file will be output, and all pages will +# contain a tag referring to it. The value of this option must be the +# base URL from which the finished HTML is served. +#html_use_opensearch = '' + +# This is the file name suffix for HTML files (e.g. ".xhtml"). +#html_file_suffix = None + +# Language to be used for generating the HTML full-text search index. +# Sphinx supports the following languages: +# 'da', 'de', 'en', 'es', 'fi', 'fr', 'hu', 'it', 'ja' +# 'nl', 'no', 'pt', 'ro', 'ru', 'sv', 'tr' +#html_search_language = 'en' + +# A dictionary with options for the search language support, empty by default. +# Now only 'ja' uses this config value +#html_search_options = {'type': 'default'} + +# The name of a javascript file (relative to the configuration directory) that +# implements a search results scorer. If empty, the default will be used. +#html_search_scorer = 'scorer.js' + +# Output file base name for HTML help builder. +htmlhelp_basename = 'mitmproxydoc' \ No newline at end of file diff --git a/docs/dev/exceptions.rst b/docs/dev/exceptions.rst new file mode 100644 index 00000000..d1e4bfe5 --- /dev/null +++ b/docs/dev/exceptions.rst @@ -0,0 +1,9 @@ +.. _exceptions: + +Exceptions +========== + +.. automodule:: libmproxy.exceptions + :show-inheritance: + :members: + :undoc-members: \ No newline at end of file diff --git a/docs/dev/models.rst b/docs/dev/models.rst new file mode 100644 index 00000000..1ac7d042 --- /dev/null +++ b/docs/dev/models.rst @@ -0,0 +1,19 @@ +.. _models: + +Models +====== + +.. warning:: + The documentation for models has not been converted to rst yet and **many attributes/features + are missing**. + Please read the source code instead. + +.. automodule:: libmproxy.models + :show-inheritance: + :members: + :undoc-members: + + +.. automodule:: netlib.http.semantics + :members: Request, Response + :undoc-members: \ No newline at end of file diff --git a/docs/dev/protocols.rst b/docs/dev/protocols.rst new file mode 100644 index 00000000..498f634d --- /dev/null +++ b/docs/dev/protocols.rst @@ -0,0 +1,15 @@ +.. _protocols: + +Protocols +========= + +.. automodule:: libmproxy.protocol + + .. autoclass:: Layer + :members: + :special-members: + + .. autoclass:: ServerConnectionMixin + :members: + + .. autoexception:: Kill \ No newline at end of file diff --git a/docs/dev/proxy.rst b/docs/dev/proxy.rst new file mode 100644 index 00000000..c0cdb259 --- /dev/null +++ b/docs/dev/proxy.rst @@ -0,0 +1,12 @@ +.. _proxy: + +Proxy Server +============ + +.. automodule:: libmproxy.proxy + + .. autoclass:: ProxyServer + .. autoclass:: DummyServer + .. autoclass:: ProxyConfig + .. autoclass:: RootContext + :members: \ No newline at end of file diff --git a/docs/favicon.ico b/docs/favicon.ico new file mode 100644 index 00000000..3c3b891c Binary files /dev/null and b/docs/favicon.ico differ diff --git a/docs/features/upstreamcerts.rst b/docs/features/upstreamcerts.rst new file mode 100644 index 00000000..a287daef --- /dev/null +++ b/docs/features/upstreamcerts.rst @@ -0,0 +1,4 @@ +.. _upstreamcerts: + +Upstream Certificates +===================== \ No newline at end of file diff --git a/docs/howmitmproxy.rst b/docs/howmitmproxy.rst new file mode 100644 index 00000000..8bc20792 --- /dev/null +++ b/docs/howmitmproxy.rst @@ -0,0 +1,238 @@ +How mitmproxy works +=================== + +Mitmproxy is an enormously flexible tool. Knowing exactly how the proxying +process works will help you deploy it creatively, and take into account its +fundamental assumptions and how to work around them. This document explains +mitmproxy's proxy mechanism in detail, starting with the simplest unencrypted +explicit proxying, and working up to the most complicated interaction - +transparent proxying of SSL-protected traffic [#ssl]_ in the presence of `Server Name Indication`_. + +Explicit HTTP +------------- + +Configuring the client to use mitmproxy as an explicit proxy is the simplest +and most reliable way to intercept traffic. The proxy protocol is codified in the +`HTTP RFC`_, so the behaviour of both +the client and the server is well defined, and usually reliable. In the +simplest possible interaction with mitmproxy, a client connects directly to the +proxy, and makes a request that looks like this: + +.. code-block:: http + + GET http://example.com/index.html HTTP/1.1 + +This is a proxy GET request - an extended form of the vanilla HTTP GET request +that includes a schema and host specification, and it includes all the +information mitmproxy needs to proceed. + +.. image:: schematics/how-mitmproxy-works-explicit.png + :align: center + +1. The client connects to the proxy and makes a request. +2. Mitmproxy connects to the upstream server and simply forwards the request on. + + +Explicit HTTPS +-------------- + +The process for an explicitly proxied HTTPS connection is quite different. The +client connects to the proxy and makes a request that looks like this: + +.. code-block:: http + + CONNECT example.com:443 HTTP/1.1 + +A conventional proxy can neither view nor manipulate an SSL-encrypted data +stream, so a CONNECT request simply asks the proxy to open a pipe between the +client and server. The proxy here is just a facilitator - it blindly forwards +data in both directions without knowing anything about the contents. The +negotiation of the SSL connection happens over this pipe, and the subsequent +flow of requests and responses are completely opaque to the proxy. + +The MITM in mitmproxy +^^^^^^^^^^^^^^^^^^^^^ + +This is where mitmproxy's fundamental trick comes into play. The MITM in its +name stands for Man-In-The-Middle - a reference to the process we use to +intercept and interfere with these theoretically opaque data streams. The basic +idea is to pretend to be the server to the client, and pretend to be the client +to the server, while we sit in the middle decoding traffic from both sides. The +tricky part is that the `Certificate Authority`_ system is +designed to prevent exactly this attack, by allowing a trusted third-party to +cryptographically sign a server's SSL certificates to verify that they are +legit. If this signature doesn't match or is from a non-trusted party, a secure +client will simply drop the connection and refuse to proceed. Despite the many +shortcomings of the CA system as it exists today, this is usually fatal to +attempts to MITM an SSL connection for analysis. Our answer to this conundrum +is to become a trusted Certificate Authority ourselves. Mitmproxy includes a +full CA implementation that generates interception certificates on the fly. To +get the client to trust these certificates, we :ref:`register mitmproxy as a trusted +CA with the device manually `. + +Complication 1: What's the remote hostname? +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +To proceed with this plan, we need to know the domain name to use in the +interception certificate - the client will verify that the certificate is for +the domain it's connecting to, and abort if this is not the case. At first +blush, it seems that the CONNECT request above gives us all we need - in this +example, both of these values are "example.com". But what if the client had +initiated the connection as follows: + +.. code-block:: http + + CONNECT 10.1.1.1:443 HTTP/1.1 + +Using the IP address is perfectly legitimate because it gives us enough +information to initiate the pipe, even though it doesn't reveal the remote +hostname. + +Mitmproxy has a cunning mechanism that smooths this over - :ref:`upstream +certificate sniffing `. As soon as we +see the CONNECT request, we pause the client part of the conversation, and +initiate a simultaneous connection to the server. We complete the SSL handshake +with the server, and inspect the certificates it used. Now, we use the Common +Name in the upstream SSL certificates to generate the dummy certificate for the +client. Voila, we have the correct hostname to present to the client, even if +it was never specified. + + +Complication 2: Subject Alternative Name +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Enter the next complication. Sometimes, the certificate Common Name is not, in +fact, the hostname that the client is connecting to. This is because of the +optional `Subject Alternative Name`_ field in the SSL certificate +that allows an arbitrary number of alternative domains to be specified. If the +expected domain matches any of these, the client will proceed, even though the +domain doesn't match the certificate Common Name. The answer here is simple: +when we extract the CN from the upstream cert, we also extract the SANs, and +add them to the generated dummy certificate. + + +Complication 3: Server Name Indication +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +One of the big limitations of vanilla SSL is that each certificate requires its +own IP address. This means that you couldn't do virtual hosting where multiple +domains with independent certificates share the same IP address. In a world +with a rapidly shrinking IPv4 address pool this is a problem, and we have a +solution in the form of the `Server Name Indication`_ extension to +the SSL and TLS protocols. This lets the client specify the remote server name +at the start of the SSL handshake, which then lets the server select the right +certificate to complete the process. + +SNI breaks our upstream certificate sniffing process, because when we connect +without using SNI, we get served a default certificate that may have nothing to +do with the certificate expected by the client. The solution is another tricky +complication to the client connection process. After the client connects, we +allow the SSL handshake to continue until just _after_ the SNI value has been +passed to us. Now we can pause the conversation, and initiate an upstream +connection using the correct SNI value, which then serves us the correct +upstream certificate, from which we can extract the expected CN and SANs. + +Putting it all together +^^^^^^^^^^^^^^^^^^^^^^^ + +Lets put all of this together into the complete explicitly proxied HTTPS flow. + +.. image:: schematics/how-mitmproxy-works-explicit-https.png + :align: center + +1. The client makes a connection to mitmproxy, and issues an HTTP CONNECT request. +2. Mitmproxy responds with a ``200 Connection Established``, as if it has set up the CONNECT pipe. +3. The client believes it's talking to the remote server, and initiates the SSL connection. + It uses SNI to indicate the hostname it is connecting to. +4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname + indicated by the client. +5. The server responds with the matching SSL certificate, which contains the CN and SAN values + needed to generate the interception certificate. +6. Mitmproxy generates the interception cert, and continues the + client SSL handshake paused in step 3. +7. The client sends the request over the established SSL connection. +8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4. + +Transparent HTTP +---------------- + +When a transparent proxy is used, the HTTP/S connection is redirected into a +proxy at the network layer, without any client configuration being required. +This makes transparent proxying ideal for those situations where you can't +change client behaviour - proxy-oblivious Android applications being a common +example. + +To achieve this, we need to introduce two extra components. The first is a +redirection mechanism that transparently reroutes a TCP connection destined for +a server on the Internet to a listening proxy server. This usually takes the +form of a firewall on the same host as the proxy server - `iptables`_ on Linux or +pf_ on OSX. Once the client has initiated the connection, it makes a vanilla HTTP request, +which might look something like this: + +.. code-block:: http + + GET /index.html HTTP/1.1 + +Note that this request differs from the explicit proxy variation, in that it +omits the scheme and hostname. How, then, do we know which upstream host to +forward the request to? The routing mechanism that has performed the +redirection keeps track of the original destination for us. Each routing +mechanism has a different way of exposing this data, so this introduces the +second component required for working transparent proxying: a host module that +knows how to retrieve the original destination address from the router. In +mitmproxy, this takes the form of a built-in set of +modules_ that know how to talk to each platform's redirection mechanism. +Once we have this information, the process is fairly straight-forward. + +.. image:: schematics/how-mitmproxy-works-transparent.png + :align: center + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically listening on a local port + of the same host. Mitmproxy then consults the routing mechanism to establish what the original + destination was. +3. Now, we simply read the client's request... +4. ... and forward it upstream. + +Transparent HTTPS +----------------- + +The first step is to determine whether we should treat an incoming connection +as HTTPS. The mechanism for doing this is simple - we use the routing mechanism +to find out what the original destination port is. By default, we treat all +traffic destined for ports 443 and 8443 as SSL. + +From here, the process is a merger of the methods we've described for +transparently proxying HTTP, and explicitly proxying HTTPS. We use the routing +mechanism to establish the upstream server address, and then proceed as for +explicit HTTPS connections to establish the CN and SANs, and cope with SNI. + +.. image:: schematics/how-mitmproxy-works-transparent-https.png + :align: center + +1. The client makes a connection to the server. +2. The router redirects the connection to mitmproxy, which is typically listening on a local port of + the same host. Mitmproxy then consults the routing mechanism to establish what the original + destination was. +3. The client believes it's talking to the remote server, and initiates the SSL connection. It uses + SNI to indicate the hostname it is connecting to. +4. Mitmproxy connects to the server, and establishes an SSL connection using the SNI hostname + indicated by the client. +5. The server responds with the matching SSL certificate, which contains the CN and SAN values + needed to generate the interception certificate. +6. Mitmproxy generates the interception cert, and continues the client SSL handshake paused in + step 3. +7. The client sends the request over the established SSL connection. +8. Mitmproxy passes the request on to the server over the SSL connection initiated in step 4. + +.. rubric:: Footnotes + +.. [#ssl] I use "SSL" to refer to both SSL and TLS in the generic sense, unless otherwise specified. + +.. _Server Name Indication: https://en.wikipedia.org/wiki/Server_Name_Indication +.. _HTTP RFC: https://tools.ietf.org/html/rfc7230 +.. _Certificate Authority: https://en.wikipedia.org/wiki/Certificate_authority +.. _Subject Alternative Name: https://en.wikipedia.org/wiki/SubjectAltName +.. _iptables: http://www.netfilter.org/ +.. _pf: https://en.wikipedia.org/wiki/PF_\(firewall\) +.. _modules: https://github.com/mitmproxy/mitmproxy/tree/master/libmproxy/platform diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 00000000..7c792ea4 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,43 @@ +.. include:: introduction.rst + + +.. toctree:: + :hidden: + :maxdepth: 1 + + introduction + install + certinstall + howmitmproxy + modes + +.. toctree:: + :hidden: + :caption: Tools + + mitmproxy + mitmdump + +.. toctree:: + :hidden: + :caption: Scripting + + scripting/inlinescripts + scripting/libmproxy + + +.. toctree:: + :hidden: + :caption: Development + + dev/protocols + dev/proxy + dev/exceptions + dev/models + +.. Indices and tables + ================== + + * :ref:`genindex` + * :ref:`modindex` + diff --git a/docs/install.rst b/docs/install.rst new file mode 100644 index 00000000..e0a572af --- /dev/null +++ b/docs/install.rst @@ -0,0 +1,100 @@ +.. _install: + +Installation +============ + +.. _install-ubuntu: + +Installation On Ubuntu +---------------------- + +Ubuntu comes with Python but we need to install pip, python-dev and several libraries. +This was tested on a fully patched installation of Ubuntu 14.04. + +>>> sudo apt-get install python-pip python-dev libffi-dev libssl-dev libxml2-dev libxslt1-dev +>>> sudo pip install mitmproxy + +Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal. + +Installation From Source (Ubuntu) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, install the dependencies as you would for a regular +mitmproxy installation (see :ref:`install-ubuntu`). +Then see the Hacking_ section of the README on GitHub. + + + +Installation On Mac OS X +------------------------ + +The easiest way to get up and running on OSX is to download the pre-built binary packages from +`mitmproxy.org`_. + +There are a few bits of customization you might want to do to make mitmproxy comfortable to use on +OSX. The default color scheme is optimized for a dark background terminal, but you can select a +palette for a light terminal background with the ``--palette`` option. +You can use the OSX **open** program to create a simple and effective ``~/.mailcap`` file to view +request and response bodies: + +.. code-block:: none + + application/*; /usr/bin/open -Wn %s + audio/*; /usr/bin/open -Wn %s + image/*; /usr/bin/open -Wn %s + video/*; /usr/bin/open -Wn %s + +Once installation is complete you can run :ref:`mitmproxy` or :ref:`mitmdump` from a terminal. + + +Installation From Source (Mac OS X) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, there are a few OS X specific things to keep in mind. + +- Make sure that XCode is installed from the App Store, and that the command-line tools have been + downloaded (XCode/Preferences/Downloads). +- If you're running a Python interpreter installed with homebrew (or similar), you may have to + install some dependencies by hand. + +Then see the Hacking_ section of the README on GitHub. + +Installation On Windows +----------------------- + +.. note:: + Please note that mitmdump is the only component of mitmproxy that is supported on Windows at + the moment. + + **There is no interactive user interface on Windows.** + + +First, install the latest version of Python 2.7 from the `Python website`_. +If you already have an older version of Python 2.7 installed, make sure to install pip_ +(pip is included in Python 2.7.9+ by default). + +Next, add Python and the Python Scripts directory to your **PATH** variable. +You can do this easily by running the following in powershell: + +>>> [Environment]::SetEnvironmentVariable("Path", "$env:Path;C:\Python27;C:\Python27\Scripts", "User") + +Now, you can install mitmproxy by running + +>>> pip install mitmproxy + +Once the installation is complete, you can run :ref:`mitmdump` from a command prompt. + +Installation From Source (Windows) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +If you would like to install mitmproxy directly from the master branch on GitHub or would like to +get set up to contribute to the project, install Python as outlined above, then see the +Hacking_ section of the README on GitHub. + + +.. _Hacking: https://github.com/mitmproxy/mitmproxy/blob/master/README.mkd#hacking +.. _mitmproxy.org: https://mitmproxy.org/ +.. _`Python website`: https://www.python.org/downloads/windows/ +.. _pip: https://pip.pypa.io/en/latest/installing.html \ No newline at end of file diff --git a/docs/introduction.rst b/docs/introduction.rst new file mode 100644 index 00000000..6ce6add9 --- /dev/null +++ b/docs/introduction.rst @@ -0,0 +1,26 @@ +Introduction +============ + +**mitmproxy** is an interactive, SSL-capable man-in-the-middle proxy for HTTP +with a console interface. + +**mitmdump** is the command-line version of mitmproxy. Think tcpdump for HTTP. + +**libmproxy** is the library that mitmproxy and mitmdump are built on. + +Documentation, tutorials and distribution packages can be found on the +mitmproxy website: `mitmproxy.org `_ + + +.. rubric:: Features + + +- Intercept HTTP requests and responses and modify them on the fly. +- Save complete HTTP conversations for later replay and analysis. +- Replay the client-side of an HTTP conversations. +- Replay HTTP responses of a previously recorded server. +- Reverse proxy mode to forward traffic to a specified server. +- Transparent proxy mode on OSX and Linux. +- Make scripted changes to HTTP traffic using Python. +- SSL certificates for interception are generated on the fly. +- And much, much more. \ No newline at end of file diff --git a/docs/mitmdump.rst b/docs/mitmdump.rst new file mode 100644 index 00000000..c56903ec --- /dev/null +++ b/docs/mitmdump.rst @@ -0,0 +1,4 @@ +.. _mitmdump: + +mitmdump +======== \ No newline at end of file diff --git a/docs/mitmproxy-long.png b/docs/mitmproxy-long.png new file mode 100644 index 00000000..f9397d1e Binary files /dev/null and b/docs/mitmproxy-long.png differ diff --git a/docs/mitmproxy.rst b/docs/mitmproxy.rst new file mode 100644 index 00000000..b39a8f87 --- /dev/null +++ b/docs/mitmproxy.rst @@ -0,0 +1,4 @@ +.. _mitmproxy: + +mitmproxy +========= \ No newline at end of file diff --git a/docs/modes.rst b/docs/modes.rst new file mode 100644 index 00000000..ccd1f3ee --- /dev/null +++ b/docs/modes.rst @@ -0,0 +1,193 @@ +.. _modes: + +Modes of Operation +================== + +Mitmproxy has four modes of operation that allow you to use mitmproxy in a +variety of scenarios: + +- **Regular** (the default) +- **Transparent** +- **Reverse Proxy** +- **Upstream Proxy** + + +Now, which one should you pick? Use this flow chart: + +.. image:: schematics/proxy-modes-flowchart.png + :align: center + +Regular Proxy +------------- + +Mitmproxy's regular mode is the simplest and the easiest to set up. + +1. Start mitmproxy. +2. Configure your client to use mitmproxy by explicitly setting an HTTP proxy. +3. Quick Check: You should already be able to visit an unencrypted HTTP site through the proxy. +4. Open the magic domain mitm.it and install the certificate for your device. + +.. note:: + Unfortunately, some applications bypass the system HTTP proxy settings - Android applications + are a common example. In these cases, you need to use mitmproxy's transparent mode. + +If you are proxying an external device, your network will probably look like this: + +.. image:: schematics/proxy-modes-regular.png + :align: center + +The square brackets signify the source and destination IP addresses. Your +client explicitly connects to mitmproxy and mitmproxy explicitly connects +to the target server. + +Transparent Proxy +----------------- + +In transparent mode, traffic is directed into a proxy at the network layer, +without any client configuration required. This makes transparent proxying +ideal for situations where you can't change client behaviour. In the graphic +below, a machine running mitmproxy has been inserted between the router and +the internet: + +.. image:: schematics/proxy-modes-transparent-1.png + :align: center + +The square brackets signify the source and destination IP addresses. Round +brackets mark the next hop on the *Ethernet/data link* layer. This distinction +is important: when the packet arrives at the mitmproxy machine, it must still +be addressed to the target server. This means that Network Address Translation +should not be applied before the traffic reaches mitmproxy, since this would +remove the target information, leaving mitmproxy unable to determine the real +destination. + +.. image:: schematics/proxy-modes-transparent-wrong.png + :align: center + +Common Configurations +^^^^^^^^^^^^^^^^^^^^^ + +There are many ways to configure your network for transparent proxying. We'll +look at two common scenarios: + +1. Configuring the client to use a custom gateway/router/"next hop" +2. Implementing custom routing on the router + +In most cases, the first option is recommended due to its ease of use. + +(a) Custom Gateway +~~~~~~~~~~~~~~~~~~ + +One simple way to get traffic to the mitmproxy machine with the destination IP +intact, is to simply configure the client with the mitmproxy box as the +default gateway. + +.. image:: schematics/proxy-modes-transparent-2.png + :align: center + +In this scenario, we would: + +1. Configure the proxy machine for transparent mode. You can find instructions + in the :ref:`transparent` section. +2. Configure the client to use the proxy machine's IP as the default gateway. +3. Quick Check: At this point, you should already be able to visit an + unencrypted HTTP site over the proxy. +4. Open the magic domain mitm.it and install the certificate + for your device. + +Setting the custom gateway on clients can be automated by serving the settings +out to clients over DHCP. This lets set up an interception network where all +clients are proxied automatically, which can save time and effort. + +.. admonition:: Troubleshooting Transparent Mode + :class: note + + Incorrect transparent mode configurations are a frequent source of + error. If it doesn't work for you, try the following things: + + - Open mitmproxy's event log (press :kbd:`e`) - do you see clientconnect messages? + If not, the packets are not arriving at the proxy. One common cause is the occurrence of ICMP + redirects, which means that your machine is telling the client that there's a faster way to + the internet by contacting your router directly (see the :ref:`transparent` section on how to + disable them). If in doubt, Wireshark_ may help you to see whether something arrives at your + machine or not. + - Make sure you have not explicitly configured an HTTP proxy on the client. + This is not needed in transparent mode. + - Re-check the instructions in the :ref:`transparent` section. Anything you missed? + + If you encounter any other pitfalls that should be listed here, please let us know! + +(b) Custom Routing +~~~~~~~~~~~~~~~~~~ + +In some cases, you may need more fine-grained control of which traffic reaches +the mitmproxy instance, and which doesn't. You may, for instance, choose only +to divert traffic to some hosts into the transparent proxy. There are a huge +number of ways to accomplish this, and much will depend on the router or +packet filter you're using. In most cases, the configuration will look like +this: + +.. image:: schematics/proxy-modes-transparent-3.png + :align: center + + +Reverse Proxy +------------- + +mitmproxy is usually used with a client that uses the proxy to access the +Internet. Using reverse proxy mode, you can use mitmproxy to act like a normal +HTTP server: + +.. image:: schematics/proxy-modes-reverse.png + :align: center + +There are various use-cases: + +- Say you have an internal API running at http://example.local/. You could now + set up mitmproxy in reverse proxy mode at http://debug.example.local/ and + dynamically point clients to this new API endpoint, which provides them + with the same data and you with debug information. Similarly, you could move + your real server to a different IP/port and set up mitmproxy in the original + place to debug and or redirect all sessions. + +- Say you're a web developer working on http://example.com/ (with a development + version running on http://localhost:8000/). You can modify your hosts file so that + example.com points to 127.0.0.1 and then run mitmproxy in reverse proxy mode + on port 80. You can test your app on the example.com domain and get all + requests recorded in mitmproxy. + +- Say you have some toy project that should get SSL support. Simply set up + mitmproxy as a reverse proxy on port 443 and you're done (``mitmdump -p 443 -R + http://localhost:80/``). Mitmproxy auto-detects TLS traffic and intercepts it dynamically. + There are better tools for this specific task, but mitmproxy is very quick and simple way to + set up an SSL-speaking server. + +- Want to add a non-SSL-capable compression proxy in front of your server? You + could even spawn a mitmproxy instance that terminates SSL (``-R http://...``), + point it to the compression proxy and let the compression proxy point to a + SSL-initiating mitmproxy (``-R https://...``), which then points to the real + server. As you see, it's a fairly flexible thing. + +.. admonition:: Caveat: Interactive Use + :class: warning + + Reverse Proxy mode is usually not sufficient to create a copy of an interactive website at + different URL. The HTML served to the client remains unchanged - as soon as the user clicks on + an non-relative URL (or downloads a non-relative image resource), traffic no longer passes + through mitmproxy. + +Upstream Proxy +-------------- + +If you want to chain proxies by adding mitmproxy in front of a different proxy +appliance, you can use mitmproxy's upstream mode. In upstream mode, all +requests are unconditionally transferred to an upstream proxy of your choice. + +.. image:: schematics/proxy-modes-upstream.png + :align: center + +mitmproxy supports both explicit HTTP and explicit HTTPS in upstream proxy +mode. You could in theory chain multiple mitmproxy instances in a row, but +that doesn't make any sense in practice (i.e. outside of our tests). + + +.. _Wireshark: https://wireshark.org/ \ No newline at end of file diff --git a/docs/schematics/architecture.pdf b/docs/schematics/architecture.pdf new file mode 100644 index 00000000..77f5ad58 Binary files /dev/null and b/docs/schematics/architecture.pdf differ diff --git a/docs/schematics/architecture.png b/docs/schematics/architecture.png new file mode 100644 index 00000000..67d6c718 Binary files /dev/null and b/docs/schematics/architecture.png differ diff --git a/docs/schematics/architecture.vsdx b/docs/schematics/architecture.vsdx new file mode 100644 index 00000000..c4ff13d2 Binary files /dev/null and b/docs/schematics/architecture.vsdx differ diff --git a/docs/schematics/how-mitmproxy-works-explicit-https.png b/docs/schematics/how-mitmproxy-works-explicit-https.png new file mode 100644 index 00000000..1f1ca023 Binary files /dev/null and b/docs/schematics/how-mitmproxy-works-explicit-https.png differ diff --git a/docs/schematics/how-mitmproxy-works-explicit.png b/docs/schematics/how-mitmproxy-works-explicit.png new file mode 100644 index 00000000..c9ba26a7 Binary files /dev/null and b/docs/schematics/how-mitmproxy-works-explicit.png differ diff --git a/docs/schematics/how-mitmproxy-works-transparent-https.png b/docs/schematics/how-mitmproxy-works-transparent-https.png new file mode 100644 index 00000000..559cddd2 Binary files /dev/null and b/docs/schematics/how-mitmproxy-works-transparent-https.png differ diff --git a/docs/schematics/how-mitmproxy-works-transparent.png b/docs/schematics/how-mitmproxy-works-transparent.png new file mode 100644 index 00000000..3994d681 Binary files /dev/null and b/docs/schematics/how-mitmproxy-works-transparent.png differ diff --git a/docs/schematics/proxy-modes-flowchart.png b/docs/schematics/proxy-modes-flowchart.png new file mode 100644 index 00000000..716b5ee2 Binary files /dev/null and b/docs/schematics/proxy-modes-flowchart.png differ diff --git a/docs/schematics/proxy-modes-regular.png b/docs/schematics/proxy-modes-regular.png new file mode 100644 index 00000000..95bada08 Binary files /dev/null and b/docs/schematics/proxy-modes-regular.png differ diff --git a/docs/schematics/proxy-modes-reverse.png b/docs/schematics/proxy-modes-reverse.png new file mode 100644 index 00000000..071d3fc8 Binary files /dev/null and b/docs/schematics/proxy-modes-reverse.png differ diff --git a/docs/schematics/proxy-modes-transparent-1.png b/docs/schematics/proxy-modes-transparent-1.png new file mode 100644 index 00000000..002e0e76 Binary files /dev/null and b/docs/schematics/proxy-modes-transparent-1.png differ diff --git a/docs/schematics/proxy-modes-transparent-2.png b/docs/schematics/proxy-modes-transparent-2.png new file mode 100644 index 00000000..41997b05 Binary files /dev/null and b/docs/schematics/proxy-modes-transparent-2.png differ diff --git a/docs/schematics/proxy-modes-transparent-3.png b/docs/schematics/proxy-modes-transparent-3.png new file mode 100644 index 00000000..ee26cb4f Binary files /dev/null and b/docs/schematics/proxy-modes-transparent-3.png differ diff --git a/docs/schematics/proxy-modes-transparent-wrong.png b/docs/schematics/proxy-modes-transparent-wrong.png new file mode 100644 index 00000000..ca501e93 Binary files /dev/null and b/docs/schematics/proxy-modes-transparent-wrong.png differ diff --git a/docs/schematics/proxy-modes-upstream.png b/docs/schematics/proxy-modes-upstream.png new file mode 100644 index 00000000..d40a6494 Binary files /dev/null and b/docs/schematics/proxy-modes-upstream.png differ diff --git a/docs/schematics/proxy-modes.pdf b/docs/schematics/proxy-modes.pdf new file mode 100644 index 00000000..f07ea05e Binary files /dev/null and b/docs/schematics/proxy-modes.pdf differ diff --git a/docs/schematics/proxy-modes.vsdx b/docs/schematics/proxy-modes.vsdx new file mode 100644 index 00000000..c78cf8d0 Binary files /dev/null and b/docs/schematics/proxy-modes.vsdx differ diff --git a/docs/screenshots/firefox3-import.jpg b/docs/screenshots/firefox3-import.jpg new file mode 100644 index 00000000..47fcd672 Binary files /dev/null and b/docs/screenshots/firefox3-import.jpg differ diff --git a/docs/screenshots/firefox3-trust.jpg b/docs/screenshots/firefox3-trust.jpg new file mode 100644 index 00000000..50a2f341 Binary files /dev/null and b/docs/screenshots/firefox3-trust.jpg differ diff --git a/docs/screenshots/firefox3.jpg b/docs/screenshots/firefox3.jpg new file mode 100644 index 00000000..6c4613b6 Binary files /dev/null and b/docs/screenshots/firefox3.jpg differ diff --git a/docs/screenshots/ios-gateway.png b/docs/screenshots/ios-gateway.png new file mode 100644 index 00000000..2489cba3 Binary files /dev/null and b/docs/screenshots/ios-gateway.png differ diff --git a/docs/screenshots/ios-installed.png b/docs/screenshots/ios-installed.png new file mode 100644 index 00000000..2071e441 Binary files /dev/null and b/docs/screenshots/ios-installed.png differ diff --git a/docs/screenshots/ios-manual.png b/docs/screenshots/ios-manual.png new file mode 100644 index 00000000..3977acfe Binary files /dev/null and b/docs/screenshots/ios-manual.png differ diff --git a/docs/screenshots/ios-profile.png b/docs/screenshots/ios-profile.png new file mode 100644 index 00000000..5bcd5a0d Binary files /dev/null and b/docs/screenshots/ios-profile.png differ diff --git a/docs/screenshots/ios-reverse.png b/docs/screenshots/ios-reverse.png new file mode 100644 index 00000000..6ab5b7c0 Binary files /dev/null and b/docs/screenshots/ios-reverse.png differ diff --git a/docs/screenshots/ios-warning.png b/docs/screenshots/ios-warning.png new file mode 100644 index 00000000..d882c514 Binary files /dev/null and b/docs/screenshots/ios-warning.png differ diff --git a/docs/screenshots/mitmproxy-flowview.png b/docs/screenshots/mitmproxy-flowview.png new file mode 100644 index 00000000..154963fe Binary files /dev/null and b/docs/screenshots/mitmproxy-flowview.png differ diff --git a/docs/screenshots/mitmproxy-intercept-filt.png b/docs/screenshots/mitmproxy-intercept-filt.png new file mode 100644 index 00000000..60556ee7 Binary files /dev/null and b/docs/screenshots/mitmproxy-intercept-filt.png differ diff --git a/docs/screenshots/mitmproxy-intercept-mid.png b/docs/screenshots/mitmproxy-intercept-mid.png new file mode 100644 index 00000000..d5b03922 Binary files /dev/null and b/docs/screenshots/mitmproxy-intercept-mid.png differ diff --git a/docs/screenshots/mitmproxy-intercept-options.png b/docs/screenshots/mitmproxy-intercept-options.png new file mode 100644 index 00000000..8dc4ad2c Binary files /dev/null and b/docs/screenshots/mitmproxy-intercept-options.png differ diff --git a/docs/screenshots/mitmproxy-intercept-result.png b/docs/screenshots/mitmproxy-intercept-result.png new file mode 100644 index 00000000..7d9f5c94 Binary files /dev/null and b/docs/screenshots/mitmproxy-intercept-result.png differ diff --git a/docs/screenshots/mitmproxy-kveditor-editmode.png b/docs/screenshots/mitmproxy-kveditor-editmode.png new file mode 100644 index 00000000..a8315ee5 Binary files /dev/null and b/docs/screenshots/mitmproxy-kveditor-editmode.png differ diff --git a/docs/screenshots/mitmproxy-kveditor.png b/docs/screenshots/mitmproxy-kveditor.png new file mode 100644 index 00000000..144b9701 Binary files /dev/null and b/docs/screenshots/mitmproxy-kveditor.png differ diff --git a/docs/screenshots/mitmproxy.png b/docs/screenshots/mitmproxy.png new file mode 100644 index 00000000..42a10e32 Binary files /dev/null and b/docs/screenshots/mitmproxy.png differ diff --git a/docs/screenshots/osx-addcert-alwaystrust.png b/docs/screenshots/osx-addcert-alwaystrust.png new file mode 100644 index 00000000..4c5cc704 Binary files /dev/null and b/docs/screenshots/osx-addcert-alwaystrust.png differ diff --git a/docs/screenshots/win7-certstore-trustedroot.png b/docs/screenshots/win7-certstore-trustedroot.png new file mode 100644 index 00000000..e15a87f5 Binary files /dev/null and b/docs/screenshots/win7-certstore-trustedroot.png differ diff --git a/docs/screenshots/win7-certstore.png b/docs/screenshots/win7-certstore.png new file mode 100644 index 00000000..f8ce54bd Binary files /dev/null and b/docs/screenshots/win7-certstore.png differ diff --git a/docs/screenshots/win7-wizard.png b/docs/screenshots/win7-wizard.png new file mode 100644 index 00000000..eff6ad09 Binary files /dev/null and b/docs/screenshots/win7-wizard.png differ diff --git a/docs/screenshots/winpythoninstaller.jpg b/docs/screenshots/winpythoninstaller.jpg new file mode 100644 index 00000000..0473c66a Binary files /dev/null and b/docs/screenshots/winpythoninstaller.jpg differ diff --git a/docs/scripting/inlinescripts.rst b/docs/scripting/inlinescripts.rst new file mode 100644 index 00000000..9b5ced5b --- /dev/null +++ b/docs/scripting/inlinescripts.rst @@ -0,0 +1,216 @@ +.. _inline-scripts: + +Inline Scripts +============== + +**mitmproxy** has a powerful scripting API that allows you to modify flows +on-the-fly or rewrite previously saved flows locally. + +The mitmproxy scripting API is event driven - a script is simply a Python +module that exposes a set of event methods. Here's a complete mitmproxy script +that adds a new header to every HTTP response before it is returned to the +client: + +.. literalinclude:: ../../examples/add_header.py + :caption: examples/add_header.py + :language: python + +The first argument to each event method is an instance of +:py:class:`~libmproxy.script.ScriptContext` that lets the script interact with the global mitmproxy +state. The **response** event also gets an instance of :py:class:`~libmproxy.script.ScriptContext`, +which we can use to manipulate the response itself. + +We can now run this script using mitmdump or mitmproxy as follows: + +>>> mitmdump -s add_header.py + +The new header will be added to all responses passing through the proxy. + +Examples +-------- + +mitmproxy comes with a variety of example inline scripts, which demonstrate many basic tasks. +We encourage you to either browse them locally or on `GitHub`_. + + +Events +------ + +.. TODO: Split this into Connection, HTTP and TCP events once we have TCP events. + +The ``context`` argument passed to each event method is always a +:py:class:`~libmproxy.script.ScriptContext` instance. It is guaranteed to be the same object +for the scripts lifetime and is not shared between multiple inline scripts. You can safely use it +to store any form of state you require. + +Events are listed in the order they usually occur. + +.. py:function:: start(context, argv) + + Called once on startup, before any other events. + + :param List[str] argv: The inline scripts' arguments. + For example, ``mitmproxy -s 'example.py --foo 42'`` sets argv to ``["--foo", "42"]``. + +.. py:function:: clientconnect(context, root_layer) + + Called when a client initiates a connection to the proxy. Note that + a connection can correspond to multiple HTTP requests. + + .. versionchanged:: 0.14 + + :param Layer root_layer: The root layer (see :ref:`protocols` for an explanation what the root + layer is), which provides transparent access to all attributes of the + :py:class:`~libmproxy.proxy.RootContext`. For example, ``root_layer.client_conn.address`` + gives the remote address of the connecting client. + + +.. py:function:: request(context, flow) + + Called when a client request has been received. The ``flow`` object is + guaranteed to have a non-None ``request`` attribute. + + :param HTTPFlow flow: The flow containing the request which has been received. + The object is guaranteed to have a non-None ``request`` attribute. + +.. py:function:: serverconnect(context, server_conn) + + Called before the proxy initiates a connection to the target server. Note that + a connection can correspond to multiple HTTP requests. + + :param ServerConnection server_conn: The server connection object. It is guaranteed to have a + non-None ``address`` attribute. + +.. py:function:: responseheaders(context, flow) + + Called when the headers of a server response have been received. + This will always be called before the response hook. + + :param HTTPFlow flow: The flow containing the request and response. + The object is guaranteed to have non-None ``request`` and + ``response`` attributes. ``response.content`` will be ``None``, + as the response body has not been read yet. + +.. py:function:: response(context, flow) + + Called when a server response has been received. + + :param HTTPFlow flow: The flow containing the request and response. + The object is guaranteed to have non-None ``request`` and + ``response`` attributes. ``response.body`` will contain the raw response body, + unless response streaming has been enabled. + +.. py:function:: error(context, flow) + + Called when a flow error has occurred, e.g. invalid server responses, or + interrupted connections. This is distinct from a valid server HTTP error + response, which is simply a response with an HTTP error code. + + :param HTTPFlow flow: The flow containing the error. + It is guaranteed to have non-None ``error`` attribute. + +.. py:function:: serverdisconnect(context, server_conn) + + Called when the proxy has closed the server connection. + + .. versionadded:: 0.14 + + :param ServerConnection server_conn: see :py:func:`serverconnect` + +.. py:function:: clientdisconnect(context, root_layer) + + Called when a client disconnects from the proxy. + + .. versionchanged:: 0.14 + + :param Layer root_layer: see :py:func:`clientconnect` + +.. py:function:: done(context) + + Called once on script shutdown, after any other events. + + +API +--- + +The canonical API documentation is the code, which you can browse here, locally or on `GitHub`_. +*Use the Source, Luke!* + +The main classes you will deal with in writing mitmproxy scripts are: + +:py:class:`~libmproxy.script.ScriptContext` + - A handle for interacting with mitmproxy's Flow Master from within scripts. +:py:class:`~libmproxy.models.ClientConnection` + - Describes a client connection. +:py:class:`~libmproxy.models.ServerConnection` + - Describes a server connection. +:py:class:`~libmproxy.models.HTTPFlow` + - A collection of objects representing a single HTTP transaction. +:py:class:`~libmproxy.models.HTTPRequest` + - An HTTP request. +:py:class:`~libmproxy.models.HTTPResponse` + - An HTTP response. +:py:class:`~libmproxy.models.Error` + - A communications error. +:py:class:`netlib.odict.ODDict` + - A dictionary-like object for managing sets of key/value data. There + is also a variant called ODictCaseless that ignores key case for some + calls (used mainly for headers). +:py:class:`netlib.certutils.SSLCert` + - Exposes information SSL certificates. +:py:class:`libmproxy.flow.FlowMaster` + - The "heart" of libmproxy, usually subclassed as :py:class:`libmproxy.dump.DumpMaster` or + :py:class:`libmproxy.console.ConsoleMaster`. + +Script Context +-------------- + +.. autoclass:: libmproxy.script.ScriptContext + :members: + :undoc-members: + +Running scripts in parallel +--------------------------- + +We have a single flow primitive, so when a script is blocking, other requests are not processed. +While that's usually a very desirable behaviour, blocking scripts can be run threaded by using the +:py:obj:`libmproxy.script.concurrent` decorator. +**If your script does not block, you should avoid the overhead of the decorator.** + +.. literalinclude:: ../../examples/nonblocking.py + :caption: examples/nonblocking.py + :language: python + +Make scripts configurable with arguments +---------------------------------------- + +Sometimes, you want to pass runtime arguments to the inline script. This can be simply done by +surrounding the script call with quotes, e.g. ```mitmdump -s 'script.py --foo 42'``. +The arguments are then exposed in the start event: + +.. literalinclude:: ../../examples/modify_response_body.py + :caption: examples/modify_response_body.py + :language: python + +Running scripts on saved flows +------------------------------ + +Sometimes, we want to run a script on :py:class:`~libmproxy.models.Flow` objects that are already +complete. This happens when you start a script, and then load a saved set of flows from a file +(see the "scripted data transformation" example `here `_). +It also happens when you run a one-shot script on a single flow through the ``|`` (pipe) shortcut +in mitmproxy. + +In this case, there are no client connections, and the events are run in the following order: +**start**, **request**, **responseheaders**, **response**, **error**, **done**. +If the flow doesn't have a **response** or **error** associated with it, the matching events will +be skipped. + +Spaces in the script path +------------------------- + +By default, spaces are interpreted as a separator between the inline script and its arguments +(e.g. ``-s 'foo.py 42'``). Consequently, the script path needs to be wrapped in a separate pair of +quotes if it contains spaces: ``-s '\'./foo bar/baz.py\' 42'``. + +.. _GitHub: https://github.com/mitmproxy/mitmproxy diff --git a/docs/scripting/libmproxy.rst b/docs/scripting/libmproxy.rst new file mode 100644 index 00000000..e263b89b --- /dev/null +++ b/docs/scripting/libmproxy.rst @@ -0,0 +1,27 @@ +.. _libmproxy: + +libmproxy +========= + +.. note:: + + We strongly encourage you to use :ref:`inline-scripts` rather than libmproxy. + - Inline Scripts are equally powerful and provide an easier syntax. + - Most examples are written as inline scripts. + - Multiple inline scripts can be used together. + - Inline Scripts can either be executed headless with mitmdump or within the mitmproxy UI. + + +All of mitmproxy's basic functionality is exposed through the **libmproxy** +library. The example below shows a simple implementation of the "sticky cookie" +functionality included in the interactive mitmproxy program. Traffic is +monitored for ``Cookie`` and ``Set-Cookie`` headers, and requests are rewritten +to include a previously seen cookie if they don't already have one. In effect, +this lets you log in to a site using your browser, and then make subsequent +requests using a tool like curl, which will then seem to be part of the +authenticated session. + + +.. literalinclude:: ../../examples/stickycookies + :caption: examples/stickycookies + :language: python diff --git a/docs/transparent.rst b/docs/transparent.rst new file mode 100644 index 00000000..fbc94e08 --- /dev/null +++ b/docs/transparent.rst @@ -0,0 +1,6 @@ +.. _transparent: + +Transparent Proxying +==================== + +TODO \ No newline at end of file -- cgit v1.2.3