diff options
51 files changed, 172 insertions, 145 deletions
@@ -221,11 +221,11 @@ * All mitmproxy tools are now Python 3 only! We plan to support Python 3.5 and higher. - * Web-Based User Interface: Mitmproxy now offically has a web-based user interface + * Web-Based User Interface: Mitmproxy now officially has a web-based user interface called mitmweb. We consider it stable for all features currently exposed in the UI, but it still misses a lot of mitmproxy’s options. - * Windows Compatibility: With mitmweb, mitmproxy is now useable on Windows. + * Windows Compatibility: With mitmweb, mitmproxy is now usable on Windows. We are also introducing an installer (kindly sponsored by BitRock) that simplifies setup. @@ -387,7 +387,7 @@ * libmproxy: Avoid double-connect in case of TLS Server Name Indication. This yields a massive speedup for TLS handshakes. - * libmproxy: Prevent unneccessary upstream connections (macmantrl) + * libmproxy: Prevent unnecessary upstream connections (macmantrl) * Inline Scripts: New API for HTTP Headers: http://docs.mitmproxy.org/en/latest/dev/models.html#netlib.http.Headers @@ -650,7 +650,7 @@ JSON, Javascript, images, XML, URL-encoded forms, as well as hexadecimal and raw views. - * Add Set Headers, analagous to replacement hooks. Defines headers that are set + * Add Set Headers, analogous to replacement hooks. Defines headers that are set on flows, based on a matching pattern. * A graphical editor for path components in mitmproxy. @@ -708,7 +708,7 @@ expanding random and generated portions, and logging a reproducible specification. - * Streamline the specification langauge. HTTP response message is now + * Streamline the specification language. HTTP response message is now specified using the "r" mnemonic. * Add a "u" mnemonic for specifying User-Agent strings. Add a set of @@ -26,7 +26,7 @@ and pathod websites. |mitmproxy_site| -The latest documentation for mitmproxy is also available on ReadTheDocs. +The latest documentation for mitmproxy is available on our website. |mitmproxy_docs| @@ -95,6 +95,12 @@ requirements installed, and you can run the full test suite (including tests for tox +To run complete tests with a full coverage report, you can use the following command: + +.. code-block:: bash + + tox -- --verbose --cov-report=term + For speedier testing, we recommend you run `pytest`_ directly on individual test files or folders: .. code-block:: bash @@ -145,7 +151,7 @@ with the following command: :alt: mitmproxy.org .. |mitmproxy_docs| image:: https://shields.mitmproxy.org/api/docs-latest-brightgreen.svg - :target: http://docs.mitmproxy.org/en/latest/ + :target: https://mitmproxy.org/docs/latest/ :alt: mitmproxy documentation .. |mitmproxy_discourse| image:: https://shields.mitmproxy.org/api/https%3A%2F%2F-discourse.mitmproxy.org-orange.svg diff --git a/docs/raw/proxy-modes.vsdx b/docs/raw/proxy-modes.vsdx Binary files differindex 0128a142..d115ea9e 100644..100755 --- a/docs/raw/proxy-modes.vsdx +++ b/docs/raw/proxy-modes.vsdx diff --git a/docs/src/content/concepts-modes.md b/docs/src/content/concepts-modes.md index 86bb7b0f..7a0b835a 100644 --- a/docs/src/content/concepts-modes.md +++ b/docs/src/content/concepts-modes.md @@ -157,20 +157,20 @@ There are various use-cases: example.com domain and get all requests recorded in mitmproxy. - Say you have some toy project that should get SSL support. Simply set up mitmproxy as a reverse proxy on port 443 and you're done (`mitmdump -p 443 - -R http://localhost:80/`). Mitmproxy auto-detects TLS traffic and intercepts + --mode reverse:http://localhost:80/`). Mitmproxy auto-detects TLS traffic and intercepts it dynamically. There are better tools for this specific task, but mitmproxy is very quick and simple way to set up an SSL-speaking server. - Want to add a non-SSL-capable compression proxy in front of your server? You - could even spawn a mitmproxy instance that terminates SSL (`-R http://...`), + could even spawn a mitmproxy instance that terminates SSL (`--mode reverse:http://...`), point it to the compression proxy and let the compression proxy point to a - SSL-initiating mitmproxy (`-R https://...`), which then points to the real + SSL-initiating mitmproxy (`--mode reverse:https://...`), which then points to the real server. As you see, it's a fairly flexible thing. ### Host Header In reverse proxy mode, mitmproxy automatically rewrites the Host header to match the upstream server. This allows mitmproxy to easily connect to existing -endpoints on the open web (e.g. `mitmproxy -R https://example.com`). You can +endpoints on the open web (e.g. `mitmproxy --mode reverse:https://example.com`). You can disable this behaviour with the `keep_host_header` option. However, keep in mind that absolute URLs within the returned document or HTTP diff --git a/docs/src/content/howto-transparent-vms.md b/docs/src/content/howto-transparent-vms.md index b186fd39..1446ede7 100644 --- a/docs/src/content/howto-transparent-vms.md +++ b/docs/src/content/howto-transparent-vms.md @@ -106,7 +106,7 @@ sudo iptables -t nat -A PREROUTING -i eth1 -p tcp --dport 443 -j REDIRECT --to-p Finally, we can run mitmproxy in transparent mode with {{< highlight bash >}} -mitmproxy -T +mitmproxy --mode transparent {{< / highlight >}} The proxied machine cannot to leak any data outside of HTTP or DNS requests. If diff --git a/docs/src/content/howto-transparent.md b/docs/src/content/howto-transparent.md index 224cb5ee..3d99e9dc 100644 --- a/docs/src/content/howto-transparent.md +++ b/docs/src/content/howto-transparent.md @@ -58,7 +58,7 @@ dropped privileges. It can be used as follows: gcc examples/complex/full_transparency_shim.c -o mitmproxy_shim -lcap sudo chown root:root mitmproxy_shim sudo chmod u+s mitmproxy_shim -./mitmproxy_shim $(which mitmproxy) -T --spoof-source-address +./mitmproxy_shim $(which mitmproxy) --mode transparent --set spoof-source-address {{< / highlight >}} @@ -112,10 +112,10 @@ something like this: You probably want a command like this: {{< highlight bash >}} -mitmproxy -T --host +mitmproxy --mode transparent --showhost {{< / highlight >}} -The `-T` flag turns on transparent mode, and the `--host` argument tells +The `--mode transparent` option turns on transparent mode, and the `--showhost` argument tells mitmproxy to use the value of the Host header for URL display. ### 6. Finally, configure your test device @@ -163,10 +163,10 @@ doas pfctl -e You probably want a command like this: {{< highlight bash >}} -mitmproxy -T --host +mitmproxy --mode transparent --showhost {{< / highlight >}} -The `-T` flag turns on transparent mode, and the `--host` argument tells +The `--mode transparent` option turns on transparent mode, and the `--showhost` argument tells mitmproxy to use the value of the Host header for URL display. ### 6. Finally, configure your test device @@ -245,10 +245,10 @@ tighten the restriction up to the user running mitmproxy. You probably want a command like this: {{< highlight bash >}} -mitmproxy -T --host +mitmproxy --mode transparent --showhost {{< / highlight >}} -The `-T` flag turns on transparent mode, and the `--host` argument tells +The `--mode transparent` flag turns on transparent mode, and the `--showhost` argument tells mitmproxy to use the value of the Host header for URL display. ### 6. Finally, configure your test device diff --git a/docs/src/content/overview-tools.md b/docs/src/content/overview-tools.md index 7612383a..0200e899 100644 --- a/docs/src/content/overview-tools.md +++ b/docs/src/content/overview-tools.md @@ -8,7 +8,7 @@ menu: # Overview -You should thin of the mitmproxy project's tools as a set of front-ends that +You should think of the mitmproxy project's tools as a set of front-ends that expose the same underlying functionality. We aim to have feature parity across all of our tooling, and all tools share a common configuration mechanism and most command-line options. diff --git a/docs/src/static/schematics/proxy-modes-reverse.png b/docs/src/static/schematics/proxy-modes-reverse.png Binary files differindex 071d3fc8..87372688 100644..100755 --- a/docs/src/static/schematics/proxy-modes-reverse.png +++ b/docs/src/static/schematics/proxy-modes-reverse.png diff --git a/docs/src/static/schematics/proxy-modes-transparent-1.png b/docs/src/static/schematics/proxy-modes-transparent-1.png Binary files differindex 002e0e76..557406f8 100644..100755 --- a/docs/src/static/schematics/proxy-modes-transparent-1.png +++ b/docs/src/static/schematics/proxy-modes-transparent-1.png diff --git a/docs/src/static/schematics/proxy-modes-transparent-2.png b/docs/src/static/schematics/proxy-modes-transparent-2.png Binary files differindex 41997b05..ecc09f5a 100644..100755 --- a/docs/src/static/schematics/proxy-modes-transparent-2.png +++ b/docs/src/static/schematics/proxy-modes-transparent-2.png diff --git a/docs/src/static/schematics/proxy-modes-transparent-3.png b/docs/src/static/schematics/proxy-modes-transparent-3.png Binary files differindex ee26cb4f..177aaaef 100644..100755 --- a/docs/src/static/schematics/proxy-modes-transparent-3.png +++ b/docs/src/static/schematics/proxy-modes-transparent-3.png diff --git a/docs/src/static/schematics/proxy-modes-transparent-wrong.png b/docs/src/static/schematics/proxy-modes-transparent-wrong.png Binary files differindex ca501e93..c69c6b70 100644..100755 --- a/docs/src/static/schematics/proxy-modes-transparent-wrong.png +++ b/docs/src/static/schematics/proxy-modes-transparent-wrong.png diff --git a/docs/src/static/schematics/proxy-modes-upstream.png b/docs/src/static/schematics/proxy-modes-upstream.png Binary files differindex d40a6494..daa2b9af 100644..100755 --- a/docs/src/static/schematics/proxy-modes-upstream.png +++ b/docs/src/static/schematics/proxy-modes-upstream.png diff --git a/examples/addons/events.py b/examples/addons/events.py index 93664954..d3c90430 100644 --- a/examples/addons/events.py +++ b/examples/addons/events.py @@ -155,7 +155,7 @@ class Events: def log(self, entry: mitmproxy.log.LogEntry): """ - Called whenver a new log entry is created through the mitmproxy + Called whenever a new log entry is created through the mitmproxy context. Be careful not to log from this event, which will cause an infinite loop! """ diff --git a/examples/complex/README.md b/examples/complex/README.md index 77dbe2f5..c53503e4 100644 --- a/examples/complex/README.md +++ b/examples/complex/README.md @@ -10,7 +10,7 @@ | mitmproxywrapper.py | Bracket mitmproxy run with proxy enable/disable on OS X | | nonblocking.py | Demonstrate parallel processing with a blocking script | | remote_debug.py | This script enables remote debugging of the mitmproxy _UI_ with PyCharm. | -| sslstrip.py | sslstrip-like funtionality implemented with mitmproxy | +| sslstrip.py | sslstrip-like functionality implemented with mitmproxy | | stream.py | Enable streaming for all responses. | | stream_modify.py | Modify a streamed response body. | | tcp_message.py | Modify a raw TCP connection | diff --git a/examples/complex/stream_modify.py b/examples/complex/stream_modify.py index 5e5da95b..46bdcb78 100644 --- a/examples/complex/stream_modify.py +++ b/examples/complex/stream_modify.py @@ -3,7 +3,7 @@ This inline script modifies a streamed response. If you do not need streaming, see the modify_response_body example. Be aware that content replacement isn't trivial: - If the transfer encoding isn't chunked, you cannot simply change the content length. - - If you want to replace all occurences of "foobar", make sure to catch the cases + - If you want to replace all occurrences of "foobar", make sure to catch the cases where one chunk ends with [...]foo" and the next starts with "bar[...]. """ diff --git a/examples/complex/xss_scanner.py b/examples/complex/xss_scanner.py index 0ee38cd4..0c0dd0f3 100755 --- a/examples/complex/xss_scanner.py +++ b/examples/complex/xss_scanner.py @@ -215,7 +215,7 @@ def get_SQLi_data(new_body: str, original_body: str, request_URL: str, injection # A qc is either ' or " def inside_quote(qc: str, substring_bytes: bytes, text_index: int, body_bytes: bytes) -> bool: - """ Whether the Numberth occurence of the first string in the second + """ Whether the Numberth occurrence of the first string in the second string is inside quotes as defined by the supplied QuoteChar """ substring = substring_bytes.decode('utf-8') body = body_bytes.decode('utf-8') @@ -246,7 +246,7 @@ def paths_to_text(html: str, string: str) -> List[str]: - Note that it does a BFS """ def remove_last_occurence_of_sub_string(string: str, substr: str) -> str: - """ Delete the last occurence of substr from str + """ Delete the last occurrence of substr from str String String -> String """ index = string.rfind(substr) @@ -274,7 +274,7 @@ def paths_to_text(html: str, string: str) -> List[str]: def get_XSS_data(body: Union[str, bytes], request_URL: str, injection_point: str) -> Optional[XSSData]: """ Return a XSSDict if there is a XSS otherwise return None """ def in_script(text, index, body) -> bool: - """ Whether the Numberth occurence of the first string in the second + """ Whether the Numberth occurrence of the first string in the second string is inside a script tag """ paths = paths_to_text(body.decode('utf-8'), text.decode("utf-8")) try: @@ -284,7 +284,7 @@ def get_XSS_data(body: Union[str, bytes], request_URL: str, injection_point: str return False def in_HTML(text: bytes, index: int, body: bytes) -> bool: - """ Whether the Numberth occurence of the first string in the second + """ Whether the Numberth occurrence of the first string in the second string is inside the HTML but not inside a script tag or part of a HTML attribute""" # if there is a < then lxml will interpret that as a tag, so only search for the stuff before it diff --git a/mitmproxy/addons/cut.py b/mitmproxy/addons/cut.py index 1c8fbc05..f9874038 100644 --- a/mitmproxy/addons/cut.py +++ b/mitmproxy/addons/cut.py @@ -61,7 +61,7 @@ class Cut: from the base of the flow object, with a few conveniences - "port" and "host" retrieve parts of an address tuple, ".header[key]" retrieves a header value. Return values converted to strings or - bytes: SSL certicates are converted to PEM format, bools are "true" + bytes: SSL certificates are converted to PEM format, bools are "true" or "false", "bytes" are preserved, and all other values are converted to strings. """ diff --git a/mitmproxy/addons/dumper.py b/mitmproxy/addons/dumper.py index 48bc8118..0df2d339 100644 --- a/mitmproxy/addons/dumper.py +++ b/mitmproxy/addons/dumper.py @@ -31,6 +31,23 @@ class Dumper: self.filter = None # type: flowfilter.TFilter self.outfp = outfile # type: typing.io.TextIO + def load(self, loader): + loader.add_option( + "flow_detail", int, 1, + """ + The display detail level for flows in mitmdump: 0 (almost quiet) to 3 (very verbose). + 0: shortened request URL, response status code, WebSocket and TCP message notifications. + 1: full request URL with response status code + 2: 1 + HTTP headers + 3: 2 + full response content, content of WebSocket and TCP messages. + """ + ) + loader.add_option( + "dumper_default_contentview", str, "auto", + "The default content view mode.", + choices = [i.name.lower() for i in contentviews.views] + ) + def configure(self, updated): if "view_filter" in updated: if ctx.options.view_filter: @@ -61,7 +78,7 @@ class Dumper: def _echo_message(self, message): _, lines, error = contentviews.get_message_content_view( - ctx.options.default_contentview, + ctx.options.dumper_default_contentview, message ) if error: diff --git a/mitmproxy/addons/intercept.py b/mitmproxy/addons/intercept.py index 9e1a283e..d39d1962 100644 --- a/mitmproxy/addons/intercept.py +++ b/mitmproxy/addons/intercept.py @@ -1,3 +1,5 @@ +import typing + from mitmproxy import flowfilter from mitmproxy import exceptions from mitmproxy import ctx @@ -7,6 +9,17 @@ class Intercept: def __init__(self): self.filt = None + def load(self, loader): + loader.add_option( + "intercept_active", bool, False, + "Intercept toggle" + ) + + loader.add_option( + "intercept", typing.Optional[str], None, + "Intercept filter expression." + ) + def configure(self, updated): if "intercept" in updated: if not ctx.options.intercept: diff --git a/mitmproxy/addons/proxyauth.py b/mitmproxy/addons/proxyauth.py index dc99d5cc..37d7d93c 100644 --- a/mitmproxy/addons/proxyauth.py +++ b/mitmproxy/addons/proxyauth.py @@ -52,6 +52,18 @@ class ProxyAuth: self.authenticated = weakref.WeakKeyDictionary() # type: MutableMapping[connections.ClientConnection, Tuple[str, str]] """Contains all connections that are permanently authenticated after an HTTP CONNECT""" + def load(self, loader): + loader.add_option( + "proxyauth", Optional[str], None, + """ + Require proxy authentication. Format: + "username:pass", + "any" to accept any user/pass combination, + "@path" to use an Apache htpasswd file, + or "ldap[s]:url_server_ldap:dn_auth:password:dn_subtree" for LDAP authentication. + """ + ) + def enabled(self) -> bool: return any([self.nonanonymous, self.htpasswd, self.singleuser, self.ldapconn, self.ldapserver]) @@ -160,7 +172,7 @@ class ProxyAuth: server = ldap3.Server(ldap_server) else: raise exceptions.OptionsError( - "Invalid ldap specfication on the first part" + "Invalid ldap specification on the first part" ) conn = ldap3.Connection( server, diff --git a/mitmproxy/addons/upstream_auth.py b/mitmproxy/addons/upstream_auth.py index ea6af337..0b90b48f 100644 --- a/mitmproxy/addons/upstream_auth.py +++ b/mitmproxy/addons/upstream_auth.py @@ -42,7 +42,7 @@ class UpstreamAuth(): # FIXME: We're doing this because our proxy core is terminally confused # at the moment. Ideally, we should be able to check if we're in # reverse proxy mode at the HTTP layer, so that scripts can put the - # proxy in reverse proxy mode for specific reuests. + # proxy in reverse proxy mode for specific requests. if "upstream_auth" in updated: if ctx.options.upstream_auth is None: self.auth = None diff --git a/mitmproxy/addons/view.py b/mitmproxy/addons/view.py index e87daf35..c0f7e7b4 100644 --- a/mitmproxy/addons/view.py +++ b/mitmproxy/addons/view.py @@ -325,7 +325,7 @@ class View(collections.Sequence): key: str ) -> None: """ - Toggle a boolean value in the settings store, seting the value to + Toggle a boolean value in the settings store, setting the value to the string "true" or "false". """ updated = [] diff --git a/mitmproxy/addons/wsgiapp.py b/mitmproxy/addons/wsgiapp.py index 155444fc..549d8c87 100644 --- a/mitmproxy/addons/wsgiapp.py +++ b/mitmproxy/addons/wsgiapp.py @@ -7,7 +7,7 @@ from mitmproxy import version class WSGIApp: """ - An addon that hosts a WSGI app withing mitproxy, at a specified + An addon that hosts a WSGI app within mitproxy, at a specified hostname and port. """ def __init__(self, app, host, port): diff --git a/mitmproxy/certs.py b/mitmproxy/certs.py index 4e10529a..6487b750 100644 --- a/mitmproxy/certs.py +++ b/mitmproxy/certs.py @@ -160,7 +160,7 @@ class CertStore: def load_dhparam(path): # mitmproxy<=0.10 doesn't generate a dhparam file. - # Create it now if neccessary. + # Create it now if necessary. if not os.path.exists(path): with open(path, "wb") as f: f.write(DEFAULT_DHPARAM) diff --git a/mitmproxy/command.py b/mitmproxy/command.py index 48968c90..45141576 100644 --- a/mitmproxy/command.py +++ b/mitmproxy/command.py @@ -54,7 +54,7 @@ class Command: self.has_positional = False for i in sig.parameters.values(): - # This is the kind for *args paramters + # This is the kind for *args parameters if i.kind == i.VAR_POSITIONAL: self.has_positional = True self.paramtypes = [v.annotation for v in sig.parameters.values()] diff --git a/mitmproxy/controller.py b/mitmproxy/controller.py index f39c1b24..beb210ca 100644 --- a/mitmproxy/controller.py +++ b/mitmproxy/controller.py @@ -56,7 +56,7 @@ class Reply: self._state = "start" # "start" -> "taken" -> "committed" - # Holds the reply value. May change before things are actually commited. + # Holds the reply value. May change before things are actually committed. self.value = NO_REPLY @property @@ -66,7 +66,7 @@ class Reply: sequentially through the following lifecycle: 1. start: Initial State. - 2. taken: The reply object has been taken to be commited. + 2. taken: The reply object has been taken to be committed. 3. committed: The reply has been sent back to the requesting party. This attribute is read-only and can only be modified by calling one of @@ -91,7 +91,7 @@ class Reply: def commit(self): """ - Ultimately, messages are commited. This is done either automatically by + Ultimately, messages are committed. This is done either automatically by if the message is not taken or manually by the entity which called .take(). """ diff --git a/mitmproxy/io/compat.py b/mitmproxy/io/compat.py index 51bd116b..231fa3b8 100644 --- a/mitmproxy/io/compat.py +++ b/mitmproxy/io/compat.py @@ -123,7 +123,7 @@ def convert_200_300(data): def convert_300_4(data): data["version"] = 4 - # Ths is an empty migration to transition to the new versioning scheme. + # This is an empty migration to transition to the new versioning scheme. return data diff --git a/mitmproxy/net/http/http1/read.py b/mitmproxy/net/http/http1/read.py index 0f70b1a7..294e8358 100644 --- a/mitmproxy/net/http/http1/read.py +++ b/mitmproxy/net/http/http1/read.py @@ -43,7 +43,7 @@ def read_request_head(rfile): Raises: exceptions.HttpReadDisconnect: No bytes can be read from rfile. exceptions.HttpSyntaxException: The input is malformed HTTP. - exceptions.HttpException: Any other error occured. + exceptions.HttpException: Any other error occurred. """ timestamp_start = time.time() if hasattr(rfile, "reset_timestamps"): @@ -82,7 +82,7 @@ def read_response_head(rfile): Raises: exceptions.HttpReadDisconnect: No bytes can be read from rfile. exceptions.HttpSyntaxException: The input is malformed HTTP. - exceptions.HttpException: Any other error occured. + exceptions.HttpException: Any other error occurred. """ timestamp_start = time.time() diff --git a/mitmproxy/options.py b/mitmproxy/options.py index 70d454fd..7a00bb87 100644 --- a/mitmproxy/options.py +++ b/mitmproxy/options.py @@ -1,7 +1,6 @@ from typing import Optional, Sequence from mitmproxy import optmanager -from mitmproxy import contentviews from mitmproxy.net import tls log_verbosity = [ @@ -57,11 +56,6 @@ class Options(optmanager.OptManager): # FIXME: Options that must be migrated to addons, but are complicated # because they're used by more than one addon, or because they're # embedded in the core code somehow. - default_contentview = None # type: str - flow_detail = None # type: int - intercept = None # type: Optional[str] - intercept_active = None # type: bool - proxyauth = None # type: Optional[str] showhost = None # type: bool verbosity = None # type: str view_filter = None # type: Optional[str] @@ -81,24 +75,9 @@ class Options(optmanager.OptManager): "Log verbosity.", choices=log_verbosity ) - self.add_option( - "default_contentview", str, "auto", - "The default content view mode.", - choices = [i.name.lower() for i in contentviews.views] - ) # Proxy options self.add_option( - "proxyauth", Optional[str], None, - """ - Require proxy authentication. Format: - "username:pass", - "any" to accept any user/pass combination, - "@path" to use an Apache htpasswd file, - or "ldap[s]:url_server_ldap:dn_auth:password:dn_subtree" for LDAP authentication. - """ - ) - self.add_option( "add_upstream_certs_to_client_chain", bool, False, """ Add all certificates of the upstream server to the certificate chain @@ -253,30 +232,8 @@ class Options(optmanager.OptManager): ) self.add_option( - "intercept_active", bool, False, - "Intercept toggle" - ) - - self.add_option( - "intercept", Optional[str], None, - "Intercept filter expression." - ) - - self.add_option( "view_filter", Optional[str], None, "Limit which flows are displayed." ) - # Dump options - self.add_option( - "flow_detail", int, 1, - """ - The display detail level for flows in mitmdump: 0 (almost quiet) to 3 (very verbose). - 0: shortened request URL, response status code, WebSocket and TCP message notifications. - 1: full request URL with response status code - 2: 1 + HTTP headers - 3: 2 + full response content, content of WebSocket and TCP messages. - """ - ) - self.update(**kwargs) diff --git a/mitmproxy/platform/windows.py b/mitmproxy/platform/windows.py index 1c90a7a0..439c6702 100644 --- a/mitmproxy/platform/windows.py +++ b/mitmproxy/platform/windows.py @@ -359,7 +359,7 @@ class TransparentProxy: packet.dst_addr, packet.dst_port = self.proxy_addr, self.proxy_port packet.direction = pydivert.consts.Direction.INBOUND - # Use any handle thats on the NETWORK layer - request_local may be + # Use any handle that's on the NETWORK layer - request_local may be # unavailable. self.response_handle.send(packet) diff --git a/mitmproxy/proxy/protocol/http.py b/mitmproxy/proxy/protocol/http.py index 076ffa62..99286fa5 100644 --- a/mitmproxy/proxy/protocol/http.py +++ b/mitmproxy/proxy/protocol/http.py @@ -481,7 +481,7 @@ class HttpLayer(base.Layer): if address != self.server_conn.address or tls != self.server_tls: self.set_server(address) self.set_server_tls(tls, address[0]) - # Establish connection is neccessary. + # Establish connection is necessary. if not self.server_conn.connected(): self.connect() else: diff --git a/mitmproxy/proxy/protocol/http2.py b/mitmproxy/proxy/protocol/http2.py index cc99a715..1e7c041d 100644 --- a/mitmproxy/proxy/protocol/http2.py +++ b/mitmproxy/proxy/protocol/http2.py @@ -269,7 +269,7 @@ class Http2Layer(base.Layer): def _handle_priority_updated(self, eid, event): if not self.config.options.http2_priority: - self.log("HTTP/2 PRIORITY frame surpressed. Use --http2-priority to enable forwarding.", "debug") + self.log("HTTP/2 PRIORITY frame suppressed. Use --http2-priority to enable forwarding.", "debug") return True if eid in self.streams and self.streams[eid].handled_priority_event is event: @@ -541,7 +541,7 @@ class Http2SingleStreamLayer(httpbase._HttpTransmissionLayer, basethread.BaseThr # only send priority information if they actually came with the original HeadersFrame # and not if they got updated before/after with a PriorityFrame if not self.config.options.http2_priority: - self.log("HTTP/2 PRIORITY information in HEADERS frame surpressed. Use --http2-priority to enable forwarding.", "debug") + self.log("HTTP/2 PRIORITY information in HEADERS frame suppressed. Use --http2-priority to enable forwarding.", "debug") else: priority_exclusive = self.priority_exclusive priority_depends_on = self._map_depends_on_stream_id(self.server_stream_id, self.priority_depends_on) diff --git a/mitmproxy/proxy/protocol/tls.py b/mitmproxy/proxy/protocol/tls.py index 876c1162..09ce87ba 100644 --- a/mitmproxy/proxy/protocol/tls.py +++ b/mitmproxy/proxy/protocol/tls.py @@ -5,7 +5,7 @@ from mitmproxy import exceptions from mitmproxy.net import tls as net_tls from mitmproxy.proxy.protocol import base -# taken from https://testssl.sh/openssl-rfc.mappping.html +# taken from https://testssl.sh/openssl-rfc.mapping.html CIPHER_ID_NAME_MAP = { 0x00: 'NULL-MD5', 0x01: 'NULL-MD5', diff --git a/mitmproxy/test/taddons.py b/mitmproxy/test/taddons.py index 5930e414..1ca8ba8d 100644 --- a/mitmproxy/test/taddons.py +++ b/mitmproxy/test/taddons.py @@ -137,7 +137,7 @@ class context: def command(self, func, *args): """ - Invoke a command function with a list of string arguments within a command context, mimicing the actual command environment. + Invoke a command function with a list of string arguments within a command context, mimicking the actual command environment. """ cmd = command.Command(self.master.commands, "test.command", func) return cmd.call(args) diff --git a/mitmproxy/tools/console/consoleaddons.py b/mitmproxy/tools/console/consoleaddons.py index c73eda42..3d76d20c 100644 --- a/mitmproxy/tools/console/consoleaddons.py +++ b/mitmproxy/tools/console/consoleaddons.py @@ -77,13 +77,18 @@ class ConsoleAddon: def load(self, loader): loader.add_option( + "console_default_contentview", str, "auto", + "The default content view mode.", + choices = [i.name.lower() for i in contentviews.views] + ) + loader.add_option( "console_layout", str, "single", "Console layout.", choices=sorted(console_layouts), ) loader.add_option( "console_layout_headers", bool, True, - "Show layout comonent headers", + "Show layout component headers", ) loader.add_option( "console_focus_follow", bool, False, @@ -110,15 +115,6 @@ class ConsoleAddon: """ return ["single", "vertical", "horizontal"] - @command.command("console.intercept.toggle") - def intercept_toggle(self) -> None: - """ - Toggles interception on/off leaving intercept filters intact. - """ - ctx.options.update( - intercept_active = not ctx.options.intercept_active - ) - @command.command("console.layout.cycle") def layout_cycle(self) -> None: """ @@ -227,7 +223,7 @@ class ConsoleAddon: ) -> None: """ Prompt the user to choose from a specified list of strings, then - invoke another command with all occurances of {choice} replaced by + invoke another command with all occurrences of {choice} replaced by the choice the user made. """ def callback(opt): @@ -253,7 +249,7 @@ class ConsoleAddon: ) -> None: """ Prompt the user to choose from a list of strings returned by a - command, then invoke another command with all occurances of {choice} + command, then invoke another command with all occurrences of {choice} replaced by the choice the user made. """ choices = ctx.master.commands.call_args(choicecmd, []) @@ -540,7 +536,7 @@ class ConsoleAddon: [ "@focus", "flowview_mode_%s" % idx, - self.master.options.default_contentview, + self.master.options.console_default_contentview, ] ) diff --git a/mitmproxy/tools/console/defaultkeys.py b/mitmproxy/tools/console/defaultkeys.py index c7876288..7f65c1f7 100644 --- a/mitmproxy/tools/console/defaultkeys.py +++ b/mitmproxy/tools/console/defaultkeys.py @@ -26,7 +26,7 @@ def map(km): km.add("ctrl f", "console.nav.pagedown", ["global"], "Page down") km.add("ctrl b", "console.nav.pageup", ["global"], "Page up") - km.add("I", "console.intercept.toggle", ["global"], "Toggle intercept") + km.add("I", "set intercept_active=toggle", ["global"], "Toggle intercept") km.add("i", "console.command.set intercept", ["global"], "Set intercept") km.add("W", "console.command.set save_stream_file", ["global"], "Stream to file") km.add("A", "flow.resume @all", ["flowlist", "flowview"], "Resume all intercepted flows") diff --git a/mitmproxy/tools/console/statusbar.py b/mitmproxy/tools/console/statusbar.py index bdb39013..8553a66f 100644 --- a/mitmproxy/tools/console/statusbar.py +++ b/mitmproxy/tools/console/statusbar.py @@ -197,10 +197,10 @@ class StatusBar(urwid.WidgetWrap): r.append("[") r.append(("heading_key", "u")) r.append(":%s]" % self.master.options.stickyauth) - if self.master.options.default_contentview != "auto": + if self.master.options.console_default_contentview != "auto": r.append("[") r.append(("heading_key", "M")) - r.append(":%s]" % self.master.options.default_contentview) + r.append(":%s]" % self.master.options.console_default_contentview) if self.master.options.has_changed("view_order"): r.append("[") r.append(("heading_key", "o")) diff --git a/mitmproxy/utils/arg_check.py b/mitmproxy/utils/arg_check.py index 873bef06..9c582c4c 100644 --- a/mitmproxy/utils/arg_check.py +++ b/mitmproxy/utils/arg_check.py @@ -11,7 +11,6 @@ DEPRECATED = """ --order --no-mouse --reverse ---socks --http2-priority --no-http2-priority --no-websocket @@ -59,6 +58,7 @@ REPLACED = """ -i -f --filter +--socks """ REPLACEMENTS = { @@ -99,7 +99,8 @@ REPLACEMENTS = { "--replace": "--replacements", "-i": "--intercept", "-f": "--view-filter", - "--filter": "--view-filter" + "--filter": "--view-filter", + "--socks": "--mode socks5" } diff --git a/pathod/pathoc.py b/pathod/pathoc.py index b177d556..18dcccf2 100644 --- a/pathod/pathoc.py +++ b/pathod/pathoc.py @@ -352,7 +352,7 @@ class Pathoc(tcp.TCPClient): timeout: If specified None may be yielded instead if timeout is reached. If timeout is None, wait forever. If timeout is 0, return - immedately if nothing is on the queue. + immediately if nothing is on the queue. finish: If true, consume messages until the reader shuts down. Otherwise, return None on timeout. @@ -434,7 +434,7 @@ class Pathoc(tcp.TCPClient): req = language.serve(r, self.wfile, self.settings) self.wfile.flush() - # build a dummy request to read the reponse + # build a dummy request to read the response # ideally this would be returned directly from language.serve dummy_req = net_http.Request( first_line_format="relative", @@ -471,7 +471,7 @@ class Pathoc(tcp.TCPClient): """ Performs a single request. - r: A language.message.Messsage object, or a string representing + r: A language.message.Message object, or a string representing one. Returns Response if we have a non-ignored response. diff --git a/release/README.md b/release/README.md index 2b709318..b2f97aab 100644 --- a/release/README.md +++ b/release/README.md @@ -16,7 +16,11 @@ Make sure run all these steps on the correct branch you want to create a new rel - Attach all files from the new release folder on https://snapshots.mitmproxy.org ## PyPi -- Upload wheel to pypi: `twine upload <mitmproxy-...-.whl` +- `tox -e rtool -- upload-release` + +## Homebrew +- `tox -e rtool -- homebrew-pr` +- The Homebrew maintainers are typically very fast and detect our new relese within a day, but we can be a nice citizen and create the PR ourself. ## Docker - Update docker-releases repo @@ -28,6 +32,13 @@ Make sure run all these steps on the correct branch you want to create a new rel * `2.0.0` for new major versions * `2.0.2` for new patch versions - Update `latest` tag [here](https://hub.docker.com/r/mitmproxy/mitmproxy/~/settings/automated-builds/) +- Check that the build for this tag succeeds [https://hub.docker.com/r/mitmproxy/mitmproxy/builds/](here) +- If build failed: + - Fix it and commit + - `git tag 3.0.2` the new commit + - `git push origin :refs/tags/3.0.2` to delete the old remote tag + - `git push --tags` to push the new tag + - Check the build details page again ## Prepare for next release diff --git a/release/rtool.py b/release/rtool.py index 9050107e..25f37e61 100755 --- a/release/rtool.py +++ b/release/rtool.py @@ -3,6 +3,7 @@ import contextlib import fnmatch import os +import sys import platform import re import runpy @@ -286,6 +287,24 @@ def upload_release(username, password, repository): ]) +@cli.command("homebrew-pr") +def homebrew_pr(): + """ + Create a new Homebrew PR + """ + if platform.system() != "Darwin": + print("You need to run this on macOS to create a new Homebrew PR. Sorry.") + sys.exit(1) + + print("Creating a new PR with Homebrew...") + subprocess.check_call([ + "brew", + "bump-formula-pr", + "--url", "https://github.com/mitmproxy/mitmproxy/archive/v{}".format(get_version()), + "mitmproxy", + ]) + + @cli.command("upload-snapshot") @click.option("--host", envvar="SNAPSHOT_HOST", prompt=True) @click.option("--port", envvar="SNAPSHOT_PORT", type=int, default=22) diff --git a/test/examples/test_xss_scanner.py b/test/examples/test_xss_scanner.py index 8cf06a2a..610bdd72 100644 --- a/test/examples/test_xss_scanner.py +++ b/test/examples/test_xss_scanner.py @@ -296,6 +296,14 @@ class TestXSSScanner(): assert xss_info == expected_xss_info assert sqli_info is None + def mocked_socket_gethostbyname(domain): + claimed_domains = ["google.com"] + if domain not in claimed_domains: + from socket import gaierror + raise gaierror("[Errno -2] Name or service not known") + else: + return '216.58.221.46' + @pytest.fixture def logger(self): class Logger(): @@ -309,6 +317,7 @@ class TestXSSScanner(): def test_find_unclaimed_URLs(self, monkeypatch, logger): logger.args = [] monkeypatch.setattr("mitmproxy.ctx.log", logger) + monkeypatch.setattr("socket.gethostbyname", self.mocked_socket_gethostbyname) xss.find_unclaimed_URLs("<html><script src=\"http://google.com\"></script></html>", "https://example.com") assert logger.args == [] diff --git a/test/mitmproxy/addons/test_allowremote.py b/test/mitmproxy/addons/test_allowremote.py index 52dae68d..69019726 100644 --- a/test/mitmproxy/addons/test_allowremote.py +++ b/test/mitmproxy/addons/test_allowremote.py @@ -1,7 +1,7 @@ from unittest import mock import pytest -from mitmproxy.addons import allowremote +from mitmproxy.addons import allowremote, proxyauth from mitmproxy.test import taddons @@ -19,7 +19,8 @@ from mitmproxy.test import taddons ]) def test_allowremote(allow_remote, ip, should_be_killed): ar = allowremote.AllowRemote() - with taddons.context(ar) as tctx: + up = proxyauth.ProxyAuth() + with taddons.context(ar, up) as tctx: tctx.options.allow_remote = allow_remote with mock.patch('mitmproxy.proxy.protocol.base.Layer') as layer: diff --git a/test/mitmproxy/addons/test_dumper.py b/test/mitmproxy/addons/test_dumper.py index 9774e131..ead6b7e7 100644 --- a/test/mitmproxy/addons/test_dumper.py +++ b/test/mitmproxy/addons/test_dumper.py @@ -14,7 +14,7 @@ from mitmproxy import http def test_configure(): d = dumper.Dumper() - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, view_filter="~b foo") assert d.filter @@ -33,7 +33,7 @@ def test_configure(): def test_simple(): sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=0) d.response(tflow.tflow(resp=True)) assert not sio.getvalue() @@ -101,7 +101,7 @@ def test_echo_body(): sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=3) d._echo_message(f.response) t = sio.getvalue() @@ -111,7 +111,7 @@ def test_echo_body(): def test_echo_request_line(): sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=3, showhost=True) f = tflow.tflow(client_conn=None, server_conn=True, resp=True) f.request.is_replay = True @@ -146,7 +146,7 @@ class TestContentView: view_auto.side_effect = exceptions.ContentViewException("") sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=4, verbosity='debug') d.response(tflow.tflow()) assert ctx.master.has_log("content viewer failed") @@ -155,7 +155,7 @@ class TestContentView: def test_tcp(): sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=3, showhost=True) f = tflow.ttcpflow() d.tcp_message(f) @@ -170,7 +170,7 @@ def test_tcp(): def test_websocket(): sio = io.StringIO() d = dumper.Dumper(sio) - with taddons.context() as ctx: + with taddons.context(d) as ctx: ctx.configure(d, flow_detail=3, showhost=True) f = tflow.twebsocketflow() d.websocket_message(f) diff --git a/test/mitmproxy/addons/test_intercept.py b/test/mitmproxy/addons/test_intercept.py index d9598101..b3d24626 100644 --- a/test/mitmproxy/addons/test_intercept.py +++ b/test/mitmproxy/addons/test_intercept.py @@ -8,7 +8,7 @@ from mitmproxy.test import tflow def test_simple(): r = intercept.Intercept() - with taddons.context() as tctx: + with taddons.context(r) as tctx: assert not r.filt tctx.configure(r, intercept="~q") assert r.filt diff --git a/test/mitmproxy/addons/test_proxyauth.py b/test/mitmproxy/addons/test_proxyauth.py index 97259d1c..9e2365cf 100644 --- a/test/mitmproxy/addons/test_proxyauth.py +++ b/test/mitmproxy/addons/test_proxyauth.py @@ -49,7 +49,7 @@ class TestProxyAuth: ]) def test_is_proxy_auth(self, mode, expected): up = proxyauth.ProxyAuth() - with taddons.context() as ctx: + with taddons.context(up) as ctx: ctx.options.mode = mode assert up.is_proxy_auth() is expected @@ -75,7 +75,7 @@ class TestProxyAuth: def test_check(self): up = proxyauth.ProxyAuth() - with taddons.context() as ctx: + with taddons.context(up) as ctx: ctx.configure(up, proxyauth="any", mode="regular") f = tflow.tflow() assert not up.check(f) @@ -133,7 +133,7 @@ class TestProxyAuth: def test_authenticate(self): up = proxyauth.ProxyAuth() - with taddons.context() as ctx: + with taddons.context(up) as ctx: ctx.configure(up, proxyauth="any", mode="regular") f = tflow.tflow() @@ -165,7 +165,7 @@ class TestProxyAuth: def test_configure(self): up = proxyauth.ProxyAuth() - with taddons.context() as ctx: + with taddons.context(up) as ctx: with pytest.raises(exceptions.OptionsError): ctx.configure(up, proxyauth="foo") @@ -223,7 +223,7 @@ class TestProxyAuth: def test_handlers(self): up = proxyauth.ProxyAuth() - with taddons.context() as ctx: + with taddons.context(up) as ctx: ctx.configure(up, proxyauth="any", mode="regular") f = tflow.tflow() diff --git a/test/mitmproxy/net/test_tcp.py b/test/mitmproxy/net/test_tcp.py index 8c012e42..e862d0ad 100644 --- a/test/mitmproxy/net/test_tcp.py +++ b/test/mitmproxy/net/test_tcp.py @@ -485,7 +485,7 @@ class TestSSLDisconnect(tservers.ServerTestBase): c = tcp.TCPClient(("127.0.0.1", self.port)) with c.connect(): c.convert_to_tls() - # Excercise SSL.ZeroReturnError + # Exercise SSL.ZeroReturnError c.rfile.read(10) c.close() with pytest.raises(exceptions.TcpDisconnect): diff --git a/test/mitmproxy/tools/console/test_master.py b/test/mitmproxy/tools/console/test_master.py index 6ea61991..5be035e8 100644 --- a/test/mitmproxy/tools/console/test_master.py +++ b/test/mitmproxy/tools/console/test_master.py @@ -1,8 +1,6 @@ import urwid from mitmproxy import options -from mitmproxy.test import tflow -from mitmproxy.test import tutils from mitmproxy.tools import console from ... import tservers @@ -24,16 +22,3 @@ class TestMaster(tservers.MasterTest): except urwid.ExitMainLoop: pass assert len(m.view) == i - - def test_intercept(self): - """regression test for https://github.com/mitmproxy/mitmproxy/issues/1605""" - m = self.mkmaster(intercept="~b bar") - f = tflow.tflow(req=tutils.treq(content=b"foo")) - m.addons.handle_lifecycle("request", f) - assert not m.view[0].intercepted - f = tflow.tflow(req=tutils.treq(content=b"bar")) - m.addons.handle_lifecycle("request", f) - assert m.view[1].intercepted - f = tflow.tflow(resp=tutils.tresp(content=b"bar")) - m.addons.handle_lifecycle("request", f) - assert m.view[2].intercepted diff --git a/test/mitmproxy/tools/console/test_statusbar.py b/test/mitmproxy/tools/console/test_statusbar.py index ac17c5c0..db8a63a7 100644 --- a/test/mitmproxy/tools/console/test_statusbar.py +++ b/test/mitmproxy/tools/console/test_statusbar.py @@ -14,7 +14,7 @@ def test_statusbar(monkeypatch): view_filter="~dst example.com", stickycookie="~dst example.com", stickyauth="~dst example.com", - default_contentview="javascript", + console_default_contentview="javascript", anticache=True, anticomp=True, showhost=True, diff --git a/test/mitmproxy/tools/test_dump.py b/test/mitmproxy/tools/test_dump.py index 952c3f4f..f303c808 100644 --- a/test/mitmproxy/tools/test_dump.py +++ b/test/mitmproxy/tools/test_dump.py @@ -11,7 +11,7 @@ from .. import tservers class TestDumpMaster(tservers.MasterTest): def mkmaster(self, flt, **opts): - o = options.Options(view_filter=flt, verbosity='error', flow_detail=0, **opts) + o = options.Options(view_filter=flt, verbosity='error', **opts) m = dump.DumpMaster(o, with_termlog=False, with_dumper=False) return m |
