v2.3.0 (#470)

* Update mypy from 0.780 to 0.781 (#379) * Add FilterByClientIpPlugin example (#381) * Update mypy from 0.781 to 0.782 (#382) * Update twine from 3.1.1 to 3.2.0 (#384) * Update tox from 3.15.2 to 3.16.0 (#385) * Update tox from 3.16.0 to 3.16.1 (#386) * Document FilterByClientIpPlugin & ModifyChunkResponsePlugin (#387) * Refactor plugin base classes for plugin specific flags (#388) * Update to latest code signing recommendations * Move HttpProtocolHandlerPlugin into separate file * Dont add subject attributes if not provided by upstream. Also handle subprocess.TimeoutExpired raised during certificate generation. Instead of retries, we simply close the connection on timeout * Remove plugin specific flag initialization methods for now * Update coverage from 5.1 to 5.2 (#390) * Core acceptor pool doc, cleanup and standalone example (#393) * Better document acceptor module and add a TCP Echo Server example * autopep8 formating * Rename ThreadlessWork --> Work class * Make initialize, is_inactive and shutdown as optional interface methods. Also introduce Readables & Writables custom types. * Move websocket code into its own module * Add websocket client example * Cleanup websocket client * Decouple SSL wrap logic into connection classes (#394) * Move wrap functionality within respective connection classes. Also decouple websocket client handshake method * Add a TCP echo client example that works with TCP echo server example * Add SSL echo server & client example (#395) * Move wrap_socket for SSL server within utils. Also complete proxy.common.pki gen_csr and sign_csr actions. Used by Makefile sign-https-certificates. * Add SSL echo server and client example * Add examples documentation * Add core pubsub eventing example and add menubar item skeleton (#396) * Initialize menu bar items with click handler and open a popover for preferences * Add Core PubSub eventing example * Remove hardcoded request ids * Move codecov.yml to top level directory (#400) * Add cross ref for how to generate SSL certs. (#401) * Add plugin "FilterByURLRegexPlugin" (#397) * Initial draft of filter_by_url_regex.py * Add FilterByURLRegexPlugin * Fix dictionary key & add logging * Add proper logging * Add better logging * Add logging * move code to handle_client_request * development logging * development * development * development * dev * dev * dev * dev * dev * dev * dev * dev * dev * dev * dev * Fix blocked log * Add to FILTER_LIST, some tidy up * Update FILTER_LIST * dev * remove scheme from url * Add to FILTER_LIST * Add to FILTER_LIST * Update FILTER_LIST * commenting * Update FILTER_LIST * After autopep8 * Fix Anomalous backslash in string (pep8) * Address code quality checks - flake8 F401 & W605 * Address flake8 errors * Attempt to fix flake8 errors * Fix linting issues * Address flake8 W292 * Attempt to create tests * Add FilterByURLRegexPlugin * Rename test * Work on tests * Work on tests * Work on tests Co-authored-by: Abhinav Singh <[email protected]> * Update tox from 3.16.1 to 3.17.0 (#402) * Update codecov from 2.1.7 to 2.1.8 (#404) * Update tox from 3.17.0 to 3.17.1 (#403) Co-authored-by: Abhinav Singh <[email protected]> * Bump lodash from 4.17.15 to 4.17.19 in /dashboard (#405) Bumps [lodash](https://github.com/lodash/lodash) from 4.17.15 to 4.17.19. - [Release notes](https://github.com/lodash/lodash/releases) - [Commits](lodash/lodash@4.17.15...4.17.19) Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Update tox from 3.17.1 to 3.18.0 (#406) * Update coverage from 5.2 to 5.2.1 (#407) * Update tox from 3.18.0 to 3.18.1 (#408) * Fix docker build by using correct pip flags (#417) * Update tox from 3.18.1 to 3.19.0 (#416) Co-authored-by: Abhinav Singh <[email protected]> * Update autopep8 from 1.5.3 to 1.5.4 (#412) Co-authored-by: Abhinav Singh <[email protected]> * Update pytest from 5.4.3 to 6.0.1 (#410) Co-authored-by: Abhinav Singh <[email protected]> * npm upgrade (#418) * Remove test for 'HttpWebServerRouteHandler' This does not exist (as fas as I can see) and it bother the linter (Mypy), when I tell it `klass` is a `type` instance. * Pass a list plugin class objects or bytes to proxy when used in embeded mode. No automated tests for the feature yet. * Tests for Flags.load_plugins method. * Ensure plugins are loaded only once. Also changed module name for plugins passed by type. * Update wheel from 0.34.2 to 0.35.0 (#421) * Allow to use types when embeding Proxy (#420) * Remove test for 'HttpWebServerRouteHandler' This does not exist (as fas as I can see) and it bother the linter (Mypy), when I tell it `klass` is a `type` instance. * Pass a list plugin class objects or bytes to proxy when used in embeded mode. No automated tests for the feature yet. * Tests for Flags.load_plugins method. * Ensure plugins are loaded only once. Also changed module name for plugins passed by type. Co-authored-by: Abhinav Singh <[email protected]> * Documentation for plugin loading in embedded mode (#422) * Update pytest-cov from 2.10.0 to 2.10.1 (#423) * Update wheel from 0.35.0 to 0.35.1 (#424) * Update typing-extensions from 3.7.4.2 to 3.7.4.3 (#428) * Update codecov from 2.1.8 to 2.1.9 (#427) Co-authored-by: Abhinav Singh <[email protected]> * Update pylint from 2.5.3 to 2.6.0 (#426) Co-authored-by: Abhinav Singh <[email protected]> * Update paramiko from 2.7.1 to 2.7.2 (#429) * Update pytest from 6.0.1 to 6.1.0 (#436) * Update coverage from 5.2.1 to 5.3 (#433) Co-authored-by: Abhinav Singh <[email protected]> * Update tox from 3.19.0 to 3.20.0 (#430) Co-authored-by: Abhinav Singh <[email protected]> * Update flake8 from 3.8.3 to 3.8.4 (#439) * Allow plugins to add custom command line flags (#438) * Allow plugins to add custom command line flags. Addresses #301 * Reduce dependency over Flags class. This will be deprecated so that adhoc flags can be added without any additional manual configuration * Fix: Argument 1 to "mock_default_args" of "TestMain" has incompatible type "Namespace"; expected "Mock" * Reduce Flags class to just the initializer. * Store list of action dest in FlagParser * Update pytest from 6.1.0 to 6.1.1 (#440) * More examples (#444) * Refactor into BaseServerHandler and BaseEchoServerHandler classes * Add connect tunnel example * Update rope from 0.17.0 to 0.18.0 (#445) * Update tox from 3.20.0 to 3.20.1 (#446) * Update codecov from 2.1.9 to 2.1.10 (#447) * Update mypy (#449) * Fix path to devtools websocket endpoint, broken after refactoring (#450) * Relax proxy auth requirement to allow mixed case for the auth type e.g. "basic", "Basic", "BaSiC" are all allowed (#451) * Go flagless to allow custom user defined flags. (#452) * Go flagless to allow custom user defined flags. Fixes #301 * Add --cache-dir flag for cache plugin (when used with on-disk store) * Enable discovery of flags from external plugins, example those that reside outside of proxy.py package and loaded on demand. This also allows external flags to surface in --help section * Define --filtered-client-ips flag for FilterByClientIpPlugin * Separate basic auth plugin outside of core server (#453) * Separate basic auth plugin outside of core * Put basic auth plugin at top * Create codeql-analysis.yml (#454) * Create SECURITY.md (#455) * Refactor (#456) * Update pytest from 6.1.1 to 6.1.2 (#457) * npm update (#460) * Refactor base server interfaces into core modules (#461) * Ensure pending buffers are flushed before shutting down in base_server.py Handle unsupported scheme cases within connect_tunnel.py * Move base implementations within core module * Update ssl_echo_server * Update wheel from 0.35.1 to 0.36.0 (#462) * Update wheel from 0.36.0 to 0.36.1 (#463) * Update pytest from 6.1.2 to 6.2.0 (#465) * Update wheel from 0.36.1 to 0.36.2 (#466) * Update pytest from 6.2.0 to 6.2.1 (#467) * Update codecov from 2.1.10 to 2.1.11 (#469) * Add version check for README.md (#471) Co-authored-by: pyup.io bot <[email protected]> Co-authored-by: Mike <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Pascal COMBES <[email protected]>
abhinavsingh · Dec 20, 2020 · 5b5c77e · 5b5c77e
1 parent 7ccdc2a
commit 5b5c77e
Show file tree

Hide file tree

Showing 88 changed files with 3,768 additions and 1,661 deletions.
diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml
@@ -0,0 +1,71 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+name: "CodeQL"
+
+on:
+  push:
+    branches: [develop, master]
+  pull_request:
+    # The branches below must be a subset of the branches above
+    branches: [develop]
+  schedule:
+    - cron: '0 14 * * 1'
+
+jobs:
+  analyze:
+    name: Analyze
+    runs-on: ubuntu-latest
+
+    strategy:
+      fail-fast: false
+      matrix:
+        # Override automatic language detection by changing the below list
+        # Supported options are ['csharp', 'cpp', 'go', 'java', 'javascript', 'python']
+        language: ['python', 'javascript']
+        # Learn more...
+        # https://docs.github.com/en/github/finding-security-vulnerabilities-and-errors-in-your-code/configuring-code-scanning#overriding-automatic-language-detection
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v2
+      with:
+        # We must fetch at least the immediate parents so that if this is
+        # a pull request then we can checkout the head.
+        fetch-depth: 2
+
+    # If this run was triggered by a pull request event, then checkout
+    # the head of the pull request instead of the merge commit.
+    - run: git checkout HEAD^2
+      if: ${{ github.event_name == 'pull_request' }}
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v1
+      with:
+        languages: ${{ matrix.language }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file. 
+        # Prefix the list here with "+" to use these queries and those in the config file.
+        # queries: ./path/to/local/query, your-org/your-repo/queries@main
+
+    # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
+    # If this step fails, then you should remove it and run the build manually (see below)
+    # - name: Autobuild
+    #   uses: github/codeql-action/autobuild@v1
+
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 https://git.io/JvXDl
+
+    # ✏️ If the Autobuild fails above, remove it and uncomment the following three lines
+    #    and modify them (or add more) to build your code if your project
+    #    uses a compiled language
+
+    #- run: |
+    #   make bootstrap
+    #   make release
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v1
diff --git a/Dockerfile b/Dockerfile
@@ -7,7 +7,7 @@ COPY README.md /app/
 COPY proxy/ /app/proxy/
 WORKDIR /app
 RUN pip install --upgrade pip && \
-    pip install --install-option="--prefix=/deps" .
+    pip install --prefix=/deps .
 
 FROM base
 

diff --git a/Makefile b/Makefile
@@ -8,6 +8,8 @@ IMAGE_TAG := $(NS)/$(IMAGE_NAME):$(VERSION)
 
 HTTPS_KEY_FILE_PATH := https-key.pem
 HTTPS_CERT_FILE_PATH := https-cert.pem
+HTTPS_CSR_FILE_PATH := https-csr.pem
+HTTPS_SIGNED_CERT_FILE_PATH := https-signed-cert.pem
 
 CA_KEY_FILE_PATH := ca-key.pem
 CA_CERT_FILE_PATH := ca-cert.pem
@@ -25,6 +27,7 @@ devtools:
 	pushd dashboard && npm run devtools && popd
 
 autopep8:
+	autopep8 --recursive --in-place --aggressive examples
 	autopep8 --recursive --in-place --aggressive proxy
 	autopep8 --recursive --in-place --aggressive tests
 	autopep8 --recursive --in-place --aggressive setup.py
@@ -40,6 +43,20 @@ https-certificates:
 		--private-key-path $(HTTPS_KEY_FILE_PATH) \
 		--public-key-path $(HTTPS_CERT_FILE_PATH)
 
+sign-https-certificates:
+	# Generate CSR request
+	python -m proxy.common.pki gen_csr \
+		--csr-path $(HTTPS_CSR_FILE_PATH) \
+		--private-key-path $(HTTPS_KEY_FILE_PATH) \
+		--public-key-path $(HTTPS_CERT_FILE_PATH)
+	# Sign CSR with CA
+	python -m proxy.common.pki sign_csr \
+		--csr-path $(HTTPS_CSR_FILE_PATH) \
+		--crt-path $(HTTPS_SIGNED_CERT_FILE_PATH) \
+		--hostname example.com \
+		--private-key-path $(CA_KEY_FILE_PATH) \
+		--public-key-path $(CA_CERT_FILE_PATH)
+
 ca-certificates:
 	# Generate CA key
 	python -m proxy.common.pki gen_private_key \
@@ -73,8 +90,8 @@ lib-clean:
 	rm -rf .hypothesis
 
 lib-lint:
-	flake8 --ignore=W504 --max-line-length=127 --max-complexity=19 proxy/ tests/ setup.py
-	mypy --strict --ignore-missing-imports proxy/ tests/ setup.py
+	flake8 --ignore=W504 --max-line-length=127 --max-complexity=19 examples/ proxy/ tests/ setup.py
+	mypy --strict --ignore-missing-imports examples/ proxy/ tests/ setup.py
 
 lib-test: lib-clean lib-version lib-lint
 	pytest -v tests/
@@ -93,7 +110,7 @@ lib-coverage:
 	open htmlcov/index.html
 
 lib-profile:
-	sudo py-spy -F -f profile.svg -d 3600 proxy.py
+	sudo py-spy record -o profile.svg -t -F -s -- python -m proxy
 
 dashboard:
 	pushd dashboard && npm run build && popd

diff --git a/README.md b/README.md
@@ -57,6 +57,8 @@ Table of Contents
         * [Cache Responses Plugin](#cacheresponsesplugin)
         * [Man-In-The-Middle Plugin](#maninthemiddleplugin)
         * [Proxy Pool Plugin](#proxypoolplugin)
+        * [FilterByClientIpPlugin](#filterbyclientipplugin)
+        * [ModifyChunkResponsePlugin](#modifychunkresponseplugin)
     * [HTTP Web Server Plugins](#http-web-server-plugins)
         * [Reverse Proxy](#reverse-proxy)
         * [Web Server Route](#web-server-route)
@@ -70,6 +72,7 @@ Table of Contents
 * [Embed proxy.py](#embed-proxypy)
     * [Blocking Mode](#blocking-mode)
     * [Non-blocking Mode](#non-blocking-mode)
+    * [Loading Plugins](#loading-plugins)
 * [Unit testing with proxy.py](#unit-testing-with-proxypy)
     * [proxy.TestCase](#proxytestcase)
     * [Override Startup Flags](#override-startup-flags)
@@ -669,6 +672,57 @@ Make a curl request via `8899` proxy:
 Verify that `8899` proxy forwards requests to upstream proxies
 by checking respective logs.
 
+### FilterByClientIpPlugin
+
+Reject traffic from specific IP addresses.  By default this
+plugin blocks traffic from `127.0.0.1` and `::1`.
+
+Start `proxy.py` as:
+
+```bash
+❯ proxy \
+    --plugins proxy.plugin.FilterByClientIpPlugin
+```
+
+Send a request using `curl -v -x localhost:8899 http://google.com`:
+
+```bash
+... [redacted] ...
+> Proxy-Connection: Keep-Alive
+>
+< HTTP/1.1 418 I'm a tea pot
+< Connection: close
+<
+* Closing connection 0
+```
+
+Modify plugin to your taste e.g. Allow specific IP addresses only.
+
+### ModifyChunkResponsePlugin
+
+This plugin demonstrate how to modify chunked encoded responses.  In able to do so, this plugin uses `proxy.py` core to parse the chunked encoded response.  Then we reconstruct the response using custom hardcoded chunks, ignoring original chunks received from upstream server.
+
+Start `proxy.py` as:
+
+```bash
+❯ proxy \
+    --plugins proxy.plugin.ModifyChunkResponsePlugin
+```
+
+Verify using `curl -v -x localhost:8899 http://httpbin.org/stream/5`:
+
+```bash
+... [redacted] ...
+modify
+chunk
+response
+plugin
+* Connection #0 to host localhost left intact
+* Closing connection 0
+```
+
+Modify `ModifyChunkResponsePlugin` to your taste. Example, instead of sending hardcoded chunks, parse and modify the original `JSON` chunks received from the upstream server.
+
 ## HTTP Web Server Plugins
 
 ### Reverse Proxy
@@ -774,6 +828,22 @@ Verify using `curl -x https://localhost:8899 --proxy-cacert https-cert.pem https
 }
 ```
 
+If you want to avoid passing `--proxy-cacert` flag, also consider signing generated SSL certificates.  Example:
+
+First, generate CA certificates:
+
+```bash
+make ca-certificates
+```
+
+Then, sign SSL certificate:
+
+```bash
+make sign-https-certificates
+```
+
+Now restart the server with `--cert-file https-signed-cert.pem` flag.  Note that you must also trust generated `ca-cert.pem` in your system keychain.
+
 TLS Interception
 =================
 
@@ -1103,6 +1173,36 @@ Note that:
    input arguments e.g. `start(['--port', '8899'])` or
    by using passing flags as kwargs e.g. `start(port=8899)`.
 
+## Loading Plugins
+
+You can, of course, list plugins to load in the input arguments list of `proxy.main`, `proxy.start` or the `Proxy` constructor. Use the `--plugins` flag as when starting from command line:
+
+```python
+import proxy
+
+if __name__ == '__main__':
+  proxy.main([
+    '--plugins', 'proxy.plugin.CacheResponsesPlugin',
+  ])
+```
+
+However, for simplicity you can pass the list of plugins to load as a keyword argument to `proxy.main`, `proxy.start` or the `Proxy` constructor:
+
+```python
+import proxy
+from proxy.plugin import FilterByUpstreamHostPlugin
+
+if __name__ == '__main__':
+  proxy.main([], plugins=[
+    b'proxy.plugin.CacheResponsesPlugin',
+    FilterByUpstreamHostPlugin,
+  ])
+```
+
+Note that it supports:
+1. The fully-qualified name of a class as `bytes`
+2. Any `type` instance for a Proxy.py plugin class. This is espacially useful for custom plugins defined locally.
+
 Unit testing with proxy.py
 ==========================
 
@@ -1603,7 +1703,7 @@ usage: proxy [-h] [--backlog BACKLOG] [--basic-auth BASIC_AUTH]
              [--static-server-dir STATIC_SERVER_DIR] [--threadless]
              [--timeout TIMEOUT] [--version]
 
-proxy.py v2.2.0
+proxy.py v2.3.0
 
 optional arguments:
   -h, --help            show this help message and exit

diff --git a/SECURITY.md b/SECURITY.md
@@ -0,0 +1,16 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 2.x     | :white_check_mark: |
+| < 2.x   | :x:                |
+
+## Reporting a Vulnerability
+
+Follow these steps:
+
+1. Start by [emailing developers](mailto:[email protected])
+2. If unresponsive, [create a public issue](https://github.com/abhinavsingh/proxy.py/issues/new/choose)
+3. [Pull requests](https://github.com/abhinavsingh/proxy.py/pulls) are always welcome
diff --git a/tests/codecov.yml → codecov.yml b/tests/codecov.yml → codecov.yml