diff --git a/RELEASE.md b/RELEASE.md index beb59747f..212759cf5 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -6,7 +6,8 @@ 1. Add new authors to the authors file if needed, by running `./tools/authors.sh` and committing the changes. 1. Update the version in `Cargo.toml`. - 1. Update the version name in `src/main.rs`. + 1. Update the version name in `retis/src/main.rs`. + 1. Update the version name in `retis-events/pyproject.toml`. 1. Run `cargo publish --dry-run` to check for any issue. 1. Open a PR and get it merged. This must have runtime tests enabled! 1. Tag the right commit in the `vx.y.z` form and push it. @@ -22,6 +23,8 @@ 1. `$ buildah push quay.io/retis/retis:x.y.z` 1. Manually tag on the web UI the image pushed with `latest`, if applicable. + 1. Build and upload the python bindings. + 1. `podman run --rm --env MATURIN_PYPI_TOKEN=$(cat ~/.my_pypi_token) -v $(pwd):/io:z ghcr.io/pyo3/maturin publish -m retis-events/Cargo.toml -F python-lib` 1. Release on [crates.io](https://crates.io): `cargo publish`. 1. Write and publish a release notes in the GitHub interface. This must be done once the rpm and the image successfully built to allow pushing last minute diff --git a/docs/CONTRIBUTING.md b/docs/CONTRIBUTING.md index 94d6b9764..d2b76a41d 100644 --- a/docs/CONTRIBUTING.md +++ b/docs/CONTRIBUTING.md @@ -122,6 +122,20 @@ flavor coding style for the BPF parts. latest `main`. This can happen at any time, e.g. when other pull-requests are being merged. +### Python bindings development + +When changing python bindings, it can be useful to quickly build and install the +current code into a virtual environment for testing. An easy way to do it is by +using [maturin](https://github.com/PyO3/maturin): + +``` +$ python -m venv venv && source venv/bin/activate # jump into a virtual env (required) +$ pip install maturin +$ maturin develop -m retis-events/Cargo.toml -F python-lib +$ python +>>> import retis +``` + ### Documentation preview HTTP documentation is automatically generated for releases and the diff --git a/docs/index.md b/docs/index.md index d21b0b027..dbd7a93a7 100644 --- a/docs/index.md +++ b/docs/index.md @@ -121,6 +121,10 @@ $ PAGER=more retis sort $ NOPAGER=1 retis sort ``` +In addition to built-in post-processing commands, it is possible to use the +python bindings to implement custom processing. See the +[python bindings documentation](python.md). + ## Profiles and customization Retis has the concept of profiles, which are a predefined set of cli arguments diff --git a/docs/python.md b/docs/python.md new file mode 100644 index 000000000..7e863bf32 --- /dev/null +++ b/docs/python.md @@ -0,0 +1,56 @@ +# Python bindings + +Besides the basic post-processing commands provided by `retis` (e.g: `sort`), +python bindings exist to enable writing custom post-processing scripts. + +These bindings can be used in two different ways: the built-in python shell and +the external python library. + +## Overview + +Python bindings currently provide 3 basic python classes that allow inspecting +retis events: + +- **Event**: Python representation of a retis event. It provides helpers to +access the event's sections and data within those sections. +- **EventSeries**: Python representation of a series of sorted events resulting +from the execution of `retis sort -o`. It implements the iterator protocol to +access the events. +- **EventReader**: Class capable of reading a file created by retis and iterate +over the events or series it contains. It implements the iterator protocol to +access the events. + +More details can be found in the `retis_events` crate documentation. + +## Builtin shell + +The builtin shell, although basic, it enables quick event inspection. +Once you drop into the shell, a global variable called `events` is available. +It is of type `EventReader`. + +``` python +$ retis python +>>> for event in events: +>>> if "skb" in event and "tcp" in event["skb"].raw(): +>>> print("TCP event with dport: {}".format( +>>> event["skb"].raw()["tcp"]["dport"]) +``` + +## Python library + +For more sophisticated programs, a python library is available in +[pypi](https://pypi.org/retis). Unlike the builtin command, in this case the +`EventReader` has to be created manually. + +```python +from retis import EventReader + +import statistics + +reader = EventReader("sorted_events.json") + +events_per_series = [len(s) for s in reader] + +print("Number of series: {}".format(len(events_per_series))) +print("Average events per series: {}".format(statistics.mean(events_per_series))) +``` diff --git a/mkdocs.yml b/mkdocs.yml index 292ee6f96..97063e1c1 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,6 +14,7 @@ nav: - Requirements: requirements.md - Contributing: CONTRIBUTING.md - External resources: resources.md + - Python bindings: python.md - Collectors: - skb: modules/skb.md - skb-drop: modules/skb_drop.md