GraphSense REST Interface

The GraphSense REST Interface provides access to denormalized views computed by the graphsense-transformation pipeline. It is used by the graphsense-dashboard component.

It is based on a server stub generated by the OpenAPI Generator project against (local) the OpenAPI specification. It uses the Connexion library on top of aiohttp.

Setup

Copy the configuration file template instance/config.yaml.template

cp instance/config.yaml.template instance/config.yaml

Open instance/config.yaml and configure the database connection.

Requirements

Make sure you are running Python > 3.10. We are using uv for packaging and dependency management. Please install uv.

Run

make install-dev

to setup the project for development.

uv sync --frozen

only installs the dependencies nessesary for running the project.

You need access to GraphSense raw and transformed keyspaces. See Graphsense Transformation Pipeline for further details.

Run the REST interface

make serve

Test the service in your browser:

http://localhost:9000

On OS X you need to install GNU sed and link it on your system as sed.

Deployment

In order to utilize multiple cores, you may run the REST interface through Gunicorn.

Install gunicorn:

uv pip install gunicorn

If unspecified gunicorn is run with a number of workers and threads both equal to the number of CPUs x 2.

Run production server, overriding number of workers and threads through command line options (4 workers, 4 threads). Specify the aiohttp specific WebWorker class to leverage the full power of lightweight threads:

uv run gunicorn -w 4 --threads 4 "gsrest:main()" --worker-class aiohttp.GunicornWebWorker

Deployment with docker

To run the server on a Docker container, please execute the following from the root directory:

Build the image

make build-docker

You may specifiy the number of workers and threads through the environments variables NUM_WORKERS and NUM_THREADS respectively.

Start up a container with 1 workers and 1 threads:

make serve-docker

As an alternative you can use our prebuild docker images.

docker run -e NUM_WORKERS=4 -e NUM_THREADS=4 -p 9000:9000 ghcr.io/graphsense/graphsense-rest:latest

Customize REST interface configuration

Flask configuration parameters can be added at the top level of instance/config.yaml. A configuration template file is provided in instance/config.yaml.template.

You may define ALLOWED_ORIGINS in instance/config.yaml if the REST interface is to be consumed by a Web browser, e.g. by graphsense-dashboard.

Logging

REST log messages can be delivered via mail. See instance/config.yaml.template for an example.

Development

Generate server stub

Install version 4.44.2 or higher of yq (https://github.com/mikefarah/yq/) and run the following command to generate the server stub:

make generate-openapi-server

The service implementation (gsrest/service) should use the generated models located in openapi_server/models.

Source code generation is based on templates. See the openapi_spec/templates directory for currently used templates. If you need more templates, retrieve them from the generator like so:

docker run --rm -v "/tmp/templates:/templates" openapitools/openapi-generator-cli author template -g python-aiohttp -o /templates

Templates are written to /tmp/templates. Copy the needed ones to this project's templates directory.

Testing

Service tests are located in gsrest/test. These are called from the generated controller tests located in tests/test_*.

To launch the integration tests run:

make test

This launches a mockup Cassandra database instance and ingests test data from tests/cassandra/data and tests/tagstore/data via the corresponding insert.py scripts in each folder.

Setup for the cassandra and postgres databases and the general configuration for the rest interface for testing is defined in conftest.py as a pytest fixture.