Skip to content
This repository has been archived by the owner on Jun 1, 2022. It is now read-only.

Commit

Permalink
Initial attempt at a data model and API for capturing availability.
Browse files Browse the repository at this point in the history
- Models for representing availability reports and availability windows.
- REST API endpoint for writing those availabiltiy reports.
- Basic test that the API endpoint works.
  • Loading branch information
nschiefer committed Mar 5, 2021
1 parent 7aefb8c commit 78a4071
Show file tree
Hide file tree
Showing 12 changed files with 538 additions and 115 deletions.
59 changes: 59 additions & 0 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,3 +83,62 @@ A tool for trying out this API is available at https://vaccinateca-preview.herok
Anything submitted using that tool will have `is_test_data` set to True in the database.

You can view test reports here: https://vaccinateca-preview.herokuapp.com/admin/core/report/?is_test_data__exact=1

## api/submitAvailabilityReport

This endpoint records an availability report for a location, which generally includes a list of known appointment windows.
It will usually be used by an automated script.

This API endpoint is called with an HTTP post, with JSON in teh POST body.
The `SCRAPER_API_KEY` should be sent in the header as `Authorization: Bearer <SCRAPER_API_KEY>`.

The JSON document must have the following keys:
- `feed_update`: an object with three keys: `uuid`, `github_url`, and `feed_provider`.
This should be the same for all reports submitted in a single session (e.g., as a result of a single feed update or scrape).
The `uuid` should be generated by the client (e.g., using Python's `uuid.uuid4()` method).
The `github_url` is a URL to our repository on GitHub where the raw data can be inspected.
The `feed_provider` is a slug that uniquely refers to this particular data source (e.g., `curative` for Curative's JSON feed).
Provider slugs are available in the `Feed provider` table.
- `location`. This is a unique identifier to the location _used by this feed provider_.
It is not the `public_id` of the location.
A pre-submitted concordance is used to map this to a location in our database.
- `availability_windows` is a list of objects, each of which has the fields `starts_at`, `ends_at`, `slots`, and `additional_restrictions`.
The `starts_at` and `ends_at` fields are timestamps that indicate the bounds of this window.
The `slots` field indicates the number of currently available slots in this window.
The `additional_restrictions` field is a list of availability tags, using their slugs (see above).

Optionally, the JSON document may include a `feed_json` key, with a value consisting of the raw JSON (e.g., from a provider's API)
used to inform the availability report _for this location_.

For example:
```json
{
"feed_update": {
"uuid": "02d63a35-5dbc-4ac8-affb-14603bf6eb2e",
"github_url": "https://example.com",
"feed_provider": "test_provider"
},
"location": "116",
"availability_windows": [
{
"starts_at": "2021-02-28T10:00:00Z",
"ends_at": "2021-02-28T11:00:00Z",
"slots": 25,
"additional_restrictions": []
},
{
"starts_at": "2021-02-28T11:00:00Z",
"ends_at": "2021-02-28T12:00:00Z",
"slots": 18,
"additional_restrictions": [
"vaccinating_65_plus"
]
}
]
}
```

If the request was successful, it returns a 201 Created HTTP response.

A common reason for an unsuccessful request is the lack of concordance between the provider's location ID and our known
locations. In general, an automated process should find and flag new locations in each scrape before publshing availability.
102 changes: 17 additions & 85 deletions vaccinate/api/migrations/0001_initial.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Generated by Django 3.1.7 on 2021-03-03 20:19
# Generated by Django 3.1.7 on 2021-03-04 05:34

import core.fields
from django.db import migrations, models
Expand All @@ -11,97 +11,29 @@ class Migration(migrations.Migration):
initial = True

dependencies = [
("core", "0031_auto_20210303_2019"),
('core', '0031_auto_20210304_0534'),
]

operations = [
migrations.CreateModel(
name="ApiLog",
name='ApiLog',
fields=[
(
"id",
models.AutoField(
auto_created=True,
primary_key=True,
serialize=False,
verbose_name="ID",
),
),
(
"created_at",
models.DateTimeField(
db_index=True, default=django.utils.timezone.now
),
),
(
"method",
core.fields.CharTextField(
help_text="The HTTP method", max_length=65000
),
),
(
"path",
core.fields.CharTextField(
help_text="The path, starting with /", max_length=65000
),
),
(
"query_string",
core.fields.CharTextField(
blank=True, help_text="The bit after the ?", max_length=65000
),
),
("remote_ip", core.fields.CharTextField(max_length=65000)),
(
"post_body",
models.BinaryField(
blank=True,
help_text="If the post body was not valid JSON, log it here as text",
null=True,
),
),
(
"post_body_json",
models.JSONField(
blank=True,
help_text="Post body if it was valid JSON",
null=True,
),
),
(
"response_status",
models.IntegerField(
help_text="HTTP status code returned by the API"
),
),
(
"response_body",
models.BinaryField(
blank=True,
help_text="If the response body was not valid JSON",
null=True,
),
),
(
"response_body_json",
models.JSONField(
blank=True, help_text="Response body if it was JSON", null=True
),
),
(
"created_report",
models.ForeignKey(
blank=True,
help_text="Report that was created by this API call, if any",
null=True,
on_delete=django.db.models.deletion.SET_NULL,
related_name="created_by_api_logs",
to="core.report",
),
),
('id', models.AutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
('created_at', models.DateTimeField(db_index=True, default=django.utils.timezone.now)),
('method', core.fields.CharTextField(help_text='The HTTP method', max_length=65000)),
('path', core.fields.CharTextField(help_text='The path, starting with /', max_length=65000)),
('query_string', core.fields.CharTextField(blank=True, help_text='The bit after the ?', max_length=65000)),
('remote_ip', core.fields.CharTextField(max_length=65000)),
('post_body', models.BinaryField(blank=True, help_text='If the post body was not valid JSON, log it here as text', null=True)),
('post_body_json', models.JSONField(blank=True, help_text='Post body if it was valid JSON', null=True)),
('response_status', models.IntegerField(help_text='HTTP status code returned by the API')),
('response_body', models.BinaryField(blank=True, help_text='If the response body was not valid JSON', null=True)),
('response_body_json', models.JSONField(blank=True, help_text='Response body if it was JSON', null=True)),
('created_availability_report', models.ForeignKey(blank=True, help_text='Availability report that was created by this API call, if any', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='created_by_api_logs', to='core.appointmentavailabilityreport')),
('created_report', models.ForeignKey(blank=True, help_text='Report that was created by this API call, if any', null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='created_by_api_logs', to='core.report')),
],
options={
"db_table": "api_log",
'db_table': 'api_log',
},
),
]
8 changes: 8 additions & 0 deletions vaccinate/api/models.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,14 @@ class ApiLog(models.Model):
on_delete=models.SET_NULL,
help_text="Report that was created by this API call, if any",
)
created_availability_report = models.ForeignKey(
"core.AppointmentAvailabilityReport",
null=True,
blank=True,
related_name="created_by_api_logs",
on_delete=models.SET_NULL,
help_text="Availability report that was created by this API call, if any"
)

class Meta:
db_table = "api_log"
Expand Down
45 changes: 45 additions & 0 deletions vaccinate/api/test-data/submitAvailabilityReport/001.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
{
"location": "recaQlVkkI1rNarvx",
"input": {
"feed_update": {
"uuid": "02d63a35-5dbc-4ac8-affb-14603bf6eb2e",
"github_url": "https://example.com",
"feed_provider": "test_provider"
},
"location": "116",
"availability_windows": [
{
"starts_at": "2021-02-28T10:00:00Z",
"ends_at": "2021-02-28T11:00:00Z",
"slots": 25,
"additional_restrictions": []
},
{
"starts_at": "2021-02-28T11:00:00Z",
"ends_at": "2021-02-28T12:00:00Z",
"slots": 18,
"additional_restrictions": ["vaccinating_65_plus"]
}
]
},
"expected_status": 201,
"expected_fields": {
"location__public_id": "recaQlVkkI1rNarvx"
},
"expected_windows": [
{
"expected_fields":
{
"slots": 25
},
"expected_additional_restrictions": []
},
{
"expected_fields":
{
"slots": 18
},
"expected_additional_restrictions": ["vaccinating_65_plus"]
}
]
}
90 changes: 90 additions & 0 deletions vaccinate/api/test_availability_report.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
from config.settings import SCRAPER_API_KEY
from core.models import Report, Location, FeedProvider, AppointmentAvailabilityWindow, AppointmentAvailabilityReport, \
LocationFeedConcordance
from api.models import ApiLog
import json
import pathlib
import pytest

tests_dir = pathlib.Path(__file__).parent / "test-data" / "submitAvailabilityReport"


@pytest.mark.django_db
def test_submit_availability_report_api_bad_token(client):
response = client.post("/api/submitAvailabilityReport")
assert response.json() == {"error": "Authorization header must start with 'Bearer'"}
assert response.status_code == 403
last_log = ApiLog.objects.order_by("-id")[0]
assert {
"method": "POST",
"path": "/api/submitAvailabilityReport",
"query_string": "",
"remote_ip": "127.0.0.1",
"response_status": 403,
"created_report_id": None,
}.items() <= last_log.__dict__.items()


@pytest.mark.django_db
def test_submit_report_api_invalid_json(client):
response = client.post(
"/api/submitAvailabilityReport",
"This is bad JSON",
content_type="text/plain",
HTTP_AUTHORIZATION="Bearer {}".format(SCRAPER_API_KEY)
)
assert response.status_code == 400
assert response.json()["error"] == "Expecting value: line 1 column 1 (char 0)"


@pytest.mark.django_db
@pytest.mark.parametrize("json_path", tests_dir.glob("*.json"))
def test_submit_report_api_example(client, json_path):
fixture = json.load(json_path.open())
assert Report.objects.count() == 0
# Ensure location exists
location, _ = Location.objects.get_or_create(
public_id=fixture["location"],
defaults={
"latitude": 0,
"longitude": 0,
"location_type_id": 1,
"state_id": 1,
"county_id": 1,
},
)
# Ensure feed provider exists
provider, _ = FeedProvider.objects.get_or_create(
name="Test feed",
slug=fixture["input"]["feed_update"]["feed_provider"]
)
# Create concordance
LocationFeedConcordance.objects.create(
feed_provider=provider,
location=location,
provider_id=fixture["input"]["location"]
)

response = client.post(
"/api/submitAvailabilityReport",
fixture["input"],
content_type="application/json",
HTTP_AUTHORIZATION="Bearer {}".format(SCRAPER_API_KEY),
)
assert response.status_code == fixture["expected_status"]
# Load new report from DB and check it
report = AppointmentAvailabilityReport.objects.order_by("-id")[0]
expected_field_values = AppointmentAvailabilityReport.objects.filter(pk=report.pk).values(
*list(fixture["expected_fields"].keys())
)[0]
assert expected_field_values == fixture["expected_fields"]

# Check the windows
for window, expected_window in zip(report.windows.all(), fixture["expected_windows"]):
expected_field_values = AppointmentAvailabilityWindow.objects.filter(pk=window.pk).values(
*list(expected_window["expected_fields"].keys())
)[0]
assert expected_field_values == expected_window["expected_fields"]

actual_tags = [tag.slug for tag in window.additional_restrictions.all()]
assert actual_tags == expected_window["expected_additional_restrictions"]
Loading

0 comments on commit 78a4071

Please sign in to comment.