Add documentation for the API for a few primary components (#727)

documentation/api/api-details.md

@@ -2,24 +2,141 @@
 
 See [Technical Overview](./technical-overview.md) for details on the technologies used.
 
-Each endpoint is configured in the [openapi.generated.yml](../../api/openapi.generated.yml) file which provides basic request validation. Each endpoint specifies an `operationId` that maps to a function defined in the code that will handle the request.
+Each endpoint is defined as a function using an APIFlask blueprint where we define the schema of the
+request and response objects, attach authentication, and otherwise configure the endpoint based on
+several decorators attached to the endpoint function.
 
-# Swagger
+An OpenAPI file is generated from this which can be found at [openapi.generated.yml](../../api/openapi.generated.yml).
+
+# Routes
+Routes and Marshmallow schema are defined in the [api/src/api](../../api/src/api) folder.
 
-The Swagger UI  can be reached locally at [http://localhost:8080/docs](http://localhost:8080/docs) when running the API. The UI is based on the [openapi.generated.yml](../../api/openapi.generated.yml) file.
-![Swagger UI](images/swagger-ui.png)
+A route should only contain what is necessary to configure the inputs/outputs, and setup the DB session.
+A route should generally:
+* Define any request/response schemas
+* Define any additional schemas (ie. headers or path params)
+* Define the authentication for the route
+* Define any DB sessions that the route will require
+* Call the handler
+* Return the response
 
-Each of the endpoints you've described in your openapi.generated.yml file will appear here, organized based on their defined tags. For any endpoints with authentication added, you can add your authentication information by selecting `Authorize` in the top right.
-![Swagger Auth](images/swagger-auth.png)
+## Defining a schema
 
-All model schemas defined can be found at the bottom of the UI.
+We define schemas using [Marshmallow](https://marshmallow.readthedocs.io/en/stable/), but use our own derived version
+of the schemas, fields, and validators to be able to produce errors in the format we want. See [error-handling.md](./error-handling.md) for more details.
 
-# Routes
+These schemas will be used to generate  the OpenAPI schema, setup validations that will run, and make it possible to have a Swagger endpoint available for use
+
+For example, if we wanted to define an endpoint with a request like:
+```json
+{
+  "name": {
+      "first_name": "Bob",
+      "last_name": "Smith",
+      "suffix": "SR"
+  },
+  "birth_date": "1990-01-01"
+}
+```
+We would define the Marshmallow schema in-python like so:
+```py
+from enum import StrEnum
+from src.api.schemas.extension import Schema, fields, validators
+
+class Suffix(StrEnum):
+    SENIOR = "SR"
+    JUNIOR = "JR"
+
+class NameSchema(Schema):
+    first_name = fields.String(
+        metadata={"description": "First name", "example": "Jane"},
+        validate=[validators.Length(max=28)],
+        required=True,
+    )
+    last_name = fields.String(
+        metadata={"description": "Last name", "example": "Doe"},
+        validate=[validators.Length(max=28)],
+        required=True,
+    )
+    suffix = fields.Enum(
+        Suffix, metadata={"description": "Suffix"}
+    )
+
+class ExampleSchema(Schema):
+    name = fields.Nested(NameSchema())
+    birth_date = fields.Date(metadata={"description": "Their birth date"})
+```
+
+Anything specified in the metadata field is passed to the OpenAPI file that backs the swagger endpoint. The values
+that can be passed through are defined in the [APIFlask docs](https://apiflask.com/openapi/#response-and-request-schema)
+but it's recommended you try to populate the following:
+- description
+- example - if you want a specific default in the swagger docs
+- type - only necessary if the default can't be determined by APIFlask
+
+You can specify validators that will be run when the request is being serialized by APIFlask
+
+Defining a response works the exact same way however field validation does not occur on response, only formatting.
+The response schema only dictates the data portion of the response, the rest of the response is defined in
+[ResponseSchema](../../api/src/api/schemas/response_schema.py) which is connected to APIFlask via the `BASE_RESPONSE_SCHEMA` config.
+
+
+### Schema tips
+
+`required=True` does not mean a field isn't nullable, it means the field must be set to something.
+If you have a field named `my_field` that is required then 
+- `{"my_field": null}` is perfectly valid
+- `{}` would fail validation
+
+If the value we need to have in the request, and what we want to call it in-code are different, you can specify a
+`data_key` for the request value and Marshmallow will map it for you.
+
+When providing an example for a value, consider what is a reasonable test default. Ideally it should pass any validations
+and be representative of real data. For example, we often specify zip codes as several components use those zip codes.
+
+Nested fields:
+- If you define a field as `my_field = fields.Nested(AnotherSchema())`, don't provide a metadata as in some cases it seems to cause an issue with openapi generation
+
+## Defining an endpoint
+
+To define an endpoint, you need to attach a route definition to a function.
+
+For example, if we wanted to use the above schema and create a `POST /example-route/<uuid:example_id>` we would
+define it like so:
+
+
+```py
+from src.auth.api_key_auth import api_key_auth
+from apiflask import APIBlueprint
+import src.api.response as response
+
+example_blueprint = APIBlueprint("Example", __name__, tag="Example")
+
+@example_blueprint.post("/example-route/<string:example_id>")
+@example_blueprint.input(ExampleSchema, arg_name="body") # As defined above, arg_name is used to map to the field in the function below
+@example_blueprint.output(ExampleResponseSchema)
+@example_blueprint.auth_required(api_key_auth) # The HTTPTokenAuth object that has a registered authentication function
+def post_example_route(example_id: str, body: dict) -> response.ApiResponse:
+    # Implement API logic
+
+    return response.ApiResponse("Example success message", data={"example_id": "abcd1234"})
+```
+
+The API itself is defined first, including any route parameters. We then specify the schema that will define
+the input and output of the request. Finally we define any authentication + specifically specify the auth used
+so that OpenAPI will give you an authentication pop-up to fill out for specific endpoints.
+
+When you define a blueprint like this, it needs to be registered with the APIFlask app, which we do in [app.py](../../api/src/app.py)
+by calling `app.register_blueprints(example_blueprint)`.
+
+# Swagger
+
+The Swagger UI  can be reached locally at [http://localhost:8080/docs](http://localhost:8080/docs) when running the API.
+![Swagger UI](./images/swagger-ui.png)
 
-## Health Check
-[GET /v1/healthcheck](../../api/src/api/healthcheck.py) is an endpoint for checking the health of the service. It verifies that the database is reachable, and that the API service itself is up and running.
+The actual openapi spec generated that backs this swagger UI can be seen at [http://localhost:8080/openapi.json](http://localhost:8080/openapi.json)
 
-Note this endpoint explicitly does not require authorization so it can be integrated with any automated monitoring you may build.
+Each of the endpoints you've described in API blueprints will appear here, organized based on their defined tags. For any endpoints with authentication added, you can add your authentication information by selecting `Authorize` in the top right.
+![Swagger Auth](./images/swagger-auth.png)
 
-### Example Response
-![Example Response](images/healthcheck-response.png)
+All model schemas defined can be found at the bottom of the UI.

documentation/api/database/database-local-usage.md

@@ -0,0 +1,31 @@
+# Overview
+
+This document describes how you can access and use your local DB.
+
+## Connecting to the DB
+
+### UI
+
+There are many different programs you can use to connect to your DB, for example: [pgAdmin](https://www.pgadmin.org/).
+
+Steps:
+1. Make sure your local DB is running, you can start it up by doing `make init-db` which will handle starting + running initial DB migrations
+2. Open your program for connecting to the DB, and add the connection parameters as shown below
+
+![Local DB Connection Parameters](../images/local-db-connection.png)
+
+Connection parameters for the database can be found in [local.env](../../api/local.env) including the password.
+
+### CLI
+
+If you have `psql` installed locally, you can also access the DB by running `psql -h localhost -d app -U app` which will require you enter the password afterwards.
+
+For example:
+![Local DB CLI](../images/local-db-cli.png)
+
+## Seeding the DB
+
+You can populate our opportunity data by running: `make db-seed-local` when you have the DB running.
+
+This script currently creates 25 new opportunities each time you run it. In the future, as we expand the amount of data we support, we'll
+add more options and data to the DB, but this should give a rough set of data to work with.

documentation/api/database/database-management.md

@@ -65,7 +65,7 @@ $ make db-migrate
 ```python
 class ExampleTable(Base):
     ...
-    my_new_timestamp = Column(TIMESTAMP(timezone=True)) # Newly added line
+    my_new_timestamp: Mapped[datetime] = mapped_column(TIMESTAMP(timezone=True)) # Newly added line
 ```
 
 2. Automatically generate a migration file with `make db-migrate-create MIGRATE_MSG="Add created_at timestamp to address table"`

documentation/api/database/database-testing.md

@@ -11,3 +11,4 @@ Note that [PostgreSQL schemas](https://www.postgresql.org/docs/current/ddl-schem
 ## Test Factories
 
 The application uses [Factory Boy](https://factoryboy.readthedocs.io/en/stable/) to generate test data for the application. This can be used to create models `Factory.build` that can be used in any test, or to prepopulate the database with persisted models using `Factory.create`. In order to use `Factory.create` to create persisted database models, include the `enable_factory_create` fixture in the test.
+

documentation/api/error-handling.md

@@ -0,0 +1,102 @@
+# Overview
+
+When an error occurs in an API endpoint, we want to return something formatted like:
+
+```json
+{
+  "data": {},
+  "errors": [
+    {
+      "field": "path.to.field_name",
+      "message": "field is required",
+      "type": "required",
+      "value": null
+    }
+  ],
+  "message": "There was a validation error",
+  "status_code": 422
+}
+```
+
+This structure allows users of the API the ability to determine what the error was
+and for which field in the request programatically. While not every error will
+specify a field, by having a consistent structure users are able to reliably
+implement error handling.
+
+However, the various libraries we use don't structure errors this way, and so
+we have several pieces of code to handle making everything fit together nicely.
+
+APIFlask supports providing your own [error processor](https://apiflask.com/error-handling/)
+which we have implemented in [restructure_error_response](../../api/src/api/response.py)
+
+# Error Handling
+
+## Throwing an exception
+
+When an exception is thrown during an API call, this will result in a 500 error
+unless it is an exception that APIFlask has configured to return a different status code
+like the ones from the `werkzeug` library. However, if you throw one of these
+exceptions, any message or other information is lost. To avoid this, instead
+use our `raise_flask_error` function which handles wrapping exceptions in
+a format that APIFlask will keep the context we add.
+
+```py
+from src.api.response import ValidationErrorDetail
+from src.api.route_utils import raise_flask_error
+from src.validation.validation_constants import ValidationErrorType
+
+
+from werkzeug.exceptions import NotFound
+
+
+raise_flask_error(NotFound.code, "Unable to find the record requested", validation_issues=[ValidationErrorDetail(message="could not find", type=ValidationErrorType.UNKNOWN)])
+```
+
+This would result in an error that looks like:
+```json
+{
+  "data": {},
+  "errors": [
+    {
+      "field": null,
+      "message": "could not find",
+      "type": "unknown"
+    }
+  ],
+  "message": "Unable to find the record requested",
+  "status_code": 404
+}
+```
+
+Note that the validation error detail list is optional, and generally only used
+for validation errors.
+
+## Marshmallow Errors
+
+By default, Marshmallow constructs its errors like:
+```json
+{
+  "path": {
+    "to": {
+      "field": ["error message1", "error message2"]
+    }
+  }
+}
+```
+
+There are two issues with this, the path structure, and error message. Fixing the path
+structure is simple and just requires flattening the dictionary, but the error messages
+unfortunately only provide a message, when we want a message AND code for the particular error.
+
+To work around this challenge, we created our own derived versions of the Marshmallow schema,
+field, and validator classes in the [extensions folder](../../api/src/api/schemas/extension).
+
+These extend the Marshmallow classes to instead output their errors as a [MarshmallowErrorContainer](../../api/src/api/schemas/extension/schema_common.py)
+
+This is done by modifying the default error message that each validation rule has to instead
+be a `MarshmallowErrorContainer` object. For most of the fields, this is just a bit of configuration,
+but the validators required re-implementing them as they handled errors directly in validation.
+
+When Marshmallow throws its errors, our [process_marshmallow_issues](../../api/src/api/response.py) function
+will get called which handles flattening the errors, and then restructuring them into
+proper format.