HHS · acouch · Jan 10, 2025
@@ -4,87 +4,56 @@ This application is dockerized. Take a look at [Dockerfile](../../api/Dockerfile
 
 A very simple [docker-compose.yml](../../docker-compose.yml) has been included to support local development and deployment.
 
-## Prerequisites
+Several components like tests, linting, and scripts can be run either inside of the Docker container, or outside on your  machine.
 
-1. Install the version of Python specified in [pyproject.toml](../../api/pyproject.toml)
-   [pyenv](https://github.com/pyenv/pyenv#installation) is one popular option for installing Python,
-   or [asdf](https://asdf-vm.com/).
-   - If using pyenv run `pyenv local <version>` to ensure that version will be used in subsequent steps
-2. Ensure that `python -V` and `python3 -V` are picking up that version.
-   - If not, run `pyenv init -` and/or restart your shell to ensure it was run automatically
-3. After installing and activating the right version of Python, install
-   [poetry](https://python-poetry.org/docs/#installation) and follow the instructions to add poetry to your path if necessary.
-
-   ```bash
-   curl -sSL https://install.python-poetry.org | python3 -
-   ```
-
-4. You'll also need [Docker Desktop](https://www.docker.com/products/docker-desktop/)
+**Running in Docker is the default**, but on some machines like the M1 Mac, running natively may be desirable for performance reasons.
 
-## **Note:** All the following commands should be run from the `/api` directory.
+## Docker
 
-## Database setup: Run Migrations/Seeds
+This section covers development using Docker. There are a number of Docker commands included in the [Makefile](../../api/Makefile) which are helpful for local development. Run `make help` for a list of commands.
 
-1. If you haven't done local development before you'll need to execute the migrations and seed the DB with data using the steps in [database-local-usage.md](database/database-local-usage.md)
+### Setup
 
-## OpenSearch setup
+Run `make init && make run logs` to start the local containers. The application will be available at `http://localhost:8080/docs`.
 
-1. Run `make init-opensearch` setup the OpenSearch Container
-2. Run `make populate-search-opportunities` to push data previously seeded in the DB into the search index
+This stands up the following services:
 
-If your DB or OpenSearch end up in an odd place, you can reset all the persistent storage using `make volume-recreate`
+* Flask API (http://localhost:8080)
+* Postgres database
+* OpenSearch node
+* OpenSearchdashboard (http://localhost:5601)
+* localstack
+* mock oauth server (http://localhost:5001)
 
-## Run the application
+### Seed data
 
-1. Make sure you have [Docker Desktop](https://www.docker.com/products/docker-desktop/) installed & running.
-2. Run `make setup-local` to install dependencies
-3. Run `make init start` to build the image and start the container.
-4. Navigate to `localhost:8080/docs` to access the Swagger UI.
-5. Run `make run-logs` to see the logs of the running API container
-6. Run `make stop` when you are done to delete the container.
+Run `make db-seed-local && populate-search-opportunities` to create local data in the database and make it available in the API.
 
-## Some Useful Commands
+### API Authenticaion
 
-`make test` will run all of the tests. Additional arguments can be passed to this command which will be passed to pytest like so: `make test args="tests/api/route -v"` which would run all tests in the route folder with verbosity increased. See the [Pytest Docs](https://docs.pytest.org/en/7.1.x/reference/reference.html#command-line-flags) for more details on CLI flags you can set.
-
-`make clean-volumes` will spin down the docker containers + delete the volumes.
-
-`make volume-recreate` Deletes the volumes and then re-initializes the persistant portions of the stack. This can be useful to reset your DB, or fix any bad states your local environment may have gotten into.
-
-See the [Makefile](../../api/Makefile) for a full list of commands you can run.
-
-The `make console` command initializes a Python REPL environment pre-configured with database connectivity. This allows developers to perform database queries, utilize factories for data generation, and interact with the application's models directly.
-
-- Writing a query: `dbs.query(Opportunity).all()`
-- Saving some factory generated data to the db: `f.OpportunityFactory.create()`
-
-## Docker and Native Development
-
-Several components like tests, linting, and scripts can be run either inside of the Docker container, or outside on your native machine.
-Running in Docker is the default, but on some machines like the M1 Mac, running natively may be desirable for performance reasons.
-
-You can switch which way many of these components are run by setting the `PY_RUN_APPROACH` env variable in your terminal.
-
-- `export PY_RUN_APPROACH=local` will run these components natively
-- `export PY_RUN_APPROACH=docker` will run these within Docker
+This API uses a very simple [ApiKey authentication approach](https://apiflask.com/authentication/#use-external-authentication-library) which requires the caller to provide a static key. This is specified with the `API_AUTH_TOKEN` environment variable.
 
-Note that even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within docker.
+### User Authentication
 
-Running in the native/local approach may require additional packages to be installed on your machine to get working.
+Run `make setup-env-override-file` to create the `override.env` file which will include the necessary JWT keys for running the user.
 
-## Environment Variables
+### Environment Variables
 
 Most configuration options are managed by environment variables.
 
 Environment variables for local development are stored in the [local.env](../../api/local.env) file. This file is automatically loaded when running. If running within Docker, this file is specified as an `env_file` in the [docker-compose](../../docker-compose.yml) file, and loaded [by a script](../../api/src/util/local.py) automatically when running most other components outside the container.
 
 Any environment variables specified directly in the [docker-compose](../../docker-compose.yml) file will take precedent over those specified in the [local.env](../../api/local.env) file.
 
-## Authentication
+### Troubleshooting
 
-This API uses a very simple [ApiKey authentication approach](https://apiflask.com/authentication/#use-external-authentication-library) which requires the caller to provide a static key. This is specified with the `API_AUTH_TOKEN` environment variable.
+Errors in standing up the API can originate from an out of date container, database syncronization, or other issues with previously created services. Helper functions are available to rebuild:
+
+* **db-check-migrations** - check if migrations are out of sync
+* **volume-recreate** - delete all existing volumes and data
+* **remake-backend** - delete all data (`volume-recreate`) and load data (`db-seed-local` and `populate-search-opportunities`)
 
-## VSCode Remote Attach Container Debugging
+### VSCode Remote Attach Container Debugging
 
 The API can be run in debug mode that allows for remote attach debugging (currently only supported from VSCode) to the container.
 
@@ -105,6 +74,52 @@ The API can be run in debug mode that allows for remote attach debugging (curren
 
 - You should now be able to hit set breakpoints throughout the API
 
+## Local (non-Docker)
+
+Run `export PY_RUN_APPROACH=local` to run API and test functions locally when running commands in the Makefile. For example, `make test` or `make format` will run outside of Docker.
+
+**Note:** even with the native mode, many components like the DB and API will only ever run in Docker, and you should always make sure that any implementations work within docker.
+
+Running in the native/local approach may require additional packages to be installed on your machine to get working.
+
+### Prerequisites
+
+1. Install the version of Python specified in [pyproject.toml](../../api/pyproject.toml)
+   [pyenv](https://github.com/pyenv/pyenv#installation) is one popular option for installing Python,
+   or [asdf](https://asdf-vm.com/).
+   - If using pyenv run `pyenv local <version>` to ensure that version will be used in subsequent steps
+2. Ensure that `python -V` and `python3 -V` are picking up that version.
+   - If not, run `pyenv init -` and/or restart your shell to ensure it was run automatically
+3. After installing and activating the right version of Python, install
+   [poetry](https://python-poetry.org/docs/#installation) and follow the instructions to add poetry to your path if necessary.
+
+   ```bash
+   curl -sSL https://install.python-poetry.org | python3 -
+   ```
+
+4. You'll also need [Docker Desktop](https://www.docker.com/products/docker-desktop/)
+
+**Note:** All the following commands should be run from the `/api` directory.
+
+### Database setup: Run Migrations/Seeds
+
+If you haven't done local development before you'll need to execute the migrations and seed the DB with data using the steps in [database-local-usage.md](database/database-local-usage.md)
+
+### Services
+
+Individual services can be run through Docker, which can be useful in concert with non-Docker application development:
+
+* **OpenSearch**
+  * Run `make init-opensearch` setup the OpenSearch Container
+  * Run `make populate-search-opportunities` to push data previously seeded in the DB into the search index
+
+  If your DB or OpenSearch end up in an odd place, you can reset all the persistent storage using `make volume-recreate`.
+
+* **Localstack (local s3)**
+   * Run `make init-localstack`
+* **Mock OAuth server**
+   * Run `make init-mock-oath2`
+
 ## Next steps
 
 Now that you're up and running, read the [application docs](../../api/README.md) to familiarize yourself with the application.
@@ -0,0 +1,86 @@
+# Development
+
+This [Next.js](https://nextjs.org) application can be run natively (or locally)
+
+**Running in locally is the default**, but it can be useful to run the built images in order to troubleshoot and to connect directly with the local API application for development.
+
+## Local (non-Docker)
+
+Run `npm install && npm run dev` to start the application.
+
+### Testing "built" version locally
+
+The Next.js frontend application is exported for production using [next build](https://nextjs.org/docs/app/api-reference/cli/next#next-build-options). To recreate this locally, outside of the container, run the following:
+
+- `npm run build` - Builds the production Next.js bundle
+- `npm start` - Runs the Next.js server, after building the production bundle
+
+### Search and Opportunity Pages
+
+The `/search` and opportunity pages rely on the application API. The API endpoint and authenticaion token are defined in `.env.development` and can be overwritten in an `.env.ocal` file.
+
+Update the `API_URL` can be set to connect to prod (`https://api.simpler.grants.gov`) or lower environment URLs to quickly develop using production or development data. This requires the correct `API_AUTH_TOKEN` variable to be set correctly.
+
+To connect to the development version of the API, run `make init && db-seed-local && populate-search-opportunities` in the `/api` folder.
+
+See [documentation/api/development.md](../api/development.md) for more details.
+
+### Authentication
+
+Running authentication locally requires running the API, directing the API redirect to the frontend, and sharing the correct JWT keys.
+
+1. Run `make setup-env-override-file` to create the `override.env` file in the `/api` folder
+2. Copy the `API_JWT_PUBLIC_KEY` value from `/api/override.env` file to `/frontend/.env.local` file which creates the necessary keys
+3. Add `LOGIN_FINAL_DESTINATION=http://localhost:3000/api/auth/callback` to the `api/override.env` so the API redirects to the correct callback route
+4. Start the API and frontend for development
+
+#### Secrets
+
+Some functionality will not work locally without supplying the application environment variables containing secrets.
+
+- New Relic
+  - `NEW_RELIC_APP_NAME`
+  - `NEW_RELIC_LICENSE_KEY`
+- Email subscription form (Sendy)
+  - `SENDY_API_KEY`
+  - `SENDY_API_URL`
+  - `SENDY_LIST_ID`
+
+If you need to access this functionality locally, contact an engineer on the team to get access to the necessary secrets.
+
+### +Docker
+
+#### Development version
+
+Alternatively, you can run the application in a Docker container.
+
+**Note**: If you are running docker locally for the first time, you need to run the API locally through Docker as well, in order to create the required `api_grants_backend` network.
+
+From the `/frontend` directory:
+
+1. Run the local development server
+   ```bash
+   make dev
+   ```
+1. Navigate to [localhost:3000](http://localhost:3000) to view the application
+
+- If installing new packages locally with npm and using `make dev` with docker to run locally, you may need to run `make build` first to bring the new packages into the container
+
+#### Production version
+
+The `make dev` command runs the `docker-compose.yml` which runs the `dev` target in the [Dockerfile](./Dockerfile). To run a production version in docker, run `docker compose up -d -f docker-compose-realease.yml` which targest the `release` stage in the docker build. This runs the production version, while still creating a network connection to the local API.
+
+#### Testing Release Target Locally
+
+To test the release target locally, run:
+
+- `make release-build OPTS="--tag [IMAGE_NAME]"` or
+- `docker buildx build --target release --tag [IMAGE_NAME]` for a faster build on OSX
+
+to build a local image. To view the site at `localhost:3000`, run: `docker run -e "HOSTNAME=0.0.0.0" -p 3000:3000 [IMAGE_NAME]`.
+
+### Search and Opportunity Pages
+
+The search page and opportunity pages needs additional setup to test locally.
+
+To