Automate certified release process

[rvermeulen]

Description

This PR adds new workflows and updates existing workflows to create an automated certified release process.

This process reuses the existing release branch approach, but automates the release using a release PR against the release branch. This allows use to reuse all the checks we already perform on PRs and add checks specific for a release.

This event driven process requires monitoring of dispatched workflows and the overall status of the release. This has been realized using check-runs.

The overall flow a release is:

flowchart TD;
  prepare-release["Prepare release (prepare-release.yml)"]
  validate-release["Validate release (validate-release.yml)"]
  compiler-validation["Compiler tests (release-engineering/release-compiler-validation.yml.)"]
  performance-testing["Performance testing (release-engineering/release-performance-testing.yml)"]
  existing-checks["Existing checks run on each PR"]
  update-release["Update release (update-release.yml)"]
  finalize-release["Finalize release (finalize-release.yml)"]

  prepare-release-->validate-release
  validate-release-->compiler-validation-->update-release
  validate-release-->performance-testing-->update-release
  prepare-release-->existing-checks-->update-release
  update-release-->finalize-release

Loading

but this is not the actually execution flow because those are event driven.

The prepare-release.yml workflow starts the release process through a dispatch that takes as inputs the version to release and the ref to release from. The workflows then

• Creates a release branch
• Creates a release PR with a commit updating the user manual to the chosen version
• Creates a draft release for the chosen version

The creation of the PR is done in such a way that it will trigger a pull_request event targeting a release branch. This will ensure it will execute all the existing tests and additional release tests defined in validate-release.yml.

The validate-release.yml workflow will dispatch the performance and compiler testing defined in the codeql-coding-standards-engineering repository. To be able to monitor these dispatched workflows and the overall release status 3 check-runs are created and updated accordingly.

On each workflow conclusion its associated checkrun concludes and the update-release-status.yml workflow is configured to validate the overall release status when triggered by an checkrun completion or re-request. A release status that was concluded could be renewed if a checkrun is re-requested.

Once a release status is complete, as determined byupdate-release-status.yml, the update-release.yml workflow will be called that will update the release body and artifacts. This workflow relies on the update-release-assets.py and update-release-notes.py. The latter is a simple wrapper to call the existing generate_release_notes.py script, but the update-release-assets.py is a new script.

The update-release-assets.py takes the head-sha of the PR, GitHub tokens to collect workflow runs, and a layout. The layout decoratively specifies the release layout. Our current layout is defined in release-layout.yml.

The layout specification builds artifacts through the supported actions. We currently have 4 actions: file, shell, workflow-log, and workflow-artifact to support the existing artifacts and the artifacts required for the certification kit.

Once the release is status is green we can merge the release PR. The closing of a release PR triggers the finalize-release.yml workflow that pushes the latest release tag and takes the release out of draft, concluding the release.

Finally, the update-check-run.yml is used to allow external repositories with dispatch authorization on this repository to update check-runs. Check runs cannot be updated with a PAT, but only by GitHub Apps. Since we cannot install the later this is a workaround that allow use to update check-runs with a PAT.

Resilience

The release process is designed to be resilient against errors by making each workflow restartable in the case of errors. The prepare-release.yml will only stop preparing a new release if there already exists a non-draft release of the provided version. When you restart the a release preparation the release branch will be recreated so any additional commits will be removed.

The developer manual will contain a list of possible failures and how to restart the release process.

Caveats

• The update-release-status.yml has only be tested with manual dispatch because the check_run event isn't triggered on workflows outside of the main branch.
• The update-release.yml isn't tested in full because the package generation and unit tests flows require a larger runner not available on the accounts used to test the new workflows.
• Workflows have been renamed so we need to update the branch protection rules.

TODOS

• Update release process description in the developer manual.
• Remove references to rvermeulen/codeql-coding-standards used for testing.
• Mint production tokens used by the release process.
• Update branch protection rules.

Change request type

• Release or process automation (GitHub workflows, internal scripts)
• Internal documentation
• External documentation
• Query files (.ql, .qll, .qls or unit tests)
• External scripts (analysis report or other code shipped as part of a release)

Rules with added or modified queries

• No rules added
• Queries have been added for the following rules:
  • rule number here
• Queries have been modified for the following rules:
  • rule number here

Release change checklist

A change note (development_handbook.md#change-notes) is required for any pull request which modifies:

• The structure or layout of the release artifacts.
• The evaluation performance (memory, execution time) of an existing query.
• The results of an existing query in any circumstance.

If you are only adding new rule queries, a change note is not required.

Author: Is a change note required?

• Yes
• No

🚨🚨🚨
Reviewer: Confirm that format of shared queries (not the .qll file, the
.ql file that imports it) is valid by running them within VS Code.

• Confirmed

Reviewer: Confirm that either a change note is not required or the change note is required and has been added.

• Confirmed

Query development review checklist

For PRs that add new queries or modify existing queries, the following checklist should be completed by both the author and reviewer:

Author

• Have all the relevant rule package description files been checked in?
• Have you verified that the metadata properties of each new query is set appropriately?
• Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
• Are the alert messages properly formatted and consistent with the style guide?
• Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
  As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
• Does the query have an appropriate level of in-query comments/documentation?
• Have you considered/identified possible edge cases?
• Does the query not reinvent features in the standard library?
• Can the query be simplified further (not golfed!)

Reviewer

• Have all the relevant rule package description files been checked in?
• Have you verified that the metadata properties of each new query is set appropriately?
• Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
• Are the alert messages properly formatted and consistent with the style guide?
• Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
  As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
• Does the query have an appropriate level of in-query comments/documentation?
• Have you considered/identified possible edge cases?
• Does the query not reinvent features in the standard library?
• Can the query be simplified further (not golfed!)

[jsinglet]

In addition to our extensive in person reviews, I think this is GTG now! Thanks for all this amazing hard work @rvermeulen 🚀