From 7bbac29bbbfcdaf501f2232f280f320778a4ddd7 Mon Sep 17 00:00:00 2001 From: "stonezdj(Daojun Zhang)" Date: Wed, 12 Jun 2024 15:20:20 +0800 Subject: [PATCH] Use the sbom_report instead of scan_report to store the sbom scan overview information (#241) Signed-off-by: stonezdj --- proposals/new/sbom_gen_scan.md | 37 +++++++++++++++++++++++----------- 1 file changed, 25 insertions(+), 12 deletions(-) diff --git a/proposals/new/sbom_gen_scan.md b/proposals/new/sbom_gen_scan.md index 5c1b3c1a..9b4b6692 100644 --- a/proposals/new/sbom_gen_scan.md +++ b/proposals/new/sbom_gen_scan.md @@ -87,10 +87,23 @@ For each project, the scanner API (/api/v2.0/projects//scanner) coul ### Database Schema Changes -The scan_report table should be updated to add the following columns +Add a new table sbom_report table, it stores the SBOM report summary information generated by the scanner. the table schema is as following: + ``` -ALTER TABLE scan_report ADD COLUMN IF NOT EXISTS media_type varchar(255); -``` +CREATE TABLE IF NOT EXISTS sbom_report +( + id SERIAL PRIMARY KEY NOT NULL, + uuid VARCHAR(64) UNIQUE NOT NULL, + artifact_id INT NOT NULL, + registration_uuid VARCHAR(64) NOT NULL, + mime_type VARCHAR(256) NOT NULL, + media_type VARCHAR(256) NOT NULL, + report JSON, + UNIQUE(artifact_id, registration_uuid, mime_type, media_type) +); +``` +The sbom_report table use the artifact_id, registration_uuid, mime_type and media_type as the unique key. because +the sbom is an accessory of the artifact, it is stored as artifact accessory for the subject artifact, it has fixed project and repository, if two artifact have same digest, they generate each sbom accessory respectively. it is different with the vulnerability scan. ### Adatper API change The adapter need to implement the following APIs to support the SBOM generation and scan. @@ -250,7 +263,7 @@ If the sbom_media_type is not provided, the scanner adapter should return 400 to 1. Add a project level configuration item auto_gen_sbom to enable auto generate SBOM on image push, the default value is false. -2. The default SBOM media_type for scanner is hardcode to `application/spdx+json`, the media_type is stored in scan_report table as a column. +2. The default SBOM media_type for scanner is hardcode to `application/spdx+json`, the media_type is stored in sbom_report table as a column. 3. Update existing scan API to allow to generate SBOM Request Method: @@ -286,11 +299,11 @@ If the scan_type is empty, it is default to "vulnerability". it will query the e 5. Update existing scan job service, add support to generate SBOM for an artifact, update the existing job service in pkg/scan/job.go to support SBOM generate in the request parameter. for a generate SBOM report request, the job service should: - 1. Delete the previous SBOM report generated with the same scanner if exist. include the information in the table scan_report and the artifact accessory in the OCI registry. if current scan type is SBOM, it will get the `sbom_digest` and `sbom_repository` information from the scan_report table's report column, if both exist, then it will delete the SBOM accessory in the OCI registry. - 1. Parse the SBOM report and push the the SBOM to the OCI registry as an artifact accessory. its media type should be "application/vnd.goharbor.harbor.sbom.v1" - 1. Store the SBOM digest information into the table scan_report. + 1. Delete the previous SBOM report generated with the same scanner if exist. include the information in the table sbom_report and the artifact accessory in the OCI registry. it gets the `sbom_digest` and `sbom_repository` information from the sbom_report table's report column, if both exist, then it deletes the SBOM accessory in the OCI registry. + 1. Parse the SBOM report and push the the SBOM to the OCI registry as an artifact accessory. its mime type should be "application/vnd.goharbor.harbor.sbom.v1" + 1. Store the SBOM digest information into the table sbom_report. - The Harbor job service stores the SBOM digest information into the database. the scan_report table's report column is updated with the following content when it is a sbom report: + The Harbor job service stores the SBOM digest information into the database. the sbom_report table's report column is updated with the following content: ``` { "sbom_digest":"sha256:51a41cec9de9d62ee60e206f5a8a615a028a65653e45539990867417cb486285", @@ -308,7 +321,7 @@ If the scan_type is empty, it is default to "vulnerability". it will query the e 5. The job service query the scan report from the scanner adapter by the given request id, until all required reports are ready. it also have a timeout to avoid the job to be blocked too long. 6. Parse the report and store the report information. -6. Update existing list artifact API to support list artifact with_sbom_overview, if with_sbom_overview is true and the sbom information is in the scan_report, the sbom_overview data should be provided. +6. Update existing list artifact API to support list artifact with_sbom_overview, if with_sbom_overview is true and the sbom information is in the sbom_report, the sbom_overview data should be provided. ``` [ { @@ -340,7 +353,7 @@ If the scan_type is empty, it is default to "vulnerability". it will query the e } ] ``` -In the list artifact page, add SBOM overview column to display the sbom_overview information of the artifact, it retrieves the scan_report table and get the digest of the artifact accessory, if the artifact accessory exists, it returns the SBOM status information. If the report in scan_report has no record related to this artifact's SBOM, the sbom_overview keeps empty or fallback to the query result of the artifact accessory for this artifact. +In the list artifact page, add SBOM overview column to display the sbom_overview information of the artifact, it retrieves the sbom_report table and get the digest of the artifact accessory, if the artifact accessory exists, it returns the SBOM status information. If the report in sbom_report has no record related to this artifact's SBOM, the sbom_overview keeps empty or fallback to the query result of the artifact accessory for this artifact. 7. Add an API to retrieve the SBOM content for the SBOM accessory. Request Method: @@ -399,7 +412,7 @@ Given a artifact digest, if the SBOM report is unavailable, it returns http code The SBOM is stored the OCI registry, it should: -1. It can be replicated by Harbor replication, except missing SBOM summary information, the target Harbor should display the same SBOM detail information. when a SBOM artifact accessory is replicated from Harbor A to Harbor B, then list the artifact in Harbor B, the SBOM accessory should be displayed in the artifact's list page. because the scan_report in Harbor B has no sbom report associate with the artifact. then it will fallback to the query result of artifact accessory for this artifact, the artifact accessory query API is +1. It can be replicated by Harbor replication, except missing SBOM summary information, the target Harbor should display the same SBOM detail information. when a SBOM artifact accessory is replicated from Harbor A to Harbor B, then list the artifact in Harbor B, the SBOM accessory should be displayed in the artifact's list page. because the sbom_report in Harbor B has no sbom report associate with the artifact. then it will fallback to the query result of artifact accessory for this artifact, the artifact accessory query API is ``` GET /api/v2.0/projects//repositories//artifacts//accessories?q=type%253D%257Bsignature.cosign%2520harbor.sbom%257D&page_size=5&page=1 ``` @@ -423,7 +436,7 @@ The response should be like this: ``` If there is a SBOM accessory for the current artifact, then the artifact's SBOM status column will display the link to the SBOM detail page. it is implemented in UI logic. 2. If the artifact is deleted, its SBOM information should be removed as well -3. If a SBOM accessory is deleted, Harbor should remove it from scan_report table by digest and repository information stored in the report column. this logic should be implemented in the internal artifact event hook. +3. If a SBOM accessory is deleted, Harbor should remove it from sbom_report table by digest and repository information stored in the report column. this logic should be implemented in the internal artifact event hook. 4. If it is generated by the current scanner in the Harbor, it should be removed when a new SBOM generation start. for each artifact, it only keeps the latest SBOM report generated by the current scanner. ### UI