Air gap environment scanning reports getting deleted automatically after scan

[tanjiro896]

Hi Guys,

I have reconfigured existing container registry as air gap environment and harbor version is 2.8.4 and trivy adaptor version 0.44, after reconfigure air gap environment I noticed trivy adaptor logs and container logs able to see only in debug mode and after scanning generated scanning reports getting deleted immediately after scanning completed, is it defult behaviour of the air gap environment??

I have downloaded trivy database and Java database and replaced with existing database.

So kindly let me know how to troubleshoot why the trivy adaptor logs can see in debug mode and why scanning reports are getting deleted.

[wy65701436]

can you provide the log of trivy?

Harbor deletes the report firstly when you repeat scanning one specific artifact, and if at this time there is something wrong at the trivy end, as you have reconfigured your harbor as air gap environment, and the new report won't be generated successfully.

[tanjiro896]

Hi Wang yan,

Thanks for the response, when we run scan if the harbor log level is INFO, trivy adaptor logs are not generating and in the core logs I could see scanning process, if we changed to DEBUG trivy adaptor logs are generating, you can see below.

tail -f trivy-adapter.log
Jul 5 07:43:28 172.30.0.1 trivy-adapter[2525]: Appending internal tls trust CA to ca-bundle ...
Jul 5 07:43:28 172.30.0.1 trivy-adapter[2525]: find: '/etc/harbor/ssl': No such file or directory
Jul 5 07:43:28 172.30.0.1 trivy-adapter[2525]: Internal tls trust CA appending is Done.
Jul 5 07:43:28 172.30.0.1 trivy-adapter[2525]: {"built_at":"unknown","commit":"none","level":"info","msg":"Starting harbor-scanner-trivy","time":"2024-07-05T11:43:28Z","version":"dev"}
Jul 5 07:43:28 172.30.0.1 trivy-adapter[2525]: {"addr":":8080","level":"warning","msg":"Starting API server without TLS","time":"2024-07-05T11:43:28Z"}

I noticed these two lines when I ran scanning, please have a look meanwhile will share the latest logs.

Jul 5 03:40:20 172.23.0.1 trivy-adapter[2525]: {"level":"debug","msg":"Saving scan report to tmp file","path":"/home/scanner/.cache/reports/scan_report_3489813556.json","time":"2024-07-05T07:40:20Z"}

Jul 5 03:48:02 172.23.0.1 trivy-adapter[2525]: {"level":"debug","msg":"Removing scan report tmp file","path":"/home/scanner/.cache/reports/scan_report_684860888.json","time":"2024-07-05T07:48:02Z"}
Ju

[tanjiro896]

Hi, just now enabled the debug logs and run the scan, please have a look into below logs.

Jul 8 05:44:09 192.168.80.1 trivy-adapter[2525]: find: '/etc/harbor/ssl': No such file or directory
Jul 8 05:44:09 192.168.80.1 trivy-adapter[2525]: Appending internal tls trust CA to ca-bundle ...
Jul 8 05:44:09 192.168.80.1 trivy-adapter[2525]: Internal tls trust CA appending is Done.
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"built_at":"unknown","commit":"none","level":"info","msg":"Starting harbor-scanner-trivy","time":"2024-07-08T09:44:10Z","version":"dev"}
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Current process","pid":1,"time":"2024-07-08T09:44:10Z"}
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"gid":10000,"home_dir":"/home/scanner","level":"debug","msg":"Current user","time":"2024-07-08T09:44:10Z","uid":10000}
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"level":"debug","mode":"drwxr-xr-x","msg":"trivy cache dir permissions","time":"2024-07-08T09:44:10Z"}
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"level":"debug","mode":"drwxr-xr-x","msg":"trivy reports dir permissions","time":"2024-07-08T09:44:10Z"}
Jul 8 05:44:10 192.168.80.1 trivy-adapter[2525]: {"addr":":8080","level":"warning","msg":"Starting API server without TLS","time":"2024-07-08T09:44:10Z"}

Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Enqueueing scan job","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Successfully enqueued scan job","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Saving scan job","redis_key":"harbor.scanner.trivy:store:scan-job:xxxxxxxxxxxxxxx","scan_job_id":"xxxxxxxxxxx","scan_job_status":"Queued","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Enqueueing scan job","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Successfully enqueued scan job","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:48 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Saving scan job","redis_key":"harbor.scanner.trivy:store:scan-job:xxxxxxxxxxxxxx","scan_job_id":"xxxxxxxxxxxxxxx","scan_job_status":"Queued","time":"2024-07-08T09:47:48Z"}
Jul 8 05:47:51 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:xxxxxxxxxxxxxxx","scan_job_id":"xxxxxxxxxx","scan_job_status":"Pending","time":"2024-07-08T09:47:51Z"}
Jul 8 05:47:51 192.168.80.1 trivy-adapter[2525]: {"image_ref":"core:8080/projectname/imagename@sha256xxxxxxxxxxxxxxxxxxx","level":"debug","msg":"Started scanning","time":"2024-07-08T09:47:51Z"}
Jul 8 05:47:51 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Saving scan report to tmp file","path":"/home/scanner/.cache/reports/scan_report_935272296.json","time":"2024-07-08T09:47:51Z"}

Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Removing scan report tmp file","path":"/home/scanner/.cache/reports/scan_report_935272296.json","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating reports for scan job","scan_job_id":"324e4f45266c7df5019813ea","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:324e4f45266c7df5019813ea","scan_job_id":"324e4f45266c7df5019813ea","scan_job_status":"Pending","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating status for scan job","new_status":"Finished","scan_job_id":"324e4f45266c7df5019813ea","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:324e4f45266c7df5019813ea","scan_job_id":"324e4f45266c7df5019813ea","scan_job_status":"Finished","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Executing enqueued scan job","scan_job_id":"0516feedd85aa9416ae7e13a","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating status for scan job","new_status":"Pending","scan_job_id":"0516feedd85aa9416ae7e13a","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:0516feedd85aa9416ae7e13a","scan_job_id":"0516feedd85aa9416ae7e13a","scan_job_status":"Pending","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"image_ref":"core:8080/project/image@sha256:3e3c92cd4f1773xxxxxxxxxxxxx","level":"debug","msg":"Started scanning","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:52 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Saving scan report to tmp file","path":"/home/scanner/.cache/reports/scan_report_2146663388.json","time":"2024-07-08T09:47:52Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"exit_code":0,"image_ref":"core:8080/project/image@sha256:3e3c92cd4f177389abd1a0e0cc0534a010d3e727cb5xxxxxxxxxxxxx","level":"debug","msg":"Running trivy finished","std_out":"2024-07-08T09:47:52.670Z\t\u001b[35mDEBUG\u001b[0m\tSeverities: ["UNKNOWN" "LOW" "MEDIUM" "HIGH" "CRITICAL"]\n2024-07-08T09:47:52.670Z\t\u001b[35mDEBUG\u001b[0m\tIgnore statuses\t{"statuses": null}\n2024-07-08T09:47:52.671Z\t\u001b[35mDEBUG\u001b[0m\tcache dir: /home/scanner/.cache/trivy\n2024-07-08T09:47:52.671Z\t\u001b[35mDEBUG\u001b[0m\tSkipping DB update...\n2024-07-08T09:47:52.671Z\t\u001b[35mDEBUG\u001b[0m\tDB Schema: 2, UpdatedAt: 2024-06-19 12:13:07.559919049 +0000 UTC, NextUpdate: 2024-06-19 18:13:07.559918778 +0000 UTC, DownloadedAt: 2024-06-19 14:08:41.895521713 +0000 UTC\n2024-07-08T09:47:52.672Z\t\u001b[34mINFO\u001b[0m\tVulnerability scanning is enabled\n2024-07-08T09:47:52.672Z\t\u001b[35mDEBUG\u001b[0m\tVulnerability type: [os library]\n2024-07-08T09:47:52.716Z\t\u001b[35mDEBUG\u001b[0m\tImage ID: sha256:143e4da4d3690405a25286064def4a614650b904e8ec9a74ca5b01e3536f0e26\n2024-07-08T09:47:52.716Z\t\u001b[35mDEBUG\u001b[0m\tDiff IDs: ]\n2024-07-08T09:47:52.764Z\t\u001b[34mINFO\u001b[0m\tDetected OS: redhat\n2024-07-08T09:47:52.764Z\t\u001b[34mINFO\u001b[0m\tDetecting RHEL/CentOS vulnerabilities...\n2024-07-08T09:47:52.764Z\t\u001b[35mDEBUG\u001b[0m\tRed Hat: os version: 8\n2024-07-08T09:47:52.764Z\t\u001b[35mDEBUG\u001b[0m\tRed Hat: the number of packages: 195\n2024-07-08T09:47:52.805Z\t\u001b[34mINFO\u001b[0m\tNumber of language-specific files: 2\n2024-07-08T09:47:52.805Z\t\u001b[34mINFO\u001b[0m\tDetecting jar vulnerabilities...\n2024-07-08T09:47:52.805Z\t\u001b[35mDEBUG\u001b[0m\tDetecting library vulnerabilities, type: jar, path: \n2024-07-08T09:47:52.868Z\t\u001b[34mINFO\u001b[0m\tDetecting python-pkg vulnerabilities...\n2024-07-08T09:47:52.868Z\t\u001b[35mDEBUG\u001b[0m\tDetecting library vulnerabilities, type: python-pkg, path: \n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-22662": no vulnerability details for CVE-2022-22662\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26700": no vulnerability details for CVE-2022-26700\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26709": no vulnerability details for CVE-2022-26709\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26710": no vulnerability details for CVE-2022-26710\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26716": no vulnerability details for CVE-2022-26716\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26717": no vulnerability details for CVE-2022-26717\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-26719": no vulnerability details for CVE-2022-26719\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-30293": no vulnerability details for CVE-2022-30293\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-32792": no vulnerability details for CVE-2022-32792\n2024-07-08T09:47:52.873Z\t\u001b[33mWARN\u001b[0m\tError while getting vulnerability details: failed to get the vulnerability "CVE-2022-32816": no vulnerability details for CVE-2022-32816\n","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Removing scan report tmp file","path":"/home/scanner/.cache/reports/scan_report_2146663388.json","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating reports for scan job","scan_job_id":"0516feedd85aa9416ae7e13a","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:0516feedd85aa9416ae7e13a","scan_job_id":"0516feedd85aa9416ae7e13a","scan_job_status":"Pending","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating status for scan job","new_status":"Finished","scan_job_id":"0516feedd85aa9416ae7e13a","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:0516feedd85aa9416ae7e13a","scan_job_id":"0516feedd85aa9416ae7e13a","scan_job_status":"Finished","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Executing enqueued scan job","scan_job_id":"a83a94adf905b2c9938d589c","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"level":"debug","msg":"Updating status for scan job","new_status":"Pending","scan_job_id":"a83a94adf905b2c9938d589c","time":"2024-07-08T09:47:53Z"}
Jul 8 05:47:53 192.168.80.1 trivy-adapter[2525]: {"expire":3600,"level":"debug","msg":"Updating scan job","redis_key":"harbor.scanner.trivy:store:scan-job:a83a94adf905b2c9938d589c","scan_job_id":"a83a94adf905b2c9938d589c","scan_job_status":"Pending","time":"2024-07-08T09:47:53Z"}

can you please let me know why trivy logs are generating when we enable debug mode only and why scanning reports are deleted, is this default behavior of airgap environment ?