🐛 Bug - Code Auditor failed on Website

[jeoffreyfischer]

Cc: @zacharykeeping

Hi

Describe the Bug

I ran Code Auditor several times this week and it:

• ran for 6h - normally it should only run for 1h
• failed

To Reproduce

Steps to reproduce the behavior:

1. Go to https://github.com/SSWConsulting/SSW.Website/actions/workflows/weekly-codeauditor-scan.yml
2. Click on 'Run workflow'
3. Wait a few hours
4. See failed job

Expected Behavior

The job should successfully complete in about 1h

Tasks

• Investigate
• Fix

More Information

Environment

• Device: MSI laptop
• Browser: Chrome
• OS: Windows

Screenshots

Figure: Code Auditor ran for hours and failed twice

Figure: Error logged

Thanks!

[Calinator444]

The bug seems to be occurring in sswlinkauditor.go. I've tried running it locally and based on the URLs in the logs it's crawling URLS that never existed on SSW's Website.

e.g. https://www.ssw.com.au/ssw/Consulting/consulting/artificial-intelligence/consulting/process-mining/consulting/microsoft-365

Note: the link above redirects to our consulting page, so link auditor may be treating the link as unique but reading the response from the consulting page.

I think the next course of action is to visit the pages where sswlinkauditor is finding these URLs to verify that they exist.

Figure: No WayBackMachine record for the URL listed above

Figure: Strange url in link auditor outputs

[Calinator444]

I believe I've diagnosed the issue:

Any traffic under /ssw/Consulting gets redirected to our current consulting page, but code auditor seems to treat redirects as unique pages.

Essentially, Code Auditor keeps trying to crawl our consulting index page after getting redirected to it and then gets redirected again after accessing the links because they redirect back to the home page. This results in an infinite loop.

[Calinator444]

I re-ran the scan after adding in the fix to handle redirects

It scanned a little over 28000 links, on the main domain
given that the last successful code auditor scan included 28,552 links, this result appears to be accurate

Figure: Last successful code auditor scan

although the number of broken links has climbed to approximately

[Calinator444]

I was able to successfully build the docker image and run a scan on the website, but need my changes approved before I can wrap this up.

There are also a large number of broken links on the website (1,032), many of which are caused because we didn't migrate the client side redirects during the v3-migration. I recommend we create PBIs to fix this soon.

https://codeauditor.com/build/b4e3dc61-a3f4-4bab-2817-33afbf8ca9dd