Logging level doesn't work on Windows

[derevnjuk]

Description:

SecTester doesn’t print any logs to the stdout, no matter what logging level was used in Configuration. Only stderr logs are shown.

pic.1

It looks like this on *nix systems:

pic.2

Steps to reproduce:

1. Write a simple test
2. Set Configuration property LogLevel to LogLevel.Trace
3. Run on a Windows machine using the latest .Net 7.0.1 using one of these commands

dotnet test -c Debug -v 4
# or
dotnet  test -c Debug -v 4
# or
dotnet  test -c Debug --logger "console;verbosity=detailed"
# or
dotnet  test -c Debug --logger "console;verbosity=detailed" -v 4

Actual result:

Test run for C:\Users\ogtux\OneDrive\Desktop\NetSecTester 2\NetSecTester\bin\Debug\net7.0\NetSecTester.dll (.NETCoreApp,Version=v7.0)
Microsoft (R) Test Execution Command Line Tool Version 17.4.0 (arm64)
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
C:\Users\ogtux\OneDrive\Desktop\NetSecTester 2\NetSecTester\bin\Debug\net7.0\NetSecTester.dll
[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.5+1caef2f33e (64-bit .NET 7.0.1)
[xUnit.net 00:00:00.86]   Discovering: NetSecTester
[xUnit.net 00:00:00.90]   Discovered:  NetSecTester
[xUnit.net 00:00:00.90]   Starting:    NetSecTester
[xUnit.net 00:02:04.32]     TestProject3.SecurityTests.Post_BrokencrystalsApiMetadata_ShouldHaveXxZ [FAIL]
[xUnit.net 00:02:04.32]       SecTester.Runner.IssueFound : Target is vulnerable

Issue in Bright UI:   https://hotel.playground.brightsec.com/scans/uiX6gjK5ScmVghgdAZZHW4/issues/x4zK7ZUtBpgiVaMUwJuPpB
[xUnit.net 00:02:04.32]       Name:                 Directory Listing
[xUnit.net 00:02:04.32]       Severity:             Medium
[xUnit.net 00:02:04.32]       Remediation:
[xUnit.net 00:02:04.32]       Configure the web server running the application to disable directory listing for all pages
[xUnit.net 00:02:04.32]       Details:
[xUnit.net 00:02:04.32]       The server allows directory listing, which enables the exposure of files that might not be available   
[xUnit.net 00:02:04.32]       via external links or might not be intended for website access.
[xUnit.net 00:02:04.32]       Extra Details:
[xUnit.net 00:02:04.32]       ? Directory listings found at the following URLs:
[xUnit.net 00:02:04.32]          https://qa.brokencrystals.com/vendor/
[xUnit.net 00:02:04.32]           - https://qa.brokencrystals.com/vendor/assets/
[xUnit.net 00:02:04.32]           - https://qa.brokencrystals.com/vendor/assets/vendor/
[xUnit.net 00:02:04.32]           - https://qa.brokencrystals.com/vendor/assets/vendor/owl.carousel/
[xUnit.net 00:02:04.32]           - https://qa.brokencrystals.com/vendor/assets/vendor/owl.carousel/assets/
[xUnit.net 00:02:04.32]
[xUnit.net 00:02:04.32]       References:
[xUnit.net 00:02:04.32]       ? https://cwe.mitre.org/data/definitions/548.html
[xUnit.net 00:02:04.32]       ? https://www.owasp.org/index.php/Top_10-2017_A6-Security_Misconfiguration
[xUnit.net 00:02:04.32]       ? https://kb.neuralegion.com/#/guide/vulnerabilities/31-default-login-location.md
[xUnit.net 00:02:04.32]       Stack Trace:
[xUnit.net 00:02:04.32]         /home/runner/work/sectester-net/sectester-net/src/SecTester.Runner/SecScan.cs(64,0): at SecTester.Runner.SecScan.Assert(IScan scan)
[xUnit.net 00:02:04.32]         /home/runner/work/sectester-net/sectester-net/src/SecTester.Runner/SecScan.cs(36,0): at SecTester.Runner.SecScan.Run(Target target, CancellationToken cancellationToken)
[xUnit.net 00:02:04.32]         /home/runner/work/sectester-net/sectester-net/src/SecTester.Runner/SecScan.cs(40,0): at SecTester.Runner.SecScan.Run(Target target, CancellationToken cancellationToken)
[xUnit.net 00:02:04.32]         /home/runner/work/sectester-net/sectester-net/src/SecTester.Runner/SecScan.cs(40,0): at SecTester.Runner.SecScan.Run(Target target, CancellationToken cancellationToken)
[xUnit.net 00:02:04.32]         C:\Users\ogtux\OneDrive\Desktop\NetSecTester 2\NetSecTester\UnitTest1.cs(74,0): at TestProject3.SecurityTests.Post_BrokencrystalsApiMetadata_ShouldHaveXxZ()
[xUnit.net 00:02:04.32]         --- End of stack trace from previous location ---
  Failed TestProject3.SecurityTests.Post_BrokencrystalsApiMetadata_ShouldHaveXxZ [2 m 3 s]
  Error Message:
   SecTester.Runner.IssueFound : Target is vulnerable

This issue ends up with the following:

• CLI test runners do not rely on capturing the console output due to internal reasons (see xUnit explanation https://xunit.net/docs/capturing-output).
• Console log appearance on *nix derived OSes could be a side effect since terminal context is shared between all the processes forked from the CLI test runner.
• CI runners could parse the CLI test runner output while running tests to provide failed test case reports for developers. The console log from our logger may overlap with CLI test runner output and introduce unwanted entries in such a report. A non-console-based logging is also possible.

To solve this issue, we should introduce a logger provider for each test framework implementing the ILoggerProvider interface. This will be addressed as part of introducing test framework-specific wrappers.