userguide: explain rule types and categorization - v8

[jufajardini]

Previous PR: #12114

Link to ticket: https://redmine.openinfosecfoundation.org/issues/
https://redmine.openinfosecfoundation.org/issues/7031

Please check the built version: https://suri-rtd-test.readthedocs.io/en/doc-sigtypes-et-properties-v8/rules/rule-types.html

Describe changes:

• Move to its own section under Suricata Rules
• Add engine analysis report examples for each rule type
  • Showcase having these examples collapsed, as some can be quite lengthy
• Another try at userguide: explain rule types and categorization - v7 #12114 (comment), I think this is more correct, but a strict read of this table, especially for stream and packet-stream rule types is very welcome
• Newer version of the flowchart for SignatureSetType(). Add the drawio file to version control - makes maintaining easier.
• Add explanation about different outcomes of using Flowvars and other similar keywords
• Add a brief explanation about the existence of rules that require real packets (more work to be done here with https://redmine.openinfosecfoundation.org/issues/7424), trying to tackle detect: don't run pkt sigs on ffr pkts #12095 (comment) -- what do you think, @catenacyber ?
• add a table/ list with human readable rule types

Many of the examples and conclusions documented here were derived from tests and checks as seen on OISF/suricata-verify#2153

[jufajardini]

Moved to draft as I see too many CI failures.
Still apt for a review, though, please.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 83.17%. Comparing base (ae10fc3) to head (c7ff988).
Report is 5 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #12184   +/-   ##
=======================================
  Coverage   83.16%   83.17%           
=======================================
  Files         912      912           
  Lines      257066   257066           
=======================================
+ Hits       213797   213818   +21     
+ Misses      43269    43248   -21     

Flag	Coverage Δ
fuzzcorpus	61.01% <ø> (+0.03%)	⬆️
livemode	19.41% <ø> (ø)
pcap	44.39% <ø> (-0.04%)	⬇️
suricata-verify	62.75% <ø> (+0.05%)	⬆️
unittests	59.18% <ø> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

WARNING:

field	baseline	test	%
	build_asan

Pipeline 23611

[inashivb]

The docs look neat and well organized. If you clean the PR, this looks pretty good to me.
Will let others review the content too.

[jufajardini]

Clang and commit errors were due to a mistakenly added libhtp build workflow? I'll never know how, but fixed.

Still trying to figure out how to fix the Almalinux 9 failure. Must find the right package for sphinx-toolbox there...

[jufajardini]

Clang and commit errors were due to a mistakenly added libhtp build workflow? I'll never know how, but fixed.

Still trying to figure out how to fix the Almalinux 9 failure. Must find the right package for sphinx-toolbox there...

Not sure I'll be able to, as I couldn't find the package for that distro. I went back to #9165 to see if @jasonish had found a solution for displaying very long json/ schema stuff, but failed to find something.

So, the next question is: do we keep large --engine-analysis output examples, or should I look for shorter ones?

[jasonish]

Doesn't look like sphinx-toolbox is universally available in system packages? And we'd at least need it on the Dist Builder which is currently Ubuntu 22.04.

[jasonish]

Not sure I'll be able to, as I couldn't find the package for that distro. I went back to #9165 to see if @jasonish had found a solution for displaying very long json/ schema stuff, but failed to find something.

I can't remember what the issue was, but this work did get merged.

[jufajardini]

Not sure I'll be able to, as I couldn't find the package for that distro. I went back to #9165 to see if @jasonish had found a solution for displaying very long json/ schema stuff, but failed to find something.

I can't remember what the issue was, but this work did get merged.

But I think with your work you didn't use anything to collapse large sections, right?

[jufajardini]

Doesn't look like sphinx-toolbox is universally available in system packages? And we'd at least need it on the Dist Builder which is currently Ubuntu 22.04.

It works locally and on read the docs, but I'd rather have our docs looking the same - as much as possible - everywhere. I'll submit a new PR removing the sphinx-toolbox dependency.

[jufajardini]

collapsible sections removed with #12209

[jasonish]

But I think with your work you didn't use anything to collapse large sections, right?

No, it didn't.