Protodetect probing mask32 7437 v2

[catenacyber]

Link to ticket: https://redmine.openinfosecfoundation.org/issues/
https://redmine.openinfosecfoundation.org/issues/7437
https://redmine.openinfosecfoundation.org/issues/7469

Describe changes:

• app-layer: improve limits on number of probing parsers (which was limited to 32 in total and worked out of luck with the current 37 protocols)
• smtp: improve probing parser in case of TLS first from client

SV_BRANCH=OISF/suricata-verify#2200

Closing #12256 to get this part first

#12233 with

• new commit for SMTP
• new SV test
• a bit more in the first commit message to explain the other accidental bugfix

This will require a QA baseline update as highlighted by the SV test

[suricata-qa]

WARNING:

field	baseline	test	%
	SURI_TLPW1_stats_chk
.app_layer.error.tls.parser	1152	1203	104.43%
	SURI_TLPR1_stats_chk
.app_layer.tx.ftp	95972	102934	107.25%
.ftp.memuse	3102	10661	343.68%

Pipeline 24024

[catenacyber]

A minimal change to get the same QA change should be

diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index a65a98c88a..b40c0adc44 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -619,7 +619,7 @@ again_midstream:
         else if (pp_port_sp)
             mask = pp_port_sp->alproto_mask;
 
-        if (alproto_masks[0] == mask) {
+        if ((alproto_masks[0] & mask) == mask) {
             FLOW_SET_PP_DONE(f, dir);
             SCLogDebug("%s, mask is now %08x, needed %08x, so done",
                     (dir == STREAM_TOSERVER) ? "toserver":"toclient",

meaning we have tried all expected probing parsers and maybe more...

[victorjulien]

A minimal change to get the same QA change should be

diff --git a/src/app-layer-detect-proto.c b/src/app-layer-detect-proto.c
index a65a98c88a..b40c0adc44 100644
--- a/src/app-layer-detect-proto.c
+++ b/src/app-layer-detect-proto.c
@@ -619,7 +619,7 @@ again_midstream:
         else if (pp_port_sp)
             mask = pp_port_sp->alproto_mask;
 
-        if (alproto_masks[0] == mask) {
+        if ((alproto_masks[0] & mask) == mask) {
             FLOW_SET_PP_DONE(f, dir);
             SCLogDebug("%s, mask is now %08x, needed %08x, so done",
                     (dir == STREAM_TOSERVER) ? "toserver":"toclient",

meaning we have tried all expected probing parsers and maybe more...

I don't understand this comment. What is the choice we have to make here? Which one is (more) correct?

[catenacyber]

I think this PR is more correct than the minimal change, but also more complex as it does more stuff, including this accidental fix during the refactoring.

But if you want to process by little steps, we can make the minimal change with its QA rebase line, and then rebase this PR which will not change anymore QA results

[victorjulien]

Will we want to backport? If so, smaller and well isolated steps would be preferred. Otherwise I'd say we can move ahead with this PR.

[catenacyber]

Will we want to backport?

Good question, I do not think we need to backport this as long as we do not add new app-layer protocols in 7.
But it is still a bug with some side effects... (not finishing protocol detection as soon as we can)

What do you think ?