[Security Solution] Normalizes filters field before rule diff comparison

[dplumlee]

Summary

Addresses #202966

The issue that causes the overarching problem mentioned in the ticket is that we add an extra alias: null property to the filter via the kibana filter utils instead of keeping the alias field unset. This is functionally the same rule but since the prebuilt rule objects are technically different (alias is set to undefined instead of null), we mark these rules as customized and causes the query fields to show as a modified field on update.

To address this, since changing the kibana util filter would be very invasive and touching a lot of code, we instead normalize the field on our side before version comparison. This fixes the bug reported and improves resiliency of rule upgrades in the future.

Testing (copied from ticket)

• Ensure the prebuiltRulesCustomizationEnabled feature flag is enabled
• Allow internal APIs via adding server.restrictInternalApis: false to kibana.dev.yaml
• Clear Elasticsearch data
• Run Elasticsearch and Kibana locally (do not open Kibana in a web browser)
• Install an outdated version of the security_detection_engine Fleet package

curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1

• Install prebuilt rules

curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform

• Open a threat_match rule for editing. For example Threat Intel Hash Indicator Match with rule_id aab184d3-72b3-4639-b242-6597c99d8bca.

With this fix, users should NOT see any extra fields in the rule upgrade flyout, nor should the rule be marked as "Modified" if opened and saved with no other modifications

Checklist

Check the PR satisfies following conditions.

Reviewers should verify this PR satisfies this list as well.

• Unit or functional tests were updated or added to match the most common scenarios

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

💔 Build Failed

• Buildkite Build
• Commit: 4e2724c

Failed CI Steps

• FTR Configs #1
• FTR Configs #1
• FTR Configs #115
• FTR Configs #115

Test Failures

• [job] [logs] FTR Configs #1 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI review prebuilt rules updates from package with mock rule assets kql_query fields when rule field has an update and a custom value that are different - scenario ABC when all versions are inline query type should show a non-solvable conflict in the upgrade/_review API response
• [job] [logs] FTR Configs #115 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI review prebuilt rules updates from package with mock rule assets kql_query fields when rule field has an update and a custom value that are different - scenario ABC when all versions are inline query type should show a non-solvable conflict in the upgrade/_review API response
• [job] [logs] FTR Configs #115 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI review prebuilt rules updates from package with mock rule assets kql_query fields when rule field has an update and a custom value that are different - scenario ABC when all versions are inline query type should show a non-solvable conflict in the upgrade/_review API response
• [job] [logs] FTR Configs #1 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI review prebuilt rules updates from package with mock rule assets kql_query fields when rule field has an update and a custom value that are different - scenario ABC when all versions are inline query type should show a non-solvable conflict in the upgrade/_review API response

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
securitySolution	22.2MB	22.2MB	+62.0B

History

cc @dplumlee