[FR] [DAC] Add Exception Duplication Checking

[eric-forte-elastic]

Pull Request

Issue link(s):

Resolves: #5664

Summary - What I changed

Note

It is unclear and needs to be determined if item_ids are unique across spaces and can function as an identifier similarly to list_ids

Important

Kibana item_id uniqueness

Conclusion: item_id is not globally unique; it is unique within an exception list (within a given list_id). So we use (list_id, item_id) as the deduplication key.

• Create API: Returns 409 when an exception list item with the same item_id already exists in that list; docs do > not state global uniqueness.
• Import API: Overwrite by item_id is list-scoped; duplicate errors include both item_id and list_id.
• Identifier: Correct composite id for an item is (list_id, item_id) (and for space-scoped lists, effectively (space, > list_id, item_id)).

References:

• Create exception list item (409 for duplicate)
• Get exception list item (identify by id or item_id)
• Import exception list (overwrite by list_id and by item_id within list)
• Export exception list (space-scoped, list_id parameter)

1. New helper: _deduplicate_comments()

• Purpose: Removes duplicate comments on an exception item by comment text.
• Behavior: Takes an exception item dict, shallow-copies it, and builds a new comments list that keeps only the first occurrence of each comment (using a seen_texts set).
• Return: The copied item with comments replaced by the deduplicated list; the original is not mutated.

2. Changes to parse_exceptions_results_from_api()

• Item deduplication by item_id:
  • Added seen_item_ids: set[str] to track already-seen exception item IDs.
  • For each API result that is an exception item: if it has an item_id and that ID is already in seen_item_ids, the result is skipped (duplicate).
  • When an item is kept and has an item_id, that ID is added to seen_item_ids.
• Comment deduplication: Before appending an exception item, the code calls _deduplicate_comments(result) and appends the returned item instead of the raw result.

Result

• Items: Duplicate exception items from the Kibana export API (same item_id) are dropped before building TOML.
• Comments: Duplicate comments within each kept item are removed so the repo doesn’t accumulate repeated comments (e.g. auto-generated “pre-filled from alert” text).

How To Test

Inject duplicate comment

Use the prepared test scripts.

./detection_rules/etc/test_remote_cli.bash

./detection_rules/etc/test_cli.bash

Results

detection-rules on  5664-dac-fr-addressing-duplicate-exceptions [$?] is  v1.5.41 via  v3.12.12 (detection-rules-build) on  eric.forte 
❯ ./detection_rules/etc/test_remote_cli.bash
Running detection-rules remote CLI tests...
Performing a quick rule alerts search...
Requires .detection-rules-cfg.json credentials file set.
Loaded config file: /home/forteea1/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

No alerts detected
Setting Up Custom Directory...
Loaded config file: /home/forteea1/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

Created directory: tmp-custom/actions
Created directory: tmp-custom/action_connectors
Created directory: tmp-custom/exceptions
Created directory: tmp-custom/rules
Created directory: tmp-custom/rules_building_block
Created directory: tmp-custom/etc
Created file with default content: tmp-custom/etc/deprecated_rules.json
Created file with default content: tmp-custom/etc/version.lock.json
Created file with default content: tmp-custom/etc/packages.yaml
Created file with default content: tmp-custom/etc/stack-schema-map.yaml
Created file with default content: tmp-custom/etc/test_config.yaml
Created file with default content: tmp-custom/_config.yaml

# For details on how to configure the _config.yaml file,
# consult: /home/forteea1/tmp/detection-rules/detection_rules/etc/_config.yaml
# or the docs: /home/forteea1/tmp/detection-rules/docs-dev/custom-rules-management.md
Performing a rule conversion from ndjson to toml files...
Loaded config file: /home/forteea1/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

[+] Building rule for tmp-custom/rules/test_kql_rule.toml
[+] Building rule for tmp-custom/rules/test_kql_with_alert_supprestion_and_investigation_fileds.toml
[+] Building rule for tmp-custom/rules/test_kql_with_alert_suppression.toml
[+] Building rule for tmp-custom/rules/test_eql_rule.toml
[+] Building rule for tmp-custom/rules/test_esql_rule_with_shared_rule_exception.toml
/home/forteea1/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://127.0.0.1:9200' using TLS with verify_certs=False is insecure
  _transport = transport_class(
/home/forteea1/tmp/detection-rules/detection_rules/index_mappings.py:366: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  response = elastic_client.esql.query(query=query)
[+] Building rule for tmp-custom/rules/test_new_terms_rule_with_shared_rule_exception.toml
[+] Building rule for tmp-custom/rules/test_indicator_match_rule_with_email_actions.toml
[+] Building rule for tmp-custom/rules/test_threshold_with_rule_exception.toml
[+] Building rule for tmp-custom/rules/test_machine_learning_rule_with_index_action_connector.toml
[+] Building exception(s) for /home/forteea1/tmp/detection-rules/tmp-custom/exceptions/1c8a1378-8f0d-4565-9ae0-abeeaf3981ca_exceptions.toml
[+] Building exception(s) for /home/forteea1/tmp/detection-rules/tmp-custom/exceptions/0a4124f8-2074-450b-8689-d7dee319c666_exceptions.toml
[+] Building action connector(s) for /home/forteea1/tmp/detection-rules/tmp-custom/action_connectors/e1b418e7-78df-4042-bfb0-1cc5fb6f7a4e_actions.toml
[+] Building action connector(s) for /home/forteea1/tmp/detection-rules/tmp-custom/action_connectors/1b8d347f-2542-4390-85de-2653518311e2_actions.toml
15 results exported
9 rules converted
4 exceptions exported
2 actions connectors exported
Performing a rule import to kibana...
Loaded config file: /home/forteea1/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

/home/forteea1/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://127.0.0.1:9200' using TLS with verify_certs=False is insecure
  _transport = transport_class(
/home/forteea1/tmp/detection-rules/detection_rules/index_mappings.py:366: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  response = elastic_client.esql.query(query=query)
5 rule(s) successfully imported
 - 2cc8f325-e1b1-4201-8b8d-88a51c94992b
 - bcbd5906-fc38-4cbe-8b54-c2dba5d4b127
 - 2c6c5352-11cb-40a5-9294-e61ef5f1954f
 - 742feb36-ac4c-45e0-b8a5-3b3cfa66b6d2
 - d46a29ca-9b5b-4cbd-b11f-35c6b59f207b
4 rule(s) failed to import!
 - 7e0f6dae-5847-465f-89e9-a6de0e9ef918: (400) Rule with rule_id: "7e0f6dae-5847-465f-89e9-a6de0e9ef918" references a non existent exception list of list_id: "1c8a1378-8f0d-4565-9ae0-abeeaf3981ca". Reference has been removed.
 - 2390c9dd-ad90-4af6-97a4-1d607ba0f092: (400) Rule with rule_id: "2390c9dd-ad90-4af6-97a4-1d607ba0f092" references a non existent exception list of list_id: "1c8a1378-8f0d-4565-9ae0-abeeaf3981ca". Reference has been removed.
 - 8a3296e2-4a74-4d51-b819-8d4e58377bf7: (400) Your license does not support machine learning. Please upgrade your license.
 - 4c589d81-2622-4036-8cc7-372ea8f0e038: (404) Rule actions reference the following missing action IDs: 1b8d347f-2542-4390-85de-2653518311e2
Performing a rule export...
Loaded config file: /home/forteea1/tmp/detection-rules/.detection-rules-cfg.json

█▀▀▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄▄▄ ▄   ▄      █▀▀▄ ▄  ▄ ▄   ▄▄▄ ▄▄▄
█  █ █▄▄  █  █▄▄ █    █   █  █ █ █▀▄ █      █▄▄▀ █  █ █   █▄▄ █▄▄
█▄▄▀ █▄▄  █  █▄▄ █▄▄  █  ▄█▄ █▄█ █ ▀▄█      █ ▀▄ █▄▄█ █▄▄ █▄▄ ▄▄█

/home/forteea1/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://127.0.0.1:9200' using TLS with verify_certs=False is insecure
  _transport = transport_class(
/home/forteea1/tmp/detection-rules/detection_rules/index_mappings.py:366: ElasticsearchWarning: No limit defined, adding default limit of [1000]
  response = elastic_client.esql.query(query=query)
9 results exported
7 rules converted
1 exceptions exported
0 action connectors exported
7 rules saved to tmp-custom
1 exception lists saved to /home/forteea1/tmp/detection-rules/tmp-custom/exceptions
0 action connectors saved to /home/forteea1/tmp/detection-rules/tmp-custom/action_connectors
Testing ESQL Rules...
============================================================================================================== test session starts ===============================================================================================================
platform linux -- Python 3.12.12, pytest-9.0.2, pluggy-1.6.0
rootdir: /home/forteea1/tmp/detection-rules
configfile: pyproject.toml
plugins: typeguard-4.4.4
collected 13 items                                                                                                                                                                                                                               

tests/test_rules_remote.py .............                                                                                                                                                                                                   [100%]

================================================================================================================ warnings summary ================================================================================================================
tests/test_rules_remote.py: 14 warnings
  /home/forteea1/tmp/detection-rules/env/detection-rules-build/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py:399: SecurityWarning: Connecting to 'https://127.0.0.1:9200' using TLS with verify_certs=False is insecure
    _transport = transport_class(

tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_endpoint_alerts_index_endpoint_fields
tests/test_rules_remote.py::TestRemoteRules::test_esql_filtered_index
tests/test_rules_remote.py::TestRemoteRules::test_esql_non_dataset_package_related_integrations
tests/test_rules_remote.py::TestRemoteRules::test_esql_non_ecs_schema_conflict_resolution
tests/test_rules_remote.py::TestRemoteRules::test_esql_related_integrations
tests/test_rules_remote.py::TestRemoteRules::test_new_line_split_index
  /home/forteea1/tmp/detection-rules/detection_rules/index_mappings.py:366: ElasticsearchWarning: No limit defined, adding default limit of [1000]
    response = elastic_client.esql.query(query=query)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
======================================================================================================== 13 passed, 22 warnings in 13.60s ========================================================================================================
Removing generated files...
Detection-rules Remote CLI tests completed!

test_cli.txt

Checklist

• Added a label for the type of pr: bug, enhancement, schema, maintenance, Rule: New, Rule: Deprecation, Rule: Tuning, Hunt: New, or Hunt: Tuning so guidelines can be generated
• Added the meta:rapid-merge label if planning to merge within 24 hours
• Secret and sensitive material has been managed correctly
• Automated testing was updated or added to match the most common scenarios
• Documentation and comments were added for features that require explanation

Contributor checklist

• Have you signed the contributor license agreement?
• Have you followed the contributor guidelines?

[github-actions]

Enhancement - Guidelines

These guidelines serve as a reminder set of considerations when addressing adding a feature to the code.

Documentation and Context

• Describe the feature enhancement in detail (alternative solutions, description of the solution, etc.) if not already documented in an issue.
• Include additional context or screenshots.
• Ensure the enhancement includes necessary updates to the documentation and versioning.

Code Standards and Practices

• Code follows established design patterns within the repo and avoids duplication.
• Ensure that the code is modular and reusable where applicable.

Testing

• New unit tests have been added to cover the enhancement.
• Existing unit tests have been updated to reflect the changes.
• Provide evidence of testing and validating the enhancement (e.g., test logs, screenshots).
• Validate that any rules affected by the enhancement are correctly updated.
• Ensure that performance is not negatively impacted by the changes.
• Verify that any release artifacts are properly generated and tested.
• Conducted system testing, including fleet, import, and create APIs (e.g., run make test-cli, make test-remote-cli, make test-hunting-cli)

Additional Checks

• Verify that the enhancement works across all relevant environments (e.g., different OS versions).
• Confirm that the proper version label is applied to the PR patch, minor, major.