Monitor audit trails in DataDog Cloud SIEM

[marcoieni]

1. Collect all audit logs from our infrastructure services
2. Ingest them into DataDog
3. Analyze them via Cloud SIEM
4. Send alerts

critical infrastructure

• Enable Rust Foundation Enterprise GitHub org audit logs
• Enable Rust Enterprise GitHub org audit logs
• Datadog audit trails events
• Configure AWS CloudTrail in the root account and send them to DataDog
  • Enable CloudTrail trusted access in AWS Organizations simpleinfra#1160
  • Allow Datadog to discover CloudTrail trails simpleinfra#1161
  • Deploy an organization CloudTrail for Datadog Cloud SIEM simpleinfra#1162
  • add wiz inventory permissions simpleinfra#1190
• Enable Wiz integration to send Wiz alerts to DataDog
• Fastly
• Google Workspace (rust-lang.org)
• Google Workspace (rustfoundation.org)
• Heroku
  • Audit trail are enterprise only (we aren't enterprise)
  • There are no Heroku integrations or content packs in Datadog
  • we already ship logs for activity in Heroku, we just need to add these logs to the filter of Cloud SIEM. the security team added the following filter to the index: OR (source:heroku @syslog.procid:api -"scheduler@addons.heroku.com")

non-critical infrastructure

• WP Engine (check if it's easy via a wordpress plugin)
• GCP (rust-lang.org and Rust Foundation organizations)
  • Useful links:
    • https://docs.datadoghq.com/integrations/google-cloud-platform/
    • https://docs.cloud.google.com/dataflow/docs/guides/templates/provided/pubsub-to-datadog
    • https://github.com/GoogleCloudPlatform/terraform-gcp-datadog-integration
    • https://docs.datadoghq.com/logs/guide/google-cloud-log-forwarding/
    • https://registry.terraform.io/providers/DataDog/datadog/latest/docs/resources/integration_gcp_sts
• Azure
• Crunchybridge
  • The three Crunchy Bridge clusters are already sending service:postgres logs.
  • There are AUDIT: SESSION pgAudit records in the logs, including read/write and administrative activity.

Discarded

• 1password: for events reporting we need 1password business plan. So I'm marking this as done because both the rust-lang and rust-foundation 1password vaults are on the team plan (so we can't send audit logs to datadog cloud siem).
• Mailgun: we want to get rid of it in the near future, so no need to monitor it.

[marcoieni]

You can only enable audit logs streaming for GitHub Enterprise. So since we just want logs for the RustLang organization (no rust analyzer, rust embedded, etc), we implemented this filter in Data Dog. In this way, logs from other organizations are not included.

If you want other organizations of the Rust Enterprise to be included, reach out to us and we will include your organization 👍

[marcoieni]

Datadog works (I evaluated a dummy rule to verify)

[marcoieni]

google workspace (Rust Project)

I enabled datadog for `rust-lang.org`. Let's check tomorrow if we see events at https://app.datadoghq.com/logs?query=source%3Agsuite&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=desc&viz=stream&from_ts=1786221507914&to_ts=1786394307914&live=true

EDIT: I see stuff!

EDIT 2:
Installed the google workspace siem content pack

Now the logs explorer shows event with the query source:gsuite datadog.cloud_siem:true ✅

Wiz

I enabled Wiz. in https://app.datadoghq.com/security/siem/content-packs?query=wiz&sp=%5B%7B%22p%22%3A%7B%22contentPackId%22%3A%22wiz%22%7D%2C%22i%22%3A%22security-content-pack-panel%22%7D%5D.

I see logs now!

fastly

I enabled the fastly cloud siem content pack. There is a warning but I don't think it's very important, since the integration is correctly configured. Probably the first log hasn't arrived yet.

Fastly account audit activity is exposed through the /events API. We should deploy an application that pulls from this API and pushes to datadog. This is too much complex, so I decided to not do it. Instead, I setup Fastly audit logs alerts: https://manage.fastly.com/observability/alerts/integrations

I received a test event after in https://app.datadoghq.com/event/explorer?query=source%3Afastly&agg_m=count&agg_m_source=base&agg_t=count&clustering_pattern_field_path=message&cols=&messageDisplay=expanded-lg&options=&refresh_mode=sliding&sort=DESC&from_ts=1786457300730&to_ts=1786458200730&live=true (filter source:fastly).

The rust foundation security team can create their own detection rules in cloud siem

Google Workspace (Rust Foundation)

Joel enabled the integration for me (I don't have admin access there).

I see logs 👍

AWS

with this script I confirmed that there is no AWS cloudtrail configured at the moment.

#!/bin/bash
profile_name="rust-root"

aws sts get-caller-identity \
  --profile "$profile_name" \
  --output json \
  --no-cli-pager

aws organizations list-aws-service-access-for-organization \
  --profile "$profile_name" \
  --query 'EnabledServicePrincipals[?ServicePrincipal==`cloudtrail.amazonaws.com`]' \
  --output json \
  --no-cli-pager

trail_arns=$(aws cloudtrail list-trails \
  --profile "$profile_name" \
  --region us-east-1 \
  --query 'Trails[].TrailARN' \
  --output text \
  --no-cli-pager)

if [ -z "$trail_arns" ] || [ "$trail_arns" = "None" ]; then
  echo "NO TRAILS FOUND"
else
  for trail_arn in $trail_arns; do
    trail_region=$(printf '%s\n' "$trail_arn" | cut -d: -f4)

    echo "=== TRAIL: $trail_arn ==="

    aws cloudtrail get-trail \
      --profile "$profile_name" \
      --region "$trail_region" \
      --name "$trail_arn" \
      --output json \
      --no-cli-pager

    aws cloudtrail get-trail-status \
      --profile "$profile_name" \
      --region "$trail_region" \
      --name "$trail_arn" \
      --output json \
      --no-cli-pager

    aws cloudtrail get-event-selectors \
      --profile "$profile_name" \
      --region "$trail_region" \
      --trail-name "$trail_arn" \
      --output json \
      --no-cli-pager
  done
fi

output:

{
    "UserId": "ID:marcoieni",
    "Account": "762490340130",
    "Arn": "arn:aws:sts::762490340130:assumed-role/AWSReservedSSO_AdministratorAccess_<ID>/marcoieni"
}
[]
NO TRAILS FOUND

After I configured cloudtrail in rust-lang/simpleinfra#1162 I see logs!

content pack works:

GCP

orgs and projects:

• Rust Foundation:
  • rust-crater
  • dev-desktops-prod
  • concrete-racer-468119-m7 (assets backup staging)
  • rust tuf https://console.cloud.google.com/welcome?pli=1&rapt=AEjHL4OWXwQv844g3Tw0MlOAcxkMdYBIPrhZHDGZsj15RHqMB3lx8VziwmwqxQz1VpBQkmc3H2dd0bsqZ0gJ3chHAZC0P0FwTDWdgUXzEqCeSrCc-YQpKAY&project=rust-tuf
  • security initiative https://console.cloud.google.com/welcome?pli=1&rapt=AEjHL4OWXwQv844g3Tw0MlOAcxkMdYBIPrhZHDGZsj15RHqMB3lx8VziwmwqxQz1VpBQkmc3H2dd0bsqZ0gJ3chHAZC0P0FwTDWdgUXzEqCeSrCc-YQpKAY&project=security-initiative
• Rust Project:
  • rustlang-gws-iac

[marcoieni]

We now monitor what we consider critical infrastructure. Now I'm unassigning this issue from me, as we can re-prioritize this issue in the future, when we have the time to provision monitoring of non-critical infrastructure as well 👍