[New Integration] Greenhouse ATS audit logs #17347

narph · 2026-02-10T13:56:43Z

Summary

This PR adds a new integration to collect audit logs from Greenhouse Applicant Tracking System (ATS) via the Greenhouse Audit Log API.

Closes #16983

Features

CEL input with OAuth 2.0 Client Credentials authentication (Harvest API V3)
Cursor-based pagination using Pit-Id and Search-After headers
Time-based filtering for incremental data collection
Full ECS mapping including user, event, and source fields
GeoIP enrichment for source IP addresses
Configurable filtering by performer IDs and event types

Data Stream

audit: Collects all Greenhouse audit log event types:
- data_change_create - Record creation events
- data_change_update - Record modification events
- data_change_destroy - Record deletion events
- harvest_access - API access events
- action - User action events

Dashboards

[Greenhouse] Audit Logs Overview

Total event count and timeline
Event type distribution
User activity breakdown
Geographic distribution (GeoIP heatmap)
Request types and target types analysis
Performer types (User vs API)

[Greenhouse] Data Changes

Create/Update/Delete metrics
Data changes over time by type
Changes by target type treemap
User activity breakdown table
Most frequently changed fields
Recent data changes table

Testing

Pipeline tests with sample Greenhouse audit log events
System tests with mock HTTP service

Checklist

I have reviewed tips for building integrations
I have verified that all data streams collect metrics or logs
I have added an entry to changelog.yml
I have verified that Kibana version constraints are set
I have verified dashboards are working correctly

Made with Cursor

Add a new integration to collect audit logs from Greenhouse Applicant Tracking System (ATS) via the Greenhouse Audit Log API. Features: - CEL input with two-step JWT authentication flow - Cursor-based pagination with Pit-Id and Search-After headers - Time-based filtering for incremental data collection - Full ECS mapping including user, event, and source fields - GeoIP enrichment for source IP addresses - Pipeline and system tests with mock service Dashboards: - Audit Logs Overview: Event counts, timeline, user activity, geographic distribution, event types, request types, target types, performer types - Data Changes: Focused view on create/update/delete operations with field-level change tracking and user activity breakdown Co-authored-by: Cursor <[email protected]>

github-actions · 2026-02-10T13:58:03Z

Vale Linting Results

Summary: 1 warning found

⚠️ Warnings (1)

File	Line	Rule	Message
packages/greenhouse/docs/README.md	124	Elastic.Latinisms	Latin terms and abbreviations are a common source of confusion. Use 'using' instead of 'via'.

The Vale linter checks documentation changes against the Elastic Docs style guide.

To use Vale locally or report issues, refer to Elastic style guide for Vale.

Co-authored-by: Cursor <[email protected]>

Migrate from Audit Log V2 (deprecated August 2026) to Harvest API V3 with OAuth 2.0 Client Credentials authentication. Changes: - Replace harvest_api_key with client_id, client_secret, and user_id - Update token endpoint from /auth/jwt_access_token to /token - Use OAuth 2.0 Client Credentials grant_type with sub parameter - Update token response parsing for expires_at field - Update mock service and system tests for V3 flow - Update documentation with V3 setup instructions Co-authored-by: Cursor <[email protected]>

Co-authored-by: Cursor <[email protected]>

Add the docs README.md template with {{fields}} and {{event}} placeholders that get expanded during package build. This matches the structure used by other integrations. The template includes additional sections for rate limiting, data retention, and troubleshooting. Co-authored-by: Cursor <[email protected]>

The ecs.yml file is only needed for backward compatibility with Kibana versions prior to 8.14. Since this integration requires Kibana 8.18+, the explicit ECS field declarations are not needed. ECS fields are automatically recognized in newer Kibana versions. Co-authored-by: Cursor <[email protected]>

elasticmachine · 2026-02-10T15:46:18Z

💚 Build Succeeded

Buildkite Build
Commit: 7499aed

History

💚 Build #37865 succeeded 313ccce
💔 Build #37854 failed 9611d58

ShourieG

🤖 AI-Generated Review | Elastic Integration PR Review Bot

⚠️ This is an automated review generated by an AI assistant. Please verify all suggestions before applying changes. This review does not represent a human reviewer's opinion.

PR Review | elastic/integrations #17347

Field Mapping

✅ Reviewed - No actionable issues found.

Pipeline

Data Stream: `audit` (package: greenhouse)

File: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml

Issue 1: Date Processor Missing Error Handling
Severity: 🟡 Medium
Location: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml lines 37-43

Problem: The date processor lacks both a tag field and an on_failure handler. Date parsing can fail with malformed timestamps, and without error handling, the entire document ingestion will fail.

Recommendation:

- date:
    field: greenhouse.audit.event_time
    target_field: "@timestamp"
    formats:
      - "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'"
      - "yyyy-MM-dd'T'HH:mm:ss.SSSXXX"
      - ISO8601
    tag: parse_timestamp
    on_failure:
      - append:
          field: error.message
          value: 'Failed to parse timestamp from greenhouse.audit.event_time: {{{ _ingest.on_failure_message }}}'

Issue 2: Convert Processors Missing on_failure Handlers
Severity: 🟡 Medium
Location: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml lines 106, 121, 154, 161

Problem: Multiple convert processors have ignore_missing: true but lack on_failure handlers. Conversions can fail when data is in an unexpected format (e.g., invalid IP address, non-numeric string for ID), causing document ingestion to fail.

Recommendation:

- convert:
    field: greenhouse.audit.performer.id
    type: string
    target_field: user.id
    ignore_missing: true
    tag: convert_performer_id
    on_failure:
      - append:
          field: error.message
          value: 'Failed to convert performer.id to string: {{{ _ingest.on_failure_message }}}'

Apply similar fixes to:

Line 121: greenhouse.audit.performer.ip_address → source.ip (type: ip)
Line 154: greenhouse.audit.organization_id → organization.id (type: string)
Line 161: greenhouse.audit.event.target_id (type: string)

Issue 3: Global on_failure Handler Missing Processor Tag
Severity: 🟡 Medium
Location: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml line 232

Problem: The global error handler doesn't include _ingest.on_failure_processor_tag, making it harder to identify which processor failed during troubleshooting.

Recommendation:

- append:
    field: error.message
    value: 'Processor {{{ _ingest.on_failure_processor_tag }}} failed: {{{ _ingest.on_failure_message }}}'

Issue 4: JSON Processor Missing Tag
Severity: 🔵 Low
Location: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml line 17

Problem: The json processor should have a tag field for error identification and traceability.

Recommendation:

- json:
    field: event.original
    target_field: json
    tag: parse_json

Issue 5: GeoIP Processors Missing Tags
Severity: 🔵 Low
Location: packages/greenhouse/data_stream/audit/elasticsearch/ingest_pipeline/default.yml lines 132, 136

Problem: The geoip processors lack tag fields for better traceability during debugging.

Recommendation:

- geoip:
    field: source.ip
    target_field: source.geo
    ignore_missing: true
    tag: geoip_source_geo
- geoip:
    database_file: GeoLite2-ASN.mmdb
    field: source.ip
    target_field: source.as
    properties:
      - asn
      - organization_name
    ignore_missing: true
    tag: geoip_source_asn

💡 Suggestions

Consider adding event.created field to track when Greenhouse created the audit log record (if the API provides this field)
Consider adding event.outcome field to distinguish successful operations from failed attempts (if the API provides status information)

Input Configuration

Data Stream: `audit` (package: greenhouse)

File: packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs

Issue 1: Missing state.with() wrapper - cursor and token state not preserved on errors
Severity: 🟠 High
Location: packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs line 35

Problem: The entire CEL program is not wrapped in state.with(), which means cursor and token state will be lost when errors occur. Without state.with(), only explicitly returned fields are preserved. The token acquisition error branch (lines 56-68) doesn't preserve any state, and error branches in the main API request don't preserve cursor state. This can lead to data loss or duplicate ingestion after errors.

Recommendation:

program: |
  state.with(
    // Check if token needs refresh (expired or not present)
    (
      !has(state.token) || state.token == "" ||
      !has(state.token_expires) || timestamp(state.token_expires) < now
    ).as(need_token,
      // ... rest of the program
    )
  )

Issue 2: CEL program not formatted according to celfmt standards
Severity: 🔵 Low
Location: packages/greenhouse/data_stream/audit/agent/stream/cel.yml.hbs line 35

Problem: CEL program formatting does not match celfmt standards. While syntactically valid, the formatting differs from canonical format, affecting readability and consistency.

Recommendation:
Run celfmt on the CEL program to apply standard formatting for improved readability and consistency.

Transform

✅ No transform files in this PR.