Skip to content

Grafana Dashboard Generator #5

Open
Open
Grafana Dashboard Generator#5

Description

@pristley
pristley
opened on Jun 27, 2026

Feature Spec: Grafana Dashboard Generator

1. Overview

Automatically generate production-ready Grafana dashboards from NeuralBudget SLO DAGs and aggregation results. Eliminate manual dashboard creation by mapping calculated metrics and SLO states directly to Grafana JSON format.

2. Problem Statement

  • SREs manually recreate dashboards for each SLO DAG (error-prone, time-consuming)
  • Metrics calculated in NeuralBudget don't automatically visualize
  • Dashboard definitions drift from actual SLO definitions
  • No standard way to share dashboard templates across teams
  • Inconsistent dashboard design patterns
  • Manual panel configuration (queries, thresholds, colors)

3. Goals

  1. Auto-Generate: Create Grafana dashboards from SLO DAGs (zero manual panels)
  2. Dynamic: Update dashboards when DAGs change
  3. Professional: Production-ready design with best practices
  4. Themeable: Customizable colors, layouts, panel types
  5. Shareable: Export as JSON for version control
  6. Integrated: Native Grafana provisioning via ConfigMaps
  7. Compliant: Full Grafana 8.x+ compatibility

4. Architecture

SLO DAG Definition
    ↓
Dashboard Generator Service
    ├─ Parse DAG nodes/edges
    ├─ Generate panel definitions
    ├─ Apply theming rules
    ├─ Map to Grafana queries
    └─ Compile to JSON
    ↓
Grafana Dashboard JSON
    ↓
Output Formats:
    ├─ Direct HTTP download
    ├─ Grafana API import
    ├─ Kubernetes ConfigMap (provisioning)
    ├─ Git commit (GitOps)
    └─ S3 bucket (archival)

5. Core Features

5.1 DAG to Dashboard Mapping

SLO DAG Structure:
├─ metric_query nodes → Grafana graph/stat panels
├─ threshold nodes → Gauge/threshold panels
├─ arithmetic nodes → Calculated metrics panels
├─ alert nodes → Alert list panels
└─ edges → Panel layout guidance

Auto-Generated:
├─ 1 dashboard per SLO DAG (configurable)
├─ 1 row per logical group (metric queries, alerts, SLO status)
├─ 1 panel per DAG node
├─ Queries pre-populated from metric definitions
├─ Thresholds from SLO targets
└─ Annotations from DAG metadata

5.2 Panel Types

Node Type → Panel Type:

metric_query (single series)     → Graph (Time Series)
metric_query (multi-series)      → Graph with legend
metric_query (rate/derivative)   → Bar chart
threshold                        → Gauge + threshold line
arithmetic                       → Stat panel with formula
alert                           → Alert list + firing state
slo_validator                   → Stat panel (SLO %age)
custom_expression               → Text panel or graph

5.3 Query Translation

DAG Node Definition:
{
  "id": "latency_check",
  "type": "metric_query",
  "config": {
    "metric": "http_request_duration_seconds",
    "selectors": {
      "service": "payment-service",
      "percentile": "0.99"
    },
    "rate_window": "5m"
  }
}

Generated Grafana Query:
{
  "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service=\"payment-service\"}[5m]))",
  "legendFormat": "p99 Latency",
  "refId": "A",
  "datasourceUid": "prometheus-uid"
}

6. API Specification

6.1 Dashboard Generation Endpoints

# Generate dashboard from SLO DAG
POST /api/v1/dashboards/generate
Request:
{
  "slo_dag_id": "payment-service-slo",
  "namespace": "observability",
  "style": "dark",  # light|dark
  "datasource": "prometheus-main",
  "panel_types": {
    "metric_query": "graph",     # Override default
    "threshold": "gauge"
  }
}

Response:
{
  "dashboard_json": {...},
  "uid": "payment-service-slo-abc123",
  "title": "Payment Service SLO Dashboard",
  "description": "Auto-generated from SLO DAG",
  "tags": ["slo", "payment-service"],
  "panels_count": 8,
  "rows_count": 3
}

# Export dashboard as JSON file
GET /api/v1/dashboards/{slo_dag_id}/export
Query params:
  ?format=json|yaml
  &include_provisioning=true
  &include_variables=true

Response: Raw Grafana dashboard JSON (download)

# Preview dashboard before saving
POST /api/v1/dashboards/preview
Request: Same as /generate
Response: Same as /generate (no persistence)

# Sync dashboard to Grafana instance
POST /api/v1/dashboards/{slo_dag_id}/sync
Request:
{
  "grafana_url": "https://grafana.company.com",
  "grafana_api_token": "...",
  "folder_id": 42,
  "overwrite": true
}

Response:
{
  "dashboard_id": 123,
  "dashboard_url": "https://grafana.company.com/d/abc123",
  "status": "created|updated"
}

# List generated dashboards
GET /api/v1/dashboards?tag=slo&namespace=observability

# Delete generated dashboard
DELETE /api/v1/dashboards/{slo_dag_id}

6.2 Customization API

# Get dashboard customization template
GET /api/v1/dashboard-templates/{slo_dag_id}

Response:
{
  "layout": {
    "columns": 12,
    "rowHeight": 8
  },
  "panels": [
    {
      "node_id": "latency_check",
      "panel_type": "graph",
      "position": {
        "h": 8,
        "w": 6,
        "x": 0,
        "y": 0
      },
      "overrides": {
        "yAxisMin": 0,
        "yAxisMax": 1000,
        "colors": ["#1f77b4"]
      }
    }
  ],
  "rows": [
    {
      "title": "Metrics",
      "panels": ["latency_check", "error_rate_check"]
    }
  ]
}

# Update customization
PUT /api/v1/dashboard-templates/{slo_dag_id}
Request: Modified template

7. Grafana JSON Schema

7.1 Dashboard Structure

{
  "dashboard": {
    "id": null,
    "uid": "payment-service-slo-abc123",
    "title": "Payment Service SLO Dashboard",
    "description": "Auto-generated from SLO DAG payment-service-slo",
    "tags": ["slo", "payment-service", "neuralbudget-generated"],
    "timezone": "browser",
    "refresh": "30s",
    "time": {
      "from": "now-7d",
      "to": "now"
    },
    "templating": {
      "list": [
        {
          "name": "datasource",
          "type": "datasource",
          "datasource": "prometheus",
          "current": {"value": "prometheus-main", "text": "Prometheus"}
        },
        {
          "name": "service",
          "type": "const",
          "value": "payment-service",
          "hide": 2
        }
      ]
    },
    "panels": [...],
    "rows": [...],
    "schemaVersion": 36,
    "version": 1
  },
  "overwrite": false
}

7.2 Panel Definitions

{
  "id": 1,
  "title": "Request Latency (p99)",
  "type": "timeseries",
  "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
  "targets": [
    {
      "expr": "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{service=\"$service\"}[5m]))",
      "legendFormat": "p99",
      "refId": "A",
      "datasourceUid": "prometheus-uid"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "color": {"mode": "palette-classic"},
      "custom": {
        "axisCenteredZero": false,
        "axisPlacement": "auto",
        "barAlignment": 0,
        "drawStyle": "line",
        "fillOpacity": 10,
        "gradientMode": "none",
        "hideFrom": {"tooltip": false, "viz": false, "legend": false},
        "lineInterpolation": "linear",
        "lineWidth": 1,
        "pointSize": 5,
        "scaleDistribution": {"type": "linear"},
        "showPoints": "never",
        "spanNulls": false,
        "stacking": {"group": "A", "mode": "none"},
        "thresholdsStyle": {"mode": "off"}
      },
      "mappings": [],
      "max": 1000,
      "min": 0,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "green", "value": null},
          {"color": "yellow", "value": 200},
          {"color": "red", "value": 500}
        ]
      },
      "unit": "ms"
    }
  },
  "options": {
    "legend": {"calcs": ["mean", "max"], "displayMode": "table", "placement": "right"},
    "tooltip": {"mode": "multi"}
  }
}

7.3 Stat Panel (for SLO percentage)

{
  "id": 5,
  "title": "SLO Compliance",
  "type": "stat",
  "gridPos": {"h": 8, "w": 6, "x": 0, "y": 8},
  "targets": [
    {
      "expr": "(1 - rate(errors_total[$__range])) * 100",
      "refId": "A",
      "datasourceUid": "prometheus-uid"
    }
  ],
  "fieldConfig": {
    "defaults": {
      "mappings": [
        {
          "type": "value",
          "options": {"1": {"text": "OK", "color": "green"}}
        }
      ],
      "max": 100,
      "min": 0,
      "thresholds": {
        "mode": "absolute",
        "steps": [
          {"color": "red", "value": null, "value": 99},
          {"color": "yellow", "value": 99},
          {"color": "green", "value": 99.5}
        ]
      },
      "unit": "percent"
    }
  },
  "options": {
    "colorMode": "background",
    "graphMode": "none",
    "justifyMode": "auto",
    "orientation": "auto",
    "reduceOptions": {"values": false, "fields": "", "calcs": ["lastNotNull"]},
    "textMode": "auto"
  }
}

7.4 Alert List Panel

{
  "id": 8,
  "title": "Active Alerts",
  "type": "alertlist",
  "gridPos": {"h": 8, "w": 12, "x": 0, "y": 16},
  "options": {
    "alertName": "",
    "dashboardAlerts": true,
    "dashboardTitle": "",
    "datasource": "-- Grafana --",
    "folderId": null,
    "maxItems": 10,
    "options": {"sortOrder": 1},
    "showOptions": "current",
    "sortOrder": 1,
    "stateFilter": ["alerting", "no_data"],
    "tags": ["slo", "$service"]
  }
}

8. Theming & Customization

8.1 Built-in Themes

themes:
  professional:
    colors:
      success: "#299c46"
      warning: "#ff9830"
      critical: "#d44a3a"
    layout:
      columns: 12
      rowHeight: 8
    panels:
      default_height: 8
      default_width: 6
      
  minimal:
    colors:
      success: "#3274d9"
      warning: "#f4d5d5"
      critical: "#ff6b6b"
    layout:
      columns: 24
      rowHeight: 4
    panels:
      default_height: 4
      default_width: 4
      
  dark:
    colors:
      background: "#1a1a1a"
      text: "#ffffff"
      success: "#73bf69"
      warning: "#fac858"
      critical: "#ee6666"

8.2 Custom Theme Definition

{
  "name": "company-branding",
  "colors": {
    "primary": "#0066cc",
    "secondary": "#ffffff",
    "success": "#00a651",
    "warning": "#ff9900",
    "critical": "#cc0000"
  },
  "fonts": {
    "family": "Inter, sans-serif",
    "sizes": {"small": 12, "medium": 14, "large": 16}
  },
  "layout": {
    "spacing": 8,
    "borderRadius": 4
  },
  "panels": {
    "defaultHeight": 8,
    "defaultWidth": 6,
    "borders": true,
    "shadows": true
  }
}

9. Template Variables

9.1 Auto-Generated Variables

From DAG selectors:
├─ $datasource → Prometheus instance
├─ $service → From spec.selector.service
├─ $region → From spec.selector.region
├─ $environment → From spec.selector.environment
├─ $interval → Time aggregation window
└─ $range → Query time range

Variable Types:
├─ const (hidden, service name)
├─ datasource (Prometheus picker)
├─ query (dynamic from metric labels)
└─ custom (SLO threshold values)

9.2 Variable Definition

{
  "name": "service",
  "type": "const",
  "current": {"value": "payment-service", "text": "payment-service"},
  "hide": 2,
  "skipUrlSync": false
}

10. Dashboard Provisioning

10.1 Kubernetes ConfigMap Provisioning

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboard-payment-slo
  namespace: observability
  labels:
    grafana_dashboard: "1"
data:
  payment-service-slo.json: |
    {
      "dashboard": {
        "title": "Payment Service SLO Dashboard",
        ...
      },
      "overwrite": true
    }
---
# Grafana watches ConfigMaps with label grafana_dashboard=1
# and auto-imports them as dashboards

10.2 Grafana Provisioning File

# /etc/grafana/provisioning/dashboards/neuralbudget.yaml
apiVersion: 1
providers:
  - name: 'NeuralBudget Dashboards'
    orgId: 1
    folder: 'SLO'
    type: file
    disableDeletion: false
    editable: true
    options:
      path: /var/lib/grafana/dashboards/neuralbudget

11. Configuration

11.1 Generation Config

dashboard_generator:
  # Default settings
  defaults:
    theme: "professional"
    refresh_interval: "30s"
    time_range: "7d"
    max_data_points: 1000
    
  # Panel defaults
  panels:
    default_type: "timeseries"  # for metric_query nodes
    line_style: "solid"
    line_width: 1
    point_size: 5
    fill_opacity: 10
    
  # Theming
  colors:
    success: "#299c46"
    warning: "#ff9830"
    critical: "#d44a3a"
    background: "#ffffff"
    
  # Layout
  layout:
    columns: 12
    row_height: 8
    panel_height: 8
    panel_width: 6
    
  # Export options
  export:
    include_uid: true
    include_version: true
    include_panels: true
    include_rows: true
    
  # Grafana sync
  grafana:
    enabled: true
    url: "https://grafana.company.com"
    api_token_secret: "grafana-api-token"
    overwrite_existing: true
    default_folder: "SLO Dashboards"

12. Implementation Details

12.1 Generation Algorithm

Algorithm: GenerateDashboard(dag, config)

1. ValidateDAG(dag)
   └─ Check syntax, references

2. GeneratePanels(dag.nodes):
   for each node in dag.nodes:
     - Determine panel type based on node.type
     - Generate Prometheus query from node.config
     - Apply thresholds from SLO targets
     - Apply styling from config
     └─ Add to panels list

3. GenerateRows(dag.edges):
   - Group related panels by edge connectivity
   - Assign positions (x, y, w, h)
   - Create row containers
   └─ Order for visual flow

4. GenerateVariables(dag.spec):
   for each selector key:
     - Create variable (const or query)
     - Set default value
     - Apply label filters
     └─ Add to variables list

5. GenerateAnnotations(dag.metadata):
   - Add SLO thresholds as annotations
   - Add alert firing patterns
   └─ Show on time series

6. CompileDashboard():
   - Merge all components
   - Apply theme
   - Validate Grafana schema
   └─ Return JSON

Time: O(n+e) where n=nodes, e=edges
Output size: ~50KB per dashboard (typical)

12.2 Query Builder

pub fn build_prometheus_query(node: &MetricQueryNode) -> String {
    let metric = &node.config.metric;
    let selectors = &node.config.selectors;
    let percentile = node.config.percentile;
    
    // Build selector string: {label1="value1", label2="value2"}
    let selector_str = selectors
        .iter()
        .map(|(k, v)| format!("{}=\"{}\"", k, v))
        .collect::<Vec<_>>()
        .join(", ");
    
    // Determine query type
    if percentile.is_some() {
        // Histogram quantile for percentiles
        format!(
            "histogram_quantile({}, rate({}bucket{{{}}}[{}]))",
            percentile.unwrap(),
            metric,
            selector_str,
            node.config.rate_window.unwrap_or("5m")
        )
    } else if node.config.is_rate {
        // Rate for counters
        format!("rate({}{{{}}}[{}])", metric, selector_str, node.config.rate_window.unwrap_or("5m"))
    } else {
        // Direct metric query
        format!("{}{{{}}} ", metric, selector_str)
    }
}

13. Export Formats

13.1 JSON Export

# Download Grafana dashboard JSON
GET /api/v1/dashboards/payment-service-slo/export?format=json

13.2 YAML Export

# Dashboard definition in YAML (for version control)
dashboard:
  uid: payment-service-slo-abc123
  title: Payment Service SLO Dashboard
  tags: [slo, payment-service]
  panels:
    - id: 1
      title: Request Latency
      type: timeseries
      targets:
        - expr: "histogram_quantile(0.99, ...)"

13.3 CloudDash Format

// Export for CloudDash provisioning
{
  "dashboards": [
    {
      "dashboard": {...},
      "overwrite": true,
      "tags": ["auto-generated"]
    }
  ]
}

14. Versioning & Change Tracking

14.1 Dashboard Versions

Dashboard Versions:
├─ v1 (created 2024-01-15T10:30Z from SLO DAG v1)
├─ v2 (created 2024-01-16T14:20Z from SLO DAG v2)
│  └─ Changes: Added error rate panel, updated thresholds
├─ v3 (created 2024-01-17T09:15Z from SLO DAG v3)
│  └─ Changes: Modified layout, changed color scheme
└─ current → v3

Track:
- SLO DAG version that generated it
- Timestamp of generation
- Changes from previous version
- Manual edits (if any)

14.2 Version Storage

{
  "uid": "payment-service-slo-abc123",
  "title": "Payment Service SLO Dashboard",
  "version": 3,
  "versions": [
    {
      "version": 1,
      "created_at": "2024-01-15T10:30:00Z",
      "created_from_dag_version": 1,
      "changes": ["initial creation"],
      "dashboard_json": {...}
    },
    {
      "version": 2,
      "created_at": "2024-01-16T14:20:00Z",
      "created_from_dag_version": 2,
      "changes": ["added error rate panel", "updated SLO threshold"],
      "dashboard_json": {...}
    },
    {
      "version": 3,
      "created_at": "2024-01-17T09:15:00Z",
      "created_from_dag_version": 3,
      "changes": ["layout reorganization", "color scheme update"],
      "dashboard_json": {...}
    }
  ]
}

15. Integration Points

15.1 SLO DAG Integration

SLO DAG Update → Trigger regeneration
  ↓
Detect changes (nodes, edges, thresholds)
  ↓
Regenerate dashboard
  ↓
Compare with previous version
  ↓
If changed: Sync to Grafana/ConfigMap
  ↓
Notify users of dashboard update

15.2 Kubernetes Operator Integration

# SLODag CRD includes dashboard generation
apiVersion: neuralbudget.io/v1
kind: SLODag
metadata:
  name: payment-service-slo
spec:
  # ... DAG spec ...
  
  # Dashboard generation
  dashboard:
    enabled: true
    theme: "professional"
    provisioning:
      type: "configmap"  # configmap | grafana_api | s3
      configmap_name: "grafana-dashboard-payment-slo"
      namespace: "observability"
    sync:
      enabled: true
      interval: "5m"
      grafana_instance: "main"

status:
  dashboard:
    uid: "payment-service-slo-abc123"
    url: "https://grafana.company.com/d/payment-service-slo-abc123"
    version: 3
    synced_at: "2024-01-17T09:15:00Z"

16. API Authentication

# API Key auth for dashboard generation
Authorization: Bearer neuralbudget-api-key-xxx

# Grafana API token auth for sync
x-grafana-api-token: glc_...

# OIDC/SAML for UI access (standard K8s)

17. Monitoring & Metrics

dashboard_generator_generations_total{status, slo_dag}
dashboard_generator_generation_duration_seconds{quantile, slo_dag}
dashboard_generator_grafana_syncs_total{status}
dashboard_generator_errors_total{error_type}
dashboard_generator_panels_created_total{panel_type}
dashboard_generator_customizations_applied{customization_type}

18. Implementation Roadmap

Phase 1 (Weeks 1-3): Core generator

  • Parse SLO DAG → Extract metrics
  • Generate Prometheus queries
  • Create basic panels (timeseries, stat)

Phase 2 (Weeks 4-5): Grafana JSON compilation

  • Build dashboard JSON schema
  • Apply theming rules
  • Template variable generation

Phase 3 (Weeks 6-7): Export & provisioning

  • JSON export API
  • Kubernetes ConfigMap provisioning
  • Version management

Phase 4 (Weeks 8-9): Grafana sync

  • Direct Grafana API integration
  • Dashboard auto-update
  • URL generation

Phase 5 (Weeks 10-12): Advanced features

  • Custom panel templates
  • Layout editor
  • Theme builder

19. CLI Interface

# Generate dashboard locally
neuralbudget dashboard generate \
  --slo-dag payment-service-slo \
  --namespace observability \
  --style dark \
  --output dashboard.json

# Preview dashboard
neuralbudget dashboard preview \
  --slo-dag payment-service-slo

# Sync to Grafana
neuralbudget dashboard sync \
  --slo-dag payment-service-slo \
  --grafana-url https://grafana.company.com \
  --grafana-api-token glc_...

# Export as ConfigMap
neuralbudget dashboard export \
  --slo-dag payment-service-slo \
  --format configmap \
  --output configmap.yaml

# List generated dashboards
neuralbudget dashboard list --namespace observability

20. Success Metrics

  • 100% of SLO DAGs have auto-generated dashboards
  • <10 second dashboard generation time
  • 95% SRE satisfaction (vs manual creation)
  • 99.5% Grafana schema compliance
  • <500KB average dashboard size
  • 90% reduction in manual dashboard creation time

21. Before vs After

Metric              Before (Manual)    After (Auto-Generated)
────────────────────────────────────────────────────
Dashboard creation  2-4 hours          <10 seconds
Panel count         1 per team         1 per DAG node
Maintenance         High (drift)       Auto (synced)
SRE effort          High               Minimal
Consistency         Low (variation)    High (templated)
Audit trail         None               Full versioning
Grafana API calls   Manual             Automated
Shareability        Screenshots        JSON + URL
Multi-cluster       Manual per cluster Automated
Threshold updates   Manual panels      Auto from DAG

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions