Skip to content

Latest commit

 

History

7 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 

Repository files navigation

toolbox

A collection of operational scripts for the data streaming team.


image-ownership-mapping

Maps every running container image across all K8s clusters back to the owning team via braze-deployments CODEOWNERS.

Scripts

Script Purpose
build-image-mapping.py Query one or more clusters and produce a JSON mapping
run-all-clusters.sh Run build-image-mapping.py across all clusters in parallel
generate-spreadsheets.py Produce two CSVs from the per-cluster JSON files
test_build_image_mapping.py Unit tests for build-image-mapping.py

Prerequisites

  • kubectl configured with contexts for the target clusters (standard Braze kubeconfig)
  • Python 3.9+, pyyaml (pip install pyyaml)
  • A local checkout of braze-deployments (default: ~/braze/braze-deployments)

Quick start — all clusters

cd image-ownership-mapping

# 1. Collect data from every cluster (~25 seconds)
./run-all-clusters.sh

# 2. Generate spreadsheets from the collected data
python3 generate-spreadsheets.py

Output files land in image-ownership-mapping/output/:

File Contents
<cluster-name>.json Per-cluster image mapping (one file per cluster)
sheet1-team-image-mapping.csv Unique (team, image, clusters) pairs
sheet2-detailed.csv One row per container instance across all clusters

Running a single cluster

cd image-ownership-mapping

python3 build-image-mapping.py \
  --cluster k8s.cluster-001.d-use-1.braze.com \
  --format json \
  --output output/dev.json \
  --repo-root ~/braze/braze-deployments

CLI options — build-image-mapping.py

Flag Default Description
--cluster (required) Cluster hostname (without role suffix)
--repo-root . Path to braze-deployments checkout
--format json Output format: json or csv
--output stdout Output file path
--workers 5 Parallel kubectl calls within a cluster
--delay 0 Seconds between worker batches
--timeout 30 Per-kubectl call timeout (seconds)
--env (all) Filter by environment (development, staging, production)
--region (all) Filter by AWS region (e.g. us-east-1)

Environment variables — run-all-clusters.sh

Variable Default Description
REPO_ROOT ~/braze/braze-deployments Path to braze-deployments checkout
OUTPUT_DIR ./output Directory for JSON and log files
PARALLEL 8 Number of clusters to query in parallel
WORKERS 5 kubectl workers per cluster (passed to build-image-mapping.py)

Output format

JSON (keyed by image name):

{
  "artifactory.infra.braze.com/docker/kirby-kafka-connect": [
    {
      "image": "artifactory.infra.braze.com/docker/kirby-kafka-connect:176-dc3fa4c",
      "image_name": "artifactory.infra.braze.com/docker/kirby-kafka-connect",
      "image_tag": "176-dc3fa4c",
      "app_name": "data-lake-user-profile-connect",
      "teams": ["@Appboy/data-lakehouse"],
      "namespace": "airflow-datalake",
      "workload": "data-lake-user-profile-connect-warpstream",
      "workload_kind": "Deployment",
      "cluster": "k8s.cluster-001.d-use-1.braze.com",
      "environment": "development",
      "region": "us-east-1",
      "matched": true,
      "source": "helm"
    }
  ]
}

source values:

  • helm — matched via Helm release labels in braze-deployments
  • kops — kops-managed cluster infrastructure (calico, coredns, etc.)
  • unknown — external workload not in braze-deployments (datadog, mongodb operator, etc.)

Running tests

cd image-ownership-mapping
python3 -m pytest test_build_image_mapping.py -v

kafka-topic-retention-config

See kafka-topic-retention-config/.

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages