Skip to content

README.md

Latest commit

 

History

History
402 lines (297 loc) · 10.5 KB

README.md

File metadata and controls

402 lines (297 loc) · 10.5 KB

🎯 Orchestration Guide

When you need things done NOW — not in 30 minutes.

Overview

OpenVox's agent-server model is great for convergence (gradually bringing systems into compliance), but sometimes you need orchestration — running commands across your fleet right now, deploying code immediately, or executing complex multi-step workflows.

This guide covers:

OpenBolt (Puppet Bolt)

Bolt — now OpenBolt (v5.3.0) in the OpenVox ecosystem — is an agentless orchestration tool. It connects to remote nodes via SSH (or WinRM for Windows) and runs commands, scripts, tasks, and plans — without requiring a Puppet agent on the target. Think of it as the "do it now" complement to Puppet's "keep it this way forever" model.

Installation

# RHEL/CentOS/Rocky (from the Vox Pupuli repo)
sudo yum install -y puppet-bolt

# Debian/Ubuntu
sudo apt-get install -y puppet-bolt

# macOS
brew install puppet-bolt

# Verify installation
bolt --version
# 5.3.0

Project Setup

Create a Bolt project:

mkdir myproject && cd myproject
bolt project init myproject

This creates:

myproject/
├── bolt-project.yaml     ← Project configuration
├── inventory.yaml         ← Target definitions
├── Puppetfile             ← Module dependencies
├── plans/                 ← Bolt plans
└── tasks/                 ← Bolt tasks (RESERVED: currently not used)

Inventory

Define your targets in inventory.yaml:

---
groups:
  - name: webservers
    targets:
      - uri: web1.example.com
      - uri: web2.example.com
    config:
      ssh:
        user: deploy
        private-key: ~/.ssh/id_ed25519
        host-key-check: false
        run-as: root

  - name: databases
    targets:
      - uri: db1.example.com
      - uri: db2.example.com
    config:
      ssh:
        user: deploy
        run-as: root

  - name: all_servers
    groups:
      - webservers
      - databases

  - name: local
    targets:
      - uri: localhost
    config:
      transport: local

Running Commands

# Run a command on all webservers
bolt command run 'systemctl status httpd' --targets webservers

# Run on specific hosts
bolt command run 'df -h' --targets web1.example.com,db1.example.com

# Run on all servers
bolt command run 'uptime' --targets all_servers

# Using PuppetDB for target discovery
bolt command run 'hostname' \
  --query 'nodes[certname] { facts.os.name = "Rocky" }'

# Limit concurrency (don't overwhelm your network)
bolt command run 'yum update -y openssl' --targets all_servers --concurrency 5

Running Scripts

# Run a local script on remote targets
bolt script run ./scripts/health_check.sh --targets webservers

# Pass arguments to the script
bolt script run ./scripts/deploy.sh --targets webservers \
  --arguments 'version=2.0 environment=production'

File Operations

# Upload a file
bolt file upload ./configs/nginx.conf /etc/nginx/nginx.conf --targets webservers

# Download files (great for log collection)
bolt file download /var/log/messages ./collected_logs/ --targets all_servers

Applying Puppet Code (Agentless!)

# Apply a manifest without requiring an agent
bolt apply manifest.pp --targets web1.example.com

# Apply inline Puppet code
bolt apply -e 'package { "vim": ensure => installed }' --targets all_servers

# Apply with module support
bolt apply --modulepath ./modules manifest.pp --targets webservers

Tasks and Plans

Tasks

Tasks are single-action scripts (Bash, Python, PowerShell, Ruby) with structured metadata. They're like scripts, but with parameter validation, documentation, and discoverability.

Creating a Task

# tasks/restart_service.sh
#!/bin/bash
# Restart a service and verify it's running

SERVICE=$PT_service_name   # Parameters are passed as environment variables with PT_ prefix

systemctl restart "$SERVICE"
sleep 2

if systemctl is-active --quiet "$SERVICE"; then
    echo "{\"status\": \"success\", \"service\": \"$SERVICE\", \"state\": \"running\"}"
else
    echo "{\"status\": \"failed\", \"service\": \"$SERVICE\", \"state\": \"stopped\"}" >&2
    exit 1
fi
// tasks/restart_service.json (metadata)
{
  "description": "Restart a system service and verify it started successfully",
  "parameters": {
    "service_name": {
      "description": "The name of the service to restart",
      "type": "String"
    }
  },
  "input_method": "environment"
}

Running a Task

bolt task run myproject::restart_service service_name=httpd --targets webservers

Plans

Plans are multi-step workflows written in Puppet language or YAML. They can run commands, tasks, other plans, and include logic (conditionals, error handling, etc.).

Puppet Language Plan

# plans/rolling_deploy.pp
plan myproject::rolling_deploy (
  TargetSpec $targets,
  String     $version,
  Integer    $batch_size = 2,
) {
  # Get the targets
  $all_targets = get_targets($targets)

  # Deploy in batches
  $all_targets.each |$batch| {
    out::message("Deploying version ${version} to batch...")

    # 1. Disable the load balancer
    run_task('myproject::lb_drain', $batch)

    # 2. Deploy the new version
    run_task('myproject::deploy', $batch,
      version => $version
    )

    # 3. Run health checks
    $results = run_task('myproject::health_check', $batch)

    # 4. Fail fast if health checks fail
    $results.each |$result| {
      unless $result['healthy'] {
        fail_plan("Health check failed on ${result.target.name}")
      }
    }

    # 5. Re-enable in load balancer
    run_task('myproject::lb_enable', $batch)

    out::message("Batch complete!")
  }

  return "Deployed version ${version} to ${all_targets.length} targets"
}

YAML Plan (Simpler)

# plans/update_packages.yaml
---
description: "Update specific packages across fleet"
parameters:
  targets:
    type: TargetSpec
    description: "Targets to update"
  packages:
    type: Array[String]
    description: "Packages to update"

steps:
  - name: check_current
    command: "rpm -qa ${packages.join(' ')}"
    targets: $targets

  - name: update_packages
    command: "yum update -y ${packages.join(' ')}"
    targets: $targets

  - name: verify_update
    command: "rpm -qa ${packages.join(' ')}"
    targets: $targets

return: $verify_update

Running a Plan

bolt plan run myproject::rolling_deploy \
  targets=webservers version=2.1.0 batch_size=2

bolt plan run myproject::update_packages \
  targets=all_servers packages='["openssl","curl"]'

r10k Code Deployment (v5.0.2)

r10k (v5.0.2) is the standard tool for deploying Puppet code from Git. It maps Git branches to Puppet environments and installs Forge modules declared in a Puppetfile. The name references Star Wars' assassin droids — because this tool is killer at deployment.

How It Works

Git Repository                          Puppet Server
─────────────                           ──────────────
main branch    ─────r10k deploy────►    production/ environment
staging branch ─────r10k deploy────►    staging/ environment
feature-x      ─────r10k deploy────►    feature-x/ environment

Typical Workflow

# 1. Make changes in your control repo
git checkout -b feature/new-monitoring
# ... edit manifests, Hiera data, Puppetfile ...
git add . && git commit -m "feat: add new monitoring profile"
git push origin feature/new-monitoring

# 2. Deploy on the Puppet server
sudo r10k deploy environment --verbose

# 3. Test on a single node
sudo puppet agent -t --environment feature/new-monitoring

# 4. Merge to production
git checkout main && git merge feature/new-monitoring
git push origin main

# 5. Deploy production
sudo r10k deploy environment production --modules --verbose

Automating Deployment

Post-receive Git Hook

#!/bin/bash
# .git/hooks/post-receive (on your Git server)
r10k deploy environment --verbose

Webhook with a simple service

#!/bin/bash
# /usr/local/bin/r10k-deploy.sh
# Called by your CI/CD pipeline or webhook

cd /etc/puppetlabs/code

# Deploy all environments
r10k deploy environment --verbose --modules 2>&1 | tee /var/log/r10k-deploy.log

# Flush the environment cache
curl -i --cert $(puppet config print hostcert) \
     --key $(puppet config print hostprivkey) \
     --cacert $(puppet config print localcacert) \
     -X DELETE \
     "https://$(puppet config print certname):8140/puppet-admin-api/v1/environment-cache"

echo "Deployment complete at $(date)"

The Puppetfile

# Puppetfile — manages module dependencies

# Forge modules (pinned versions — always pin in production!)
mod 'puppetlabs-stdlib',     '9.6.0'
mod 'puppetlabs-apache',     '12.1.0'
mod 'puppetlabs-mysql',      '15.0.0'
mod 'puppetlabs-postgresql',  '10.0.3'
mod 'puppetlabs-firewall',   '8.0.3'
mod 'puppetlabs-concat',     '9.0.2'
mod 'puppetlabs-ntp',        '10.1.0'
mod 'puppet-nginx',          '6.0.1'

# Git modules
mod 'custom_profiles',
  git:    'git@github.com:myorg/custom_profiles.git',
  tag:    'v1.5.0'

# Branch tracking (for development — not recommended in production!)
mod 'experimental_module',
  git:    'git@github.com:myorg/experimental_module.git',
  branch: 'development'

Warning: Always pin module versions in production Puppetfiles! Using :latest or branch tracking in production is like juggling chainsaws — impressive until it goes wrong.

Orchestration Best Practices

  1. Start small: Test orchestration commands on one node before running against the fleet
  2. Use --noop and --concurrency: Always dry-run first, and limit concurrency to avoid overwhelming your infrastructure
  3. Pin module versions: In your Puppetfile, always pin to specific versions or tags
  4. Use Bolt for ad-hoc work, Puppet for convergence: They complement each other
  5. Version your plans and tasks: Treat them like code — they ARE code
  6. Log everything: r10k deployments, Bolt runs, and plan outputs should all be logged

Next up: Troubleshooting & FAQ →

This document was created with the assistance of AI (Grok, xAI). All technical content has been reviewed and verified by human contributors.