quickstart

Quickstart Guide

Get your first Stellar data pipeline running in 2 minutes.

Prerequisites

• Go 1.21+ - Install Go
• Git - For cloning the repository
• Docker - For running containerized components

Installation

# Clone the repository
git clone https://github.com/withobsrvr/flowctl.git
cd flowctl

# Build flowctl
make build

# Verify installation
./bin/flowctl version

Create Your First Pipeline

Option 1: Interactive Mode (Recommended)

./bin/flowctl init

Follow the prompts:

1. Network: Select testnet (recommended for learning) or mainnet
2. Destination: Select where to store data:
   • duckdb - Embedded analytics database (easiest)
   • postgres - PostgreSQL database
   • csv - CSV files

This creates a stellar-pipeline.yaml file.

Option 2: Non-Interactive Mode

For automation or CI/CD:

# Create a testnet pipeline with DuckDB sink
./bin/flowctl init --non-interactive --network testnet --destination duckdb

# Create a mainnet pipeline with PostgreSQL sink
./bin/flowctl init --non-interactive --network mainnet --destination postgres -o mainnet-pipeline.yaml

Run the Pipeline

./bin/flowctl run stellar-pipeline.yaml

What happens:

1. flowctl downloads required components from Docker Hub (first run only)
2. Starts the embedded control plane
3. Launches components: source -> contract-events-processor -> sink
4. Data flows from Stellar network to your chosen destination

Press Ctrl+C to stop.

Verify Data

DuckDB

# Query the DuckDB file for contract events
duckdb stellar-pipeline.duckdb "SELECT * FROM contract_events LIMIT 5"

PostgreSQL

# Connect and query contract events
psql -h localhost -U postgres -d stellar_events -c "SELECT * FROM contract_events LIMIT 5"

CSV

# Check the CSV files
ls -la data/
head data/contract_events.csv

Sample Pipelines

This directory contains sample pipeline configurations generated by flowctl init:

File	Network	Sink	Description
testnet-duckdb-pipeline.yaml	testnet	DuckDB	Easiest setup for learning
testnet-postgres-pipeline.yaml	testnet	PostgreSQL	Production-like setup

Understanding the Generated Configuration

A typical generated pipeline looks like:

apiVersion: flowctl/v1
kind: Pipeline
metadata:
  name: stellar-pipeline
  description: Process stellar contract events on testnet

spec:
  driver: process

  sources:
    - id: stellar-source
      type: stellar-live-source@v1.0.0
      config:
        network_passphrase: "Test SDF Network ; September 2015"
        backend_type: RPC
        rpc_endpoint: https://soroban-testnet.stellar.org
        start_ledger: 54000000

  processors:
    - id: contract-events
      type: contract-events-processor@v1.0.0
      config:
        network_passphrase: "Test SDF Network ; September 2015"
      inputs: ["stellar-source"]

  sinks:
    - id: duckdb-sink
      type: duckdb-consumer@v1.0.0
      config:
        database_path: ./stellar-pipeline.duckdb
      inputs: ["contract-events"]

Key points:

• driver: process runs components as local processes
• Components are automatically downloaded from Docker Hub
• Pipeline has three stages: source → processor → sink
• The contract-events processor extracts Soroban events from ledgers
• inputs connects each component to its upstream data source

Troubleshooting

"Component not found" or "Image pull failed"

Check Docker is running:

docker ps

Manually pull the image:

docker pull docker.io/withobsrvr/stellar-live-source:v1.0.0

"Connection refused" to control plane

Ensure port 8080 is available:

lsof -i :8080

No data appearing

1. Check component logs:

   ./bin/flowctl run stellar-pipeline.yaml --log-level=debug

2. Verify network connectivity to Stellar Horizon

DuckDB file not created

Check the working directory and ensure the path is writable:

ls -la .

Next Steps

• Add processors: Transform data with processors between source and sink
• Monitor pipelines: Use flowctl dashboard for real-time monitoring
• Deploy to production: Use flowctl translate for Docker Compose or Kubernetes

Resources

• flowctl init Reference
• Processor Discovery
• Building Components
• Configuration Guide
• Getting Started Guide