Skip to content

Latest commit

 

History

History
481 lines (363 loc) · 11.6 KB

developer-guide.md

File metadata and controls

481 lines (363 loc) · 11.6 KB

Concourse Developer Guide

This guide provides instructions for developers who want to build, modify, or contribute to the Concourse codebase.

Table of Contents

Prerequisites

Required Software

  • Java 8+: Concourse is built with Java 8 as the minimum version

    java -version
# Should show 1.8.x or higher

  • Gradle: The build system (wrapper included)

    ./gradlew --version

  • Git: For version control

    git --version

Optional Tools

  • Docker: For running containerized tests or the server
  • Thrift Compiler: For modifying the Thrift interface definitions
  • Ruby: For generating manual pages with ronn

System Requirements

  • At least 4 GB of RAM (8 GB recommended for running tests)
  • At least 2 GB of free disk space

Building from Source

Clone the Repository

git clone https://github.com/cinchapi/concourse.git
cd concourse

Install Git Hooks

./utils/install-git-hooks.sh

This installs pre-push hooks that ensure code formatting before pushes.

Build the Project

# Full build with tests
./gradlew build

# Build without tests (faster)
./gradlew build -x test

# Build the installer
./gradlew installer

# Clean and rebuild
./gradlew clean build

Build Artifacts

After building, key artifacts are located at:

Artifact Location
Server Installer concourse-server/build/distributions/concourse-server-*.bin
Java Driver JAR concourse-driver-java/build/libs/concourse-driver-java-*.jar
Shell JAR concourse-shell/build/libs/concourse-shell-*.jar

Project Structure

concourse/
├── build.gradle              # Root build configuration
├── settings.gradle           # Gradle settings and module definitions
├── gradlew                   # Gradle wrapper script
│
├── concourse-server/         # Core database server
│   ├── src/main/java/        # Server source code
│   ├── src/test/java/        # Server unit tests
│   ├── conf/                 # Configuration templates
│   └── scripts/              # Server management scripts
│
├── concourse-driver-java/    # Java client driver
│   ├── src/main/java/        # Driver source code
│   └── src/test/java/        # Driver tests
│
├── concourse-shell/          # Interactive shell (CaSH)
├── concourse-cli/            # CLI tools
├── concourse-plugin-core/    # Plugin framework
├── concourse-import/         # Import tools
├── concourse-export/         # Export tools
│
├── concourse-driver-python/  # Python driver
├── concourse-driver-php/     # PHP driver
├── concourse-driver-ruby/    # Ruby driver
│
├── concourse-integration-tests/  # Integration tests
├── concourse-ete-test-core/      # ETE test framework
├── concourse-ete-tests/          # End-to-end tests
├── concourse-unit-test-core/     # Unit test base classes
│
├── interface/                # Thrift interface definitions
│   ├── concourse.thrift      # Main API definition
│   ├── data.thrift           # Data type definitions
│   ├── exceptions.thrift     # Exception definitions
│   └── shared.thrift         # Shared types
│
├── docs/                     # Documentation
│   ├── guide/                # User guide
│   ├── shell/                # Shell command docs
│   └── dev/                  # Developer docs
│
├── utils/                    # Development utilities
├── examples/                 # Example projects
└── fixtures/                 # Test fixtures

Development Workflow

Running the Server Locally

Option 1: From Built Installer

# Build the installer
./gradlew installer

# Install
sh concourse-server/build/distributions/concourse-server-*.bin

# Start the server
concourse start

# Connect with shell
concourse shell

Option 2: Using Docker

# Build and run
./gradlew dockerize
docker run -p 1717:1717 cinchapi/concourse

Option 3: IDE Launch Configuration

Import the project into your IDE and use the launch configurations in:

  • concourse-server/launch/
  • concourse-shell/launch/

Using the Shell

Connect to a running server:

# Default connection (localhost:1717)
concourse shell

# With custom connection
concourse shell --host myserver --port 1717 --username admin --password admin

Running Tests

# Run all tests
./gradlew test

# Run tests for a specific module
./gradlew :concourse-server:test

# Run a specific test class
./gradlew :concourse-server:test --tests "*.EngineTest"

# Run integration tests
./gradlew :concourse-integration-tests:test

# Run with test output
./gradlew test --info

Code Formatting

Concourse uses Spotless for code formatting:

# Check formatting
./gradlew spotlessCheck

# Apply formatting
./gradlew spotlessApply

Coding Standards

Java Conventions

Naming Conventions

  • Classes: PascalCase, clear and descriptive (e.g., BufferedStore, TransactionManager)
  • Methods: camelCase, verbs for actions (e.g., add, remove, find)
  • Getters: Avoid get prefix (e.g., size() not getSize())
  • Variables: camelCase, concise but descriptive
  • Constants: UPPER_SNAKE_CASE

Code Style

// Good: Clear, well-documented method
/**
 * Add {@code key} as {@code value} to {@code record} if the mapping does
 * not already exist.
 *
 * @param key the field name
 * @param value the value to add
 * @param record the record id
 * @return {@code true} if the value is added
 */
public boolean add(String key, Object value, long record) {
    // Implementation
}

// Avoid: Vague names, missing documentation
public boolean a(String k, Object v, long r) {
    // Implementation
}

Immutability

Favor immutable classes where possible:

// Good: Immutable value class
public final class Position implements Byteable {
    private final Identifier record;
    private final int index;

    private Position(Identifier record, int index) {
        this.record = record;
        this.index = index;
    }

    public static Position of(Identifier record, int index) {
        return new Position(record, index);
    }
}

Documentation Style

JavaDoc Rules

  1. Line Length: Do not exceed 80 characters per line
  2. Purpose: Document "what" and "why", not "how"
  3. Contract: Describe inputs, outputs, and side effects
  4. Links: Use {@link ClassName} for cross-references
  5. Return: Always document return values with @return
/**
 * Return a {@link Set} containing all the records that have a value stored
 * for {@code key} that satisfies the {@code operator} in relation to
 * {@code value}.
 * <p>
 * This method supports ad-hoc queries with automatic index utilization.
 * </p>
 *
 * @param key the field name to query
 * @param operator the {@link Operator} to use for comparison
 * @param value the value to compare against
 * @return the matching record ids
 */
public Set<Long> find(String key, Operator operator, Object value) {
    // Implementation
}

Inline Comments

Use inline comments sparingly, only for non-obvious implementation details:

// Good: Explains WHY, not WHAT
// We must acquire the lock before reading to ensure visibility
// of writes from other threads
lock.readLock().lock();

// Avoid: States the obvious
// Get the value from the map
value = map.get(key);

Import Ordering

Imports are organized by Spotless in this order:

  1. java.*
  2. javax.*
  3. Third-party libraries
  4. Project imports (com.cinchapi.*)

Testing Standards

  • Every new feature should have corresponding tests
  • Use meaningful test method names: testAddReturnsFalseWhenValueExists
  • Register test variables for debugging: Variables.register("key", key)

Common Tasks

Adding a New API Method

  1. Define in Thrift (interface/concourse.thrift):

    TObject myNewMethod(
    1: string key,
    2: i64 record,
    3: shared.AccessToken creds,
    4: shared.TransactionToken transaction,
    5: string environment
) throws (
    1: exceptions.SecurityException ex,
    2: exceptions.TransactionException ex2
);

  2. Compile Thrift:

    ./utils/compile-thrift-java.sh

  3. Implement in ConcourseServer:

    @Override
public TObject myNewMethod(String key, long record,
        AccessToken creds, TransactionToken transaction,
        String environment) throws TException {
    // Implementation
}

  4. Add to Java Driver (Concourse.java):

    public abstract Object myNewMethod(String key, long record);

  5. Write Tests:

    • Unit test in concourse-server
    • Integration test in concourse-integration-tests

Creating a Plugin

  1. Create plugin class:

    public class MyPlugin extends Plugin {

    @PluginEndpoint
    public String myMethod(String arg) {
        return "Result: " + arg;
    }
}

  2. Build as bundle:

    • Create a Gradle project with concourse-plugin-core dependency
    • Package as ZIP with all dependencies

  3. Install:

    concourse plugin install /path/to/my-plugin.zip

Modifying the Storage Engine

  1. Understand the component hierarchy:

    • Engine → coordinates all storage
    • Buffer → write-optimized storage
    • Database → read-optimized storage
    • Segment → immutable storage unit

  2. Key classes to study:

    • concourse-server/src/main/java/com/cinchapi/concourse/server/storage/

  3. Always add tests:

    • Unit tests for isolated behavior
    • Integration tests for end-to-end behavior

Debugging Tips

Logging

Concourse uses SLF4J with Logback:

import com.cinchapi.concourse.util.Logger;

// Use the Logger utility
Logger.debug("Processing record {}", record);
Logger.info("Server started on port {}", port);
Logger.error("Failed to process", exception);

Configure logging in conf/logback.xml.

Test Debugging

Use the Variables class to track test state:

@Test
public void testMyFeature() {
    String key = Variables.register("key", TestData.getString());
    long record = Variables.register("record", TestData.getLong());

    // If test fails, registered variables are dumped to console
    assertTrue(concourse.add(key, "value", record));
}

JVM Configuration

Configure memory and garbage collection in conf/concourse.yaml:

# Set heap size (see concourse.yaml for sizing recommendations)
heap_size: 4g

# Force G1 garbage collector on JDK 8 (recommended for large heaps)
force_g1gc: true

Remote Debugging

Enable remote debugging by setting the remote_debugger_port in conf/concourse.yaml:

# Enable remote debugging on port 5005
remote_debugger_port: 5005

Then attach your IDE debugger to the configured port.

Logging

Configure log verbosity in conf/concourse.yaml:

# Options: ERROR, WARN, INFO, DEBUG
log_level: DEBUG

# Also print logs to console
enable_console_logging: true

Next Steps