Claude/plan build system 8 d nj m#1375
Open
aparcar wants to merge 121 commits into
Open
Conversation
aparcar
force-pushed
the
claude/plan-build-system-8DNjM
branch
from
3200b32 to
27c9b9d
Compare
Comprehensive plan for replacing the buildroot-based build system with a modern architecture featuring: - Pre-built toolchains via Docker/Nix (eliminates 20-60 min toolchain builds) - Declarative YAML package definitions (replaces complex Make macros) - Ninja-based build orchestration (better parallelism) - Content-addressable caching (cross-build artifact reuse) - Simplified SDK generation Includes proof of concept scope for ARM target with ~10 core packages, detailed implementation timeline, and migration strategy.
Implements a complete PoC for the modernized OpenWrt build system: Build System Core (owrt_build/): - Python CLI with click for build orchestration - YAML-based package and target definitions - Toolchain builder using OpenWrt sources (binutils, gcc, musl) - Kernel builder with full patch support (backport/pending/hack/target) - Device Tree compilation from OpenWrt DTS files - Package builder supporting autotools/cmake/meson/make - Image generator for initramfs/ext4/squashfs Target: armsr-armv8 (aarch64): - ARM SystemReady compliant 64-bit target - GCC 14.3.0, Binutils 2.44, musl 1.2.5 - Linux 6.12.65 with OpenWrt patches - EFI boot support Packages: - base-files: System structure and essential configs - busybox: Core userspace utilities - linux: Kernel with patch/DTS handling Infrastructure: - Docker environment for reproducible builds - JSON schemas for YAML validation - Convenience build script - Comprehensive documentation
aparcar
force-pushed
the
claude/plan-build-system-8DNjM
branch
from
27c9b9d to
f5a04a0
Compare
Move YAML target definitions and kernel configuration from owrt/targets/ to their canonical locations in target/linux/<board>/<subtarget>/: - target.yaml files for armsr-armv8, mediatek-filogic, x86-64 - Kernel config fragments to config/ subdirectory - Kernel patches to patches-6.12/ subdirectory Remove duplicate base-files that already exist in target/linux/. Update config.py to load targets from the new location: - Parse target name (e.g., mediatek-filogic) into board and subtarget - Look for target/linux/<board>/<subtarget>/target.yaml - Fix root_dir calculation (4 levels up from subtarget dir) This integrates the PoC build system more closely with the existing OpenWrt tree structure, avoiding unnecessary file duplication. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Update relative paths to reflect the new directory structure: - config/foo.config instead of kernel/config/foo.config - patches-6.12 instead of kernel/patches-6.12 The files were moved directly under the target directory, not in a kernel/ subdirectory. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Fix test imports to use owrt.tests.conftest instead of conftest - Add same-file check to CopyKernelStep and CopyRootfsStep to avoid shutil.SameFileError when source and destination are identical - Add missing mock_config attributes (name, arch, toolchain) required by resolver's _compute_cache_key method All 43 tests now pass. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- tool.py: Use owrt/tools/ instead of root/tools/ - __main__.py: Use owrt/toolchain/ for patches - toolchain.py: Fix binutils/gcc/musl patch paths to owrt/toolchain/ The POC patches are maintained in owrt/toolchain/, not the original OpenWrt toolchain/ directory at the repository root. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add unit-tests job running pytest on owrt/tests/ - Replace simple QEMU boot test with openwrt-tests framework - Uses labgrid for proper device control and pytest for test execution - Runtime tests check: shell access, SSH, ubus, kernel errors, memory usage - Upload test results as artifacts
Add 52 unit tests covering: - APKPackager: initialization, apk binary detection, arch mapping, metadata building, package creation, dependencies, conflicts, provides, dev packages, binary stripping - APKRootfs: initialization, rootfs creation, directory structure, symlinks, list file generation, postinst script handling - APKRepository: initialization, package addition, index generation All 146 tests in the test suite now pass.
Add QEMU module (owrt/qemu.py) with: - QEMURunner: Manages QEMU instances for firmware testing - QEMUTestRunner: Runs smoke tests against booted firmware - Support for armsr-armv8, x86-64, and malta-be targets - SSH-based command execution for test validation - System info collection and kernel error checking Add CLI commands: - test qemu: Run QEMU smoke tests on firmware - test list-targets: List supported QEMU targets - test run: Auto-detect firmware and run tests - test run --full: Integration with openwrt-tests framework Add 31 unit tests for the QEMU module covering: - QEMUConfig and target configurations - QEMURunner initialization and lifecycle - QEMUTestRunner test execution and reporting - Error handling for boot/SSH failures Total test count: 177 tests passing
- Use astral-sh/setup-uv action for cleaner uv installation - Add firmware detection step with proper outputs - Run openwrt-tests pytest with labgrid environment - Generate JUnit XML test results - Publish test results using action-junit-report - Upload test artifacts (XML + logs) for 30 days - Skip tests gracefully if no firmware found
The test was failing in CI because QEMU isn't installed. Mock the subprocess.run call to simulate QEMU being available so we can test the firmware file existence check.
When running in a container that already has toolchain environment variables set (CC, CXX pointing to cross-compiler), the toolchain build would fail because configure would find the cross-compiler instead of the host compiler. Add _create_host_env() method that creates a clean environment with: - CC=gcc, CXX=g++, AR=ar, etc. explicitly set to host tools - CROSS_COMPILE and other cross-compiler variables removed Use this host environment for all toolchain build steps (binutils, gcc-initial, kernel-headers, gcc-final) to ensure we always use the host compiler when building the toolchain itself. Also fix test_start_missing_firmware test to mock QEMU availability.
The build-firmware job was mounting ${{ github.workspace }}/build:/build
which shadows the entire /build directory in the container, including
the toolchain that was baked into the image at /build/toolchain/.
Fix by mounting only the specific subdirectories needed (output, packages,
staging, kernel, rootfs) while leaving /build/toolchain from the image
intact.
Add 'Build host tools' step that runs 'python3 -m owrt tool build-all' to build apk, mkimage, fwtool, squashfs4, and other host tools needed for firmware packaging. Include host-staging directory in the toolchain Docker image alongside the cross-compiler toolchain. Update PATH to include /build/host-staging/bin. This fixes the 'apk binary not found' error during firmware build.
Use multi-stage Docker build to include host tools (apk, mkimage, fwtool, squashfs4, mtd-utils, lua) in the base image. The Dockerfile now invokes 'python3 -m owrt tool build-all' during image creation. This ensures: - Host tools are always available in the base container - Toolchain images inherit host tools from base - No CI-specific workarounds needed - Consistent behavior between local and CI builds Simplify CI workflow by removing separate host tool build steps.
Iteration 1: - Add docs/README.md with architecture overview - Add docs/getting-started.md with build instructions - Add owrt/tests/test_tool.py with 24 tests for ToolConfig and ToolBuilder
Iteration 2: - Add docs/package-format.md with comprehensive package.yaml reference - Add owrt/tests/test_download.py with 27 tests for download manager
Iteration 3: - Add docs/cli-reference.md with complete command documentation - Add owrt/tests/test_container.py with 27 tests for container management
Iteration 4: - Add docs/target-config.md with target.yaml reference - Expand test_config.py with 20 new tests for Config, VariantConfig, and compute_package_content_hash (29 total)
- Add docs/build-pipeline.md documenting build stages, Ninja integration, caching strategy, and parallel execution - Add owrt/tests/test_ninja_gen.py with 19 tests covering NinjaGenerator and NinjaRunner classes - Tests cover initialization, file generation, rules, pools, targets, cached target handling, and runner command construction
- Add docs/toolchain.md documenting the cross-compilation toolchain build process, including 5-stage build order, directory structure, environment variables, ccache support, and troubleshooting - Add owrt/tests/test_toolchain.py with 36 tests covering ToolchainBuilder initialization, host environment, status checks, downloads, environment setup, versions, build process, and architecture handling
- Add docs/package-building.md documenting the package build process, build systems (autotools, cmake, meson, make, custom), fakechroot isolation, change detection, subpackages, variants, and target overlays - Add owrt/tests/test_package.py with 35 tests covering PackageBuilder initialization, paths, clean, dependency resolution, stamps, hash computation, isolation, build commands, and integration
- Add docs/kernel.md documenting the Linux kernel build process, patches, configuration, kmod integration, vermagic, DTBs/DTBOs, initramfs support, and architecture mapping - Add owrt/tests/test_kernel.py with 33 tests covering KernelBuilder initialization, architecture mapping, status/clean, vermagic, kmod config, user config, config hash, paths, and build environment
- Add docs/image-building.md documenting firmware image generation, image types (sysupgrade, initramfs, EFI, UBI, JFFS2), pipelines, rootfs assembly, filesystem formats, FIT images, and EFI boot - Add owrt/tests/test_image.py with 18 tests covering ImageBuilder initialization, naming, size parsing, symlinks, permissions, release files, user/group creation, tree copy, and integration
- Add docs/index.md as main documentation entry point, linking to all other documentation with quick start guide and key concepts overview - Add owrt/tests/test_fit.py with 27 tests covering FITBuilder, UBIFSBuilder, UBIBuilder, JFFS2Builder, and MetadataBuilder classes - Total test coverage: 443 tests across 14 test modules Documentation iteration complete: - docs/README.md - Architecture overview - docs/getting-started.md - Build instructions - docs/cli-reference.md - CLI commands - docs/package-format.md - Package YAML reference - docs/target-config.md - Target YAML reference - docs/build-pipeline.md - Ninja integration - docs/toolchain.md - Cross-compiler building - docs/kernel.md - Kernel building - docs/package-building.md - Package compilation - docs/image-building.md - Firmware images - docs/index.md - Documentation index
Lua's 'linux' make target requires readline headers. Without this dependency, the build fails with: luaconf.h:275:10: fatal error: readline/readline.h: No such file or directory
Refactor Docker build system into separate layers: - Dockerfile.base: Ubuntu 24.04 + system packages - Dockerfile.tools: Host tools (apk, mkimage, squashfs4, etc.) - Dockerfile.toolchain: Cross-toolchain for specific targets Add owrt container CLI commands: - container build: Build images with automatic caching - container status: Show image status and hashes - container hash: Compute content hashes for CI - container clean: Remove all owrt images Images are tagged with content hashes derived from: - Dockerfile contents - Tool definitions (tool.yaml files) - Toolchain patches and target configuration This enables efficient CI caching - images only rebuild when their inputs actually change. Update CI workflow to use content-based caching: - Compute hashes upfront with 'owrt container hash' - Check registry for existing images before building - Skip builds when cached image exists Remove: - docker/Dockerfile (redundant all-in-one) - docker/build-images.sh (replaced by CLI) - docker/docker-compose.yml (unused)
Add core bootstrap tools that OpenWrt builds first: - libdeflate: Fast gzip decompression for archives - patch: GNU patch for consistent patching - tar: GNU tar for consistent archive extraction - zstd: Zstandard compression for modern archives Add compression libraries needed by other tools: - xz: LZMA compression (needed by squashfs4, kernel) - zlib: Basic compression (needed by many tools) Update existing tools: - squashfs4: Add xz and zlib as dependencies - lua: Fix symlinks so meson can find lua5.1 Mark core tools with metadata.core_tool flag for future prioritization in build order. All 13 host tools now build successfully in the container: - apk, arm-trusted-firmware-tools, fwtool, libdeflate, lua, mkimage, mtd-utils, patch, squashfs4, tar, xz, zlib, zstd
Add notes for Claude workflow: - Always commit after successful testing - Container build commands reference - Host tools overview - Testing commands
Add -o/--output option to specify the output directory for the toolchain build. This is needed by Dockerfile.toolchain which builds the toolchain to /toolchain for later copying to the final image. When specified, the option overrides config.toolchain_dir before creating the ToolchainBuilder instance.
aparcar
force-pushed
the
claude/plan-build-system-8DNjM
branch
from
a7ba0ee to
a94718b
Compare
Add OWRT_IN_CONTAINER environment variable as the primary detection method. This is explicitly set in Dockerfile.toolchain during the RUN command to ensure the build system doesn't try to launch Docker when already running inside a container. Detection methods (in order of reliability): 1. OWRT_IN_CONTAINER env var - explicitly set in our Dockerfiles 2. /.dockerenv file - created by Docker runtime 3. 'container' env var - set by Podman, systemd-nspawn 4. OWRT_TOOLS_DIR/OWRT_TARGET - set in our container images Also fix Dockerfile.toolchain to: - Copy target/ directory (contains target.yaml definitions) - Create /toolchain output directory with proper permissions Update unit tests to match new detection methods and add workflow guideline to always run tests before committing.
- Add --push option to 'container build' CLI command - Add push parameter to build methods in docker_build.py - Add check_remote parameter to check registry before building - Consolidate CI from 5 build jobs to 2 (build-containers, build-firmware) - 'owrt container build --push' now handles hash computation, registry checking, building, and pushing in a single command The new workflow: 1. unit-tests: runs pytest 2. build-containers: uses 'owrt container build --push' to build all images (base, tools, toolchain) with content-based caching 3. build-firmware: uses toolchain image to build firmware This removes the need for separate compute-hashes, build-base, build-tools, and build-toolchain jobs.
aparcar
force-pushed
the
claude/plan-build-system-8DNjM
branch
from
a94718b to
4d36981
Compare
Split the single build-containers job into three separate jobs: - build-base: builds and pushes the base image - build-tools: builds and pushes the tools image - build-toolchain: builds and pushes the toolchain image Each job uses docker/build-push-action@v6 for building and pushing instead of the owrt --push flag. Images are tagged with content-based hashes computed by 'owrt container hash' for cache-friendly tagging. This allows faster feedback when only part of the container hierarchy changes, and makes the build progress more visible in the CI UI.
The toolchain Dockerfile final stage inherits from the tools image which runs as 'builder' user. Writing to /etc/environment requires root permissions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thanks for your contribution to OpenWrt!
To help keep the codebase consistent and readable,
and to help people review your contribution,
we ask you to follow the rules you find in the wiki at this link
https://openwrt.org/submitting-patches
Please remove this message before posting the pull request.