This file provides guidance to agents when working with code in this repository.
This repo ships two PyPI packages from one repository:
awkward— pure Python, insrc/awkward. Released frequently.awkward-cpp— compiled C++ kernels (pybind11 + scikit-build-core), inawkward-cpp/. Versioned independently, released rarely (compilation takes hours in CI).pyproject.tomlpins the exactawkward-cppversion thatawkwardrequires.
Parts of awkward-cpp are generated; building awkward-cpp from the repository generates them automatically (CMake runs the dev/ generation scripts at configure time).
git clone --recursive https://github.com/scikit-hep/awkward.git # rapidjson submodule
uv sync # uv workspace: builds awkward-cpp (generating headers, kernel signatures, and kernel tests) and installs both packages editableAny uv run ... command does the same setup on demand and rebuilds awkward-cpp when its C++ sources or the kernel spec change (see cache-keys in awkward-cpp/pyproject.toml). The dev dependency group holds the test dependencies. Without uv:
python -m pip install -v ./awkward-cpp
python -m pip install -e .nox -s prepareruns the generation scripts without building; it accepts--headers --signatures --tests --docsto select targets (--docsdata is the only target the build doesn't cover).noxfile.pyis a uv script (./noxfile.py -s preparealso works without nox installed);nox -R -s ...reuses the session venv.- Python-only changes need no rebuild (editable install). Rebuilding
awkward-cppis only needed when C++ or the kernel spec changes; for fast iteration install build deps (includingpyyamlandnumpy) and usepip install --no-build-isolation --check-build-dependencies ./awkward-cpp.
python -m pytest -n auto tests # main test suite
python -m pytest tests/test_XXXX-description.py # single test file
python -m pytest -n auto awkward-cpp/tests-spec # generated kernel-spec tests (Python reference impls)
python -m pytest -n auto awkward-cpp/tests-cpu-kernels # generated tests against compiled kernelsCUDA suites (tests-cuda, tests-cuda-kernels) require an Nvidia GPU + CuPy. The generated kernel test dirs are created by building awkward-cpp from the repository (or by nox -s prepare -- --tests).
Test files are named tests/test_XXXX-description.py where XXXX is the GitHub issue or PR number (enforced by dev/validate-test-names.py). New code needs a new test file following this convention.
Warnings are errors in pytest (filterwarnings = ["error", ...]).
prek -a --quiet # or: nox -s lint (or pre-commit run -a if prek is missing)There is also a custom flake8 plugin (dev/flake8_awkward.py) and nox -s pylint.
highlevel.py— user-facingak.Array,ak.Record,ak.ArrayBuilder. These wrap a low-level layout.operations/ak_*.py— one file per publicak.*function. Each goes through_dispatch.py(@high_level_function), which enables third-party overload dispatch; the function body is split into adispatchgenerator yielding array arguments, then the_implfunction.contents/— the layout node types (NumpyArray,ListOffsetArray,RecordArray,IndexedOptionArray,UnionArray, etc.), forming a tree that represents nested/ragged data columnar-wise.index.pyholds the integer index buffers._do.pycontains cross-cutting operations on layouts.forms/— metadata-only mirrors of each content type (the "form" = type + buffer structure without data), used for serialization and typetracer._meta/— shared base logic between contents and forms._backends/and_nplikes/— abstraction over array libraries: NumPy, CuPy, JAX, and typetracer (shape-only arrays with possibly-unknown lengths, used by dask-awkward to compute without data). Code incontents/must go through the nplike API, not call NumPy directly._kernels.py+awkward_cpp._kernel_signatures— how layouts invoke compiled kernels via ctypes-style signatures, per backend._connect/— integrations: numba, jax, pyarrow (Arrow/Parquet), numexpr, RDataFrame, cling/cppyy, cuda.behaviors/— built-in behaviors (e.g. strings as character lists) layered on theak.behaviorregistry, which maps record names/parameters to Python mixin classes._broadcasting.py,_slicing.py,_reducers.py— the core algorithms behind ufuncs,__getitem__, and reductions.
kernel-specification.yml (repo root) is the source of truth for every low-level kernel: signatures plus a Python reference implementation. From it:
dev/generate-kernel-signatures.py→awkward-cpp/include/awkward/kernels.h,awkward_cpp/_kernel_signatures.py, and the CUDA signature table insrc/awkward/_connect/cuda/.dev/generate-tests.py+kernel-test-data.json→ thetests-spec*andtests-cpu-kernels*suites.
The hand-written C++ implementations live in awkward-cpp/src/cpu-kernels/ (one file per kernel) and must match the spec; nox -s diagnostics checks kernel definitions. Adding/changing a kernel means touching the YAML spec, the C++ implementation, and (sometimes) test data, then rebuilding awkward-cpp (which regenerates) or running nox -s prepare.
kernel-specification.yml is the single source of truth for all backends — the canonical signature, semantics, edge-case behavior, and Python reference. CPU C++ and CUDA kernels must conform to it identically: Awkward guarantees cross-backend consistency (NumPy, CuPy, JAX, typetracer, compiled kernels), so any divergence breaks slicing, broadcasting, masking, union, and Dask/JAX typetracer guarantees. Do not introduce CUDA-only behavior, reorder arguments, change pointer/buffer types, alter edge cases, add kernels directly in C++/CUDA, or bypass the generation pipeline. GPU optimizations are fine only if semantics stay bit-for-bit identical.
To change a kernel: (1) edit kernel-specification.yml; (2) run the generation pipeline (rebuild awkward-cpp, or nox -s prepare -- --signatures --tests); (3) update both the CPU C++ and CUDA implementations to match the reference; (4) confirm tests-spec, tests-cpu-kernels, and tests-cuda-kernels all pass.
header-only/ contains standalone C++ headers (LayoutBuilder, GrowableBuffer) used by downstream C++ projects; dev/copy-cpp-headers.py (part of prepare) copies them into awkward-cpp and src/awkward/_connect/header-only. Never edit the copies.
- PR titles follow Conventional Commits (
feat:,fix:,docs:, ...) — PRs are squash-merged, so the title becomes the commit message. Branches are named<github-userid>/description. mainmust always be in a working state and stays close to the latest release. Do not commit directly tomain(PRs only).main-v1is a separate diverged branch for Awkward 1.x fixes.- Changes to core dependencies (NumPy, Numba, Pandas, PyArrow) or Python version support must be discussed with maintainers first.
- If a PR requires C++ changes, the
awkward-cppversion must be bumped in bothawkward-cpp/pyproject.tomland the pin inpyproject.toml, and anawkward-cpprelease made before theawkwardrelease. - AI-assisted contributions must be disclosed in the PR description (see CONTRIBUTING.md).
nox -s prepare -- --docs --headers
nox -s docs -- --serve # requires Doxygen; serves at localhost:8000