Multiprocessing treats --ext filesystem paths as module names, breaking nested and absolute extension paths

[jpn--]

Multiprocessing treats --ext filesystem paths as module names, breaking nested and absolute extension paths

activitysim run --ext model/extensions imports the extension successfully in the parent process, but fails when multiprocessing reimports it in a worker. The same extension and configuration work in a single-process run. Absolute extension paths fail too.

This is deterministic. The apparent dependency on the launch directory arises because changing into model allows the argument to be shortened to the bare package name extensions. Even from that directory, --ext ./extensions fails while --ext extensions succeeds.

Environment

• Verified against unmodified upstream ActivitySim commit 8ca6741c83e44e70fbfe718cb91066dece989805 (upstream main when checked).
• Python 3.10.20; macOS 26.6.2, Apple Silicon.
• Actual ActivitySim console entry point, using its multiprocessing worker path. One worker is sufficient to reproduce the failure.
• No Lighthouse code, model inputs, skims, or downloads are used by the reproducer.

Standalone reproducer

Save the following as reproduce.py and run python reproduce.py in an environment with this ActivitySim revision installed and activitysim on PATH. Alternatively run python reproduce.py /absolute/path/to/activitysim.

To install the tested revision in a separate environment:

python -m venv .venv
. .venv/bin/activate
python -m pip install 'git+https://github.com/ActivitySim/activitysim.git@8ca6741c83e44e70fbfe718cb91066dece989805'
python reproduce.py

The script creates a temporary directory, a tiny extension with one printing step, empty data directories, and one settings file. Three extension overrides disable unused shared-data loading so no model data are needed. Configuration, data, and output arguments are absolute to isolate extension handling. Full logs and generated files are retained at the printed location.

"""Standalone ActivitySim CLI reproducer; no model data or repository required.

Run in an environment with ActivitySim installed:
    python reproduce.py
Or select a console script explicitly:
    python reproduce.py /absolute/path/to/activitysim
"""

import shutil
import subprocess
import sys
import tempfile
from pathlib import Path

EXTENSION = '''\
import multiprocessing
from activitysim.core import workflow

print(f"EXTENSION IMPORTED: {multiprocessing.current_process().name}", flush=True)

# Disable shared model-data allocation; this toy model has no input data.
@workflow.step(cache=True, kind="cached_object", overloading=True)
def network_los_preload(state: workflow.State):
    return None

@workflow.step(cache=True, kind="cached_object", overloading=True)
def shadow_pricing_info(state: workflow.State):
    return None

@workflow.step(cache=True, kind="cached_object", overloading=True)
def shadow_pricing_choice_info(state: workflow.State):
    return None

@workflow.step
def hello(state: workflow.State):
    print(f"HELLO: {multiprocessing.current_process().name}", flush=True)
'''

SETTINGS = '''\
models: [hello]
multiprocess: false
num_processes: 1
multiprocess_steps:
  - name: mp_hello
    begin: hello
    num_processes: 1
check_model_settings: false
memory_profile: false
sharrow: false
use_shadow_pricing: false
'''


def main():
    executable = shutil.which(sys.argv[1] if len(sys.argv) > 1 else "activitysim")
    if executable is None:
        raise SystemExit("Install ActivitySim or supply the path to its console script.")
    executable = str(Path(executable).absolute())
    root = Path(tempfile.mkdtemp(prefix="activitysim-ext-repro-"))
    model = root / "model"
    for directory in ("configs", "data", "extensions"):
        (model / directory).mkdir(parents=True)
    (model / "extensions" / "__init__.py").write_text(EXTENSION)
    (model / "configs" / "settings.yaml").write_text(SETTINGS)
    print(f"Reproducer and full logs: {root}", flush=True)

    # One subprocess is enough: -m still uses ActivitySim's worker setup.
    cases = [
        ("single-relative", root, "model/extensions", []),
        ("mp-relative", root, "model/extensions", ["-m"]),
        ("mp-absolute", root, str(model / "extensions"), ["-m"]),
        ("mp-bare", model, "extensions", ["-m"]),
        ("mp-dot-slash", model, "./extensions", ["-m"]),
        ("mp-working-dir", root, "extensions", ["-m", "-w", str(model)]),
    ]
    for name, cwd, extension, extra in cases:
        command = [executable, "run", "-c", str(model / "configs"),
                   "-d", str(model / "data"), "-o", str(root / name),
                   "--ext", extension, *extra]
        result = subprocess.run(command, cwd=cwd, text=True,
                                stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                                timeout=120)
        (root / f"{name}.log").write_text(result.stdout)
        print(f"\n{name}: exit={result.returncode}", flush=True)
        for line in result.stdout.splitlines():
            if any(token in line for token in
                   ("EXTENSION IMPORTED:", "HELLO:", "ModuleNotFoundError:",
                    "TypeError:", "RuntimeError:")):
                print(line, flush=True)


if __name__ == "__main__":
    main()

Expected and actual behavior

All six cases should run the same hello step successfully. Observed:

CWD	Extension argument	Mode	Actual result
temporary root	model/extensions	Single process	Exit 0; HELLO: MainProcess
temporary root	model/extensions	Multiprocessing	Exit 99; ModuleNotFoundError
temporary root	absolute path to model/extensions	Multiprocessing	Exit 99; ModuleNotFoundError
model	extensions	Multiprocessing	Exit 0; HELLO: mp_hello
model	./extensions	Multiprocessing	Exit 99; TypeError
temporary root, with -w /absolute/path/to/model	extensions	Multiprocessing	Exit 0; HELLO: mp_hello

The nested-path failure includes:

EXTENSION IMPORTED: MainProcess
  File ".../activitysim/core/mp_tasks.py", line 937, in setup_injectables_and_logging
    importlib.import_module(e)
ModuleNotFoundError: No module named 'model/extensions'

For ./extensions, the worker instead raises:

TypeError: the 'package' argument is required to perform a relative import for './extensions'

The generated minimal commands are equivalent to:

# From the generated temporary root: succeeds.
activitysim run -c model/configs -d model/data -o output-single --ext model/extensions

# Same CWD and extension: fails during worker setup.
activitysim run -c model/configs -d model/data -o output-mp --ext model/extensions -m

# From the extension's parent directory: succeeds.
cd model
activitysim run -c configs -d data -o ../output-control --ext extensions -m

Cause

The CLI parent loader splits the filesystem path into basepath and extpath, temporarily adds the absolute parent directory to sys.path, and calls importlib.import_module(extpath). It then saves the original argument in imported_extensions.

The worker loader retrieves that original argument and splits it in the same way, but calls importlib.import_module(e) instead of importlib.import_module(extpath). For model/extensions, it therefore tries to import the literal module name model/extensions. Adding model to sys.path cannot make that filesystem path a valid package name. An absolute path has the same problem; a leading ./ is interpreted as a relative Python import.

This is a mismatch between the parent and worker loaders, beyond the normal rule that relative filesystem paths are resolved against a working directory. Bare names also can resolve through Python's normal module search path, so extensions are not inherently required to physically reside in the launch directory.

The immediate CLI defect is the use of e instead of extpath in the worker. A robust fix should share path-resolution/import logic between parent and workers and preserve an absolute search directory plus a module name, so worker imports do not depend on their current directory. Regression coverage should include relative, absolute, ./, and bare-name inputs, plus an explicit working directory.

Python API scope

State.import_extensions() also imports the basename in the parent and records the original argument. It therefore does not generally bypass this defect: I separately verified that state.import_extensions("model/extensions") succeeds, records ["model/extensions"], and passing that record to setup_injectables_and_logging() produces the same ModuleNotFoundError. That additional check called worker setup directly; the six CLI cases above exercised actual subprocesses.

Workaround

Use a bare extension package name from its parent directory, or set that directory with -w:

activitysim run -w /absolute/path/to/model --ext extensions -m

This uses the usual configs, data, and output subdirectories of the selected working directory. Supplying an absolute extension path does not work around the bug.

Related report: Lighthouse PR discussion.