Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
123 commits
Select commit Hold shift + click to select a range
3d27d0c
feat: major refactor
martjanz May 27, 2026
645f2e0
fix: unregister DuckDB views and include dia in trip groupby
martjanz May 29, 2026
fc7562b
fix: guard save_legs against empty DataFrame early return
martjanz May 29, 2026
2bcaa02
docs: add step CLI design spec
martjanz May 29, 2026
417c7e3
docs: add step CLI implementation plan
martjanz May 29, 2026
094ce35
docs: add incremental run behaviour notes to step CLI spec
martjanz May 29, 2026
f05c6f5
refactor: extract run_ingest/legs/outputs/dashboard public step funct…
martjanz May 29, 2026
52b3a7f
feat: add check_prerequisites step guard
martjanz May 29, 2026
5c796c5
feat: add --step and --through CLI flags for single-step execution
martjanz May 29, 2026
86fdc06
fix: use h3 column name and polygon centroid correctly
martjanz May 29, 2026
951de58
docs: add scaling considerations and memory guidelines to step CLI spec
martjanz May 29, 2026
529cb30
feat: auto-tune n_batches from available RAM when not configured
martjanz May 29, 2026
8ffa687
feat: improve n_batches logging with RAM stats, source, and decision …
martjanz May 29, 2026
bb9ae93
merge: integrate validar_velocidad into feat/refactor-clean
martjanz May 31, 2026
b05b747
docs: add merge report for validar_velocidad into refactor-clean
martjanz May 31, 2026
7d09822
fix: remove dead VV code block using undefined conn in create_trips_f…
martjanz May 31, 2026
ea2b5b9
fix: align run_basic_kpi to use kmh_route_veh_h (renamed by VV helper…
martjanz May 31, 2026
7c4e850
fix: align compute_basic_kpi_line_typeday/hr_typeday schema to match …
martjanz May 31, 2026
ebe0789
fix: wire assign_time_distances into _enrich_all_legs; safe DELETE fo…
martjanz May 31, 2026
1c0c0ee
fix: drop travel_times tables before recreating to enforce current sc…
martjanz May 31, 2026
fc54f17
fix: update travel_times schema to 12/10 cols matching assign_time_di…
martjanz May 31, 2026
2a5eb17
fix: rename old column names (distancia, travel_speed, dmt_mean, ipk,…
martjanz May 31, 2026
d0ae531
fix: restore metric_cols definition and fix all/all_linea/all_ramal i…
martjanz May 31, 2026
59bb791
fix: update services schema and callers to use distance_route instead…
martjanz May 31, 2026
fc4bb7d
feat: add --config flag to point to an alternative YAML config file
martjanz May 31, 2026
7ea699f
feat: separate alias_db (outputs) from alias_db_insumos, add db_path …
martjanz May 31, 2026
47298bb
fix: remove legacy SQLite block from borrar_corridas that crashed on …
martjanz May 31, 2026
fb93158
fix: URBANTRIPS_CONFIG env var only overrides autogenerado=False read…
martjanz May 31, 2026
48d93f9
refactor: bypass autogenerado config in run_process — derive filename…
martjanz May 31, 2026
d5d177d
fix: remove stray compute_distance_km_gps call with invalid use_panda…
martjanz Jun 1, 2026
33f142e
fix: add id_servicio, distance_servicio_mts, distance_servicio_mts_ag…
martjanz Jun 1, 2026
68d89dc
fix: use autogenerado=False throughout pipeline — eliminates dependen…
martjanz Jun 1, 2026
d971510
fix: add distance_servicio_mts to GPS query in compute_speed_by_day_v…
martjanz Jun 1, 2026
9705d39
fix: replace pd.read_sql(conn_data) with ctx.data.query in compute_kp…
martjanz Jun 1, 2026
1dee4d1
fix: use h3_o instead of h3 in outputs prerequisite check (etapas has…
martjanz Jun 1, 2026
f38f0ec
refactor(dashboard): support URBANTRIPS_CONFIG env var and resolve DB…
martjanz Jun 1, 2026
549a76e
fix: add observed=True to all groupby calls on categorical columns
martjanz Jun 1, 2026
536d01a
Restore deleted test files
martjanz Jun 1, 2026
4863180
Restore deleted file
martjanz Jun 1, 2026
cac54ee
add Makefile and document dev workflow in README
martjanz Jun 1, 2026
17aec00
Fix tests
martjanz Jun 1, 2026
3cd492a
Restrict tests on push to specific branches
martjanz Jun 1, 2026
a1a69e7
Simplify card ID update to direct column assignment
martjanz Jun 1, 2026
a526476
Update gitignore
martjanz Jun 1, 2026
aa02629
Update pyproject and test fixture files
martjanz Jun 1, 2026
345f339
Fix database alias
martjanz Jun 2, 2026
d61823d
computa distancias con compute_distance_km_gps
alephcero Jun 2, 2026
cd4e31e
Catch `BaseException` in duration decorator to log timing on Keyboard…
martjanz Jun 2, 2026
1065641
Rewrite README.md and README_en.md with structured docs
martjanz Jun 2, 2026
2377192
Fix id_etapa missing from travel times and polygon centroid CRS
martjanz Jun 2, 2026
92fd1b4
Add tests to test_legs
martjanz Jun 2, 2026
cf6ea61
Add performance improvements design spec (B1-B6)
martjanz Jun 2, 2026
1897ea5
Add implementation plans for performance improvements B1-B6
martjanz Jun 2, 2026
bd43977
perf(b6): persistent DuckDB connection in all adapters
martjanz Jun 2, 2026
6b59989
perf(b2): remove defensive DataFrame copies at orchestration level
martjanz Jun 2, 2026
11aea11
perf(b4): replace point-grid with h3.geo_to_cells in select_h3_from_p…
martjanz Jun 2, 2026
2e8922c
perf(b3): DBSCAN grid reduction, early stopping, and tuning config
martjanz Jun 2, 2026
a0c75da
perf(b5): push scalar derived columns into DuckDB SQL in load_and_pro…
martjanz Jun 2, 2026
d1415eb
perf(b1): replace 9 pandas groupby scans with DuckDB queries in const…
martjanz Jun 3, 2026
518fc05
Merge perf/b2-remove-copies (resolve test file conflict)
martjanz Jun 3, 2026
d273744
Merge perf/b4-h3-polygon-fill (resolve test file conflict)
martjanz Jun 3, 2026
9b82b54
Merge perf/b3-dbscan-grid (resolve test file conflict)
martjanz Jun 3, 2026
ffe1214
Merge perf/b5-derived-columns-sql (resolve test file conflict)
martjanz Jun 3, 2026
e3dea3f
Merge perf/b1-construyo-indicadores (resolve test file conflict)
martjanz Jun 3, 2026
9e2d6dc
Fix typo in B5 test fixture (outros -> otros)
martjanz Jun 3, 2026
ba5c3e0
perf(micro): fix remaining apply(h3.cell_to_parent) in viz.py, mark P…
martjanz Jun 3, 2026
87f7149
fix_destinations_legs
sanapolsky Jun 3, 2026
b2ad30c
add batches to config
sanapolsky Jun 3, 2026
3ead46a
fix_destinations_legs
sanapolsky Jun 3, 2026
8540a05
add batches to config
sanapolsky Jun 3, 2026
e942417
refactor: chunk-based parallel leg building with DuckDB memory cap
martjanz Jun 4, 2026
f34b3e9
refactor(run_process): align n_batches to cpu_workers multiple
martjanz Jun 4, 2026
71cc800
perf(trips): replace DELETE+INSERT with in-place UPDATE for trip ID r…
martjanz Jun 4, 2026
96ddaeb
update_configuraciones.xlsx
sanapolsky Jun 4, 2026
d3065d3
perf(legs): defer h3 distance calc after time filter, dedupe pairs
martjanz Jun 7, 2026
f5b1267
perf(legs): parallelize GPS destination imputation
martjanz Jun 8, 2026
8d32346
fix: macOS GPS serial fallback, selective destination UPDATE
martjanz Jun 8, 2026
43784ef
fix: correct hora_inicio/hora_fin type to TEXT, add migration
martjanz Jun 8, 2026
60b3f47
Merge feat/refactor-clean-merge-vv-improv into feat/improvements-tmp
martjanz Jun 8, 2026
a1f3b54
fix dashboard & varios
martjanz Jun 8, 2026
32c0d6d
docs: add base-dir isolation design spec
martjanz Jun 9, 2026
a8f20f8
docs: add base-dir isolation implementation plan
martjanz Jun 9, 2026
eee1064
feat: add Paths singleton module for base-dir resolution
martjanz Jun 9, 2026
9256f45
fix: propagate YAML errors in init_paths, fix test assertions
martjanz Jun 9, 2026
f074d2c
feat: add optional input_dir/db_dir/output_dir to Config
martjanz Jun 9, 2026
3bf1728
feat: add --base-dir CLI flag, call init_paths at startup
martjanz Jun 9, 2026
bb34ef8
refactor: use get_paths() in leer_configs_generales and leer_configs_…
martjanz Jun 9, 2026
97414ae
refactor: use get_paths() in fs.create_directories and check_configs
martjanz Jun 9, 2026
a058732
fix: correct logging statement in check_configs_file after refactor
martjanz Jun 9, 2026
61547f1
refactor: use get_paths() for db and input paths in run_process
martjanz Jun 9, 2026
57ae9a9
refactor: use get_paths().input_dir in carto, datamodel, viz
martjanz Jun 9, 2026
196f995
refactor: use get_paths().output_dir in viz modules
martjanz Jun 9, 2026
7157d39
refactor: use get_paths().output_dir in kpi, cluster, preparo_dashboard
martjanz Jun 9, 2026
df400b0
refactor: use get_paths() in dashboard storage and utils
martjanz Jun 9, 2026
874fe06
refactor: replace hardcoded db_dir in dash_utils get_db_path
martjanz Jun 9, 2026
4cb6412
fix: preserve input_dir/db_dir/output_dir through config rewrite
martjanz Jun 9, 2026
37939f3
fix: stop overwriting configuraciones_generales.yaml on each run
martjanz Jun 9, 2026
b180f1a
feat: auto-fallback to .csv.zip when .csv not found
martjanz Jun 9, 2026
5bb8d0b
fix: zip fallback in check_configs file existence checks
martjanz Jun 9, 2026
0352003
fix: zip fallback in check_lineas and lineas file read
martjanz Jun 9, 2026
3ab570a
fix: handle macOS __MACOSX entries in zip files via open_csv helper
martjanz Jun 9, 2026
5e3c63a
fix: fill nombre_linea from id_linea when null; zip support in routes
martjanz Jun 9, 2026
c020f9b
feat: accept explicit config_file path in init_paths
martjanz Jun 9, 2026
5e71b93
perf: process destinations day-by-day; vectorize h3 distance
martjanz Jun 9, 2026
2b697ac
fix dash
sanapolsky Jun 9, 2026
a6d6757
fix: fill nombre_ramal from id_ramal when absent
martjanz Jun 10, 2026
75d75f0
perf: narrow frames and free memory early in dashboard/kpi
martjanz Jun 10, 2026
55db403
fix: pin DuckDB global connection memory limits at startup
martjanz Jun 10, 2026
66dc4ff
perf: process lineas_deseo day-by-day to cap working set
martjanz Jun 10, 2026
2c119cb
fix: add @duracion decorators and progress logging
martjanz Jun 10, 2026
2164965
fix: use human-readable label for NONE polygon in log
martjanz Jun 10, 2026
e8b1f37
perf: hoist zonificaciones lat/lon and cuenca_ids out of inner loop
martjanz Jun 10, 2026
d9cbaed
perf: cache creo_h3_equivalencias per (polygon, zona) before day loop
martjanz Jun 10, 2026
31ad947
perf: replace 4 h3_coords merges with O(N) map() dict lookups
martjanz Jun 10, 2026
d3b3d8d
perf: precompute poly_centroid and poly_h3_set once per zona, not per…
martjanz Jun 10, 2026
24e894e
perf: precompute h3->parent maps per resolution before zona loop
martjanz Jun 10, 2026
cd82d74
perf: replace copy+8 assignments with single assign() in zona loop
martjanz Jun 10, 2026
5db8087
fix: add progress logging to select_cases_from_polygons and h3_equiv …
martjanz Jun 10, 2026
49a0d2c
Potential fix for pull request finding
martjanz Jun 12, 2026
e8f4e77
Potential fix for pull request finding
martjanz Jun 12, 2026
bfeb373
Fix F821 lint error: replace undefined line_od with df_agg parameter …
Copilot Jun 12, 2026
87267cb
Potential fix for pull request finding
martjanz Jun 12, 2026
97f040e
Potential fix for pull request finding
martjanz Jun 12, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 21 additions & 95 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
@@ -1,102 +1,28 @@
# This workflow will install Python dependencies, run tests and lint with a variety of Python versions
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions
# For more information see: https://docs.github.com/en/actions/automating-builds-and-tests/building-and-testing-python
name: Publish

name: Build
on:
push:
branches:
- main
- dev
pull_request:
types: [opened, synchronize, reopened]
branches:
- main
- dev
tags:
- "v*"

jobs:
sonarcloud:
name: SonarCloud
publish:
name: Publish Python distributions
runs-on: ubuntu-latest
permissions:
id-token: write
steps:
- uses: actions/checkout@v2
with:
fetch-depth: 0 # Shallow clones should be disabled for a better relevancy of analysis
- name: Setup Python
uses: actions/setup-python@v2
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python }}
- name: Install tox and any other packages
run: pip install tox
- name: Run tox
run: tox -e py
- name: SonarCloud Scan
uses: SonarSource/sonarcloud-github-action@master
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # Needed to get PR information, if any
SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}
docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: ammaraskar/sphinx-action@master
with:
docs-folder: "docs/"
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]

steps:
- uses: actions/checkout@v3
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Lint with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Test with pytest
run: |
pytest

build-n-publish:
name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@master
- name: Set up Python 3.10
uses: actions/setup-python@v3
with:
python-version: "3.10"

- name: Install pypa/build
run: >-
python -m
pip install
build
--user
- name: Build a binary wheel and a source tarball
run: >-
python -m
build
--sdist
--wheel
--outdir dist/
.

- name: Publish distribution 📦 to PyPI
if: startsWith(github.ref, 'refs/tags')
uses: pypa/gh-action-pypi-publish@release/v1
with:
password: ${{ secrets.PYPI_API_TOKEN }}
python-version: "3.12"
cache: pip
- name: Install build tooling
run: |
python -m pip install --upgrade pip
python -m pip install build
- name: Build distributions
run: python -m build --sdist --wheel --outdir dist/ .
- name: Publish distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
20 changes: 20 additions & 0 deletions .github/workflows/docs.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
name: Docs
on:
push:
branches:
- main
- dev
pull_request:
types: [opened, synchronize, reopened]
branches:
- main
- dev

jobs:
docs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: ammaraskar/sphinx-action@master
with:
docs-folder: "docs/"
87 changes: 87 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,87 @@
name: CI

on:
push:
branches:
- main
- dev
pull_request:

jobs:
lint:
name: Lint
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Run Ruff
run: ruff check urbantrips

unit-tests:
name: Unit Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Run unit tests
run: pytest urbantrips/tests/unit -q --cov=urbantrips --cov-report=xml:coverage-unit.xml
- name: Upload coverage report
uses: actions/upload-artifact@v4
with:
name: coverage-unit
path: coverage-unit.xml

integration-tests:
name: Integration Tests
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Run integration tests
run: pytest urbantrips/tests/integration -q --cov=urbantrips --cov-report=xml:coverage-integration.xml
- name: Upload coverage report
uses: actions/upload-artifact@v4
with:
name: coverage-integration
path: coverage-integration.xml

package-build:
name: Package Build
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: "3.12"
cache: pip
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install -e ".[dev]"
- name: Build package
run: python -m build
Loading
Loading