Skip to content

[ADD] oca-upstream-sync: weekly drift check against upstream OCA #241

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AungKoKoLin1997 wants to merge 1 commit into
base: 16.0
Choose a base branch
from
Open

[ADD] oca-upstream-sync: weekly drift check against upstream OCA #241

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
163 changes: 163 additions & 0 deletions .github/workflows/oca-upstream-sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,163 @@
name: OCA upstream sync

# Weekly drift check against the upstream OCA repos the vendored modules came
# from. The scheduled run only ever reports -- it opens or updates one issue and
# changes no code. Writing is a deliberate act: run the workflow by hand with
# mode=apply (safe fast-forwards) or mode=merge (a named patched module).

on:
schedule:
# Mondays, 02:20 UTC = 09:20 Bangkok.
- cron: "20 2 * * 1"
workflow_dispatch:
inputs:
mode:
description: "What to do"
required: true
default: check
type: choice
options:
- check
- discover-baselines
- apply
- merge
modules:
description: "Module(s) for apply/merge — space separated; blank = all syncable"
required: false
type: string

permissions:
contents: write
issues: write
pull-requests: write

concurrency:
group: oca-upstream-sync
cancel-in-progress: false

env:
SERIES: "16.0"
MODE: ${{ inputs.mode || 'check' }}

jobs:
sync:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v7
with:
ref: ${{ env.SERIES }}

- uses: actions/setup-python@v5
with:
python-version: "3.11"

- run: pip install --quiet pyyaml

# Upstream clones are blobless and reused across runs; a stale cache only
# costs a fetch, never correctness, since every ref is resolved by sha.
- uses: actions/cache@v4
with:
path: .oca-sync-cache
key: oca-sync-${{ env.SERIES }}-${{ hashFiles('.oca-sync.yml') }}
restore-keys: oca-sync-${{ env.SERIES }}-

- name: Report drift
if: env.MODE == 'check'
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
python3 scripts/sync_oca_modules.py --check \
--workdir .oca-sync-cache \
--report-file drift.md

# One issue, reused week to week, so the drift lives at a stable URL
# instead of accumulating a pile of near-identical issues.
- name: Open or update the drift issue
if: env.MODE == 'check'
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
title="OCA upstream drift — ${SERIES}"
existing=$(gh issue list --state open --label oca-upstream-sync \
--search "$title in:title" --json number --jq '.[0].number // empty')
gh label create oca-upstream-sync --color ededed \
--description "Weekly OCA upstream drift report" 2>/dev/null || true
if [ -n "$existing" ]; then
gh issue edit "$existing" --body-file drift.md
echo "updated issue #$existing"
else
gh issue create --title "$title" --body-file drift.md \
--label oca-upstream-sync
fi

- name: Discover baselines
if: env.MODE == 'discover-baselines'
env:
GITHUB_TOKEN: ${{ github.token }}
run: |
python3 scripts/sync_oca_modules.py --discover-baselines \
--workdir .oca-sync-cache

- name: Apply safe syncs
if: env.MODE == 'apply'
env:
GITHUB_TOKEN: ${{ github.token }}
# Via env, not interpolated into the script: a dispatch input is
# attacker-controlled text as far as the shell is concerned.
MODULES: ${{ inputs.modules }}
run: |
# shellcheck disable=SC2086
python3 scripts/sync_oca_modules.py --apply $MODULES \
--workdir .oca-sync-cache

- name: Merge upstream into a patched module
if: env.MODE == 'merge'
env:
GITHUB_TOKEN: ${{ github.token }}
MODULES: ${{ inputs.modules }}
# Conflicts are the expected outcome often enough that they must not
# fail the job: the half-merged tree with its markers is the deliverable,
# and the PR is where someone resolves it.
continue-on-error: true
run: |
python3 scripts/sync_oca_modules.py --merge "$MODULES" \
--workdir .oca-sync-cache

- name: Open a draft PR with whatever changed
if: env.MODE == 'apply' || env.MODE == 'merge' || env.MODE == 'discover-baselines'
env:
GH_TOKEN: ${{ github.token }}
run: |
set -euo pipefail
rm -rf .oca-sync-cache
if [ -z "$(git status --porcelain)" ]; then
echo "nothing changed."
exit 0
fi
branch="bot/oca-sync-${MODE}-$(date -u +%Y%m%d-%H%M)"
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
git checkout -b "$branch"
git add -A
git commit -m "[IMP] oca-sync: ${MODE}"
git push -u origin "$branch"
{
echo "Opened by the \`OCA upstream sync\` workflow (mode: \`${MODE}\`)."
echo
if [ "$MODE" = "merge" ]; then
echo "**This is a three-way merge into a locally patched module.**"
echo "Check for conflict markers before reviewing anything else:"
echo
echo '```'
git grep -n '^<<<<<<< ' -- . || echo "none"
echo '```'
echo
fi
echo "Review the diff as a normal upstream sync. CI does not run on"
echo "branches pushed with \`GITHUB_TOKEN\`, so close and reopen the PR"
echo "(or push a commit) if you need the test workflow to fire."
} > body.md
gh pr create --draft --base "$SERIES" --head "$branch" \
--title "[IMP] oca-sync: ${MODE}" --body-file body.md
rm -f body.md
25 changes: 25 additions & 0 deletions .oca-sync-baselines.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Generated by scripts/sync_oca_modules.py -- do not edit by hand.
#
# Each entry records the upstream commit a vendored copy was taken from, and a
# fingerprint of the copy as it stood at that moment. The fingerprint is what
# lets the weekly job prove a copy is still untouched before overwriting it.
#
# exact: true -- the copy was byte-identical to that upstream commit, so an
# overwrite can lose nothing, and the module can auto-sync.
# exact: false -- the copy carried files of its own even then, so upstream
# changes have to be merged rather than copied.

OCA/purchase-workflow/purchase_exception:
exact: true
fingerprint: 0554f370f5c5c187
how: "discovered: byte-identical to this upstream commit"
recorded_at: "2026-09-11"
upstream_sha: 5e919619a6a21fb00b1b526ee41e889c9f9270e4
OCA/purchase-workflow/purchase_invoice_plan:
exact: false
fingerprint: 0bf057bcb601f3e1
how:
"discovered: code matches this upstream commit, but the copy carries files of its
own"
recorded_at: "2026-09-11"
upstream_sha: 0b68034402a33154dff17d766a2110e45c5edac8
32 changes: 32 additions & 0 deletions .oca-sync.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Configuration for scripts/sync_oca_modules.py — see README-oca-upstream-sync.md
#
# Every module vendored into this repo may be listed here. A module is only ever
# written to automatically when its recorded baseline proves the local copy is an
# untouched copy of upstream; anything else is reported for a human to decide.

series: "16.0"

# Paths compared but reported separately, so translation and generated-file churn
# does not drown out real code changes. Globs are relative to the module root.
noise:
- "i18n/*.po"
- "i18n/*.pot"
- "README.rst"
- "static/description/index.html"

sources:
- repo: OCA/server-tools
modules:
# Locally patched: qrtl #172 (res.groups write-override test patch) and
# tests/test_multi_company.py, which does not exist upstream. Deliberately
# left without a baseline so it can never be auto-synced.
- name: auditlog
overlays: []

- repo: OCA/purchase-workflow
modules:
- name: purchase_exception
overlays: []
# Carries a local i18n/ja.po that upstream does not have.
- name: purchase_invoice_plan
overlays: []
160 changes: 160 additions & 0 deletions README-oca-upstream-sync.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
# Keeping vendored OCA modules in step with upstream

This repo is a flat collection of addons copied out of various OCA repositories. Copies
drift: OCA fixes a bug, nobody notices for eight months, and the copy here quietly falls
behind. Noticing by hand every week does not happen.

`.github/workflows/oca-upstream-sync.yml` runs every Monday and reports the drift into a
single GitHub issue. It changes no code on a schedule — writing is always something a
person triggers.

## The problem this design exists to solve

The obvious version of this tool diffs each vendored module against upstream's branch
tip and opens a PR when they differ. That version is dangerous, because a two-way diff
cannot answer the only question that matters:

> Is this difference something OCA changed, or something _we_ changed?

Several modules here are patched on purpose. `auditlog` carries qrtl #172 and a
`tests/test_multi_company.py` that does not exist upstream; `purchase_invoice_plan`
carries a `i18n/ja.po` that upstream does not ship. A tool that cannot see the
difference will revert all of it, inside a bot PR that looks like every other routine
sync and gets waved through.

So every module gets a **baseline** in `.oca-sync-baselines.yml`: the upstream commit
the copy was taken from, plus a fingerprint of the copy as it stood at that moment. That
turns one useless diff into two useful ones:

```
diff(upstream@baseline, local copy) -> what we changed
diff(upstream@baseline, upstream@tip) -> what OCA changed
```

Safety stops being a promise someone wrote in a config file and becomes a measurement
taken on every run. Nothing is auto-written unless the first diff is empty.

## Statuses

| status | meaning | what happens |
| -------------- | --------------------------------------------------------------- | -------------------------------------------------- |
| `up to date` | matches upstream | nothing |
| `syncable` | upstream moved; copy is byte-identical to its baseline | `--apply` can update it; nothing local can be lost |
| `needs merge` | upstream moved **and** the copy is patched | left alone; use `--merge` |
| `local only` | copy differs but upstream has not moved | consider contributing it upstream |
| `unattributed` | no baseline, so the two diffs cannot be separated | run `--discover-baselines` |
| `blocked` | an overlay PR was closed unmerged, vanished, or stopped merging | left alone; a human decides |
| `not found` | configured but missing here or upstream | fix the config |

`unattributed` is the honest state for a copy nobody has attributed yet, not an error.
Modules leave it as they are adopted.

## Configuration

`.oca-sync.yml` is hand-written and says _what to track_:

```yaml
series: "16.0"

noise:
- "i18n/*.po"
- "README.rst"

sources:
- repo: OCA/purchase-workflow
modules:
- name: purchase_invoice_plan
overlays: []
```

`.oca-sync-baselines.yml` is machine-written and says _what was last synced_. Keeping
them apart means the generator never eats the comments in the file humans maintain.

### Noise

`noise` paths are still compared and still block an automatic sync — they are only
_counted separately_ in the report. `auditlog` differs from upstream in 84 files, of
which 73 are Transifex translation churn. Without the split, the 11 that are real code
are invisible.

### Overlay PRs

When a copy came from an OCA pull request that had not merged yet, say so:

```yaml
- name: purchase_invoice_plan
overlays:
- pr: 2500
```

Each run then re-checks that PR:

- **open** — merged onto the branch tip before comparing, so the copy is compared
against what it is actually supposed to be.
- **merged** — not applied separately, but only after confirming the merge commit is
genuinely an ancestor of this branch. "Merged" alone is not enough: a PR can merge to
a different branch, or merge and be reverted.
- **closed without merging** — the module is marked `blocked` and left untouched. The
code came from somewhere that no longer exists, and deleting functionality someone is
using is not a decision a cron job should make.

## Usage

The weekly run needs nothing. By hand, from the Actions tab, _Run workflow_ with a mode;
or locally:

```bash
pip install pyyaml

# What has drifted?
python3 scripts/sync_oca_modules.py --check

# Adopt modules whose copies still match some upstream commit.
python3 scripts/sync_oca_modules.py --discover-baselines

# Update copies proved untouched (all syncable ones if none named).
python3 scripts/sync_oca_modules.py --apply purchase_exception

# Bring upstream into a patched copy, keeping the local work.
python3 scripts/sync_oca_modules.py --merge auditlog
```

`--apply` refuses anything not `syncable`, even when named explicitly.

`--merge` builds a real three-way merge — baseline as the common ancestor, our copy
against upstream's — so local changes survive and genuine clashes come back as conflict
markers rather than being silently resolved in upstream's favour. It is expected to
conflict sometimes; that is the tool working.

## Adopting the rest of the repo

Only the modules listed in `.oca-sync.yml` are looked at. Adding the other ~94 is the
actual work, and `--discover-baselines` does most of it: it walks upstream history
looking for the commit whose tree matches the local copy, so a pristine copy attributes
itself. A copy that matches nothing is patched, and says so.

Suggested order:

1. Add a batch of modules to `.oca-sync.yml` with `overlays: []`.
2. Run `--discover-baselines`, commit the result.
3. Run `--check` and read the report before enabling anything.

For a module that matches nothing, someone has to decide what the local changes are — a
deliberate patch worth keeping, or drift worth discarding — and either record a baseline
by hand or push the change upstream. There is no way to automate that judgement, which
is why the tool reports it instead of guessing.

## Known rough edges

- **CI does not run on the bot's PRs.** Branches pushed with `GITHUB_TOKEN` do not
trigger other workflows. Close and reopen the PR, or push a commit, to make the test
workflow fire. A PAT in a secret would avoid this at the cost of holding a PAT.
- **Regenerated files flip a module out of `syncable`.** If `oca-gen-addon-readme`
rewrites `README.rst` after a sync, the copy no longer matches its baseline. The
report shows it as `needs merge` with the churn confined to the `i18n/gen` column.
Re-run `--apply` after committing the regeneration to re-baseline.
- **The right answer is sometimes to stop vendoring.** A module that is a pure upstream
copy with no patches and no overlay does not need syncing so much as it needs to not
be a copy: aggregate `OCA/<repo>` in `repos.yml` and upstream fixes arrive with
`git pull`. That only applies to modules genuinely free of local changes, which is
exactly what the report identifies.
Loading
Loading