Merge from upstreram#17
Open
aramikm wants to merge 177 commits into
Open
Conversation
Bumps [actions/setup-python](https://github.com/actions/setup-python) from 6.0.0 to 6.2.0. - [Release notes](https://github.com/actions/setup-python/releases) - [Commits](actions/setup-python@e797f83...a309ff8) --- updated-dependencies: - dependency-name: actions/setup-python dependency-version: 6.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.32.3 to 4.32.4. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@f5c2471...89a39a4) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.32.4 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 3.1.5 to 4.8.3. - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@c74b580...05fe457) --- updated-dependencies: - dependency-name: actions/dependency-review-action dependency-version: 4.8.3 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [microsoft/security-devops-action](https://github.com/microsoft/security-devops-action) from 1.11.0 to 1.12.0. - [Release notes](https://github.com/microsoft/security-devops-action/releases) - [Commits](microsoft/security-devops-action@cc007d0...08976cb) --- updated-dependencies: - dependency-name: microsoft/security-devops-action dependency-version: 1.12.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/github-script](https://github.com/actions/github-script) from 7.0.1 to 8.0.0. - [Release notes](https://github.com/actions/github-script/releases) - [Commits](actions/github-script@60a0d83...ed59741) --- updated-dependencies: - dependency-name: actions/github-script dependency-version: 8.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [azure/login](https://github.com/azure/login) from 2.1.1 to 2.3.0. - [Release notes](https://github.com/azure/login/releases) - [Commits](Azure/login@6c25186...a457da9) --- updated-dependencies: - dependency-name: azure/login dependency-version: 2.3.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.7.1 to 3.12.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@c47758b...8d2750c) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 3.12.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/cache](https://github.com/actions/cache) from 4.2.0 to 5.0.3. - [Release notes](https://github.com/actions/cache/releases) - [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md) - [Commits](actions/cache@1bd1e32...cdf6c1f) --- updated-dependencies: - dependency-name: actions/cache dependency-version: 5.0.3 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/checkout](https://github.com/actions/checkout) from 4.2.2 to 6.0.2. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](actions/checkout@11bd719...de0fac2) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: 6.0.2 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 4.0.1 to 5.1.0. - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](actions/setup-dotnet@6bd8b7f...baa11fb) --- updated-dependencies: - dependency-name: actions/setup-dotnet dependency-version: 5.1.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps python from `9e01bf1` to `f3fa41d`. --- updated-dependencies: - dependency-name: python dependency-version: 3.12-windowsservercore dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps python from `3de9a8d` to `f50f56f`. --- updated-dependencies: - dependency-name: python dependency-version: 3.13-windowsservercore dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat: Add UK passport and vehicle registration recognizers Add two new UK-specific recognizers: - UK_PASSPORT: detects 2-letter + 7-digit passport numbers (2015+ format) - UK_VEHICLE_REGISTRATION: detects current (2001+), prefix (1983-2001), and suffix (1963-1983) number plate formats with age identifier validation * style: fix ruff formatting in UK recognizer tests --------- Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Initial plan * Add presidio meta-package for PyPI release Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Update presidio/README.md to clarify meta-package role and credit papercloudtech/presidio Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Add CHANGELOG entry for presidio meta-package crediting papercloudtech/presidio Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Add presidio-anonymizer dependency and credit Sakthi Santhosh Anumand and Harsha Vardhan Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Credit Sakthi Santhosh Anumand and Harsha Vardhan, add presidio-anonymizer dependency and update tests Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Remove presidio/tests/ directory (smoke tests not needed in repo) Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Remove dev dependencies and coverage config from presidio/pyproject.toml (no tests to run) Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Trigger CI re-run (ARM64 runner was killed by shutdown signal, not a code issue) Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com>
…1885) Bumps python from `3de9a8d` to `f50f56f`. --- updated-dependencies: - dependency-name: python dependency-version: 3.13-slim dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
…parameter passing (#1884) * Initial plan * Add ONNX Runtime backend support to GLiNERRecognizer Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> * Refactor ONNX tests to use parameterized testing Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> * Improve documentation and add edge case test for ONNX parameters Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> * Add model_kwargs support and update GLiNER documentation Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: omri374 <3776619+omri374@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
#1899) The regex for U1-format Italian driver licenses had an erroneous ^ anchor that prevented matching when the license number was not at the start of the string. This fix removes the anchor so licenses like U1K711J11M are correctly recognized regardless of position in text. Co-authored-by: root <root@C20251020184286.local>
#1898) Bumps [actions/dependency-review-action](https://github.com/actions/dependency-review-action) from 4.8.3 to 4.9.0. - [Release notes](https://github.com/actions/dependency-review-action/releases) - [Commits](actions/dependency-review-action@05fe457...2031cfc) --- updated-dependencies: - dependency-name: actions/dependency-review-action dependency-version: 4.9.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
) Bumps [docker/setup-buildx-action](https://github.com/docker/setup-buildx-action) from 3.12.0 to 4.0.0. - [Release notes](https://github.com/docker/setup-buildx-action/releases) - [Commits](docker/setup-buildx-action@8d2750c...4d04d5d) --- updated-dependencies: - dependency-name: docker/setup-buildx-action dependency-version: 4.0.0 dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [actions/setup-dotnet](https://github.com/actions/setup-dotnet) from 5.1.0 to 5.2.0. - [Release notes](https://github.com/actions/setup-dotnet/releases) - [Commits](actions/setup-dotnet@baa11fb...c2fa09f) --- updated-dependencies: - dependency-name: actions/setup-dotnet dependency-version: 5.2.0 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 4.32.4 to 4.32.6. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](github/codeql-action@89a39a4...0d579ff) --- updated-dependencies: - dependency-name: github/codeql-action dependency-version: 4.32.6 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Initial plan * Add authentication and authorization note to main README Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Add auth note to faq.md deployment section; replace lock with warning in README Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Remove auth/authorization note from main README Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com>
) * Initial plan * Add 60-second timeouts to regex operations to prevent ReDoS Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Narrow TimeoutError scope to regex calls only; add exc_info to warnings Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Revert pattern_recognizer.py logic changes; keep original structure with timeout only Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Revert iban_recognizer.py to original lazy iterator structure; add exc_info=True to warnings Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Add IBAN recognizer timeout and empty-match tests to fix CI coverage failure Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Allow REGEX_TIMEOUT_SECONDS to be overridden via environment variable Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Initial plan * Update CHANGELOG.md unreleased section with unified dependabot entry Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Bump versions to 2.2.362 / 0.0.58 and update CHANGELOG Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Revert CHANGELOG.md to [unreleased] as requested Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Promote [unreleased] to [2.2.362] keeping individual entry descriptions Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Update presidio PyPI meta-package changelog entry with attribution Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> * Update CHANGELOG.md version comparison links for 2.2.362 Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: SharonHart <15013757+SharonHart@users.noreply.github.com>
* Add clarity cookie consent to docs site * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Table initially said IP_ADDRESS has to pass checksum, however IP addresses only use checksum at the packet level. Also, the IPRecognizer uses regex pattern matching, context words, and structural validation via Python's ipaddress library, but no sign of checksum function. Removed 'and checksum' from the documentation table. Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Allow one-off CLI threshold tuning without config edits * Cover CLI threshold boundaries and stdin arg shape --------- Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
…2110) The district-code validity check compares the parsed code against state_rto_district_map. Some states store their district codes single-digit ("1".."9"), but the parser produces the zero-padded form ("DL01" -> "01") when both characters after the state are digits. So a valid Delhi/Gujarat plate written in the standard zero-padded form (DL01..DL09, GJ01..GJ09) failed the district lookup and validate_result kept it at the base 0.5 score instead of promoting it to 1.0 — even though "DL3" (single-digit) and "DL13" (two-digit) were promoted correctly. (Other states, e.g. OD, store the padded form and already worked.) Normalise the leading zero (`str(int(dist_code))`) as an additional lookup, so a zero-padded district still matches a single-digit entry. The check is purely additive — states that store the padded form keep matching, and invalid districts (e.g. DL99, DL00) are still not promoted. Adds DL01/GJ09 cases (previously scored 0.5). Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
The Finnish personal identity code pattern used the character class `[+-ABCDEFYXWVU]` for the century separator (the 7th character). Because the `-` sits between `+` and `A`, the regex engine reads `+-A` as a character *range* (U+002B..U+0041), which silently admits `, . / 0-9 : ; < = > ? @` as "separators". validate_result only re-checks the date and control character, never the separator, so an 11-char string with a junk separator but an otherwise-correct control digit (e.g. `131052/308T`) is wrongly accepted — a false positive. Per the DVV spec the separator must be exactly one of `+ - A B C D E F Y X W V U`. Move the `-` to the start of the class so it is a literal, in both the Medium and Very Weak patterns. Adds regression cases: `131052/308T`, `131052:308T`, `131052.308T` (valid date+control, illegal separator) are now rejected; the valid `131052-308T` is still detected. Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
The TNIN pattern excludes province codes (digits N2N3) that are not real Thai provinces, but the shared character class `[25][0134567]` is over-narrow: it drops N3=2 and N3=8 for *both* N2=2 and N2=5. Per the recognizer's own docstring, only 28, 29 (for N2=2) and 59 (for N2=5) should be excluded, so the class wrongly rejected the valid province codes 22 (Chanthaburi), 52 (Lampang) and 58 (Mae Hong Son). IDs with those codes never matched the pattern, so a genuinely valid TNIN (correct mod-11 checksum) was never detected. Split the shared class into `2[0-7]|5[0-8]`, which matches exactly the set of N2N3 combinations the docstring documents as valid. Verified exhaustively across all 90 N2N3 combinations: the regex now accepts every documented-valid code and still rejects all 13 forbidden ones. Adds regression cases for valid TNINs with province codes 22/52/58. Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
NgNinRecognizer validated the Verhoeff checksum on `int(value)`, which drops a leading zero and shortens the digit string. An 11-digit NIN beginning with `0` (e.g. `00000000005`, a genuinely valid Verhoeff number) was therefore checked on only 10 digits and wrongly rejected — a false negative for ~10% of the NIN space. Pass the digit string straight to the Verhoeff check instead of `int(value)`, so all 11 digits are validated. Non-leading-zero NINs are unaffected. Adds an analyze-level case for a leading-zero NIN (fails before the fix) and a direct Verhoeff regression; also corrects the prior all-zeros unit test, which passed `int(0)` (a single "0", which is valid) rather than the 11-digit all-zero string (which is not a valid Verhoeff number). Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
The ABN checksum remapped a leading-zero first digit to 9 (`abn_list[0] = 9 if abn_list[0] == 0 else abn_list[0] - 1`). The official ABR algorithm simply subtracts 1 from the first (left-most) digit, so a leading 0 becomes -1. Remapping it to 9 instead shifts the weighted sum by 100 (= 11 mod 89) and admits some invalid 11-digit numbers beginning with 0 as valid ABNs — a false positive (e.g. 00000000560). Valid ABNs never start with 0. Use the literal subtract-1. This is identical for first digits 1-9, so no valid ABN is affected; only invalid leading-zero numbers, which the official algorithm rejects, are now correctly rejected. Adds a regression case (00000000560) that was wrongly accepted before. Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
… IDs are detected (#2076) Co-authored-by: John Kearney <johndanielkearney@gmail.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
…2138) The class docstring gives "B0" (Berlin) as an example of a Behördenkürzel and lists `B012345678A` in the Examples line, but the regex `[A-Z]{2}\d{8}[A-Z0-9]` requires both leading characters to be letters, so `B012345678A` never matches. The test file's docstring is the authoritative one — it explicitly notes that single-letter Kfz codes are used in 2-letter authority forms (e.g. `BO`, `KN`) and that `B0`-style single-letter-plus-digit combinations are out of scope, and `test_when_all_de_fuehrerschein_numbers_then_succeed` asserts `B12345678A` is rejected. Update the class docstring to match the tests: replace the `"B0" Berlin` example with `"BO" Bochum`, add a short clarification about single-letter Kfz codes, and change the misleading `B012345678A` Examples entry to `BO12345678A`. Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Fix ISO 8601 date pattern accepting impossible month/day values The "ISO 8601 datetime" pattern in DateRecognizer used `[01]\d` for the month and `[0-3]\d` for the day. These ranges admit impossible values: month `00` and `13`-`19`, and day `00` and `32`-`39`. As a result strings such as `2024-13-15T14:30:00Z` and `2024-12-32T14:30Z` were detected as DATE_TIME. Every other date pattern in this same file already constrains the month to `01`-`12` and the day to `01`-`31`; only the ISO 8601 pattern was loose. Tighten the ISO month/day fields to match (using non-capturing groups so existing capture-group positions are unaffected). No valid ISO 8601 datetime is lost, since those values are not valid dates to begin with. Adds parametrized cases for invalid month (00, 13) and day (00, 32). * Address review: apply word boundary across all alternatives, rename pattern Two review points from #2113: 1. `|` has lower precedence than concatenation, so the pattern `\b A | B | C \b` was parsed as `(\b A) | B | (C \b)`. The leading `\b` only guarded the first alternative (the full-fractional form) and the trailing `\b` only guarded the last (the minutes-only form). The seconds-only alternative in the middle had no word-boundary anchor at all, so a valid seconds/minutes datetime could match mid-word (e.g. `Today is2024-03-15T14:30:00+02:00`). Wrap the alternation in a non-capturing group so both `\b` anchors apply to every alternative. 2. Rename the pattern from "ISO 8601 datetime" to "Datetime (yyyy-mm-ddThh:mm[:ss[.f]] with timezone)" — the pattern doesn't fully validate ISO 8601 (e.g. the hour field admits 24–29). The new name honestly describes the shape it accepts. --------- Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
…A-537c-gmf6-5ccf (#2144) * Initial plan * chore: bump cryptography upper bound to <49.0.0 to allow patched 48.x --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
#2146) isdigit was referenced without being called, so the digit guard was always truthy and never actually checked anything. Under the recognizer's default pattern this stays hidden since the middle characters are already constrained to digits, but a custom patterns argument (which the constructor explicitly supports) can let non-digit text reach the checksum line further down and raise a ValueError instead of returning False. Fixes #2145
* ci: use uv for test dependency installation + add timeout safety net The analyzer test job runs `poetry install --all-extras` with no committed lock, so Poetry performs a universal resolve of the full optional-dependencies graph on every run. That resolve backtracks for many minutes — effectively hanging — driven by the langextract extra's deep, loosely-bounded transitive tree. Reproduced locally: Poetry >6 min (never completed); uv ~2s for the same 158-package resolution. Switch the test jobs to uv: - Install uv via astral-sh/setup-uv. - Create a per-component venv with `uv venv --seed` (seed keeps pip for the wheel-build step) and expose it via $GITHUB_PATH / $VIRTUAL_ENV. - `uv pip install -e '.<extras>'` plus the test tooling that previously came from the Poetry dev group (uv does not read that group). - Add a job-level `timeout-minutes: 60` (and 30 on the install step) so a bad resolve fails fast instead of hanging. No package metadata changed — pyproject.toml is untouched; this only swaps the CI install tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: run tests via uv run inside the uv-managed venv Use "uv run --no-sync" for spaCy model downloads, pytest, and diff-cover instead of exposing the venv on $GITHUB_PATH. "uv sync" builds the project's .venv (resolving the analyzer --all-extras graph in seconds where Poetry's lockless universal resolve hangs), and the subsequent steps execute inside it idiomatically. The --all-extras selection stays in the job matrix, unchanged from main. Wheel-build keeps using the system interpreter, matching the previous Poetry behaviour. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install uv via pip instead of the setup-uv action The astral-sh/setup-uv action is not on the org's allowed-actions list, so adding it made the CI workflow fail at startup (startup_failure) before any job ran. Install a pinned uv from PyPI with the already present setup-python instead; this needs no new third-party action and keeps the resolver benefits that motivated the Poetry -> uv switch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: add pip to the uv venv for spaCy model downloads spaCy's `download` command shells out to `python -m pip install`, but uv-created virtualenvs ship without pip, so runtime/first-use model downloads (e.g. the presidio-cli conftest fetching en_core_web_lg) failed with SystemExit: 1. Install pip alongside the test tooling. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: pre-install tomli for the wheel build on Python < 3.11 `python -m build` imports `tomli` on Python < 3.11 to read pyproject.toml, but it is not in the hashed requirements-build.txt (pip-compiled on 3.11+ where tomllib is stdlib). Poetry previously pulled tomli onto the runner as one of its own dependencies, so `pip install --require-hashes` found it already satisfied. uv has no Python dependencies, so the 3.10 wheel-build jobs began failing with "requirements must be pinned with ==". Pre-install a marker-guarded tomli to restore that condition. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: build service Docker images with uv to fix the E2E resolve hang The local-build-and-E2E job builds the analyzer/anonymizer/image-redactor images via `docker compose build`. Each Dockerfile ran `poetry install --no-root --only=main -E server`, which triggered the same lockless Poetry backtracking that hangs the test jobs: the analyzer image sat on "Resolving dependencies" for ~40 minutes until the job was cancelled. Switch the dependency install to `uv pip install --system -r pyproject.toml --extra server`, which resolves the same graph in seconds (verified locally: the analyzer install layer completes in ~9s and imports spacy/flask/gunicorn while correctly omitting the project itself, matching --no-root). Drop the now-unused POETRY_VIRTUALENVS_CREATE env and switch `poetry run python install_nlp_models.py` to a direct call. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * fix: run gunicorn directly in container entrypoints (drop poetry run) The Docker images no longer install Poetry (deps come from uv), so the `exec poetry run gunicorn ...` entrypoints crashed at startup with "exec: poetry: not found". The analyzer/anonymizer/image-redactor containers never came up, so every E2E test failed with connection refused on ports 5002/5003. uv installs gunicorn system-wide, so invoke it directly. Verified locally: the anonymizer container now starts healthy and /health returns 200. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: keep uv's wheel cache on the /mnt ephemeral disk Restore the intent of the old POETRY_CACHE_DIR=/mnt/poetry_cache setup for uv: point UV_CACHE_DIR at the runner's large ephemeral disk (/mnt, ~65 GB) so the heavy analyzer --all-extras wheel cache does not fill the smaller root volume. /mnt is root-owned, so a prep step creates the directory and hands it to the runner user. UV_LINK_MODE=copy silences the cross-filesystem hardlink warning, since the project venv lives on the root volume under the checkout while the cache is on /mnt. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: broaden CHANGELOG/CI comment to cover the Docker image changes Address review feedback that the PR scope grew beyond the test jobs: the CHANGELOG now notes the analyzer/anonymizer/image-redactor image builds and entrypoints also moved from Poetry to uv, and clarifies pyproject.toml metadata is unchanged. Also clarify the ci.yml install-step comment that the wheel-build step deliberately uses the system interpreter (not uv run). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * build: adopt PEP 735 dependency groups and commit uv.lock files Migrate each CI-tested package's development tooling from Poetry's [tool.poetry.group.dev.dependencies] to the standard PEP 735 [dependency-groups] table, and commit a uv.lock per package. This makes pyproject.toml the single source of truth for dev tooling and lets CI and Docker install a pinned, reproducible dependency graph instead of resolving on every run. The [project] metadata and the poetry-core build backend are unchanged, so wheels/sdists are byte-for-byte identical. Covers presidio-analyzer, presidio-anonymizer, presidio-cli, presidio-image-redactor, presidio-structured. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: install from the committed uv.lock in CI and Docker builds CI now runs `uv sync --locked --group dev`, which installs exactly the locked graph and fails if pyproject.toml and uv.lock disagree rather than silently resolving new versions. The dev group provides the test tooling (and pip, which spaCy's model download shells out to), so the manual `uv pip install pytest ...` line is gone. The uv cache is persisted across runs via actions/cache keyed on uv.lock and trimmed with `uv cache prune --ci`. The analyzer/anonymizer/image-redactor images consume the same lockfile with `uv sync --locked --no-default-groups --extra server --no-install-project` into UV_PROJECT_ENVIRONMENT=/usr/local, guaranteeing images use the same dependency graph as CI. Splitting the dependency layer from the source COPY keeps it cached until pyproject/uv.lock change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: allow pre-existing transformers advisories in dependency review The newly committed uv.lock files pin exact versions, so dependency-review now sees two transformers advisories that already exist on main (capped at transformers <5 by spacy-huggingface-pipelines and tracked as Dependabot alerts). They are unrelated to the Poetry->uv migration and will be addressed in separate security PRs, so allow-list the two GHSAs here to keep this PR's dependency review unblocked. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * docs: require uv.lock review and commits alongside pyproject changes Address review feedback from @SharonHart: - CODEOWNERS: require presidio-administrators approval for **/uv.lock, the same as **/pyproject.toml, so lockfile changes get dependency review. - copilot-instructions: switch the local dev commands to uv and add an explicit rule that editing pyproject.toml dependencies requires regenerating and committing that package's uv.lock in the same change, since CI installs with `uv sync --locked` and fails on drift. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: reference UV_CACHE_DIR in cache path; align dev-docs to uv Address review feedback: - ci.yml: use ${{ env.UV_CACHE_DIR }} for the cache action path instead of hardcoding /mnt/uv-cache, so the cache and uv stay in sync if the dir changes. - copilot-instructions: update the "Technology Stack" bullet to name uv as the dependency/install tool (poetry-core retained only as build backend), matching the uv-based dev commands already documented. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * ci: shorten workflow comments; move transformers allow-ghsas out Trim the verbose explanatory comments in the test job to 1-3 lines each, and drop the dependency-review `allow-ghsas` entry for the transformers advisories. That suppression is a security concern rather than part of the Poetry->uv migration, so it moves to the stacked security PR alongside the .trivyignore. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * ci: drop stale pip cache; trim CHANGELOG entry Remove `cache: 'pip'` (keyed on pyproject.toml) from the test job's setup-python: it is leftover Poetry-era config. Project dependencies now use the uv cache, and the only remaining pip installs (uv, tomli, build tools) are tiny and unrelated to pyproject.toml. Also condense the unreleased CHANGELOG entry to a single concise line; the implementation detail lives in the PR description. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> * small commit to trigger the pipeline again. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…izer (#2150) Fixes #1942. Merge language_model_params into provider_kwargs so timeout, num_ctx and other LLM provider params configured in YAML actually reach the provider (e.g. Ollama) instead of being silently dropped. Also fixes TypeError when kwargs: or language_model_params: is null in YAML. - Merge language_model_params into provider_kwargs via setdefault (so explicit kwargs: entries still win for backwards compatibility) - Guard against null YAML values with 'or {}' - Strengthen regression tests to assert params reach ModelConfig.provider_kwargs - Add test for kwargs: null edge case - Document fix in CHANGELOG Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped) Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
Add healthcheck and restart: unless-stopped to all services in docker-compose.yml so containers auto-recover on crash and Docker can detect service readiness. - ollama: add restart: unless-stopped (healthcheck already existed) - presidio-anonymizer: add healthcheck + restart: unless-stopped - presidio-analyzer: add healthcheck + restart: unless-stopped - presidio-image-redactor: add healthcheck + restart: unless-stopped Closes #2050 Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
#2147) * Initial plan * feat: add Python 3.14 compatibility to all remaining Presidio packages - Update requires-python to <3.15 in anonymizer, image-redactor, cli, structured, and umbrella packages - Add Python 3.14 PyPI classifier to all affected packages - Split spacy dependency markers in image-redactor to exclude 3.8.14 on Python 3.14 (no compatible wheel), matching analyzer pattern - Extend CI matrix to test all components on Python 3.14 - Document Python 3.14 as a supported version in installation docs - Add CHANGELOG entry under [unreleased] Closes #2096 * ci: move Python 3.14 into main python-version matrix All packages now support 3.14, so add it directly to the matrix array instead of duplicating each component in `include` entries. * build: regenerate uv.lock files to match updated requires-python bounds * fix: merge duplicate Changed headings in CHANGELOG.md Unreleased section --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
validate_result checked the date with datetime.strptime(date_part, "%d%m%y"),
which only sees the 2-digit year and maps it to the 2000s. The century is
actually carried by the 7th character (the separator), which strptime never
receives. So 29 Feb of a non-leap century is wrongly accepted: "290200+311B"
denotes 29 February 1800 ("+" = 1800s), but 1800 is not a leap year, yet
strptime validates it as 29 Feb 2000 (which is a leap year).
Resolve the century from the separator and validate the full date. Unknown
separators fall back to the previous strptime check, so this is independent of
the separator character-class handling.
Adds a regression case (290200+311B) that was wrongly accepted at score 1.0.
Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
…Ns (#2074) The invalidate_result() sample-SSN blocklist was matched with str.startswith(), and the literal "98765432" is only 8 digits (a truncation of the canonical fake "987654320"). The prefix match therefore invalidated the entire 987-65-4320 .. 987-65-4329 family -- ten distinct SSN-shaped values, including the widely-printed 987-65-4321 -- so real SSNs were dropped to no result and left unredacted. Split the two concerns: keep 000/666 as a never-issued area-number (first-group) check, and match the full 9-digit sample SSNs (123456789 / 987654320 / 078051120) by exact equality so they no longer over-block neighbours. NeMo Guardrails' PII rail inherits this recognizer, so the leak reached that deployed guardrail too. Co-authored-by: John Kearney <johndanielkearney@gmail.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
… registry (#2157) * fix(analyzer): honour language_model_params in BasicLangExtractRecognizer Fixes #1942. Merge language_model_params into provider_kwargs so timeout, num_ctx and other LLM provider params configured in YAML actually reach the provider (e.g. Ollama) instead of being silently dropped. Also fixes TypeError when kwargs: or language_model_params: is null in YAML. - Merge language_model_params into provider_kwargs via setdefault (so explicit kwargs: entries still win for backwards compatibility) - Guard against null YAML values with 'or {}' - Strengthen regression tests to assert params reach ModelConfig.provider_kwargs - Add test for kwargs: null edge case - Document fix in CHANGELOG Related: #1943 (lsternlicht/fix-basic-langextract-language-model-params-dropped) * fix(analyzer): honour config_path for LangExtract recognizers in YAML registry LM recognizers (BasicLangExtractRecognizer, AzureOpenAILangExtractRecognizer) configured via a recognizer registry YAML silently ignored config_path: the strict PredefinedRecognizerConfig schema has no config_path field and forbids extras, so Pydantic dropped it and the recognizer fell back to its bundled default model configuration. Add a LangExtractRecognizerConfig model (extra=allow, explicit config_path field) mirroring the existing HuggingFace/GLiNER configs, and register both LM recognizer class names in CONFIG_MODEL_MAP so config_path (and other recognizer-specific kwargs) survive validation and reach the constructor. Adds regression tests covering config_path preservation via both the model and the full ConfigurationValidator registry path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* Match batch deanonymization with existing traversal * Keep README examples exercised by CI * Align batch deanonymize dict API with documented usage * Keep batch deanonymize docs in contract order * Keep batch deanonymization from truncating or over-forwarding --------- Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* feat: add Philippine UMID (PH_UMID) recognizer - Add PhUmidRecognizer using PatternRecognizer - Support dashed (0111-1234567-8) and plain 12-digit formats - Add Filipino context words and COUNTRY_CODE = "ph" * feat: add PH_UMID recognizer tests - Cover valid/invalid formats, context words, and edge cases - Include multiple UMID detection in single text * feat: register PhUmidRecognizer in config and exports - Add import and __all__ entry in predefined_recognizers - Register in default_recognizers.yaml with enabled: false * docs: add PH_UMID to CHANGELOG and supported_entities.md - Add entry under Unreleased section - Add Philippines section between Turkey and Germany * fix: correct test score ranges for plain 12-digit UMID format * feat: export predefined recognizers in a new package initialization file * chore: remove CHANGELOG.md modifications per reviewer request --------- Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Allow per-recognizer analyzer tuning from YAML * Keep boolean flags from slipping into threshold config * Keep analyzer docs aligned with the new threshold contract * Keep direct threshold config aligned with YAML validation * Preserve lenient recognizers when duplicate spans collide * Keep threshold validation errors tied to the config key * Keep constructor threshold hints aligned with shared validation * fix(analyzer): reject boolean score thresholds * fix(analyzer): move thresholds onto recognizers (#1572) * fix(analyzer): tighten threshold consumer contract * Retry external CI flakes * Apply suggestions from code review Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> --------- Co-authored-by: Omri Mendels <omri374@users.noreply.github.com>
* docs: quote pip extras install examples * Apply suggestions from code review Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> * docs: address pip extras review comments --------- Co-authored-by: nyxst4ck <289980115+nyxst4ck@users.noreply.github.com> Co-authored-by: Omri Mendels <omri374@users.noreply.github.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Sharon Hart <sharonh.dev@gmail.com>
* Stabilize CI disk usage and Ollama CPU backend Place both uv cache and project environments on /mnt, and pin Ollama while disabling unstable AVX-512/AMX CPU backends before E2E inference on AMD64 runners. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b * Harden Ollama CPU backend diagnostics Log CPU metadata and enabled backends for ARM runners too, and leave previously disabled libraries untouched. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b * fix(ci): stabilize uv and Ollama jobs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 15a357e4-85bf-4b12-bca8-a9c3e3e45b2c * refactor(ci): run Ollama inference on arm64 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 15a357e4-85bf-4b12-bca8-a9c3e3e45b2c * chore(ci): finalize arm64 Ollama workflow Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b * fix(ci): stabilize Ollama CPU backend selection Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b * refactor(ci): keep Ollama E2E on arm64 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b * fix(ci): reclaim uv cache before coverage Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: ca3a8dc1-1c88-4c27-88e1-07f7e18b3e6b --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Description
Describe your changes
Issue reference
Fixes #XX
Checklist