Skip to content

fix(secrets): re-read findings when the file or the rule set changes - #363

Open
thefron wants to merge 6 commits into
mainfrom
fix/secrets-rescan
Open

thefron wants to merge 6 commits into
mainfrom
fix/secrets-rescan

Conversation

@thefron

@thefron thefron commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

TL;DR

  • bdrive status and the agent hook on Liam's mini still warn "an OpenAI API key" on three team/projects/searchd files that the current detector does not flag (the helpdesk-… / URL-slug false positives fixed in fix(secrets): stop flagging slugs and URL paths as OpenAI keys #322, v0.18.1). Scanning those files with the current rules finds nothing.
  • Two causes:
    1. A detector fix never re-evaluates recorded findings. A finding stands until the file changes, and README.md has not changed since 2026-10-26.
    2. Only this device's own edits are scanned (scanBlob runs on the local-scan branch only). log.md was edited by a teammate on 2026-11-11, and the edit was pulled and materialized here, so it was never read and its finding never cleared.
  • Fix: each recorded path keeps the blob it was read from, and the record keeps the rule-set version. At the end of every cycle, any entry whose file now holds another blob, or whose rules differ, is read again from the blob; entries for untracked paths are dropped. Cost: one blob read per stale finding, nothing on a quiet cycle.
  • The private-key rule now needs a key body after the header. -----BEGIN PRIVATE KEY-----\n... quoted as a placeholder in a doc (the fourth finding on the mini, in a Notion export) is no longer flagged; real PEM keys, JSON-escaped keys and encrypted PEMs with Proc-Type: headers still are.

Changes

  • internal/secrets: RulesVersion() (hash of rule ids and patterns); private_key requires 32 base64 characters after the header line and any RFC 1421 headers, counted across line breaks (real, indented, or JSON-escaped), so a body wrapped at any width still matches.
  • internal/store/secrets.go: SecretRecord{Rules, Paths: {rel: {Blob, Found}}} with LoadSecretRecord / SaveSecretRecord. The bare per-path file shape still loads (no rules, no blobs, so every path is read again). LoadSecrets / SaveSecrets keep their signatures for status and the hook.
  • internal/syncer/secrets.go: secretLog holds the record; set stores the blob; revalidate(store, cache) runs in finish. A blob that cannot be read keeps its findings, and under other rules keeps the record's rules, so the next cycle retries.
  • architecture/cli-sync.md: Store note on the record.

Tests

  • TestSecretClearsWhenAPeerFixesTheFile: A flags a file, B removes the key, A pulls: no finding.
  • TestSecretFindingsFromOtherRulesAreReadAgain: a false positive recorded under other rules for the current blob is gone after a quiet cycle; a real key stays; the record carries the current version.
  • Both fail without revalidate.
  • TestSecretFindingSurvivesAnUnreadableBlob: with the blob removed, the finding and the old rules stay; once readable, the next cycle re-reads it under the current rules.
  • TestShareRefusesAPrivateKeyWrappedInShortLines (webapp): the share gate refuses a key wrapped at 16 columns and passes the placeholder; Scan cases for 16- and 8-column (indented) wrapping.
  • TestLoadSecretRecordReadsTheBarePerPathShape (store), new private-key cases in TestScanSecrets (placeholder and header-only not flagged; JSON-escaped and encrypted PEM flagged).
  • TestSyncHookModeReportsSecrets / ...MultipleMounts now write files that hold the keys instead of seeding findings for files that do not exist (such findings are dropped as untracked).
  • go test ./internal/secrets ./internal/store ./internal/syncer ./cmd/bdrive ./internal/webapp passes; -race passes on the store and syncer secret tests.

Architecture changes

architecture/cli-sync.md: the Store note on secrets-mount-id.json gains the blob and rules stamps and the revalidation in finish.

classDiagram
    class secretLog {
        ✅ -store.SecretRecord rec
        ❌ ~~-map found~~
        ✅ revalidate(store, cache)
    }
    note for secretLog "✅ entries re-read when the file's blob or the rule set changed"
Loading

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thefron
thefron requested a review from runbear-bot November 14, 2026 07:20

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: not-ready

  • 개인키 탐지 변경이 유효한 PEM을 놓쳐 공유 확인 절차를 우회할 수 있습니다.
  • 재검증 중 blob 읽기에 실패하면 기존 경고가 영구적으로 사라집니다.
  • 이전 형식의 기록 로드와 peer 수정/규칙 변경 시 재검증 경로를 확인했습니다. 기존 리뷰 차단 사항은 없습니다.
  • 깨끗한 체크아웃에 테스트 도구가 없어 로컬 패키지 테스트는 실행하지 않았습니다.

Unanchored Findings (outside changed diff lines)

  • internal/secrets/secrets.go:58: [P1] PEM의 base64 본문은 임의 길이의 줄로 나뉠 수 있으므로, 유효한 개인키를 16자씩 줄바꿈하면 연속된 32자가 없어 이 규칙이 놓칩니다. secrets.Scan은 공유 게이트에도 사용되어 이전에는 확인을 요구하던 키가 경고 없이 공유됩니다. 줄바꿈을 사이에 둔 누적 base64 문자를 검증하고, 짧은 줄로 감싼 실제 개인키 및 공유 거부 회귀 테스트를 추가해 주세요.

Comment thread internal/syncer/secrets.go Outdated
if !rescanAll && l.rec.Paths[rel].Blob == c.Blob {
continue
}
l.drop(rel)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 재검증 전에 기존 finding을 삭제하면 OpenBlob/읽기가 일시적으로 실패해도 scanBlob이 아무 결과 없이 반환하고 삭제된 기록이 저장됩니다. 이후 해당 경로는 재검증 대상이 아니므로 실제 키가 남아도 status/hook 경고가 파일을 다시 수정할 때까지 복구되지 않습니다. 읽기 성공 시에만 finding을 교체·삭제하고, 실패 시 이전 finding과 재시도 가능한 규칙/blob 상태를 유지해 주세요.

…n an unreadable blob

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thefron

thefron commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Addressed both in 0c26f17: [P1] the private-key body is 32 base64 characters counted across line breaks (real, indented, or JSON-escaped), so a key wrapped at any width matches; new share-gate test TestShareRefusesAPrivateKeyWrappedInShortLines plus 16/8-column Scan cases. [P2] revalidate replaces or drops a finding only after a successful read; an unreadable blob keeps the finding and the record's old rules so the next cycle retries (TestSecretFindingSurvivesAnUnreadableBlob).

@thefron
thefron requested a review from runbear-bot November 14, 2026 08:14

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: ready-with-notes

  • 이전 리뷰의 두 차단 사항을 HEAD에서 확인했습니다. 개인키 본문을 줄바꿈 너머로 누적해 검사하므로 짧은 PEM 줄도 탐지하고, share gate 회귀 테스트가 추가되어 있습니다.
  • revalidate는 blob 읽기 성공 후에만 finding을 교체/삭제하며, 읽기 실패 시 기존 finding과 이전 rules를 유지해 다음 cycle에 재시도합니다.
  • peer가 materialize한 새 blob, detector rules 변경, legacy bare per-path record, untracked path 정리 경로를 확인했습니다. 누락된 blob/파일은 기존 경고를 보존하고 정상적인 파일 삭제·symlink·untracked 상태는 정리합니다.
  • 변경된 규칙 버전과 blob stamp가 저장되고 finish에서 재검증되므로 quiet cycle에서도 peer 수정과 rules 변경을 반영합니다.
  • 깨끗한 체크아웃에 패키지 테스트 도구가 없어 로컬 패키지 테스트는 실행하지 않았습니다.

thefron and others added 2 commits November 14, 2026 00:23
…sion folder

Since the hook no longer cycles mounts below the session folder, those
mounts' warnings come only from the record their daemon wrote.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thefron

thefron commented Nov 14, 2026

Copy link
Copy Markdown
Member Author

Merged main (#361). The hook no longer cycles mounts below the session folder, so the multi-mount hook test seeds each daemon's record instead of writing files and expecting a scan. Re-requesting review.

@thefron
thefron requested a review from runbear-bot November 14, 2026 08:25

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: not-ready

  • 개인키 탐지 정규식이 유효한 PEM의 줄 끝 공백을 처리하지 못해 공유 확인을 우회할 수 있습니다.
  • 이전 차단 사항인 짧은 줄의 일반 PEM 탐지와 읽을 수 없는 blob의 기존 finding 보존 및 재시도 경로를 확인했습니다.
  • legacy 기록 로드와 blob/rules 변경 시 재검증 경로를 확인했습니다.
  • 깨끗한 체크아웃에 테스트 도구가 없어 로컬 패키지 테스트는 실행하지 않았습니다.

Comment thread internal/secrets/secrets.go Outdated
// or JSON-escaped line break and any RFC 1421 headers (Proc-Type:,
// DEK-Info:) an encrypted PEM carries: 32 base64 characters, counted
// across the line breaks a PEM may wrap its body at, at any width.
{"private_key", regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----(?:\s|\\[rn])+(?:[A-Za-z-]+:[^\n\\]*(?:\s|\\[rn])+)*(?:[A-Za-z0-9+/=](?:\r?\n[ \t]*|\\[rn])*){32}`)},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 본문 줄 끝의 공백이나 탭은 허용하지 않아, 예를 들어 base64 본문을 16자씩 줄바꿈하고 각 줄 끝에 공백을 넣은 유효한 PEM 개인키를 놓칩니다(OpenSSL로 해당 형식의 키 로드 확인). 기존 헤더 전용 규칙은 이를 탐지했지만 이제 share gate에서 경고 없이 공유됩니다. 줄바꿈 앞의 수평 공백도 허용해 base64 문자를 누적하고, 짧은 줄 끝 공백이 있는 실제 키의 Scan 및 공유 거부 회귀 테스트를 추가해 주세요.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thefron

thefron commented Nov 14, 2026

Copy link
Copy Markdown
Member Author
  • [P1] trailing blanks: the body pattern now allows spaces or tabs on both sides of every line break (LF and CRLF). Scan cases for 16-column lines with trailing blanks, and a share-gate case (409), fail on the previous pattern and pass now.
  • [P2] internal/syncer/secrets.go:111 points at code that is gone: scanBlob and revalidate replace or drop a path's findings only after readFindings succeeds; a failed read keeps the old entry (old blob, so the next revalidate retries) and keeps the old rules stamp. Covered by TestSecretFindingSurvivesAnUnreadableBlob.
  • macOS: TestCollabProcessesConvergeOverRedis (internal/webapp, ycollab) is unrelated to this change; the new push reruns it.

@thefron
thefron requested a review from runbear-bot November 14, 2026 08:40

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: not-ready

  • JSON 문자열에 직렬화된 줄 끝 공백 포함 개인키는 여전히 탐지되지 않아 공유 확인을 우회합니다.
  • 일반 PEM의 짧은 줄 및 줄 끝 공백 탐지, blob 읽기 실패 시 기존 finding과 재시도 상태 유지, legacy 기록 및 규칙 변경 재검증 경로를 확인했습니다.
  • 깨끗한 체크아웃에 테스트 도구가 없어 로컬 패키지 테스트는 실행하지 않았습니다.

Comment thread internal/secrets/secrets.go Outdated
// DEK-Info:) an encrypted PEM carries: 32 base64 characters, counted
// across the line breaks a PEM may wrap its body at, at any width, with
// blanks on either side of each break.
{"private_key", regexp.MustCompile(`-----BEGIN [A-Z ]*PRIVATE KEY-----(?:\s|\\[rn])+(?:[A-Za-z-]+:[^\n\\]*(?:\s|\\[rn])+)*(?:[A-Za-z0-9+/=](?:[ \t]*\r?\n[ \t]*|\\[rn])*){32}`)},

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] 줄 끝 공백 처리는 실제 줄바꿈 분기에만 적용되어 JSON 이스케이프 줄바꿈에는 적용되지 않습니다. 예를 들어 16자씩 감싼 유효한 PEM의 각 줄 끝에 공백을 넣고 JSON 문자열로 직렬화하면 본문이 ...16자 \n...16자 \n이 되어 32자를 누적하지 못합니다. 이전 헤더 규칙은 이를 감지했지만 이제 공유 확인 없이 키가 노출됩니다. 이스케이프된 줄바꿈의 양쪽에도 공백/탭을 허용하고 해당 JSON 키에 대한 Scan 및 공유 거부 회귀 테스트를 추가해 주세요.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@thefron

thefron commented Nov 14, 2026

Copy link
Copy Markdown
Member Author
  • [P1] escaped breaks: blanks (space, tab, or an escaped \t) are now allowed on both sides of every break, real (LF/CRLF) or escaped (\n, \r), and an escaped \t may follow the header. New Scan cases (JSON key with trailing blanks; JSON key with escaped-tab indentation) and a share-gate case (JSON creds file, 409); the first Scan case fails on the previous pattern.
  • [P2] at internal/syncer/secrets.go:111 is the same carried-over comment; the current code replaces or drops findings only after a successful read (TestSecretFindingSurvivesAnUnreadableBlob).

@thefron
thefron requested a review from runbear-bot November 14, 2026 08:56

@runbear-bot runbear-bot left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bobb's bot(Totoro) Review

Conclusion: ready-with-notes

  • 이전 차단 사항인 짧은 PEM 줄과 줄 끝 공백, JSON 이스케이프 줄바꿈 처리를 확인했습니다. 관련 탐지 및 공유 거부 회귀 테스트가 추가되었습니다.
  • blob 읽기 실패 시 기존 finding을 유지하고 재시도하며, 기존 형식 기록과 blob·규칙 변경 시 재검증 경로를 확인했습니다.
  • 깨끗한 체크아웃에 패키지 테스트 도구가 없어 로컬 패키지 테스트는 실행하지 않았습니다.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants