Skip to content

fix: Hindi/Hinglish approval answers cite the source, never splice it - #92

Merged
charan-rathore merged 1 commit into
mainfrom
fix/hindi-citation-splice
Nov 12, 2026
Merged

charan-rathore merged 1 commit into
mainfrom
fix/hindi-citation-splice

Conversation

@charan-rathore

Copy link
Copy Markdown
Owner

REVIEW-2026-11-11 P0 ("Hindi template splice"): The finance approval from November 2 में मंज़ूरी दर्ज है of $42,000 — English record labels spliced into Hindi sentence grammar.

Fix (citation-splice only)

  • The four HI (src/conversation.mjs) and four HL (src/vernacular.mjs) templates now keep record text, values, negation and dates verbatim and trail the source as a citation: (स्रोत: X) / (source: X).
  • Yes. X records the approval (of Y). split into two ordered templates so the broad (.*) suffix capture can't mangle the value.
  • docLabel() (src/spoken.mjs, now exported) and docLabelOf() (src/spoken-routes.mjs) strip : from titles, so the Per <label>: citation delimiter is unambiguous by construction. New regression test pins the contract end to end.

No who-answer, claim-status, actor or gloss behaviour changes.

Evidence

REVIEW-2026-11-11 P0: "The finance approval from November 2 में मंज़ूरी
दर्ज है of $42,000" - English record labels spliced into Hindi grammar.
The four HI and four HL templates now keep record text, values, negation
and dates verbatim and trail the source as (स्रोत: X) / (source: X).
docLabel/docLabelOf strip ":" from titles so the "Per <label>:"
citation delimiter is unambiguous by construction.

RED on base: vernacular tests #9, #10, #14 fail; GREEN 14/14 with the
fix. Full battery green: main 341, boundaries 302, evaluation 536,
ingest 51, clinical 119, vault, graph, browser/UI checks, ingest bench
131/131.
@vercel

vercel Bot commented Nov 12, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
voice-memory Ready Ready Preview Nov 12, 2026 8:19pm UTC

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 12, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-12T20:20:14.239587Z d1feafb PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

Paraphrase bench (32d72c8, Xenova/bge-small-en-v1.5@q8, dense on)

{
 "development": {
  "n": 15,
  "keyword": {
   "recall1": 0.333,
   "recall3": 0.4,
   "abstain": 0.4
  },
  "hybrid": {
   "recall1": 0.467,
   "recall3": 0.533,
   "abstain": 0.4
  },
  "dense": {
   "recall1": 0.4,
   "recall3": 0.8
  },
  "nAbstain": 5,
  "units": {
   "instructed": {
    "recall1": 0.267,
    "recall3": 0.8
   },
   "plain": {
    "recall1": 0.4,
    "recall3": 0.8
   }
  }
 },
 "heldout": {
  "n": 15,
  "keyword": {
   "recall1": 0.333,
   "recall3": 0.333,
   "abstain": 0.8
  },
  "hybrid": {
   "recall1": 0.333,
   "recall3": 0.333,
   "abstain": 0.8
  },
  "dense": {
   "recall1": 0.533,
   "recall3": 0.867
  },
  "nAbstain": 5,
  "units": {
   "instructed": {
    "recall1": 0.467,
    "recall3": 0.733
   },
   "plain": {
    "recall1": 0.4,
    "recall3": 0.733
   }
  }
 },
 "tuning": {
  "instructed": {
   "chosenOnDevelopment": {
    "minCosine": 0.56,
    "minMargin": 0.02
   },
   "development": {
    "right": 1,
    "wrong": 0,
    "falseAccept": 0,
    "recall1WithRescue": 0.533
   },
   "heldout": {
    "right": 2,
    "wrong": 1,
    "falseAccept": 0,
    "recall1WithRescue": 0.467
   }
  },
  "plain": {
   "chosenOnDevelopment": {
    "minCosine": 0.605,
    "minMargin": 0
   },
   "development": {
    "right": 1,
    "wrong": 0,
    "falseAccept": 0,
    "recall1WithRescue": 0.533
   },
   "heldout": {
    "right": 1,
    "wrong": 1,
    "falseAccept": 0,
    "recall1WithRescue": 0.4
   }
  }
 },
 "shipped": {
  "gate": {
   "minCosine": 0.72,
   "minMargin": 0.02,
   "agree": 0.5
  },
  "development": {
   "right": 0,
   "wrong": 0,
   "falseAccept": 0,
   "recall1WithRescue": 0.467
  },
  "heldout": {
   "right": 0,
   "wrong": 0,
   "falseAccept": 0,
   "recall1WithRescue": 0.333
  }
 },
 "units": 25
}
Per question: split | kind | hybrid top-1 hit | top-3 dense cosine (* = right passage) | unit cosine/margin | question
d | a | 0 | 0.5121 0.5048 0.4847* | 0.4575/0.0239 | How much was the first quote for the project?
h | a | - | 0.6324* 0.5856 0.5806 | 0.6255/0.0559* | What does the work cost now that the scope was cut back?
d | a | - | 0.5315 0.5302* 0.4892 | 0.4916/-0.0157 | How is the reduced price split between before and after delivery?
h | a | 0 | 0.5935 0.5553 0.5331 | 0.6046/0.0269 | What was the original finish date?
d | a | 1 | 0.5277* 0.5103 0.4308 | 0.5419/0.012* | Which areas were taken out of the first phase?
h | a | 1 | 0.5394 0.5311 0.5151 | 0.5834/0.0255* | How often will the customer hear about progress?
d | a | 1 | 0.5848 0.5546 0.5173* | 0.6323/0.072 | Who has to okay changes to the money terms?
h | a | - | 0.5764* 0.5002 0.4945 | 0.5525/0.0295* | Did the leadership group object to the price?
d | a | 1 | 0.6068* 0.5504 0.5137 | 0.6464/0.0799* | Was there a ballot on how payments are staged?
h | a | 0 | 0.604* 0.5346 0.5281 | 0.5556/-0.0597 | When does our own team expect to be ready?
d | a | 1 | 0.6296 0.6181* 0.556 | 0.6367/0.0076* | Is the later ready date something the customer agreed to?
h | a | - | 0.5556 0.5524* 0.517 | 0.5544/-0.0077 | What could make the ready date slip?
d | a | 1 | 0.562 0.5565* 0.5556 | 0.6016/0.0495* | How far behind is the customer in handing over their data?
h | a | 0 | 0.5695* 0.5347 0.5072 | 0.5152/0.0185 | What did the customer say about keeping the first end date?
d | a | 0 | 0.6198* 0.5827 0.5617 | 0.6117/-0.0377 | Has the customer accepted a new end date?
h | a | 1 | 0.5694 0.5549* 0.5338 | 0.623/0.0488* | What does the customer want put in writing before the next leadership session?
d | a | 0 | 0.6471 0.5993 0.5832 | 0.7133/0.0464 | Is there a countersigned contract for the new terms?
h | a | 1 | 0.5337* 0.4844 0.4582 | 0.573/0.0488* | Has going live in production been authorised?
d | a | 1 | 0.5386 0.5333 0.5209 | 0.5611/0.0216* | What has to happen before the first offer becomes binding?
h | a | - | 0.6242 0.6033 0.5538* | 0.4697/0.0093 | How was the original price going to be billed?
d | a | - | 0.6494* 0.6118 0.5934 | 0.5887/0.0194 | Does the price change note mention a new end date?
h | a | - | 0.5737 0.5716* 0.5325 | 0.6121/0.1306 | Until when is the cheaper price only an offer?
d | a | - | 0.5358 0.5311* 0.5285 | 0.5302/0.0045 | Where do unresolved money questions get tracked?
h | a | - | 0.5255* 0.5215 0.518 | 0.563/0.0089 | Which two areas does the current plan cover?
d | a | 0 | 0.5263 0.5207 0.5188 | 0.5416/0.0093 | What are the two competing ways of staging payment?
h | a | - | 0.5931* 0.5854 0.5593 | 0.6033/0.029* | Which end dates are in conflict?
d | a | 1 | 0.6068* 0.5666 0.5326 | 0.6148/0.0658 | Which consultancy is running this engagement?
h | a | 1 | 0.5386* 0.484 0.4728 | 0.5685/0.0766 | What kind of study is being run for the retailer?
d | a | - | 0.6139* 0.5956 0.5053 | 0.6092/0.0522* | Had the accountants signed off the reworked pricing by the time leadership met?
h | a | 1 | 0.6515 0.6049* 0.567 | 0.6596/0.0331 | After the first meeting, was the money approval done?
d | x | 0 | 0.5424 0.5163 0.5081 | 0.5031/-0.0325 | What is the penalty for finishing late?
h | x | - | 0.509 0.4968 0.4946 | 0.5331/0.0118 | What is the client's head office address?
d | x | 0 | 0.5142 0.5016 0.494 | 0.514/0.0179 | How many analysts are on the delivery team?
h | x | - | 0.4587 0.4507 0.4391 | 0.4963/0.0105 | Which software licences are included?
d | x | 0 | 0.5644 0.5623 0.5471 | 0.5457/0.0854 | What interest is charged on late invoices?
h | x | - | 0.5112 0.5075 0.4945 | 0.5432/0.0039 | Who is the client's chief executive?
d | x | - | 0.5044 0.4954 0.4878 | 0.5001/0.0141 | What is the travel budget?
h | x | 0 | 0.4684 0.456 0.4172 | 0.5274/0.0542 | Which cloud provider hosts the data?
d | x | - | 0.536 0.5163 0.5025 | 0.5248/0.0008 | What discount applies for paying early?
h | x | - | 0.5686 0.538 0.5352 | 0.5563/0.001 | How long is the warranty period?
### Outside-authored retrieval bench (32d72c8, Xenova/bge-small-en-v1.5@q8)
Set answerable / not chance top-1 keyword top-1 / top-3 hybrid top-1 / top-3 dense top-1 / top-3 closest-line gate (answerable) shown when absent
QMSum meetings 197 / 0 5.3% 40.6% / 62.4% 41.6% / 60.4% 40.1% / 61.4% 86.3% shown, 46.5% right -
CUAD contracts 184 / 236 1.1% 44.6% / 66.8% 48.4% / 68.5% 41.3% / 61.4% 90.2% shown, 48.8% right 84.3%

Shipped discovery gate {"minCosine":0.7,"minMargin":0.03,"lexTop":3}

Set answerable shown when absent
QMSum 21.3% shown, 64.3% right -
CUAD 6.5% shown, 83.3% right 1.7%

Calibration: no rule reaches 90% right with <= 10% absent shown on dev

min coverage (dev) best rule on dev dev held-out
5% {"minCosine":0.5,"minMargin":0.07,"lexTop":3} 12/203 shown, 83.3% right, absent shown 1/106 10/178 shown, 100.0% right, absent shown 1/130
10% {"minCosine":0.66,"minMargin":0.05,"lexTop":3} 21/203 shown, 76.2% right, absent shown 1/106 20/178 shown, 85.0% right, absent shown 4/130
20% {"minCosine":0.5,"minMargin":0.025,"lexTop":1} 41/203 shown, 70.7% right, absent shown 7/106 32/178 shown, 87.5% right, absent shown 13/130
30% {"minCosine":0.6,"minMargin":0,"lexTop":1} 64/203 shown, 64.1% right, absent shown 8/106 56/178 shown, 76.8% right, absent shown 20/130

Cross-encoder rerank of the fused top 20 (Xenova/ms-marco-MiniLM-L-6-v2)

Set hybrid top-1 / top-3 reranked top-1 / top-3 MRR@10
QMSum 41.6% / 60.4% 48.7% / 66.5% 0.53 → 0.596
CUAD 48.4% / 68.5% 57.1% / 76.1% 0.6 → 0.666

Reranker gate: no reranker rule reaches 90% right with <= 10% absent shown on dev

min coverage (dev) best rule on dev dev held-out
5% {"minProb":0.83,"minMargin":0.26,"maxFusedRank":null} 14/203 shown, 85.7% right, absent shown 1/106 31/178 shown, 80.6% right, absent shown 3/130
10% {"minProb":0.02,"minMargin":0.44,"maxFusedRank":null} 24/203 shown, 83.3% right, absent shown 1/106 38/178 shown, 78.9% right, absent shown 6/130
20% {"minProb":0.02,"minMargin":0.2,"maxFusedRank":3} 42/203 shown, 73.8% right, absent shown 5/106 52/178 shown, 73.1% right, absent shown 14/130
30% {"minProb":0.19,"minMargin":0.12,"maxFusedRank":5} 62/203 shown, 67.7% right, absent shown 7/106 63/178 shown, 69.8% right, absent shown 17/130
40% {"minProb":0.23,"minMargin":0,"maxFusedRank":5} 86/203 shown, 64.0% right, absent shown 7/106 82/178 shown, 67.1% right, absent shown 22/130

Cross-encoder rerank of the fused top 20 (jinaai/jina-reranker-v1-tiny-en)

Set hybrid top-1 / top-3 reranked top-1 / top-3 MRR@10
QMSum 41.6% / 60.4% 44.2% / 65.5% 0.53 → 0.567
CUAD 48.4% / 68.5% 40.8% / 62.5% 0.6 → 0.539

Reranker gate: no reranker rule reaches 90% right with <= 10% absent shown on dev

min coverage (dev) best rule on dev dev held-out
5% {"minProb":0.02,"minMargin":0.16,"maxFusedRank":3} 28/203 shown, 75.0% right, absent shown 2/106 28/178 shown, 82.1% right, absent shown 8/130
10% {"minProb":0.02,"minMargin":0.16,"maxFusedRank":3} 28/203 shown, 75.0% right, absent shown 2/106 28/178 shown, 82.1% right, absent shown 8/130
20% {"minProb":0.02,"minMargin":0.12,"maxFusedRank":3} 44/203 shown, 70.5% right, absent shown 3/106 39/178 shown, 74.4% right, absent shown 14/130
30% {"minProb":0.34,"minMargin":0.06,"maxFusedRank":3} 69/203 shown, 62.3% right, absent shown 7/106 55/178 shown, 72.7% right, absent shown 25/130
40% {"minProb":0.46,"minMargin":0.02,"maxFusedRank":3} 83/203 shown, 56.6% right, absent shown 6/106 72/178 shown, 63.9% right, absent shown 24/130

Hybrid retrieval A/B

Bare chunk embeddings:


Hybrid (BM25 + Xenova/bge-small-en-v1.5@q8, RRF), 15 documents, 16 chunks, 4.9 ms/query
all         n=30  R@1 0.77  R@3 0.83  R@5 0.83  MRR 0.79
literal     n=23  R@1 0.91  R@3 0.96  R@5 0.96  MRR 0.94
paraphrase  n= 7  R@1 0.29  R@3 0.43  R@5 0.43  MRR 0.33
hardneg     n= 4  R@1 0.75  R@3 1.00  R@5 1.00  MRR 0.88
abstain     n=10  gate-correct 0.10
  FALSE-CONFIDENCE: What penalty applies if the pilot misses the 15 November deadline? -> served 06_Client_Email_Extract.md, 07_Internal_Action_Log.md, 01_Northstar_Proposal_v1.md
  FALSE-CONFIDENCE: What is R. Kapoor's direct phone number? -> served 06_Client_Email_Extract.md
  FALSE-CONFIDENCE: Which vendor is supplying the analytics hardware? -> served 01_Northstar_Proposal_v1.md
  FALSE-CONFIDENCE: What severance terms were agreed for the delivery team? -> served 05_Project_Delivery_Update.md, 01_Northstar_Proposal_v1.md, 03_Commercial_Revision.md
  FALSE-CONFIDENCE: What was the patient's platelet count? -> served lab-table.pdf
  FALSE-CONFIDENCE: What did generator D record during the test? -> served repeated-headings.pdf, digital.pdf, lab-table.pdf
  FALSE-CONFIDENCE: When was the cold-chain pallet price approved? -> served minutes.docx
  FALSE-CONFIDENCE: How much funding did the solar array request ask for? -> served chrome-columns.pdf
  FALSE-CONFIDENCE: What is the replacement weather mast's serial number? -> served chrome-columns.pdf
wrote evaluation/results/retrieval-hybrid-2026-11-12-32d72c8.json

Contextual chunk headers:


Hybrid (BM25 + Xenova/bge-small-en-v1.5@q8, RRF), 15 documents, 16 chunks, 4.7 ms/query
all         n=30  R@1 0.77  R@3 0.83  R@5 0.83  MRR 0.79
literal     n=23  R@1 0.91  R@3 0.96  R@5 0.96  MRR 0.94
paraphrase  n= 7  R@1 0.29  R@3 0.43  R@5 0.43  MRR 0.33
hardneg     n= 4  R@1 0.75  R@3 1.00  R@5 1.00  MRR 0.88
abstain     n=10  gate-correct 0.10
  FALSE-CONFIDENCE: What penalty applies if the pilot misses the 15 November deadline? -> served 06_Client_Email_Extract.md, 07_Internal_Action_Log.md, 01_Northstar_Proposal_v1.md
  FALSE-CONFIDENCE: What is R. Kapoor's direct phone number? -> served 06_Client_Email_Extract.md
  FALSE-CONFIDENCE: Which vendor is supplying the analytics hardware? -> served 01_Northstar_Proposal_v1.md
  FALSE-CONFIDENCE: What severance terms were agreed for the delivery team? -> served 05_Project_Delivery_Update.md, 01_Northstar_Proposal_v1.md, 03_Commercial_Revision.md
  FALSE-CONFIDENCE: What was the patient's platelet count? -> served lab-table.pdf
  FALSE-CONFIDENCE: What did generator D record during the test? -> served repeated-headings.pdf, digital.pdf, lab-table.pdf
  FALSE-CONFIDENCE: When was the cold-chain pallet price approved? -> served minutes.docx
  FALSE-CONFIDENCE: How much funding did the solar array request ask for? -> served chrome-columns.pdf
  FALSE-CONFIDENCE: What is the replacement weather mast's serial number? -> served chrome-columns.pdf
wrote evaluation/results/retrieval-hybrid-2026-11-12-32d72c8.json

@charan-rathore
charan-rathore merged commit a49e724 into main Nov 12, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant