diff --git a/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-design.md b/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-design.md new file mode 100644 index 0000000..09429d7 --- /dev/null +++ b/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-design.md @@ -0,0 +1,61 @@ +# Hebrew Translator — PDF positional overlay (block-reflow) design + +**Date:** 2026-06-04 +**Status:** Approved (design-lock) +**Builds on:** DOCX in-place (live). This is the second/last pixel-faithful file sub-phase. Scanned-PDF OCR remains a separate future phase. + +## Goal +For digital PDF uploads, produce a translated PDF that preserves images, vector graphics, and overall layout by copying the original pages and replacing each text block in place with its translation (reflowed + auto-fit within the block region). Scanned PDFs (no text layer) gracefully fall back to the existing flat path. + +## Decisions (locked) +| Topic | Decision | +|-------|----------| +| Overlay granularity | **Block-level reflow** (group lines→paragraphs; translate block; white-out region; draw wrapped, auto-fit translation) | +| Layout preservation | Copy original pages (images/vectors/other blocks intact); only text-block regions replaced | +| Translation passes | One — block extraction feeds both viewer and overlay | +| Libs | `pdf.js-extract` (present) for extraction; `pdf-lib` + `@pdf-lib/fontkit` for render; `DejaVuSans.ttf` (present) embedded | +| Scanned PDFs | Out of scope → fallback flat (OCR future) | + +## Architecture + +### Extraction — `server/services/pdfOverlay.js: extractBlocks(buffer)` +- `pdf.js-extract` → per page: items `{x, y, w, h, str, fontName}` + `pageWidth/pageHeight` (top-left origin). +- Group items into **lines** (cluster by y within tolerance; sort by x). Group lines into **blocks/paragraphs** (vertical-gap threshold + left-x alignment). +- Each block: `{ page, bbox:{x,y,w,h} (union of its lines, top-left origin), content (lines joined), lines:[{bbox,text}] }`. +- If total text items across all pages is 0 → return a `noTextLayer: true` signal (caller falls back). + +### Translation +- blocks → `buildSegments` → `buildTranslationDocument` (Groq primary, one pass). Per-block target = joined sentence targets. Reused by the viewer. + +### Render — `renderOverlay(buffer, blocksWithTargets)` +- `pdf-lib`: load the original PDF (preserves images/vectors); register `@pdf-lib/fontkit`; embed `DejaVuSans.ttf` once (subset). +- For each block with a translation: on its page, convert bbox to pdf-lib coords (bottom-left: `y' = pageHeight - y - h`). Draw a white rectangle over the block region (cover original text). Draw the translated text wrapped to the region width with **auto-fit font size**: start near the original size, shrink to fit width+height, floor ≈ 6pt; if it still overflows, allow overflow (never clip — don't lose text). +- Save → PDF buffer. + +### Integration (worker) +- `.pdf` input → `extractBlocks`; if `noTextLayer` or any error → **fallback to the current flat path**. Else translate (one pass; `saveResult` for viewer) → `renderOverlay` → output PDF. `.docx` path (in-place) unchanged. + +## Error handling +- No text layer (scanned) → fallback flat (note: OCR is a future phase). +- Any extraction/render error → fallback flat (download never breaks, job never fails). +- Missing translation for a block → leave the original (do not white-out). +- Missing glyph (e.g., retained Hebrew) → DejaVu lacks Hebrew; output is en/ru so fine — documented edge case. + +## Guardrails (design-guardrails-audit) — 🔴 = acceptance criteria +1. 🔴 **Page cap** — enforce existing `MAX_PAGES` before/at pdf.js extraction (DoS). +2. 🔴 **No-text-layer detection → fallback** — never emit a blank/garbage overlay PDF. +3. 🔴 **Render error → fallback flat** — job never fails; never emit a corrupt PDF. +4. 🟡 Coordinate conversion correctness (top-left ↔ bottom-left) — unit-tested on a known bbox. +5. 🟡 Overflow policy — shrink-to-floor then allow overflow (no clipping), explicit. +6. 🟡 Memory — bounded by existing file-size + page caps. +7. 🟡 One translation pass (cost). +8. 🟢 Font embedded once (subset) per document. + +## Testing +- Unit: line/block grouping (synthetic items → expected blocks); coordinate conversion; auto-fit font calc (short text → base size, long text → shrinks to floor). +- Render: a generated 1-page PDF + block mapping → `renderOverlay` returns a valid PDF (pdf-lib re-load succeeds, page count preserved). +- Integration: `extractBlocks` on a generated text PDF → blocks; fake-translate → `renderOverlay` → re-loads valid. Fallback path on a no-text PDF. + +## Scope (YAGNI) +**In:** digital-PDF block-reflow overlay (images/vectors preserved), auto-fit, fallback flat on error/scanned, one translation pass. +**Out:** scanned-PDF OCR, pixel-exact per-line positioning, Hebrew-glyph output, table structure reconstruction (drawn vectors stay as-is). diff --git a/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-implementation.md b/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-implementation.md new file mode 100644 index 0000000..963c33a --- /dev/null +++ b/docs/plans/2026-06-04-hebrew-translator-pdf-overlay-implementation.md @@ -0,0 +1,225 @@ +# PDF positional overlay (block-reflow) — Implementation Plan + +> **For Claude:** REQUIRED SUB-SKILL: Use superpowers:executing-plans to implement this plan task-by-task. + +**Goal:** For digital PDF uploads, output a translated PDF that keeps images/vectors/layout by copying original pages and replacing each text block in place with its reflowed, auto-fit translation. Scanned/no-text PDFs fall back to the flat path. + +**Architecture:** `pdf.js-extract` → group items into lines→blocks (pure, testable) → existing segment translator (one pass) → `pdf-lib`+`@pdf-lib/fontkit` copy original pages, white-out each block region, draw wrapped auto-fit translation (embedded DejaVuSans). Worker tries overlay; any error / no-text-layer → flat fallback. + +**Tech Stack:** pdf.js-extract (present), pdf-lib, @pdf-lib/fontkit, DejaVuSans.ttf (present at `server/assets/fonts/`), existing pipeline, vitest. + +**Design:** `docs/plans/2026-06-04-hebrew-translator-pdf-overlay-design.md` (🔴 guardrails = acceptance criteria). **Branch:** `feat/pdf-overlay`. **Test:** `npm test -- `. + +--- + +### Task 1: Add deps + +```bash +npm install pdf-lib @pdf-lib/fontkit --legacy-peer-deps +node -e "require('pdf-lib'); require('@pdf-lib/fontkit'); console.log('ok')" +git add package.json package-lock.json +git commit -m "build: add pdf-lib + @pdf-lib/fontkit for PDF overlay" +``` + +--- + +### Task 2: Block grouping (pure) + `extractBlocks` + +**Files:** Create `server/services/pdfOverlay.js` + `server/services/__tests__/pdfOverlay.group.test.js`. + +Export a **pure** `groupBlocks(items, opts)` and an async `extractBlocks(buffer)`. + +`groupBlocks(items, { yTol = 3, blockGapFactor = 1.6 } = {})`: +- `items`: `[{ x, y, width, height, str }]` (one page, top-left origin). +- Drop items whose `str.trim()` is empty. +- Sort by `y` then `x`. Cluster into **lines**: a new line when `|item.y - currentLineY| > yTol`. Within a line, sort items by `x`, `text = items.map(i=>i.str).join(' ')`, `bbox` = union (minX,minY,maxX-minX,maxY-minY), `lineHeight` = max item height. +- Group consecutive lines into **blocks**: start a new block when the vertical gap `line.y - (prevLine.y + prevLine.height) > prevLine.height * blockGapFactor`. Block `bbox` = union of its lines; `content` = lines' text joined by `' '`. +- Return `[{ bbox:{x,y,w,h}, content, lines:[{bbox,text}] }]`. + +**Step 1 — failing test:** +```javascript +import { describe, it, expect } from 'vitest'; +import { groupBlocks } from '../pdfOverlay.js'; + +const it_ = (x,y,w,h,str) => ({ x, y, width:w, height:h, str }); + +it('groups items into lines and blocks', () => { + const items = [ + it_(50, 100, 20, 10, 'Hello'), it_(75, 100, 20, 10, 'world'), // line 1 + it_(50, 112, 30, 10, 'second'), // line 2 (same block, small gap) + it_(50, 200, 30, 10, 'far'), // line 3 (new block, big gap) + ]; + const blocks = groupBlocks(items, { yTol: 3, blockGapFactor: 1.6 }); + expect(blocks.length).toBe(2); + expect(blocks[0].content).toBe('Hello world second'); + expect(blocks[1].content).toBe('far'); + expect(blocks[0].bbox.x).toBe(50); +}); + +it('drops empty items', () => { + expect(groupBlocks([it_(0,0,5,5,' '), it_(0,0,5,5,'x')]).length).toBe(1); +}); +``` + +**Step 2:** `npm test -- server/services/__tests__/pdfOverlay.group` → FAIL. +**Step 3:** implement `groupBlocks`. Then implement `extractBlocks(buffer)`: +```javascript +const { PDFExtract } = require('pdf.js-extract'); +function extractBlocks(buffer) { + return new Promise((resolve, reject) => { + new PDFExtract().extractBuffer(buffer, {}, (err, data) => { + if (err) return reject(err); + const pages = (data && data.pages) || []; + let total = 0; + const blocks = []; + pages.forEach((pg, pi) => { + const items = (pg.content || []).map(c => ({ x:c.x, y:c.y, width:c.width, height:c.height, str:c.str })); + total += items.filter(i => i.str && i.str.trim()).length; + const pw = (pg.pageInfo && pg.pageInfo.width) || pg.width; + const ph = (pg.pageInfo && pg.pageInfo.height) || pg.height; + groupBlocks(items).forEach(b => blocks.push({ ...b, page: pi, pageWidth: pw, pageHeight: ph })); + }); + resolve({ blocks, noTextLayer: total === 0 }); + }); + }); +} +module.exports = { groupBlocks, extractBlocks }; +``` +**Step 4:** PASS (2). **Step 5:** commit `feat: PDF block grouping + extractBlocks`. + +--- + +### Task 3: Coordinate conversion + wrap/auto-fit (pure) + +**Files:** Modify `server/services/pdfOverlay.js`; create `server/services/__tests__/pdfOverlay.layout.test.js`. + +Add and export: +- `toPdfRect(bbox, pageHeight)` → `{ x: bbox.x, y: pageHeight - bbox.y - bbox.h, w: bbox.w, h: bbox.h }` (top-left → pdf-lib bottom-left). +- `wrapAndFit(text, boxW, boxH, measure, { max=14, min=6, lineGap=1.15 })` where `measure(str, size)` → width in pt. Returns `{ size, lines:[string] }`. Algorithm: for `size` from `max` down to `min` step 1: greedily word-wrap `text` to `boxW` using `measure`; if `lines.length * size * lineGap <= boxH` → return. If none fit, return the `min`-size wrap (overflow allowed, never clip). + +**Step 1 — failing test:** +```javascript +import { describe, it, expect } from 'vitest'; +import { toPdfRect, wrapAndFit } from '../pdfOverlay.js'; + +it('converts top-left bbox to pdf bottom-left', () => { + expect(toPdfRect({ x:10, y:20, w:30, h:8 }, 100)).toEqual({ x:10, y:72, w:30, h:8 }); +}); + +it('wraps and fits within the box', () => { + const measure = (s, size) => s.length * size * 0.5; // fake: 0.5pt per char per size unit + const r = wrapAndFit('aaaa bbbb cccc dddd', 40, 100, measure, { max:12, min:6 }); + expect(r.lines.length).toBeGreaterThan(1); + expect(r.size).toBeLessThanOrEqual(12); + expect(r.size).toBeGreaterThanOrEqual(6); +}); + +it('falls back to min size on tiny box (overflow, no crash)', () => { + const measure = (s, size) => s.length * size; + const r = wrapAndFit('verylongword', 5, 5, measure, { max:12, min:6 }); + expect(r.size).toBe(6); + expect(Array.isArray(r.lines)).toBe(true); +}); +``` + +**Step 2:** FAIL. **Step 3:** implement. **Step 4:** PASS (3). **Step 5:** commit `feat: PDF coord conversion + wrap/auto-fit helpers`. + +--- + +### Task 4: `renderOverlay` (pdf-lib) + +**Files:** Modify `server/services/pdfOverlay.js`; create `server/services/__tests__/pdfOverlay.render.test.js`. + +`async renderOverlay(buffer, blocks)` where each block = `{ page, bbox, pageHeight, target }` (target = translated text; skip blocks without a non-empty string target): +- `const { PDFDocument, rgb } = require('pdf-lib'); const fontkit = require('@pdf-lib/fontkit');` +- `const pdf = await PDFDocument.load(buffer); pdf.registerFontkit(fontkit);` +- `const fontBytes = require('fs').readFileSync(path.join(__dirname,'assets','fonts','DejaVuSans.ttf')); const font = await pdf.embedFont(fontBytes, { subset:true });` +- group blocks by page; for each block: `const page = pdf.getPages()[block.page];` `const r = toPdfRect(block.bbox, block.pageHeight || page.getHeight());` + - white-out: `page.drawRectangle({ x:r.x, y:r.y, width:r.w, height:r.h, color: rgb(1,1,1) });` + - `const { size, lines } = wrapAndFit(block.target, r.w, r.h, (s,sz)=>font.widthOfTextAtSize(s,sz), { max: Math.min(14, r.h), min:6 });` + - draw lines from the top of the box down: `let ty = r.y + r.h - size;` for each line: `page.drawText(line, { x:r.x, y:ty, size, font, color: rgb(0,0,0) }); ty -= size*1.15;` +- `return Buffer.from(await pdf.save());` + +**Step 1 — failing test** (generate a 1-page PDF with pdf-lib, render, re-load): +```javascript +import { describe, it, expect } from 'vitest'; +import { renderOverlay } from '../pdfOverlay.js'; +import { PDFDocument } from 'pdf-lib'; + +async function makePdf() { + const d = await PDFDocument.create(); + const p = d.addPage([200, 200]); + p.drawText('original', { x: 20, y: 170, size: 12 }); + return Buffer.from(await d.save()); +} + +it('renders an overlay PDF that re-loads, page count preserved', async () => { + const buf = await makePdf(); + const blocks = [{ page:0, pageHeight:200, bbox:{ x:20, y:20, w:160, h:14 }, target:'Привет мир' }]; + const out = await renderOverlay(buf, blocks); + expect(Buffer.isBuffer(out)).toBe(true); + const re = await PDFDocument.load(out); + expect(re.getPageCount()).toBe(1); + expect(out.length).toBeGreaterThan(0); +}); + +it('skips blocks without a string target without crashing', async () => { + const buf = await makePdf(); + const out = await renderOverlay(buf, [{ page:0, pageHeight:200, bbox:{x:0,y:0,w:10,h:10}, target:null }]); + await expect(PDFDocument.load(out)).resolves.toBeTruthy(); +}); +``` + +**Step 2:** FAIL. **Step 3:** implement (require `path`). **Step 4:** PASS (2). **Step 5:** commit `feat: PDF renderOverlay (copy pages, white-out, fitted text)`. + +--- + +### Task 5: Wire into the worker (pdf branch + fallback) + +**Files:** Modify `server/api/translate.js`. Create `server/services/__tests__/pdfOverlay.integration.test.js`. + +In the processor, add a PDF branch parallel to the existing docx branch (read the current handler — it has `ext === 'docx'` then `if (!usedInplace)` flat path): +```javascript +} else if (ext === 'pdf') { + try { + const buffer = await fs.readFile(filePath); + const { blocks: pblocks, noTextLayer } = await extractBlocks(buffer); + if (noTextLayer || pblocks.length === 0) throw new Error('no text layer (scanned?)'); + const blocks = pblocks.map(b => ({ type:'paragraph', content:b.content })); + const { blocks: docBlocks, segments } = buildSegments(blocks); + if (docBlocks.length !== pblocks.length) throw new Error('block/segment mismatch'); + await job.progress(40); + doc = await buildTranslationDocument({ blocks: docBlocks, segments }, + (chunk) => aiProvider.translateBatchAligned(chunk, sourceLang||'he', targetLang), + { sourceLang: sourceLang||'he', targetLang, maxSegments: MAX_SEGMENTS, concurrency:2, maxPerChunk:8, maxTokens:1200, owner:'anon', jobId:String(job.id), ts: Date.now(), onCap:(i)=>console.warn(`cap ${i.total}>${i.cap}`) }); + await job.progress(80); + const overlayBlocks = pblocks.map((b, i) => ({ page:b.page, bbox:b.bbox, pageHeight:b.pageHeight, target: docBlocks[i].sentences.map(s=>s.target).join(' ') })); + const outBuf = await renderOverlay(buffer, overlayBlocks); + outputPath = path.join(path.dirname(filePath), `translated_${crypto.randomUUID()}.pdf`); + await fs.writeFile(outputPath, outBuf); + usedInplace = true; + } catch (e) { console.warn('PDF overlay failed, falling back to flat:', e.message); } +} +``` +Place this `else if` right after the `if (ext === 'docx') {...}` block and before the `if (!usedInplace)` flat fallback (so PDF overlay failures fall through to flat). Add import `const { extractBlocks, renderOverlay } = require('../services/pdfOverlay');`. The existing page-cap (`MAX_PAGES`) is enforced inside `documentProcessor.processDocument` on the flat path; for the overlay path, pdf.js-extract reads all pages — add a guard: after extract, if `pblocks` spans more pages than `MAX_PAGES` (max `b.page`+1 > MAX_PAGES) → throw to fall back (which enforces the cap). Implement that guard. + +**Integration test** (`pdfOverlay.integration.test.js`): generate a text PDF (pdf-lib) → `extractBlocks` → if it yields blocks, fake-translate (uppercase) → `renderOverlay` → re-load valid + page count preserved. (If `extractBlocks` yields no blocks for a pdf-lib-generated PDF in this environment, assert `noTextLayer===false` is not guaranteed — in that case skip the extract assertion and test `renderOverlay` directly with hand-built blocks. Note which path you took.) + +**Verify:** `npm test -- server/services/__tests__/pdfOverlay` (all), `npm test` full (no new failures), `node -e "require('./server/api/translate.js'); console.log('loads'); process.exit(0)"`, grep `extractBlocks`/`renderOverlay` in translate.js. + +**Commit:** `feat: wire PDF overlay into worker with flat fallback + page cap`. + +--- + +### Task 6: Deploy + verify + +- Merge `feat/pdf-overlay` → main (PR). On sec: `git pull && docker compose -f docker-compose.prod.yml up -d --build app`. +- Verify on https://translator.creatman.site: upload the real Hebrew geotech PDF (he→en), download — confirm it's a PDF that opens, text translated, images/layout preserved (vs the old flat bare text). Confirm DOCX still in-place; confirm a scanned/image-only PDF falls back without error. Log to memory. + +## Acceptance criteria (design 🔴) +- [ ] Page cap enforced on overlay path (Task 5) +- [ ] No-text-layer → fallback flat (Task 5) +- [ ] Any render/extract error → fallback flat; job never fails; no corrupt PDF (Task 5) +- [ ] Coordinate conversion correct (Task 3 test) +- [ ] Overflow: shrink-to-min then allow (no clip) (Task 3) +- [ ] One translation pass; PDF output preserves images/layout diff --git a/package-lock.json b/package-lock.json index 30b1321..aa63153 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,6 +9,7 @@ "version": "1.0.0", "dependencies": { "@google-cloud/translate": "^7.0.0", + "@pdf-lib/fontkit": "^1.1.1", "@xmldom/xmldom": "^0.9.10", "bull": "^4.12.0", "cors": "^2.8.5", @@ -26,6 +27,7 @@ "mammoth": "^1.6.0", "mime-types": "^2.1.35", "multer": "^1.4.5-lts.1", + "pdf-lib": "^1.17.1", "pdf-parse": "^1.1.1", "pdf.js-extract": "^0.2.1", "pdfkit": "^0.14.0", @@ -2270,6 +2272,33 @@ "dev": true, "license": "MIT" }, + "node_modules/@pdf-lib/fontkit": { + "version": "1.1.1", + "resolved": "https://registry.npmjs.org/@pdf-lib/fontkit/-/fontkit-1.1.1.tgz", + "integrity": "sha512-KjMd7grNapIWS/Dm0gvfHEilSyAmeLvrEGVcqLGi0VYebuqqzTbgF29efCx7tvx+IEbG3zQciRSWl3GkUSvjZg==", + "license": "MIT", + "dependencies": { + "pako": "^1.0.6" + } + }, + "node_modules/@pdf-lib/standard-fonts": { + "version": "1.0.0", + "resolved": "https://registry.npmjs.org/@pdf-lib/standard-fonts/-/standard-fonts-1.0.0.tgz", + "integrity": "sha512-hU30BK9IUN/su0Mn9VdlVKsWBS6GyhVfqjwl1FjZN4TxP6cCw0jP2w7V3Hf5uX7M0AZJ16vey9yE0ny7Sa59ZA==", + "license": "MIT", + "dependencies": { + "pako": "^1.0.6" + } + }, + "node_modules/@pdf-lib/upng": { + "version": "1.0.1", + "resolved": "https://registry.npmjs.org/@pdf-lib/upng/-/upng-1.0.1.tgz", + "integrity": "sha512-dQK2FUMQtowVP00mtIksrlZhdFXQZPC+taih1q4CvPZ5vqdxR/LKBaFg0oAfzd1GlHZXXSPdQfzQnt+ViGvEIQ==", + "license": "MIT", + "dependencies": { + "pako": "^1.0.10" + } + }, "node_modules/@pkgjs/parseargs": { "version": "0.11.0", "resolved": "https://registry.npmjs.org/@pkgjs/parseargs/-/parseargs-0.11.0.tgz", @@ -10749,6 +10778,24 @@ "node": "*" } }, + "node_modules/pdf-lib": { + "version": "1.17.1", + "resolved": "https://registry.npmjs.org/pdf-lib/-/pdf-lib-1.17.1.tgz", + "integrity": "sha512-V/mpyJAoTsN4cnP31vc0wfNA1+p20evqqnap0KLoRUN0Yk/p3wN52DOEsL4oBFcLdb76hlpKPtzJIgo67j/XLw==", + "license": "MIT", + "dependencies": { + "@pdf-lib/standard-fonts": "^1.0.0", + "@pdf-lib/upng": "^1.0.1", + "pako": "^1.0.11", + "tslib": "^1.11.1" + } + }, + "node_modules/pdf-lib/node_modules/tslib": { + "version": "1.14.1", + "resolved": "https://registry.npmjs.org/tslib/-/tslib-1.14.1.tgz", + "integrity": "sha512-Xni35NKzjgMrwevysHTCArtLDpPvye8zV/0E4EyYn43P7/7qvQwPh9BGkHewbMulVntbigmcT7rdX3BNo9wRJg==", + "license": "0BSD" + }, "node_modules/pdf-parse": { "version": "1.1.1", "resolved": "https://registry.npmjs.org/pdf-parse/-/pdf-parse-1.1.1.tgz", diff --git a/package.json b/package.json index aec231b..b1ad19b 100644 --- a/package.json +++ b/package.json @@ -23,6 +23,7 @@ }, "dependencies": { "@google-cloud/translate": "^7.0.0", + "@pdf-lib/fontkit": "^1.1.1", "@xmldom/xmldom": "^0.9.10", "bull": "^4.12.0", "cors": "^2.8.5", @@ -40,6 +41,7 @@ "mammoth": "^1.6.0", "mime-types": "^2.1.35", "multer": "^1.4.5-lts.1", + "pdf-lib": "^1.17.1", "pdf-parse": "^1.1.1", "pdf.js-extract": "^0.2.1", "pdfkit": "^0.14.0", diff --git a/server/api/translate.js b/server/api/translate.js index af394a2..1c44a1c 100644 --- a/server/api/translate.js +++ b/server/api/translate.js @@ -16,6 +16,7 @@ const { validateMagicBytes } = require('../middleware/fileValidation'); const { emitToSession } = require('../socket/rooms'); const { extractParagraphs, writeBack } = require('../services/docxInplace'); const { assertDocxSafe } = require('../services/zipGuard'); +const { extractBlocks, renderOverlay } = require('../services/pdfOverlay'); // Инициализируем сервисы const documentProcessor = new DocumentProcessor(); @@ -24,6 +25,9 @@ const aiProvider = new LiteLLMProvider(); // Жёсткий предел на число сегментов в одном документе (DoS-guard + бюджет). const MAX_SEGMENTS = Number(process.env.MAX_SEGMENTS) || 1500; +// Предел числа страниц PDF для positional overlay (DoS-guard + бюджет рендера). +const MAX_PAGES = Number(process.env.MAX_PAGES) || 50; + // Предел распакованного размера DOCX (zip-bomb guard) для воркера. const MAX_DOCX_UNCOMPRESSED = Number(process.env.MAX_DOCX_UNCOMPRESSED_MB || 100) * 1024 * 1024; @@ -105,6 +109,43 @@ documentQueue.process('translate', async (job) => { } catch (e) { console.warn('DOCX in-place failed, falling back to flat:', e.message); } + } else if (ext === 'pdf') { + // PDF: positional overlay поверх копии оригинала (сохраняем картинки/вёрстку). + // Сканы (нет текстового слоя) или любая ошибка → плоский путь ниже. + try { + const buffer = await fs.readFile(filePath); + const { blocks: pblocks, noTextLayer } = await extractBlocks(buffer); + if (noTextLayer || pblocks.length === 0) throw new Error('no text layer (scanned?)'); + const maxPage = pblocks.reduce((m, b) => Math.max(m, b.page), 0); + if (maxPage + 1 > MAX_PAGES) throw new Error(`too many pages: ${maxPage + 1} > ${MAX_PAGES}`); + const blocks = pblocks.map(b => ({ type: 'paragraph', content: b.content })); + const { blocks: docBlocks, segments } = buildSegments(blocks); + if (docBlocks.length !== pblocks.length) { + throw new Error(`block/segment mismatch ${docBlocks.length} != ${pblocks.length}`); + } + await job.progress(40); + doc = await buildTranslationDocument( + { blocks: docBlocks, segments }, + (chunk) => aiProvider.translateBatchAligned(chunk, sourceLang || 'he', targetLang), + { sourceLang: sourceLang || 'he', targetLang, maxSegments: MAX_SEGMENTS, + concurrency: 2, maxPerChunk: 8, maxTokens: 1200, + owner: 'anon', jobId: String(job.id), ts: Date.now(), + onCap: (info) => console.warn(`Segment cap hit: ${info.total} > ${info.cap} (job ${job.id})`) } + ); + await job.progress(80); + const overlayBlocks = pblocks.map((b, i) => ({ + page: b.page, + bbox: b.bbox, + pageHeight: b.pageHeight, + target: docBlocks[i].sentences.map(s => s.target).join(' ') + })); + const outBuf = await renderOverlay(buffer, overlayBlocks); + outputPath = path.join(path.dirname(filePath), `translated_${crypto.randomUUID()}.pdf`); + await fs.writeFile(outputPath, outBuf); + usedInplace = true; + } catch (e) { + console.warn('PDF overlay failed, falling back to flat:', e.message); + } } // Плоский путь (PDF всегда, DOCX как fallback): извлекаем текст и собираем diff --git a/server/services/__tests__/pdfOverlay.group.test.js b/server/services/__tests__/pdfOverlay.group.test.js new file mode 100644 index 0000000..a2eead5 --- /dev/null +++ b/server/services/__tests__/pdfOverlay.group.test.js @@ -0,0 +1,23 @@ +import { describe, it, expect } from 'vitest'; +import { groupBlocks } from '../pdfOverlay.js'; + +const it_ = (x, y, w, h, str) => ({ x, y, width: w, height: h, str }); + +describe('groupBlocks', () => { + it('groups items into lines and blocks', () => { + const items = [ + it_(50, 100, 20, 10, 'Hello'), it_(75, 100, 20, 10, 'world'), // line 1 + it_(50, 112, 30, 10, 'second'), // line 2, same block (small gap) + it_(50, 200, 30, 10, 'far'), // line 3, new block (big gap) + ]; + const blocks = groupBlocks(items, { yTol: 3, blockGapFactor: 1.6 }); + expect(blocks.length).toBe(2); + expect(blocks[0].content).toBe('Hello world second'); + expect(blocks[1].content).toBe('far'); + expect(blocks[0].bbox.x).toBe(50); + }); + + it('drops empty items', () => { + expect(groupBlocks([it_(0, 0, 5, 5, ' '), it_(0, 0, 5, 5, 'x')]).length).toBe(1); + }); +}); diff --git a/server/services/__tests__/pdfOverlay.integration.test.js b/server/services/__tests__/pdfOverlay.integration.test.js new file mode 100644 index 0000000..23bdc8a --- /dev/null +++ b/server/services/__tests__/pdfOverlay.integration.test.js @@ -0,0 +1,32 @@ +import { describe, it, expect } from 'vitest'; +import { PDFDocument } from 'pdf-lib'; +import { extractBlocks, renderOverlay } from '../pdfOverlay.js'; +import { buildSegments } from '../translationDocument.js'; +import { buildTranslationDocument } from '../pipeline.js'; + +async function makeTextPdf() { + const d = await PDFDocument.create(); + const p = d.addPage([300, 300]); + p.drawText('hello world', { x: 30, y: 250, size: 14 }); + p.drawText('second line', { x: 30, y: 220, size: 14 }); + return Buffer.from(await d.save()); +} +const fakeBatch = async (chunk) => ({ items: chunk.map(s => ({ id: s.id, target: s.source.toUpperCase(), align: [] })), usage: null }); + +it('extract -> translate(fake) -> renderOverlay produces a valid PDF', async () => { + const buf = await makeTextPdf(); + const { blocks: pblocks, noTextLayer } = await extractBlocks(buf); + if (noTextLayer || pblocks.length === 0) { + // environment did not expose a text layer for the generated PDF; + // fall back to testing renderOverlay directly with a hand-built block. + const out = await renderOverlay(buf, [{ page:0, pageHeight:300, bbox:{x:30,y:50,w:240,h:16}, target:'HELLO WORLD' }]); + expect((await PDFDocument.load(out)).getPageCount()).toBe(1); + return; + } + const blocks = pblocks.map(b => ({ type:'paragraph', content:b.content })); + const { blocks: docBlocks, segments } = buildSegments(blocks); + const doc = await buildTranslationDocument({ blocks: docBlocks, segments }, fakeBatch, { sourceLang:'he', targetLang:'en' }); + const overlay = pblocks.map((b, i) => ({ page:b.page, bbox:b.bbox, pageHeight:b.pageHeight, target: docBlocks[i].sentences.map(s=>s.target).join(' ') })); + const out = await renderOverlay(buf, overlay); + expect((await PDFDocument.load(out)).getPageCount()).toBe(1); +}); diff --git a/server/services/__tests__/pdfOverlay.layout.test.js b/server/services/__tests__/pdfOverlay.layout.test.js new file mode 100644 index 0000000..9a4b641 --- /dev/null +++ b/server/services/__tests__/pdfOverlay.layout.test.js @@ -0,0 +1,33 @@ +import { describe, it, expect } from 'vitest'; +import { toPdfRect, wrapAndFit } from '../pdfOverlay.js'; + +describe('toPdfRect', () => { + it('converts top-left bbox to pdf bottom-left', () => { + expect(toPdfRect({ x: 10, y: 20, w: 30, h: 8 }, 100)).toEqual({ x: 10, y: 72, w: 30, h: 8 }); + }); +}); + +describe('wrapAndFit', () => { + it('wraps and fits within the box', () => { + const measure = (s, size) => s.length * size * 0.5; // fake metric + const r = wrapAndFit('aaaa bbbb cccc dddd', 40, 100, measure, { max: 12, min: 6 }); + expect(r.lines.length).toBeGreaterThan(1); + expect(r.size).toBeLessThanOrEqual(12); + expect(r.size).toBeGreaterThanOrEqual(6); + // every line fits the width at the chosen size + r.lines.forEach((line) => expect(measure(line, r.size)).toBeLessThanOrEqual(40 + 1e-9)); + }); + + it('falls back to min size on a tiny box (overflow, no crash)', () => { + const measure = (s, size) => s.length * size; + const r = wrapAndFit('verylongword', 5, 5, measure, { max: 12, min: 6 }); + expect(r.size).toBe(6); + expect(Array.isArray(r.lines)).toBe(true); + expect(r.lines.length).toBeGreaterThanOrEqual(1); + }); + + it('empty text -> no lines', () => { + const r = wrapAndFit(' ', 40, 100, (s, sz) => s.length * sz, { max: 12, min: 6 }); + expect(r.lines).toEqual([]); + }); +}); diff --git a/server/services/__tests__/pdfOverlay.render.test.js b/server/services/__tests__/pdfOverlay.render.test.js new file mode 100644 index 0000000..955b223 --- /dev/null +++ b/server/services/__tests__/pdfOverlay.render.test.js @@ -0,0 +1,32 @@ +import { describe, it, expect } from 'vitest'; +import { renderOverlay } from '../pdfOverlay.js'; +import { PDFDocument } from 'pdf-lib'; + +async function makePdf() { + const d = await PDFDocument.create(); + const p = d.addPage([200, 200]); + p.drawText('original', { x: 20, y: 170, size: 12 }); + return Buffer.from(await d.save()); +} + +it('renders an overlay PDF that re-loads; page count preserved', async () => { + const buf = await makePdf(); + const blocks = [{ page: 0, pageHeight: 200, bbox: { x: 20, y: 20, w: 160, h: 14 }, target: 'Привет мир' }]; + const out = await renderOverlay(buf, blocks); + expect(Buffer.isBuffer(out)).toBe(true); + const re = await PDFDocument.load(out); + expect(re.getPageCount()).toBe(1); + expect(out.length).toBeGreaterThan(0); +}); + +it('skips blocks without a string target without crashing', async () => { + const buf = await makePdf(); + const out = await renderOverlay(buf, [{ page: 0, pageHeight: 200, bbox: { x: 0, y: 0, w: 10, h: 10 }, target: null }]); + await expect(PDFDocument.load(out)).resolves.toBeTruthy(); +}); + +it('ignores a block whose page index is out of range', async () => { + const buf = await makePdf(); + const out = await renderOverlay(buf, [{ page: 5, pageHeight: 200, bbox: { x: 0, y: 0, w: 10, h: 10 }, target: 'x' }]); + await expect(PDFDocument.load(out)).resolves.toBeTruthy(); +}); diff --git a/server/services/pdfOverlay.js b/server/services/pdfOverlay.js new file mode 100644 index 0000000..ed80d9a --- /dev/null +++ b/server/services/pdfOverlay.js @@ -0,0 +1,231 @@ +'use strict'; + +const fs = require('fs'); +const path = require('path'); +const { PDFExtract } = require('pdf.js-extract'); +const { PDFDocument, rgb } = require('pdf-lib'); +const fontkit = require('@pdf-lib/fontkit'); + +/** + * Union the bounding boxes of a list of items/boxes. + * Accepts items shaped as { x, y, width, height } or { x, y, w, h }. + * Returns { x, y, w, h }. + */ +function unionBbox(boxes) { + let minX = Infinity; + let minY = Infinity; + let maxX2 = -Infinity; + let maxY2 = -Infinity; + for (const b of boxes) { + const w = b.width != null ? b.width : b.w; + const h = b.height != null ? b.height : b.h; + if (b.x < minX) minX = b.x; + if (b.y < minY) minY = b.y; + if (b.x + w > maxX2) maxX2 = b.x + w; + if (b.y + h > maxY2) maxY2 = b.y + h; + } + return { x: minX, y: minY, w: maxX2 - minX, h: maxY2 - minY }; +} + +/** + * Pure function: group positioned text items into lines then blocks. + * + * @param {Array<{x:number,y:number,width:number,height:number,str:string}>} items - items for ONE page (top-left origin). + * @param {{yTol?:number, blockGapFactor?:number}} [opts] + * @returns {Array<{bbox:{x,y,w,h}, content:string, lines:Array<{bbox:{x,y,w,h}, text:string}>}>} + */ +function groupBlocks(items, { yTol = 3, blockGapFactor = 1.6 } = {}) { + // Drop empty/missing strings. + const clean = (items || []).filter((i) => i && typeof i.str === 'string' && i.str.trim() !== ''); + + // Sort by y asc, then x asc. + const sorted = clean.slice().sort((a, b) => (a.y - b.y) || (a.x - b.x)); + + // Cluster into lines. + const lines = []; + let current = null; + let currentLineY = null; + for (const item of sorted) { + if (current === null || Math.abs(item.y - currentLineY) > yTol) { + current = []; + currentLineY = item.y; // line anchored on its first item's y + lines.push(current); + } + current.push(item); + } + + const builtLines = lines.map((lineItems) => { + const byX = lineItems.slice().sort((a, b) => a.x - b.x); + const bbox = unionBbox(byX); + const height = byX.reduce((m, i) => Math.max(m, i.height), 0); + return { + bbox, + text: byX.map((i) => i.str).join(' '), + height, + }; + }); + + // Group consecutive lines into blocks. + const blocks = []; + let currentBlock = null; + let prevLine = null; + for (const line of builtLines) { + if ( + currentBlock === null || + line.bbox.y - (prevLine.bbox.y + prevLine.height) > prevLine.height * blockGapFactor + ) { + currentBlock = []; + blocks.push(currentBlock); + } + currentBlock.push(line); + prevLine = line; + } + + return blocks.map((blockLines) => ({ + bbox: unionBbox(blockLines.map((l) => l.bbox)), + content: blockLines.map((l) => l.text).join(' '), + lines: blockLines.map((l) => ({ bbox: l.bbox, text: l.text })), + })); +} + +/** + * Convert a top-left-origin bbox (pdf.js-extract) to a bottom-left-origin + * rectangle (pdf-lib). y grows downward in source, upward in pdf-lib. + * + * @param {{x:number,y:number,w:number,h:number}} bbox + * @param {number} pageHeight + * @returns {{x:number,y:number,w:number,h:number}} + */ +function toPdfRect(bbox, pageHeight) { + return { x: bbox.x, y: pageHeight - bbox.y - bbox.h, w: bbox.w, h: bbox.h }; +} + +/** + * Greedily word-wrap `text` into lines that each fit `boxW` at `size`. + * A single word wider than boxW still goes on its own line (overflow allowed). + */ +function wrapLines(text, boxW, size, measure) { + const words = text.split(/\s+/).filter(Boolean); + const lines = []; + let line = ''; + for (const word of words) { + const candidate = line ? `${line} ${word}` : word; + if (line && measure(candidate, size) > boxW) { + lines.push(line); + line = word; + } else { + line = candidate; + } + } + if (line) lines.push(line); + return lines; +} + +/** + * Pure word-wrap + auto-fit: pick the largest font size in [min, max] (step 1) + * at which the wrapped text fits inside boxW x boxH. If none fit, return the + * wrap computed at `min` (overflow allowed, never throws, never clips). + * + * @param {string} text + * @param {number} boxW + * @param {number} boxH + * @param {(str:string, size:number)=>number} measure - width in pt + * @param {{max?:number, min?:number, lineGap?:number}} [opts] + * @returns {{size:number, lines:string[]}} + */ +function wrapAndFit(text, boxW, boxH, measure, { max = 14, min = 6, lineGap = 1.15 } = {}) { + if (!text || text.trim() === '') return { size: max, lines: [] }; + + let fallback = null; + for (let size = max; size >= min; size -= 1) { + const lines = wrapLines(text, boxW, size, measure); + if (size === min) fallback = { size, lines }; + if (lines.length * size * lineGap <= boxH) return { size, lines }; + } + // No size fit: return the wrap at min (overflow allowed). + return fallback || { size: min, lines: wrapLines(text, boxW, min, measure) }; +} + +/** + * Extract blocks (with positions) from a PDF buffer using its text layer. + * @param {Buffer} buffer + * @returns {Promise<{blocks:Array, noTextLayer:boolean}>} + */ +function extractBlocks(buffer) { + return new Promise((resolve, reject) => { + new PDFExtract().extractBuffer(buffer, {}, (err, data) => { + if (err) return reject(err); + const pages = (data && data.pages) || []; + let total = 0; + const blocks = []; + pages.forEach((pg, pi) => { + const items = (pg.content || []).map((c) => ({ + x: c.x, + y: c.y, + width: c.width, + height: c.height, + str: c.str, + })); + total += items.filter((i) => i.str && i.str.trim()).length; + const pw = (pg.pageInfo && pg.pageInfo.width) || pg.width; + const ph = (pg.pageInfo && pg.pageInfo.height) || pg.height; + groupBlocks(items).forEach((b) => + blocks.push({ ...b, page: pi, pageWidth: pw, pageHeight: ph }) + ); + }); + resolve({ blocks, noTextLayer: total === 0 }); + }); + }); +} + +/** + * Render translations onto a copy of the original PDF. Loading the original + * buffer preserves its images/vectors; for each block we white-out the source + * text region then draw the wrapped + auto-fitted translation in a Unicode font + * (DejaVuSans covers Latin + Cyrillic). + * + * @param {Buffer} buffer - original PDF bytes + * @param {Array<{page:number, pageHeight?:number, bbox:{x,y,w,h}, target:string}>} blocks + * @returns {Promise} overlaid PDF bytes + */ +async function renderOverlay(buffer, blocks) { + const pdf = await PDFDocument.load(buffer); + pdf.registerFontkit(fontkit); + + const fontBytes = fs.readFileSync( + path.join(__dirname, '..', 'assets', 'fonts', 'DejaVuSans.ttf') + ); + const font = await pdf.embedFont(fontBytes, { subset: true }); + + const pages = pdf.getPages(); + + for (const block of blocks || []) { + if (typeof block.target !== 'string' || !block.target.trim()) continue; + const page = pages[block.page]; + if (!page) continue; + + const ph = block.pageHeight || page.getHeight(); + const r = toPdfRect(block.bbox, ph); + + // White-out the original text region. + page.drawRectangle({ x: r.x, y: r.y, width: r.w, height: r.h, color: rgb(1, 1, 1) }); + + const { size, lines } = wrapAndFit( + block.target, + r.w, + r.h, + (s, sz) => font.widthOfTextAtSize(s, sz), + { max: Math.max(6, Math.min(14, r.h)), min: 6, lineGap: 1.15 } + ); + + let ty = r.y + r.h - size; + for (const line of lines) { + page.drawText(line, { x: r.x, y: ty, size, font, color: rgb(0, 0, 0) }); + ty -= size * 1.15; + } + } + + return Buffer.from(await pdf.save()); +} + +module.exports = { groupBlocks, extractBlocks, unionBbox, toPdfRect, wrapAndFit, renderOverlay };