fix(mobile): crop label scans to a real framing guide; add library upload to label mode - #2095
Merged
CodeWithCJ merged 2 commits intoAug 13, 2026
Merged
Conversation
…load The label capture uploaded the full camera frame. Vision models misread labels that occupy a small share of the frame -- the encoder's downscale makes the digits illegible and the model fills gaps with plausible values instead of failing (CodeWithCJ#2093). Label mode also lacked the gallery upload photo mode has, so an existing photo could not be used at all. - Label mode gets a portrait framing guide (300x400), and the capture is now CROPPED to it (plus an 8% tolerance margin) before upload, via expo-image-manipulator: cover-fit inverse mapping from view points to photo pixels, clamped to bounds, then capped at 1600px longest edge. The existing preview screen therefore shows exactly the region that will be analyzed, so a missed framing is a visible Retake, not silently wrong nutrition. - Library button added to label mode with allowsEditing, so the system crop UI covers photos taken outside the app. The mapping is a pure function (src/utils/labelCrop.ts) with 6 unit tests: round-trips against the forward cover mapping, aspect preservation, padding symmetry, bounds clamping, degenerate input. Full mobile suite passes (282 suites, 4718 tests) and lint is clean at --max-warnings 0. Pipeline-level evidence that cropping is the fix, measured through the real scan endpoint (self-hosted ollama vision service): a real 4032x3024 photo with the label at ~1/4 frame returned invalid JSON once and a hallucinated product the next run; the same photo cropped to the label region returned every field exactly, 3/3 runs, deterministic. Simulator verification was attempted and is honestly incomplete: the only available hardware has VT-x disabled, and an unaccelerated qemu emulator ANR-looped in system_server across API 35 and API 31 images before the app reached interactive UI. Details in the PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PR Validation ResultsChange Detection
✅ All checks passed. Thank you! |
Contributor
|
Important Review skippedNo new commits to review since the last review. ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughNutrition-label scanning now supports editable camera and library photo selection. Selected images are resized, compressed, and encoded before use. Label previews preserve the full image within a black container, and barcode guide spacing uses a shared constant. ChangesNutrition label capture
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant User
participant FoodScanScreen
participant SystemImagePicker
participant ExpoImageManipulator
User->>FoodScanScreen: select camera or library
FoodScanScreen->>SystemImagePicker: launch editable picker
SystemImagePicker-->>FoodScanScreen: return selected photo
FoodScanScreen->>ExpoImageManipulator: resize and compress photo
ExpoImageManipulator-->>FoodScanScreen: return prepared image data
FoodScanScreen-->>User: display label preview
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
SparkyFitnessMobile/src/screens/FoodScanScreen.tsx (1)
664-666: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winShow the complete processed crop in the preview.
The label crop has a 3:4 aspect ratio. The full-screen preview is much taller.
resizeMode="cover"crops the left and right sides again, so the user cannot verify the complete image thatscanNutritionLabelreceives.Use
resizeMode="contain"and a background color for this preview.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@SparkyFitnessMobile/src/screens/FoodScanScreen.tsx` around lines 664 - 666, Update the capturedPhoto preview in the FoodScanScreen render to use resizeMode="contain" instead of "cover", and add a background color to the surrounding preview or image styling so the complete processed crop remains visible without side-cropping.
🧹 Nitpick comments (1)
SparkyFitnessMobile/src/screens/FoodScanScreen.tsx (1)
375-388: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winCap library-selected label images before analysis.
The picker path skips
LABEL_MAX_DIMENSION, so high-resolution images can create unnecessarily large in-memory base64 values and requests. Setbase64: false, processasset.uriwithImageManipulatorusing the same dimension cap, JPEG format, and compression, then store the processed base64. The picker already documents JPEG base64 output, and server dispatch corrects sniffable MIME types, so MIME relabeling is not the issue.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@SparkyFitnessMobile/src/screens/FoodScanScreen.tsx` around lines 375 - 388, Update the library-picker flow in FoodScanScreen around launchImageLibraryAsync and asset handling to request base64: false, then process asset.uri through ImageManipulator with LABEL_MAX_DIMENSION, JPEG output, and the existing compression setting before storing the resulting base64 and URI in setCapturedPhoto. Preserve the current validation and error behavior while using the processed image data for analysis.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@SparkyFitnessMobile/src/utils/labelCrop.ts`:
- Around line 68-77: Update the endpoint calculations in the label-cropping
function so x0 and x1 are both clamped to [0, photoWidth], while y0 and y1 are
both clamped to [0, photoHeight]. Add tests covering guides fully outside the
left, right, top, and bottom edges, while preserving the existing minimum crop
dimensions.
---
Outside diff comments:
In `@SparkyFitnessMobile/src/screens/FoodScanScreen.tsx`:
- Around line 664-666: Update the capturedPhoto preview in the FoodScanScreen
render to use resizeMode="contain" instead of "cover", and add a background
color to the surrounding preview or image styling so the complete processed crop
remains visible without side-cropping.
---
Nitpick comments:
In `@SparkyFitnessMobile/src/screens/FoodScanScreen.tsx`:
- Around line 375-388: Update the library-picker flow in FoodScanScreen around
launchImageLibraryAsync and asset handling to request base64: false, then
process asset.uri through ImageManipulator with LABEL_MAX_DIMENSION, JPEG
output, and the existing compression setting before storing the resulting base64
and URI in setCapturedPhoto. Preserve the current validation and error behavior
while using the processed image data for analysis.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 4370b64a-1d34-43a2-8408-34d730251281
⛔ Files ignored due to path filters (1)
pnpm-lock.yamlis excluded by!**/pnpm-lock.yaml
📒 Files selected for processing (4)
SparkyFitnessMobile/__tests__/utils/labelCrop.test.tsSparkyFitnessMobile/package.jsonSparkyFitnessMobile/src/screens/FoodScanScreen.tsxSparkyFitnessMobile/src/utils/labelCrop.ts
JonZavialov
marked this pull request as draft
August 11, 2026 10:01
…ll preview Field feedback from real use of the first revision: a fixed on-screen guide still constrains how precisely the label can be framed, and differs from the library path's adjustable system crop for no user-visible reason. Both paths now share one shape: get an image (system camera or library), crop it to the label with the system editor's draggable handles, downscale, scan. This DELETES the custom coordinate machinery the first revision added -- labelCrop.ts, its 6 unit tests, the guide overlay, and the camera-layout tracking. The system editor crops in image space, so there is no view-to-photo mapping left to get wrong; the open review finding about asymmetric clamping in computeGuideCropRect is resolved by that removal. The pre-scan preview now renders resizeMode="contain" on a black letterbox instead of "cover": what the user confirms is exactly the full image the model receives, not a screen-filling crop of it. Shared prep helper caps the longest edge at 1600px and guarantees base64 output for assets that arrive without it. Verified: lint clean at --max-warnings 0; all 281 suites pass on this code -- 274 in the parallel run plus the 7 whose workers were SIGTERMed by the build host's OOM guard (26 kills logged during the run), re-run serially: 135/135. The FoodScanScreen suite passes in isolation. Flow exercised end-to-end on a Windows-hosted emulator against a live self-hosted instance: capture -> crop -> scan returned correct label values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
JonZavialov
marked this pull request as ready for review
August 11, 2026 12:59
Contributor
Author
|
@CodeRabbit review |
Contributor
|
CodeWithCJ
approved these changes
Aug 13, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
What problem does this PR solve?
Label scans upload the full camera frame, so a label occupying a modest share of the frame reaches the vision model too small to read after encoder downscaling — and instead of failing, models fill the gaps with plausible wrong values (#2093, maintainer-invited PR). Label mode also lacked the gallery upload photo mode already has.
How did you implement the solution?
Both label paths now share one flow: get an image (system camera or library), crop it to the label with the system crop editor's adjustable handles, downscale (longest edge capped at 1600 px), scan. The pre-scan preview renders the entire image (
containon a black letterbox), so what the user confirms is exactly what the model receives.Linked Issue: Closes #2093
How to Test
PR Type
Checklist
All PRs:
New features only:
Frontend changes (
SparkyFitnessFrontend/):pnpm run validateand it passes.en) translation file.Backend changes (
SparkyFitnessServer/):rls_policies.sqlfor any new user-specific tables.UI changes (components, screens, pages):
Mobile changes (
SparkyFitnessMobile/):Screenshots
Click to expand
Notes for Reviewers
Verified on an Android emulator against a live self-hosted instance (ollama vision service,
gemma3:4b): capture → system crop → scan returned every label value correctly; the same photo scanned full-frame previously returned invalid JSON or hallucinated products. Pipeline-level reproduction with a public image is in #2093.Tests: lint clean at
--max-warnings 0; all 281 suites pass on this code (274 in a parallel run; 7 suites whose jest workers were SIGTERMed by the build host's OOM killer — unrelated to the change — pass 135/135 when re-run serially;FoodScanScreen.test.tsxpasses in isolation).Dependency note:
expo-image-manipulatoris added for the post-crop downscale (longest edge 1600 px), which bounds upload size without costing accuracy.Summary by CodeRabbit
New Features
Bug Fixes