AI extraction layer - API Contracts

[chetanr25]

AI extraction (start + poll)

Part of #541

Depends on: Async job infrastructure (#545), Text input (#546)

Contract reference

• path/extraction.yaml#L7-L156
• schemas/extraction-record.yaml
• schemas/incident-contract.yaml

Endpoints

POST /api/v1/extract/{input_id}

Takes the narrative from a previously submitted input and sends it to the local Ollama LLM to extract incident fields into the structured incident contract JSON schema.

This is async (LLM can take 30-120 seconds). Returns 202 with extract_id and job_id for polling. Details on request/error handling in the contract reference above.

GET /api/v1/extract/{extract_id}

Returns the extraction result. When completed, the response contains the full incident contract JSON under incident_contract. When still processing, returns status with retry_after_seconds.

See schemas/incident-contract.yaml for the JSON structure.

[abhishek-8081]

AI extraction (start + poll)

Part of #541

Depends on: Async job infrastructure (#545), Text input (#546)

Contract reference

• path/extraction.yaml#L7-L156
• schemas/extraction-record.yaml
• schemas/incident-contract.yaml

Endpoints

POST /api/v1/extract/{input_id}

Takes the narrative from a previously submitted input and sends it to the local Ollama LLM to extract incident fields into the structured incident contract JSON schema.

This is async (LLM can take 30-120 seconds). Returns 202 with extract_id and job_id for polling. Details on request/error handling in the contract reference above.

GET /api/v1/extract/{extract_id}

Returns the extraction result. When completed, the response contains the full incident contract JSON under incident_contract. When still processing, returns status with retry_after_seconds.

See schemas/incident-contract.yaml for the JSON structure.

It maps closely onto the #547 voice pattern (async Job + Celery task +
poll), so most of the structure carries over - Option A Job with job_type="extraction",
no migration (Extraction model exists from #544).

A few choices I'm making, shout if you'd prefer otherwise:

• llm.py's LLM class is field-loop-specific (one Ollama call per field), so I'm adding a
  standalone call_ollama(prompt, model) in llm.py for the single-shot extraction call -
  same shape as call_whisper_asr. Not reusing the per-field loop.
• Malformed LLM JSON → fail-fast: status=failed, error_type=LLM_PARSE_ERROR, store the raw
  text (truncated). The contract's error_type/error_detail fields look designed for this.
  I'll strip markdown fences/preamble before json.loads so it doesn't fail on formatting.
• One question I do want your call on: EXTRACTION_EXISTS. The contract says 409 if an
  extraction already exists. Should a failed extraction be re-runnable, or strict 409
  for any existing one? I'll implement strict 409 for now and we can add re-run as a
  follow-up unless you'd rather allow it here.

@marcvergees @vharkins1 @chetanr25

[marcvergees]

Your approach seems reasonable. I like it.

[chetanr25]

Hey @abhishek-8081, thanks for the explanation.
You explained things at a high level, but I have a few things in mind before we start work on this issue.

---

1. We need to lock the incident contract first

In our last community meet, we were reviewing the incident contract. As per the current contracts, this is roughly the JSON we would get once extraction is fully done: https://fireform-gsoc.chetanr25.in/incident-contract

But we came to the conclusion that the incident contract should be updated as per FireForm's current requirements, and some fields need to change. This issue is built completely around that JSON, so I think we should finish and lock the contract first, then start this issue.

---

2. We need to test if the model can actually do this well

This is a very challenging issue. The real challenge is how good the Ollama model is at this job. A few things I want us to think about first:

• Can the model fill 100+ fields correctly in one shot? We should try this on real sample narratives before we build around it. If it does not work well, we may need to split the extraction into a few smaller calls instead of one big call.
• How much wrong or made up data are we okay with? We need to agree on a hallucination limit.
• If the model is not sure about a field, should it leave it empty or should it guess? We need a clear rule here, not just an overall confidence score.
• Should we build a small test set of narratives with known correct answers? This way we can check accuracy every time we change the model or prompt, instead of just checking by eye.
• If a field has low confidence, should the form generation stop, or should it just get flagged for someone to check later? This should be decided before the extraction logic is written.

I think You will be a right person to address this issue, and inputs from @vharkins1 would be great

---

3. We need to decide what to call this JSON

I sometimes address this as "incident contract" and in sometimes - as "incident record". Before we go further, we should agree on one clear name for this extracted JSON output, so we do not end up confused of what we are talking about.

[abhishek-8081]

Hey @abhishek-8081, thanks for the explanation. You explained things at a high level, but I have a few things in mind before we start work on this issue.

1. We need to lock the incident contract first

In our last community meet, we were reviewing the incident contract. As per the current contracts, this is roughly the JSON we would get once extraction is fully done: https://fireform-gsoc.chetanr25.in/incident-contract

But we came to the conclusion that the incident contract should be updated as per FireForm's current requirements, and some fields need to change. This issue is built completely around that JSON, so I think we should finish and lock the contract first, then start this issue.

2. We need to test if the model can actually do this well

This is a very challenging issue. The real challenge is how good the Ollama model is at this job. A few things I want us to think about first:

• Can the model fill 100+ fields correctly in one shot? We should try this on real sample narratives before we build around it. If it does not work well, we may need to split the extraction into a few smaller calls instead of one big call.
• How much wrong or made up data are we okay with? We need to agree on a hallucination limit.
• If the model is not sure about a field, should it leave it empty or should it guess? We need a clear rule here, not just an overall confidence score.
• Should we build a small test set of narratives with known correct answers? This way we can check accuracy every time we change the model or prompt, instead of just checking by eye.
• If a field has low confidence, should the form generation stop, or should it just get flagged for someone to check later? This should be decided before the extraction logic is written.

I think You will be a right person to address this issue, and inputs from @vharkins1 would be great

3. We need to decide what to call this JSON

I sometimes address this as "incident contract" and in sometimes - as "incident record". Before we go further, we should agree on one clear name for this extracted JSON output, so we do not end up confused of what we are talking about.

Agreed - and this is a better framing . I was treating #550 as async plumbing
(reuse the voice pattern, wire the task, parse the JSON). That's the easy part. The real
question is the one I skipped: can a local model extract 100+ fields accurately, not just
return valid JSON? My fail-fast plan only catches malformed JSON - it does nothing about
hallucinated or wrong values, which is the actual risk. So I agree we don't build the
extraction logic yet.

Proposed order before any extraction code:

1. Lock the incident contract - the whole issue is built on that JSON. Is there an
   issue/PR tracking the changes from the community meet, or should we open one?
2. Settle the name (contract vs record) while locking it - one name everywhere.
3. Model viability test - run real sample narratives against the model first. Can it do
   one-shot 100+ fields, or do we need to split into smaller calls? The answer changes
   the architecture, so it has to come before code. I can start this with the current
   draft schema.
4. Accuracy rules the code must enforce: empty-vs-guess on uncertain fields, hallucination
   tolerance, and whether low-confidence fields block form generation or just get flagged.

[chetanr25]

1. I believe we should discuss about this in our next meeting and should definitely open discussion on github.
2. We can finalise this in the same discussion thread.

3 & 4. I’ll leave it to you (@abhishek-8081) and @vharkins1 since you both have more experience with this. I just wanted to point out that this could become an issue.