8 min read

From an invoice photo to clean JSON: a vision model instead of OCR

AIOpenAILangChainNode.jsOCR

The task sounds simple: you have a photo of a goods delivery note and you need structured data out of it. Supplier, buyer, every line of the goods table with prices, totals, VAT. Not as text, but as clean JSON you can push straight into a database or down the pipeline.

A few years ago this would have meant Tesseract, regexes and a lot of pain: classic OCR returns the table as a mush of lines, and every supplier prints the form differently. Modern vision models remove that problem: the model sees the whole document, understands the table structure and answers straight in the schema you define. It works the same on OpenAI and Claude. I built mine on OpenAI, so that's the version I'll show: NestJS + LangChain + gpt-4o-mini.

The schema is a contract

The first and most important decision: don't ask the model to “return JSON”, pin the response shape with structured output instead. I describe the schema in Zod and hand it to the model. LangChain converts it to JSON Schema, and the API guarantees the response validates against it:

typescript
const itemSchema = z.object({
  position: z.number().int().describe("Row number from the # column"),
  name: z.string().describe("Product name exactly as in the document"),
  quantity: z.number(),
  unit: z.string().nullable().describe("Unit of measure: pcs, cans, sheets"),
  priceWithoutVat: z.number().nullable().describe("Unit price excl. VAT"),
  amountWithoutVat: z.number().nullable().describe("Line amount excl. VAT"),
  vatRate: z.number().nullable().describe("VAT rate in percent, e.g. 20"),
  vatAmount: z.number().nullable(),
  priceWithVat: z.number().nullable(),
  amountWithVat: z.number().nullable(),
});

const invoiceSchema = z.object({
  title: z.string().nullable().describe('Title verbatim, e.g. "Delivery Note"'),
  number: z.string().nullable().describe("Document number without the # sign"),
  date: z.string().nullable().describe("Document date in YYYY-MM-DD format"),
  currency: z.string().nullable().describe("Currency in ISO format, e.g. UAH"),
  supplier: partySchema,
  buyer: partySchema,
  contract: z.object({
    number: z.string().nullable(),
    date: z.string().nullable().describe("YYYY-MM-DD"),
  }),
  items: z.array(itemSchema),
  totals: totalsSchema,
});

Two non-obvious details I didn't get right on the first try:

  • nullable, not optional. Strict structured output requires every key to be present in the response. “Not in the document” is passed explicitly as null, so the parser never breaks on a missing field.
  • describe() is a mini-prompt. Field descriptions are exactly where the model learns that unit means “pcs, cans, sheets” and that dates must be normalized. The more precise the descriptions, the fewer surprises in the response.

A prompt with domain rules

The schema says *what* to return. The system prompt says *how* to read the document (in production I run it in Ukrainian, the language of my documents; here it's translated):

typescript
const SYSTEM_PROMPT = [
  "You extract data from a photo or scan of a goods delivery invoice.",
  "Transcribe values exactly as printed: do not rephrase product names or fix spelling.",
  "Return numbers as numbers: the decimal separator is a dot, strip thousands separators (1 975.00 → 1975.00).",
  "Normalize dates to YYYY-MM-DD.",
  "Anything missing from the document is null. Never invent values or add rows that are not in the table.",
  "Field labels on such forms can be wrong: when a label contradicts the arithmetic, trust the numbers and the standard 20% VAT rate.",
].join(" ");

The last rule matters most. Real-world forms come with mislabeled fields: on my test invoice the line “Amount excl. VAT: 395.00” is actually the VAT amount, while 1975.00 labeled “Total” is the amount excluding VAT. Without that instruction the model faithfully copies the wrong labels, and your database gets 395 UAH of VAT stored as the net amount.

Feeding the photo to the model

The model has no access to your file system: it accepts either a public URL or a data URI. A local file you read yourself and pack into base64:

typescript
async function toImageUrl(source: string): Promise<string> {
  if (/^https?:\/\//i.test(source)) return source;

  const path = resolve(process.cwd(), source);
  const mime = MIME_BY_EXTENSION[extname(path).toLowerCase()];
  const file = await readFile(path);

  return `data:${mime};base64,${file.toString("base64")}`;
}

The call itself is a regular chat completion, just with an image in the message. temperature: 0, because creativity is not welcome here:

typescript
const model = new ChatOpenAI({
  apiKey,
  model: "gpt-4o-mini",
  temperature: 0,
}).withStructuredOutput(invoiceSchema, {
  name: "invoice",
  includeRaw: true,
});

const { raw, parsed } = await model.invoke([
  new SystemMessage(SYSTEM_PROMPT),
  new HumanMessage({
    content: [
      {
        type: "text",
        text: "Extract everything from this invoice: the header, both parties, every table row and the totals.",
      },
      { type: "image_url", image_url: { url: imageUrl, detail: "high" } },
    ],
  }),
]);

Note detail: "high". Without it the API downscales the image and the fine print in the table gets misread. The price is significantly more input tokens, but for documents this is not an option, it's a requirement.

The result

Scan of delivery note No. 3 of 04.01.2024: supplier Edelweiss LLC, buyer Orchid LLC, two line items, totals with VAT
The test invoice: a typical scan with a table, fine print and a trap in the totals

Here's what gpt-4o-mini returns for this scan in a single request:

json
{
  "title": "DELIVERY NOTE",
  "number": "3",
  "date": "2024-01-04",
  "currency": "UAH",
  "supplier": {
    "name": "Edelweiss LLC",
    "address": "14032, Chernihiv, 17 First Tank Brigade St.",
    "phone": "+380(462)151554",
    "iban": "UA813003350000026003333333333",
    "bank": "Aval Bank",
    "taxCode": null
  },
  "buyer": {
    "name": "Orchid LLC",
    "address": "14032, Chernihiv, 35 Donechka St.",
    "phone": "+380(462)181864",
    "iban": "UA813003350000026003333333333",
    "bank": "PrivatBank",
    "taxCode": null
  },
  "contract": { "number": "1", "date": "2024-01-03" },
  "items": [
    {
      "position": 1,
      "name": "acrylic-polyurethane furniture lacquer Trae Lyx Moebel lak (0.25 l)",
      "quantity": 10,
      "unit": "cans",
      "priceWithoutVat": 150,
      "amountWithoutVat": 1500,
      "vatRate": 20,
      "vatAmount": 300,
      "priceWithVat": 180,
      "amountWithVat": 1800
    },
    {
      "position": 2,
      "name": "fibreboard ST-40 (2.5 mm × 2440 mm × 1220 mm)",
      "quantity": 5,
      "unit": "sheets",
      "priceWithoutVat": 95,
      "amountWithoutVat": 475,
      "vatRate": 20,
      "vatAmount": 95,
      "priceWithVat": 114,
      "amountWithVat": 570
    }
  ],
  "totals": {
    "itemsCount": 2,
    "amountWithoutVat": 1975,
    "vatAmount": 395,
    "amountWithVat": 2370,
    "amountInWords": "Two thousand three hundred seventy hryvnias 00 kopecks",
    "vatInWords": "Three hundred ninety-five hryvnias 00 kopecks"
  },
  "signedBySupplier": "Sadovnyk V.S.",
  "signedByBuyer": "Dubyna M.V."
}

The totals came out right: 1975 / 395 / 2370, despite the swapped labels on the form. Not without a fly in the ointment though: mini is not perfect on fine print. The surname “Sadchykov” turned into “Sadovnyk”, and Dotsenka Street became “Donechka”. For dense scans where every letter matters, step up to gpt-4o: it costs more but reads small text noticeably better.

Trust, but verify

The model can get any digit wrong, so I validate the result with arithmetic, not with trust:

  • quantity × price = line amount, for every row;
  • sum of line amounts excl. VAT = the “excl. VAT” total;
  • total excl. VAT + VAT = total incl. VAT;
  • item count in the totals = number of recognized rows.
typescript
items.forEach((item) => {
  if (item.priceWithoutVat === null || item.amountWithoutVat === null) return;

  const expected = item.quantity * item.priceWithoutVat;

  if (Math.abs(expected - item.amountWithoutVat) > AMOUNT_TOLERANCE) {
    problems.push(
      `row ${item.position}: ${item.quantity} × ${item.priceWithoutVat} = ${expected}, but the document says ${item.amountWithoutVat}`,
    );
  }
});

A failed check is a signal to retry the request or route the document to a human. For accounting data this safeguard is mandatory: a single wrong digit costs far more than a repeated request.

What it costs

Structured output in LangChain hides the raw model message, and the token counters live exactly there, hence includeRaw: true in the config above:

typescript
const usage = isAIMessage(raw) ? raw.usage_metadata : undefined;

const cost =
  (usage.input_tokens * PRICE_PER_1M_TOKENS.input +
    usage.output_tokens * PRICE_PER_1M_TOKENS.output) /
  1_000_000;

My request: 28,336 input tokens (almost all of it the image in detail: "high") and 449 output tokens:

console
request cost: $0.004520 (gpt-4o-mini, input 28336, output 449 tokens)
Less than half a cent for a fully structured invoice, with line items, totals, VAT and arithmetic checks included.

Takeaways

  • Structured output with a schema instead of “return JSON”: the model physically can't return anything outside the contract.
  • nullable fields are the honest way to say “not in the document”, with no invented values.
  • Domain rules belong in the system prompt: number and date normalization, distrust of mislabeled form fields.
  • Arithmetic validation of the result is a mandatory safeguard for financial data.
  • A cheap vision model can be trusted with digits, but not with fine print: for names and addresses in dense scans, take a bigger model.

The whole runner is a single NestJS command of ~300 lines: nest-commander, LangChain and no OCR engine at all. The same approach works unchanged with Claude: both providers support vision and structured output.