Processing Pipelines

By Olivier Mironescu Added: 2026-07-17

Processing Pipelines

What each job type does, which tools it uses, and how the OCR routing logic works.

Pipeline Overview

Pipeline	Job Types	Primary Tools	Queue

Image	Compress, resize, format convert, thumbnail	ImageMagick, libvips	`images`
Document	PDF preview, text extraction, page rendering	Ghostscript, pdfimages, pdftotext	`documents`
OCR	Text extraction, table detection, AI escalation	PaddleOCR, Surya, Tesseract	`ocr`
OpenXML	DOCX/XLSX/PPTX processing, conversion	PHPOffice, LibreOffice	`documents`
Media	Video/audio transcode, thumbnail, waveform	FFmpeg	`media`

Image Pipeline

Handles all static image operations. Fast, high-throughput, low memory.

Operations

Operation

Description

Tools

`image.thumbnail`	Generate thumbnail at specified dimensions	libvips (fast) or ImageMagick
`image.compress`	Reduce file size while maintaining visual quality	ImageMagick, libvips
`image.convert`	Format conversion (PNG → WebP, JPEG → AVIF, etc.)	ImageMagick
`image.resize`	Resize to specific dimensions	libvips
`image.strip_metadata`	Remove EXIF, GPS, camera data	ImageMagick

Processing Flow

1. Download source image from S3

2. Validate: check magic bytes, dimensions, color space

3. Apply operation (via ImageMagick or libvips CLI)

4. Optionally generate multiple sizes in one pass

5. Upload derived image(s) to S3

6. Register outputs in database

Multi-Size Generation

When a user uploads a photo, the app typically needs multiple sizes. Instead of separate jobs, a single image.thumbnail job can generate all sizes in one pass:

{

  "job_type": "image.thumbnail",

  "params": {

    "sizes": [

      { "name": "thumb", "width": 200, "height": 200, "fit": "cover" },

      { "name": "medium", "width": 800, "height": 600, "fit": "inside" },

      { "name": "large", "width": 1920, "height": 1080, "fit": "inside" }

    ],

    "format": "webp",

    "quality": 85

  }

}

This produces 3 outputs from a single job — much more efficient than 3 separate jobs (one download, 3 resizes, 3 uploads).

Format Strategy

Format

Use Case

Quality Setting

WebP	Default for all web-served images	85 (lossy)
AVIF	Next-gen format for modern browsers	80 (lossy)
JPEG XL	Archival / high-quality	90 (lossy)
PNG	Screenshots, UI elements, transparency	Lossless
Original	Preserved untouched in S3	—

Originals are never modified or deleted. Derived images are stored alongside the original with deterministic keys.

Document Pipeline

Handles PDF and text-based document operations.

Operations

Operation

Description

Tools

`document.pdf_preview`	Render first N pages as images	Ghostscript → ImageMagick
`document.pdf_pages`	Render all pages as individual images	Ghostscript → ImageMagick
`document.pdf_text`	Extract text from PDF (non-OCR)	pdftotext (poppler-utils)
`document.pdf_info`	Extract metadata (page count, dimensions, fonts)	pdfinfo
`document.thumbnail`	Generate document thumbnail (first page)	Ghostscript → ImageMagick

PDF Preview Flow

1. Download PDF from S3

2. pdfinfo → get page count, dimensions

3. Ghostscript: render pages at 150 DPI → PNG files

4. ImageMagick: convert PNGs to WebP, generate thumbnails

5. Upload page images to S3:

   derived/{pool_id}/{object_id}/pdf-preview:v2/page-001.webp

   derived/{pool_id}/{object_id}/pdf-preview:v2/page-002.webp

   ...

6. Upload thumbnail to S3:

   derived/{pool_id}/{object_id}/pdf-preview:v2/thumbnail.webp

7. Register outputs

Text Extraction (Non-OCR)

For PDFs with embedded text (not scanned images), pdftotext is fast and accurate. This is the preferred method when available — no OCR needed.

The worker detects whether a PDF has embedded text by running pdftotext on the first page. If the output contains meaningful text (> 50 characters), full extraction proceeds without OCR. If not, the job is re-routed to the OCR pipeline.

OCR Pipeline

The most complex pipeline. Uses a two-pass approach: cheap classical OCR first, AI OCR only when needed.

OCR Engines

Engine

Type

Strengths

When Used

PaddleOCR (PP-Structure)	Classical, layout-aware	Table detection, multi-language, fast	Default first pass
Surya	Classical, layout-aware	90+ languages, table recognition	Alternative first pass
Tesseract	Classical	Fast, lightweight, good for clean scans	Fallback for simple documents
AI OCR (via core.ai)	LLM-based	Complex tables, handwriting, degraded scans	Escalation only

Routing Logic

The key insight: not every document needs AI OCR. A clean, single-column typed letter works perfectly with classical OCR at a fraction of the cost. Only complex documents (tables, multi-column layouts, degraded scans) benefit from AI.

#### First Pass: Classical OCR + Layout Analysis

Every document goes through PaddleOCR PP-Structure (or Surya) first. This produces:

Extracted text

Layout analysis (text blocks, tables, images)

Table detection with bounding boxes

Confidence scores per region

#### Routing Decision

From the first-pass output, compute features per page:

def compute_routing_features(paddle_result, page_width, page_height):

    tables = paddle_result.get("tables", [])

    page_area = page_width * page_height



    features = {

        "n_tables": len(tables),

        "table_area_ratio": sum(t["cell_area_sum"] for t in tables) / page_area if tables else 0,

        "max_table_rows": max((len(t.get("rows", [])) for t in tables), default=0),

        "max_table_cols": max((len(t.get("cols", [])) for t in tables), default=0),

        "n_columns": detect_column_count(paddle_result),

        "avg_confidence": paddle_result.get("avg_confidence", 0),

        "skew_angle": paddle_result.get("skew_angle", 0),

        "junk_token_ratio": compute_junk_ratio(paddle_result),

    }

    return features



def should_escalate_to_ai(features):

    """Route to AI OCR if any of these conditions are met."""



    # Significant table content

    if features["n_tables"] >= 1 and features["table_area_ratio"] > 0.15:

        if features["max_table_rows"] >= 6 and features["max_table_cols"] >= 4:

            return True, "large_table"



    # Very large tables (even if area is small)

    if features["max_table_rows"] >= 10 or features["max_table_cols"] >= 8:

        return True, "complex_table"



    # Multi-column layout

    if features["n_columns"] >= 2:

        return True, "multi_column"



    # Poor quality scan with low confidence

    if features["skew_angle"] > 3 and features["avg_confidence"] < 0.85:

        return True, "low_quality"



    # High junk ratio (OCR producing nonsense)

    if features["junk_token_ratio"] > 0.20:

        return True, "high_junk"



    return False, "classical_ok"

#### Routing Decision Flow

Document uploaded

    │

    ▼

Classical OCR (PaddleOCR PP-Structure)

    │

    ├── Extract text + layout + tables

    │

    ▼

Compute routing features

    │

    ├── Features indicate simple document

    │   └── DONE — use classical OCR result

    │       (cheap, fast, good enough)

    │

    └── Features indicate complex document

        │

        ▼

    AI OCR (via core.ai → vision LLM)

        │

        ├── Send page images + routing context

        ├── Receive structured extraction

        │

        ▼

    Merge results

        │

        └── DONE — use AI OCR result

            (expensive, accurate, handles complexity)

#### Per-Page Routing

Routing is per-page, not per-document. A 50-page lab report might have 48 pages of clean text (classical OCR) and 2 pages with complex tables (AI OCR). Only the 2 complex pages are escalated.

Page 1:  text → classical OCR ✓ (cheap)

Page 2:  text → classical OCR ✓

Page 3:  table → AI OCR ✓ (expensive, but needed)

Page 4:  text → classical OCR ✓

...

Page 50: table → AI OCR ✓

#### Cost Impact

Method

Cost per Page

When

pdftotext (embedded text)	~0 credits	PDF has embedded text
Classical OCR (PaddleOCR)	Low	Clean, single-column documents
AI OCR (via core.ai)	High	Tables, multi-column, degraded scans

Typical split: 70–90% of pages stay classical, 10–30% escalate to AI.

OCR Output Format

Both classical and AI OCR produce the same output structure:

{

  "pages": [

    {

      "page_number": 1,

      "method": "paddleocr",

      "confidence": 0.94,

      "text": "Full extracted text...",

      "blocks": [

        { "type": "text", "bbox": [x, y, w, h], "content": "..." },

        { "type": "table", "bbox": [x, y, w, h], "rows": [...], "cols": [...] }

      ],

      "language": "fra"

    }

  ],

  "routing": {

    "total_pages": 50,

    "classical_pages": 47,

    "ai_pages": 3,

    "escalation_reasons": ["large_table", "multi_column"]

  }

}

The output is stored as a JSON file in S3: derived/{pool_id}/{object_id}/ocr-standard:v2/result.json.

OpenXML Pipeline

Handles Microsoft Office formats. Uses native PHP libraries for modern formats, LibreOffice for legacy conversion.

Format Support

Format

Extension

Method

Library

Word (modern)	.docx	Native OpenXML	PhpWord
Word (legacy)	.doc	LibreOffice → .docx → process	LibreOffice headless
Excel (modern)	.xlsx	Native OpenXML	PHPSpreadsheet
Excel (legacy)	.xls	LibreOffice → .xlsx → process	LibreOffice headless
PowerPoint (modern)	.pptx	Native OpenXML	PHPPresentation
PowerPoint (legacy)	.ppt	LibreOffice → .pptx → process	LibreOffice headless

Operations

Operation

Description

Tools

`openxml.extract_text`	Extract all text content	PhpWord / PHPSpreadsheet / PHPPresentation
`openxml.to_html`	Convert to HTML for web display	PhpWord → HTML
`openxml.to_pdf`	Convert to PDF	LibreOffice headless (best fidelity)
`openxml.extract_data`	Extract structured data (spreadsheet rows, form fields)	PHPSpreadsheet
`openxml.thumbnail`	Generate first-page thumbnail	LibreOffice → PDF → Ghostscript
`openxml.convert_legacy`	Convert .doc → .docx, .xls → .xlsx, .ppt → .pptx	LibreOffice headless

Legacy Conversion Flow

1. User uploads legacy file (.doc, .xls, .ppt)

2. Job created: openxml.convert_legacy

3. Worker:

   a. Download source from S3

   b. Validate file signature (not just extension)

   c. Check for macros — reject .docm/.xlsm/.pptm

   d. Run LibreOffice headless:

      soffice --headless --convert-to docx --outdir /work/jobs/123/ input.doc

   e. Validate output

   f. Upload converted file to S3

   g. Register output

4. Subsequent operations use the converted modern file

PDF Export (High Fidelity)

LibreOffice headless produces the best PDF conversion quality for Office documents:

soffice --headless --convert-to pdf:writer_pdf_Export \

  --outdir /work/jobs/123/ \

  /work/jobs/123/input.docx

This is used for:

Generating printable versions

Creating PDF thumbnails

Archival conversion

Security

Macro rejection: .docm, .xlsm, .pptm files are rejected at job creation. No exceptions.

Sandboxed execution: LibreOffice runs in a restricted environment with no network access and a clean user profile per invocation.

File validation: Magic bytes are checked. A .docx file that isn't a valid ZIP archive is rejected.

Temp isolation: Each conversion gets a fresh temp directory. LibreOffice profile data is not shared between jobs.

Media Pipeline

Handles audio and video processing via FFmpeg. The most resource-intensive pipeline.

Operations

Operation

Description

Tools

`media.transcode`	Convert between formats (e.g., MOV → MP4)	FFmpeg
`media.compress`	Reduce file size (bitrate, resolution)	FFmpeg
`media.thumbnail`	Extract frame at timestamp as image	FFmpeg → ImageMagick
`media.waveform`	Generate audio waveform visualization	FFmpeg → custom renderer
`media.extract_audio`	Extract audio track from video	FFmpeg
`media.trim`	Trim video/audio to time range	FFmpeg

Transcode Profiles

Profile

Codec

Use Case

`video-h264-web`	H.264 + AAC, CRF 23	Web playback (universal)
`video-h265-archive`	H.265 + AAC, CRF 28	Storage-efficient archive
`video-webm`	VP9 + Opus	Modern browsers, smaller size
`audio-mp3`	MP3, 192kbps	Universal audio playback
`audio-aac`	AAC, 256kbps	High-quality audio

Processing Flow

1. Download source from S3

2. ffprobe → get duration, resolution, codecs, bitrate

3. Estimate processing time (based on duration × complexity factor)

4. If estimated time > 30 minutes: route to spot/burst worker

5. Run FFmpeg with progress reporting

6. Generate thumbnail from first frame

7. Upload transcoded file + thumbnail to S3

8. Register outputs

Progress Reporting

FFmpeg reports progress to stderr. The worker parses this and updates the job status periodically:

FFmpeg stderr: frame= 1234 fps=30 q=28.0 size=   45678kB time=00:00:41.23 ...

                ▲                                                  ▲

                │                                                  │

         parsed by worker                                   current position

                │                                                  │

                ▼                                                  ▼

    UPDATE processing_jobs SET progress_json =

    {"frame": 1234, "time_s": 41.23, "percent": 34.4}

Apps poll the job status endpoint and display a progress bar.

Output Key Structure

All derived files follow a deterministic key structure:

derived/{pool_id}/{object_id}/{profile_key}:{profile_version}/{output_file}

Examples:

derived/pool-abc/42/pdf-preview:v2/page-001.webp

derived/pool-abc/42/pdf-preview:v2/thumbnail.webp

derived/pool-abc/42/ocr-standard:v2/result.json

derived/pool-abc/42/image-thumbnail:v3/thumb.webp

derived/pool-abc/42/image-thumbnail:v3/medium.webp

derived/pool-abc/42/video-h264-web:v1/output.mp4

Benefits:

Idempotent: re-running the same job produces the same keys (overwrites harmlessly)

Versioned: changing a profile creates a new version — old outputs coexist

Queryable: all derived files for an object are under a single prefix

Cleanable: deleting an object's derived files is a prefix delete

Worker Architecture — How jobs flow through the system

Database Schemas — Table definitions

Scheduler — How workers are provisioned

Core Docs

Processing Pipelines

Processing Pipelines

Pipeline Overview

Image Pipeline

Operations

Processing Flow

Multi-Size Generation

Format Strategy

Document Pipeline

Operations

PDF Preview Flow

Text Extraction (Non-OCR)

OCR Pipeline

OCR Engines

Routing Logic

OCR Output Format

OpenXML Pipeline

Format Support

Operations

Legacy Conversion Flow

PDF Export (High Fidelity)

Security

Media Pipeline

Operations

Transcode Profiles

Processing Flow

Progress Reporting

Output Key Structure

Related