Worker Architecture

By Olivier Mironescu Added: 2026-07-17

Worker Architecture

How processing jobs flow from creation to completion through the queue and worker system.

Overview

core.hypermedia uses a producer-consumer pattern. The web tier (PHP-FPM) produces jobs; workers consume them. The queue is the only coupling between the two — the web tier never calls a worker directly, and workers never call the web tier.

This means:

Web responses are instant (enqueue and return)

Workers can be added, removed, or replaced without touching the web tier

A worker crash never loses a job (the queue redelivers it)

Throughput scales linearly with worker count

Queue Design

Technology: Redis

Redis serves as the durable queue. Chosen over RabbitMQ for simplicity and because it's likely already running on the VPS for caching. Uses Redis Streams (or a sorted-set-based priority queue) for job delivery with at-least-once semantics.

Queue Separation

Separate queues for different workload types. This prevents a long-running video transcode from blocking a quick image thumbnail.

Queue	Job Types	Typical Duration	Worker Profile

`images`	Compress, resize, format convert, thumbnail	0.1–5s	High concurrency (4–8 jobs)
`documents`	PDF preview, DOCX conversion, text extraction	1–30s	Medium concurrency (2–4 jobs)
`ocr`	OCR, table detection, AI OCR escalation	5–120s	Low concurrency (1–2 jobs)
`media`	Video/audio transcode, FFmpeg	30s–30min	Very low concurrency (1 job)
`bulk`	Batch regeneration, backfill, migration	Variable	Spot/preemptible only

Priority

Within each queue, jobs are ordered by priority (1–10) then by creation time. Interactive user-facing jobs get priority 1–3. Background regeneration gets 7–10.

Queue: images

  ┌─────────────────────────────────────────┐

  │ priority=1  job_abc  (user thumbnail)   │ ← next

  │ priority=1  job_def  (user preview)     │

  │ priority=5  job_ghi  (app-requested)    │

  │ priority=8  job_jkl  (bulk regenerate)  │ ← last

  └─────────────────────────────────────────┘

Job Lifecycle

1. Job Creation (Web Tier)

App → POST /v1/hypermedia/process

        { object_id, operations: ["pdf_preview", "ocr"] }



core.hypermedia web tier:

  1. Validate object exists in core.storage

  2. Check user access (via object_ownership)

  3. Look up processing_profile for each operation

  4. Estimate credits: profile.base_cost + (profile.per_mb × file_size_mb)

  5. Check quota via core.auth: "Does user have enough hypermedia credits?"

  6. Create processing_jobs rows (status: 'created')

  7. For each job:

     - Set status to 'queued'

     - Push to Redis queue: RPUSH queue:{queue_name} {job_id}

  8. Return { job_ids, estimated_credits }

Cost estimation happens before enqueue. If the user doesn't have enough credits, the request is rejected with 402 Payment Required. No S3 or compute resources are consumed.

2. Job Claiming (Worker)

Worker loop:

  1. BLPOP queue:{assigned_queue} timeout=5s

  2. If job_id received:

     a. UPDATE processing_jobs SET

          status = 'claimed',

          claimed_by = worker_id,

          claimed_at = NOW(),

          visibility_timeout = NOW() + INTERVAL timeout_seconds SECOND,

          attempt_count = attempt_count + 1

        WHERE job_id = ? AND status = 'queued'

     b. If update affected 0 rows: job was already claimed, skip

     c. INSERT INTO job_attempts (job_id, worker_id, attempt_number, status)

     d. Set status = 'processing'

     e. Execute the job

The UPDATE ... WHERE status = 'queued' pattern is the atomic claim. Two workers can't claim the same job — only one UPDATE will affect rows.

3. Job Execution (Worker)

Worker executes job:

  1. Get presigned GET URL from core.storage for the source file

  2. Download source to local temp directory: /work/jobs/{job_id}/input/

  3. Run the processing pipeline (see Processing Pipelines page)

  4. Upload outputs to S3 via presigned PUT URLs

  5. Register outputs in core.storage (create objects rows)

  6. INSERT INTO job_outputs for each derived file

  7. UPDATE processing_jobs SET status = 'completed', completed_at = NOW(),

     actual_credits = calculated_cost

  8. UPDATE job_attempts SET status = 'completed', finished_at = NOW()

  9. Charge credits via core.auth

 10. Clean up temp directory

4. Failure and Retry

If worker fails (crash, timeout, error):

  1. UPDATE job_attempts SET status = 'failed', error_message = ?, error_code = ?

  2. If attempt_count < max_attempts:

     - UPDATE processing_jobs SET status = 'retry_wait'

     - After backoff: SET status = 'queued', push back to queue

     - Backoff: 2^attempt_count minutes (1min, 2min, 4min, 8min...)

  3. If attempt_count >= max_attempts:

     - UPDATE processing_jobs SET status = 'failed'

     - Notify user (optional webhook or polling status)

Visibility timeout safety net: if a worker crashes without updating the job status (hard kill, power loss), the visibility_timeout eventually expires. A reaper process scans for jobs where status = 'processing' AND visibility_timeout < NOW() and returns them to queued.

5. Cancellation

Users can cancel queued jobs (not processing ones). Cancellation sets status = 'cancelled' and removes the job from the queue. Already-processing jobs can be cancelled by setting a cancel_requested flag that the worker checks periodically.

Worker Types

VPS Worker (Always-On)

The baseline worker runs on the existing VPS alongside Plesk. It handles the typical daily load.

# docker-compose.yml (on VPS)

worker-images:

  image: registry.medkronos.com/hypermedia-worker:latest

  command: php worker.php --queues=images,documents --concurrency=4

  restart: always

  mem_limit: 2g

  cpus: 2

  volumes:

    - /tmp/hypermedia:/work

  environment:

    - REDIS_URL=redis://localhost:6379

    - STORAGE_API_URL=https://core.storage.medkronos.com

    - WORKER_ID=vps-images-01

Concurrency: 2–4 jobs (limited by VPS CPU/RAM)

Queues: images, documents (fast jobs)

Cost: €0 (already paying for the VPS)

Always running: Yes

Burst Cloud Workers (On-Demand)

Provisioned by the scheduler when queue depth exceeds thresholds. Terminated after an idle period.

Concurrency: 1–2 jobs (sized for specific workloads)

Queues: media, ocr, bulk (heavy jobs)

Cost: Per-second billing (AWS EC2, Google Compute, IONOS)

Always running: No — provisioned and terminated by the scheduler

See Scheduler for provisioning logic.

Worker Image

All workers run the same Docker image: registry.medkronos.com/hypermedia-worker:latest. The image contains:

Component

Purpose

PHP 8.4 CLI	Worker daemon, job management
FFmpeg 7.x	Video/audio transcoding
ImageMagick / libvips	Image processing
Ghostscript	PDF rendering
LibreOffice (headless)	Legacy document conversion, PDF export
PaddleOCR / Surya	Classical OCR with layout analysis
Tesseract	Fallback OCR
Python 3.12	OCR microservice, scripting

Build once, deploy everywhere. The same image runs on the VPS, AWS EC2, Google Compute, or IONOS. Only environment variables differ (queue assignment, concurrency, storage credentials).

Workers do NOT have permanent S3 credentials. They request short-lived presigned URLs from core.storage for each job's input and output files.

Security & Sandboxing

Workers process untrusted user files. This is the most attack-exposed component in the UNVRSL ecosystem.

Isolation

Workers run as non-root user inside Docker containers

Each job gets a fresh temporary directory: /work/jobs/{job_id}/ — deleted after completion (success or failure)

No network access from within the processing sandbox (except to S3 and core.storage APIs)

No filesystem access outside the job's temp directory

Resource Limits (Per Job)

Resource

Limit

Enforcement

CPU time	`timeout_seconds` from profile	`setrlimit(RLIMIT_CPU)`
Memory	2 GB default, 8 GB for video	`setrlimit(RLIMIT_AS)` + Docker `mem_limit`
Disk	10 GB temp space	`setrlimit(RLIMIT_FSIZE)` + quota on `/work`
Processes	16 max child processes	`setrlimit(RLIMIT_NPROC)`
Wall clock	`timeout_seconds` × 1.5	Worker-side timer

Input Validation

MIME type verification by content inspection (magic bytes), not just file extension

File signature validation — PDFs start with %PDF, PNGs with \x89PNG, etc.

Macro rejection — .docm, .xlsm, .pptm files are rejected at job creation

Size limits — enforced per profile (max_input_bytes)

Virus scanning — originals can be scanned before processing (optional, adds latency)

Shell Safety

Never interpolate user-controlled filenames into shell commands

Use proc_open() with explicit argument arrays, not shell_exec() with string interpolation

Filenames are treated as opaque byte strings — no path traversal possible

Idempotency

Jobs may be executed more than once (at-least-once queue delivery, worker retries). The system must be idempotent:

Deterministic output keys. Output S3 keys are derived from the job: derived/{pool_id}/{object_id}/{profile_version}/{output_type}. Processing the same job twice produces the same keys — the second write overwrites the first harmlessly.

Database uniqueness. job_outputs.uk_job_type_profile prevents duplicate output registrations. The second attempt's INSERT fails silently.

Atomic completion. The job is only marked completed after all outputs are uploaded and registered. A crash mid-upload means the job retries and produces the same outputs again.

Monitoring

Each worker reports:

Heartbeat every 30 seconds → workers.last_heartbeat

Metrics every 60 seconds → worker_metrics (CPU, memory, disk, job throughput)

Job completion → job_attempts.duration_ms, peak_memory_mb

The scheduler uses this data for scaling decisions. See Scheduler.

Database Schemas — Table definitions

Processing Pipelines — What each job type does

Scheduler — Dynamic cloud instance management

Core Docs

Worker Architecture

Worker Architecture

Overview

Queue Design

Technology: Redis

Queue Separation

Priority

Job Lifecycle

1. Job Creation (Web Tier)

2. Job Claiming (Worker)

3. Job Execution (Worker)

4. Failure and Retry

5. Cancellation

Worker Types

VPS Worker (Always-On)

Burst Cloud Workers (On-Demand)

Worker Image

Security & Sandboxing

Isolation

Resource Limits (Per Job)

Input Validation

Shell Safety

Idempotency

Monitoring

Related