Scheduler

By Olivier Mironescu Added: 2026-07-17

Scheduler

Dynamic worker pool management. Monitors queue depth and job age, provisions cloud instances when demand exceeds VPS capacity, and terminates them when demand subsides.

Overview

The scheduler is a background process (runs every 60 seconds) that answers one question: do we need more workers?

Every 60 seconds:

  1. Read queue depths and oldest-job ages

  2. Read active worker health and utilization

  3. Evaluate scaling rules from config

  4. If scale-up needed: provision cloud instance(s)

  5. If scale-down needed: drain and terminate instance(s)

  6. Log all decisions with reasoning

Default behavior: VPS-only mode. No cloud instances are created unless explicitly enabled in the config. This ensures zero unexpected cloud costs.

Configuration

The scheduler is configured via a secure settings file on the core.hypermedia filesystem. Same pattern as storage providers — config lives on the filesystem, not in the database, and is never exposed via API.

Settings File

Path: /etc/hypermedia/scheduler.json (or equivalent, loaded at startup)

{

  "mode": "vps_only",

  "evaluation_interval_seconds": 60,

  "queues": {

    "images": {

      "baseline_workers": 1,

      "max_workers": 1,

      "cloud_enabled": false

    },

    "documents": {

      "baseline_workers": 1,

      "max_workers": 1,

      "cloud_enabled": false

    },

    "ocr": {

      "baseline_workers": 0,

      "max_workers": 3,

      "cloud_enabled": false,

      "cloud_provider": null

    },

    "media": {

      "baseline_workers": 0,

      "max_workers": 3,

      "cloud_enabled": false,

      "cloud_provider": null

    },

    "bulk": {

      "baseline_workers": 0,

      "max_workers": 5,

      "cloud_enabled": false,

      "cloud_provider": null,

      "spot_preferred": true

    }

  },

  "cloud_providers": {

    "aws": {

      "enabled": false,

      "region": "eu-west-3",

      "instance_type": "c6i.xlarge",

      "ami_id": "ami-...",

      "max_instances": 5,

      "spot_enabled": true,

      "spot_max_price": "0.05"

    },

    "google": {

      "enabled": false,

      "zone": "europe-west1-b",

      "machine_type": "c2-standard-4",

      "image": "projects/.../global/images/hypermedia-worker-latest",

      "max_instances": 5,

      "preemptible": true

    },

    "ionos": {

      "enabled": false,

      "datacenter": "de/fra",

      "server_type": "c6i.xlarge",

      "image": "hypermedia-worker-latest",

      "max_instances": 3

    }

  },

  "scaling_rules": {

    "scale_up": {

      "queue_depth_threshold": 10,

      "oldest_job_age_seconds": 300,

      "cooldown_seconds": 120

    },

    "scale_down": {

      "idle_minutes": 10,

      "queue_depth_below": 2,

      "cooldown_seconds": 300

    }

  },

  "cost_limits": {

    "max_hourly_spend_cents": 500,

    "max_daily_spend_cents": 5000,

    "alert_threshold_cents": 3000

  }

}

Modes

Mode	Behavior

`vps_only`	Only the VPS worker processes jobs. No cloud instances. Zero cloud costs.
`hybrid`	VPS handles baseline. Cloud instances provisioned for burst load.
`cloud_first`	Minimal VPS worker. Most processing on cloud instances.

Default: vps_only. Olivier must explicitly enable cloud bursting by changing the config file. There is no API endpoint to change the scheduler mode — it requires filesystem access to the server.

Scaling Logic

Scale-Up Decision

For each queue:

  depth = LLEN queue:{queue_name}

  oldest_age = now - oldest_job.created_at



  If depth >= scale_up.queue_depth_threshold

     OR oldest_age >= scale_up.oldest_job_age_seconds:



    If queue.cloud_enabled AND active_workers < queue.max_workers:

      If cooldown elapsed since last scale-up:

        Provision new cloud worker

        Log: "Scaling up {queue}: depth={depth}, oldest={oldest_age}s"

Scale-Down Decision

For each cloud worker:

  If worker.current_jobs == 0

     AND worker.last_job_completed_at < (now - idle_minutes):



    If queue_depth for worker's queues < scale_down.queue_depth_below:

      Drain worker (stop accepting new jobs)

      Wait for current jobs to finish (or timeout after 5 min)

      Terminate instance

      Log: "Scaling down worker {worker_id}: idle for {idle_time} minutes"

Scaling Policy Example

With hybrid mode and default thresholds:

Queue Depth

Oldest Job Age

Action

0–5	< 2 min	Nothing — VPS handles it
6–20	2–5 min	Provision 1 burst worker
21–100	5–10 min	Provision up to 3 burst workers
> 100	> 10 min	Provision up to max, alert operator
0 for 10 min	—	Terminate all burst workers

Worker Provisioning

Provisioning Flow

Scheduler decides to scale up

    │

    ▼

1. Select cloud provider from config

   (priority: cheapest available spot price, or configured preference)

    │

    ▼

2. Launch instance from pre-built image

   AWS: RunInstances with user-data script

   Google: instances.insert with startup-script

   IONOS: POST /servers with cloud-init

    │

    ▼

3. Instance boots, runs worker startup script:

   - Pulls latest Docker image

   - Starts worker container with queue assignment

   - Registers in workers table (INSERT INTO workers)

   - Begins heartbeat loop

    │

    ▼

4. Worker starts claiming jobs from queue

    │

    ▼

5. Scheduler detects new worker in workers table

   Updates internal state

Termination Flow

Scheduler decides to scale down

    │

    ▼

1. Set worker status to 'draining'

   Worker stops claiming new jobs

    │

    ▼

2. Wait for current jobs to complete

   (timeout: 5 minutes — then force-kill remaining)

    │

    ▼

3. Terminate cloud instance

   AWS: TerminateInstances

   Google: instances.delete

   IONOS: DELETE /servers/{id}

    │

    ▼

4. Update workers table: status = 'terminated', terminated_at = NOW()

    │

    ▼

5. If any jobs were force-killed:

   Return them to queue (visibility timeout expired anyway)

Spot / Preemptible Instances

For non-urgent workloads (bulk queue, background regeneration), spot instances offer 60–90% cost savings.

Spot Safety

Jobs are retryable. If a spot instance is terminated mid-job, the visibility timeout expires and another worker retries.

Checkpointing for long jobs. Video transcodes longer than 30 minutes should checkpoint progress (write intermediate output to S3, resume from checkpoint on retry).

Diversification. Request multiple instance types to reduce interruption risk.

Routing Rules

Job Type

Priority

Target

Interactive preview (< 5 min)	1–3	On-demand / VPS
Standard user conversion	4–6	On-demand or mixed
Bulk backfill / regeneration	7–10	Spot-preferred
Long transcode (> 30 min)	Any	Spot with checkpointing

Cost Control

Limits

The scheduler enforces hard spending limits:

max_hourly_spend_cents — if hourly cloud spend exceeds this, no new instances are provisioned regardless of queue depth

max_daily_spend_cents — if daily spend exceeds this, all cloud workers are terminated and mode reverts to vps_only until manual reset

alert_threshold_cents — notification sent when daily spend approaches this level

Cost Tracking

Each cloud worker's cost_per_hour_cents is recorded in the workers table. The scheduler sums active cloud workers' costs and compares against limits. Actual billing reconciliation happens via provider APIs (not real-time — checked every 15 minutes).

Cost Attribution

Processing costs are attributed to the requesting user via hypermedia credits (charged through core.auth). The cloud compute cost is a platform cost, not a user cost. Users pay credits; the platform pays the cloud provider. Credit pricing should be set to cover expected compute costs with margin.

Monitoring & Alerts

Metrics to Watch

Metric

Source

Alert Threshold

Queue depth (per queue)	Redis LLEN	> 50 sustained for 10 min
Oldest job age	processing_jobs.created_at	> 15 min for priority 1–3
Worker CPU utilization	worker_metrics.cpu_percent	> 90% sustained for 10 min
Job failure rate	job_attempts.status	> 10% in last 100 jobs
Cloud spend rate	workers.cost_per_hour_cents	> 80% of daily limit
Worker heartbeat staleness	workers.last_heartbeat	> 2 minutes

Alert Channels

Alerts are sent via core.console's notification system (or webhook to a monitoring endpoint). Alert levels:

Info: scaling event (worker provisioned/terminated)

Warning: queue depth sustained above threshold, worker unhealthy

Critical: daily cost limit reached, all workers unhealthy, job failure rate spike

Deployment Stages

Stage

Compute Model

When to Use

1. VPS only	One worker container on the existing VPS	Early production, low volume
2. Hybrid	VPS baseline + manual cloud burst	Growing volume, occasional spikes
3. Auto-scaled	VPS baseline + automatic cloud burst	Production with variable load
4. Managed batch	AWS Batch / Google Cloud Batch	High volume, cloud-native

Start at Stage 1. Move to Stage 2 when queues regularly back up. Move to Stage 3 when manual scaling becomes burdensome. Stage 4 is a future consideration, not a current need.

Worker Architecture — Queue design, job lifecycle, security

Database Schemas — workers, worker_metrics tables

Processing Pipelines — What workers actually do

Core Docs

Scheduler

Scheduler

Overview

Configuration

Settings File

Modes

Scaling Logic

Scale-Up Decision

Scale-Down Decision

Scaling Policy Example

Worker Provisioning

Provisioning Flow

Termination Flow

Spot / Preemptible Instances

Spot Safety

Routing Rules

Cost Control

Limits

Cost Tracking

Cost Attribution

Monitoring & Alerts

Metrics to Watch

Alert Channels

Deployment Stages

Related