Core Docs

Storage Architecture

By Olivier Mironescu Added: 2026-06-17 Updated: 2026-07-03

Storage Architecture



How core.storage manages files, S3 integration, pools, sharding, ownership, and quotas.

> Status: Architecture defined. Implementation pending.

Overview



core.storage provides a unified API for file operations across all application projects. It handles upload, download, deletion, metadata tracking, access control, and comments. Files are stored in S3-compatible object storage, with metadata tracked in core.storage's SQL.

core.storage owns the storage infrastructure — providers, pools (buckets), and objects. It exposes a provider catalog to core.console so that dev users can provision pools through the console UI, but core.console never touches S3 credentials directly.

Architecture Diagram



┌──────────────────────────────────────────────────────────────┐

│                        core.console                          │

│                                                              │

│  Dev user creates project                                    │

│  Dev user provisions pool (picks provider + region)          │

│  core.console → GET /api/storage/providers (catalog)         │

│  core.console → POST /api/storage/pools (create bucket)      │

└──────────────────────────┬───────────────────────────────────┘

                           │ internal API (no credentials exposed)

                           ▼

┌──────────────────────────────────────────────────────────────┐

│                        core.storage                          │

│                                                              │

│  ┌─────────────────────┐  ┌──────────────────────────────┐  │

│  │  Provider Config     │  │  SQL Database                │  │

│  │  (env / JSON file)   │  │  pools, objects, shards,     │  │

│  │  credentials, limits │  │  ownership, comments         │  │

│  └─────────────────────┘  └──────────────────────────────┘  │

│                                                              │

│  ┌──────────────────────────────────────────────────────┐   │

│  │  S3 Client Layer                                      │   │

│  │  Translates DB operations → S3 API calls              │   │

│  │  Handles sharding, multipart, provider differences    │   │

│  └──────────────────────────────────────────────────────┘   │

│                           │                                  │

└───────────────────────────┼──────────────────────────────────┘

                            │ S3 API

                            ▼

              ┌──────────────────────────┐

              │  S3-Compatible Storage    │

              │  AWS, Backblaze, MinIO... │

              └──────────────────────────┘


Pages




Design Principles



  • Soft delete everywhere. Every table with mutable state has deleted_at (nullable). No data is ever hard-deleted by user action. A background cleanup job handles actual S3 object removal.

  • Soft revoke for access. object_ownership uses expired_at instead of hard delete. Access history is preserved for auditing.

  • Providers are config, not data. S3 credentials live in env vars or a secure config file on the core.storage filesystem. Never in the database. Never exposed via API.

  • Pools are logical, buckets are physical. Users create "pools" (display name, provider, region). The actual S3 bucket name is auto-generated and never shown to users.

  • Sharding is transparent. Large files are automatically split into shards to respect per-provider upload limits. Clients don't need to know about sharding — core.storage handles it.

  • core.console is the UI, core.storage is the brain. Console presents options and provisions pools, but all storage logic, credentials, and S3 operations live in core.storage.

Related