Core Docs

Upload & Download

By Olivier Mironescu Added: 2026-07-03

Upload & Download



How files are uploaded (with sharding), downloaded (with reassembly), and how the flow integrates with core.console and core.auth.

Upload Flow



Simple Upload (Single Shard)



For files smaller than the provider's max_single_put_bytes limit:

Client (core.backend SDK)

    │

    │  POST /v1/storage/upload

    │  Headers: Authorization: Bearer <api_key>

    │  Body: multipart/form-data { file, pool_id, owner_type, owner_id }

    │

    ▼

core.console (API Gateway)

    │

    ├── 1. Validate API key

    ├── 2. Check: is "storage" in this key's allowed_services?

    ├── 3. Check: has the project exceeded its storage quota?

    ├── 4. Route to core.storage

    │

    ▼

core.storage

    │

    ├── 1. Validate session via core.auth (if user-owned)

    ├── 2. Verify pool exists and is active

    ├── 3. Determine file size

    │

    ├── File size ≤ max_single_put_bytes (single shard):

    │   │

    │   ├── Create objects row { pool_id, original_filename, mime_type, shard_count: 1, upload_status: 'uploading' }

    │   ├── Create object_shards row { shard_index: 0, s3_key: "shards/{object_id}/0-{uuid}", upload_status: 'uploading' }

    │   ├── Upload file to S3: PutObject(internal_bucket_name, s3_key, file_data)

    │   ├── On success: update shard status → 'complete', object status → 'complete'

    │   ├── Compute checksums

    │   ├── Update objects: total_size_bytes, checksum

    │   ├── Create object_ownership { owner_type, owner_id, access_level: 'owner', is_creator: true }

    │   └── Return { object_id, status: 'complete' }

    │

    └── On failure: update shard status → 'failed', object status → 'failed'

        └── Return error


Sharded Upload (Multiple Shards)



For files larger than max_single_put_bytes:

core.storage

    │

    ├── File size > max_single_put_bytes (sharded):

    │

    │   ── Phase 1: Initiate ──

    │   ├── Create objects row { upload_status: 'uploading' }

    │   ├── Calculate number of shards: ceil(file_size / max_single_put_bytes)

    │   ├── Create object_shards rows for each shard { status: 'pending' }

    │   ├── Return { object_id, shard_count, upload_urls: [presigned_url_0, presigned_url_1, ...] }

    │

    │   ── Phase 2: Client uploads each shard ──

    │   ├── Client PUTs shard 0 to presigned_url_0

    │   │   ├── On success: update shard[0] status → 'complete'

    │   │   └── On failure: shard[0] status → 'failed' (client can retry)

    │   ├── Client PUTs shard 1 to presigned_url_1

    │   │   └── ...same pattern...

    │   └── Client PUTs shard N to presigned_url_N

    │

    │   ── Phase 3: Finalize ──

    │   ├── Client calls POST /v1/storage/objects/{object_id}/finalize

    │   ├── core.storage verifies all shards are 'complete'

    │   ├── Computes whole-object checksum (hash of concatenated shard checksums)

    │   ├── Updates objects: status → 'complete', shard_count, total_size_bytes, checksum

    │   ├── Creates object_ownership

    │   └── Returns { object_id, status: 'complete' }

    │

    └── If any shard is still 'failed' after finalize attempt:

        └── Return error with list of failed shards. Client can retry those shards.


Presigned URLs



For sharded uploads, core.storage generates presigned PUT URLs for each shard. This lets the client upload directly to S3 without routing through core.storage — better performance, less server load.

  • Presigned URLs expire after a configurable timeout (default: 1 hour).

  • Each URL is scoped to a specific S3 key and HTTP method (PUT only).

  • The client uploads shards in parallel if desired.

Multipart Upload (S3 Native)



For very large files, core.storage can use S3's native multipart upload API instead of presigned URLs. Each shard itself could be a multipart upload if even a single shard exceeds S3's limits.

The choice between presigned URLs and S3 multipart is an implementation detail — the data model (objects + object_shards) is the same either way.

Quota Enforcement



Storage quotas are enforced at upload time:

core.storage → core.auth: "Does user X have Y bytes of storage quota remaining?"

    │

    ├── Yes → proceed with upload

    └── No → return 429, quota exceeded


Quota is calculated as total_size_bytes across all non-deleted objects owned by the user. Since objects use soft delete, deleted-but-uncleaned objects still count toward quota until the cleanup job runs and hard-deletes the S3 data.

Core.backend SDK Example



use Unvrsl\Console\Client;



$client = new Client([

    'api_key' => 'mk_live_abc123...',

    'base_url' => 'https://core.console.medkronos.com/v1'

]);



// Simple upload (single shard, handled transparently)

$object = $client->storage()->upload('pool_123', 'report.pdf', $fileContents);



// Sharded upload (also handled transparently by the SDK)

$object = $client->storage()->upload('pool_123', 'large-video.mp4', $fileHandle);



// The SDK reads the object size, checks if sharding is needed,

// and handles the multi-step flow automatically.


Download Flow



Client (core.backend SDK)

    │

    │  GET /v1/storage/objects/{object_id}

    │  Header: Authorization: Bearer <api_key>

    │

    ▼

core.console → routes to core.storage

    │

    ▼

core.storage

    │

    ├── 1. Validate ownership via object_ownership (user must have at least 'viewer' access)

    ├── 2. Load object metadata (WHERE deleted_at IS NULL)

    ├── 3. Load all object_shards (WHERE deleted_at IS NULL, ORDER BY shard_index)

    │

    ├── Single shard:

    │   └── Fetch from S3: GetObject(internal_bucket_name, shard.s3_key)

    │       └── Stream to client

    │

    └── Multiple shards:

        ├── Fetch shard 0 from S3 → stream to client

        ├── Fetch shard 1 from S3 → stream to client

        ├── ...

        └── Fetch shard N from S3 → stream to client

        (sequential streaming, in order of shard_index)


Presigned Download URLs



For large files, core.storage can return presigned GET URLs for each shard, ordered by shard_index. The client fetches each shard directly from S3.

GET /v1/storage/objects/{object_id}/download-urls



Response:

{

  "object_id": 42,

  "original_filename": "large-video.mp4",

  "total_size_bytes": 10737418240,

  "shards": [

    { "shard_index": 0, "url": "https://s3.eu-west-3.amazonaws.com/...?X-Amz-Signature=...", "size_bytes": 5368709120 },

    { "shard_index": 1, "url": "https://s3.eu-west-3.amazonaws.com/...?X-Amz-Signature=...", "size_bytes": 5368709120 }

  ]

}


The client downloads shards in order and concatenates them to reconstruct the original file.

Integrity Verification



On download, core.storage can optionally verify each shard's checksum before streaming. If a shard's checksum doesn't match, the download fails with an integrity error. This catches S3 data corruption.

Metadata Queries



List Objects in a Pool



GET /v1/storage/pools/{pool_id}/objects?page=1&limit=50



Response:

{

  "objects": [

    {

      "object_id": 42,

      "original_filename": "report.pdf",

      "mime_type": "application/pdf",

      "total_size_bytes": 2048000,

      "shard_count": 1,

      "upload_status": "complete",

      "created_at": "2026-07-03T14:00:00Z"

    }

  ],

  "total": 127,

  "page": 1,

  "limit": 50

}


Only returns objects where deleted_at IS NULL.

Get Object Details



GET /v1/storage/objects/{object_id}



Response:

{

  "object_id": 42,

  "pool_id": 7,

  "original_filename": "report.pdf",

  "mime_type": "application/pdf",

  "shard_count": 1,

  "total_size_bytes": 2048000,

  "checksum": "sha256:abc123...",

  "upload_status": "complete",

  "ownership": [

    { "owner_type": "user", "owner_id": "usr_123", "access_level": "owner", "is_creator": true }

  ],

  "created_at": "2026-07-03T14:00:00Z",

  "updated_at": "2026-07-03T14:00:00Z"

}


Related