Media and File Storage

laterite-media stores files: hashed on arrival, one copy per identical bytes, each recorded as a row your tables point at. Its table migrates with the admin; the crate is a dependency only for code that calls it.

[dependencies]
laterite-media = "0.9"

Replacing a file makes a new blob; the record’s id and every reference to it stay.

Take a file in

use laterite_media::{ingest, LocalDisk};

let disk = LocalDisk::new("storage/private");
let stored = ingest(&disk, stream, 10 * 1024 * 1024).await?;

let id = laterite_media::record::create(&db, NewMedia {
    disk: "private".into(),
    hash: stored.hash,
    size: stored.size as i64,
    content_type: stored.content_type,
    original_name: filename,
    in_library: false,
    created_by: Some(user.id),
}).await?;

ingest(disk, stream, limit) takes any byte stream: a browser upload, a server-side fetch, a URL import. It hashes as the bytes pass, aborts and discards past the limit, and sniffs the content type from the bytes.

Receive an upload

async fn receive(mut upload: VerifiedUpload) -> impl IntoResponse {
    while let Some(field) = upload.next_field().await? {
        // the token field is already consumed and checked
    }
}

<form method="post" action="{{ action }}" enctype="multipart/form-data">
  <input type="hidden" name="_csrf" value="{{ shell.csrf_token() }}">
  {% let upload_id = "file" %}
  {% let upload_accept = ".csv" %}
  {% let upload_multiple = false %}
  {% let upload_url = String::new() %}
  {% let upload_fields = String::new() %}
  {% let upload_hint = shell.t("A CSV file.") %}
  {% include "_upload.html" %}
  <button class="lat-btn lat-btn--primary" type="submit" data-upload-submit>{{ shell.t("Upload") }}</button>
</form>

_csrf comes first in the form, or arrives as the X-CSRF-Token header. A route that parses the upload without VerifiedUpload has its response refused.

A route the zone posts to answers JSON:

{"stored": [{"id": 7, "name": "cover.png", "href": "/admin/media/7", "url": "/storage/media/<hash>/cover.png"}],
 "refused": [{"name": "huge.mov", "reason": "Larger than the upload limit."}]}

Declare disks

let public = LocalDisk::new("storage/media").public_at("/storage/media");
let private = LocalDisk::new("storage/private");

fn register(&self, registry: &mut Registry) {
    registry.add(DiskReg::new("private", Arc::new(private)).label("Uploads kept out of the library"));
}

Each record stores its disk. Deduplication is per disk. A module that stores files reads the disks through ctx.contributions::<DiskReg>() and picks one by name. The admin’s import keeps uploads on private: the registered disk of that name, else private under backend.storage.

An S3 disk

[dependencies]
laterite-media = { version = "0.9", features = ["s3"] }

use laterite_media::{DiskReg, S3Config, S3Disk};

let config: S3Config = settings.get("storage.disks.s3")?;
let disk = Arc::new(S3Disk::from_config(&config).await);
registry.add(DiskReg::new("s3", disk));

[storage.disks.s3]
bucket = "acme-media"
region = "auto"
endpoint = "https://<account>.r2.cloudflarestorage.com"
access_key = "..."
secret_key = "..."
prefix = "media"
public_url = "https://media.acme.test"

A write under 16 MiB is one request; past it, a multipart upload in 16 MiB parts, aborted if a part fails. signed_url presigns a GET, presigned_put a PUT for a direct upload, and copy is server-side.

Direct uploads

POST /admin/media/direct            name=clip.mp4&content_type=video/mp4&size=52428800
→ {"direct": true, "url": "<presigned PUT>", "path": "quarantine/<id>/clip.mp4", "headers": {"Content-Type": "video/mp4"}}
PUT <presigned PUT>                 the bytes, from the browser
POST /admin/media/direct/complete   path=quarantine/<id>/clip.mp4&name=clip.mp4&folder=3
→ {"stored": [{"id": 7, "name": "clip.mp4", "href": "/admin/media/7", "url": "..."}], "refused": []}

The uploader does this on its own when the field carries upload_direct. A path outside quarantine/, or an object larger than the upload limit, is refused. Abandoned quarantine objects are the bucket’s to expire: a lifecycle rule on the quarantine/ prefix of one day.

The library

The Media tab lists the records with in_library set, in virtual folders, with upload, search, a kind filter, sorting, folders to make, rename, move and remove, and a file page to retitle, rename, move and delete. Library uploads go to the disk named public (the registered one, else media under backend.storage).

use laterite_media::library::{self, Listing, Sort};
use laterite_media::{folder, link, Kind, Owner};

let photos = folder::create(&db, None, "Photos").await?;
let year = folder::create(&db, Some(photos.id), "2026").await?;   // path /Photos/2026
folder::rename(&db, year.id, "2027").await?;
folder::move_to(&db, year.id, None).await?;
folder::delete(&db, year.id).await?;                                // empty folders only
let crumbs = folder::ancestors(&db, year.id).await?;

let page = library::list(&db, &Listing {
    folder: Some(photos.id),
    search: None,
    kind: Some(Kind::Image),
    sort: Sort::Newest,
    page: 1,
    per_page: 60,
}).await?;

let owner = Owner { owner_type: "acme_posts", owner_id: "7", field: "gallery" };
link::set(&db, owner, &[first, second]).await?;                     // in that order
let held = link::of_owner(&db, owner).await?;
let users = link::users_of(&db, first).await?;
link::clear_owner(&db, "acme_posts", "7").await?;

record::update_details(&db, id, Some("Our cat"), Some("A tabby")).await?;
record::rename(&db, id, "cat.png").await?;
record::move_to_folder(&db, id, Some(photos.id)).await?;
record::set_in_library(&db, id, true, None).await?;

Listing.search looks across every folder; Kind::of(content_type) gives Image, Video, Audio, Document or Other.

Serve

Every record serves at /storage/media/<hash>/<name>:

A record on a disk registered .private() serves only at a signed URL:

registry.add(DiskReg::new("vault", Arc::new(private)).private());
let url = laterite_admin::medialib::file_url(&record);
let timed = laterite_admin::medialib::signed_file_url(&ctx, &record, Duration::from_secs(3600));
// /storage/media/<hash>/<name>?e=<expiry>&t=<signature>

The admin’s own links to a private file carry a one-hour signature.

The media field

fields:
  cover_id: { type: media, label: Cover, mode: image }
  gallery: { type: media, label: Gallery, max: 6 }

A save writes the record’s links under the field’s name (media_links: owner_type is the form’s entity, owner_id the record’s id, field the field’s name), so the file page shows who uses a file and a delete of a linked file needs media.delete_linked. The field’s picker is the library at /admin/media/pick and its tiles come from /admin/media/resolve?ids=.

Crop

POST /admin/media/<id>/crop   x=100&y=50&w=200&h=100&mode=new

The format is kept (PNG, JPEG, WebP; anything else becomes PNG). The file page offers the dialog to an operator with media.manage on an image.

let out = laterite_media::image::crop(&bytes, Rect { x, y, width, height }, orientation)?;
laterite_media::record::replace_blob(&db, id, &hash, size, &content_type, width, height).await?;

Variants

use laterite_media::{Format, Preset};

registry.add(Preset::new("card").width(600).height(400).crop());
registry.add(Preset::new("hero").width(1600).format(Format::WebP).quality(85));

<img src="/storage/media/{{ hash }}/card/{{ name }}" width="600" height="400">

A registered preset with a built-in name replaces it. An undeclared name is a 404. A file that is not an image serves as it is under any preset. The first request renders the variant behind a lock per file and preset and stores it at variants/<hash prefix>/<hash>/<preset>.<ext> beside the original; the ETag is "<hash>/<preset>".

On upload, a JPEG, PNG or WebP loses its EXIF segment before it is hashed, its orientation is read and kept in meta.orientation, and width and height are recorded. A variant bakes the orientation in. From code:

let prepared = laterite_media::image::prepare(bytes);
let rendered = laterite_media::image::render(&bytes, &preset, prepared.orientation)?;
let made = variants.ensure(&db, disk.as_ref(), &record, &preset).await?;

Collect and verify

lat media gc                 # files outside the library that nothing links to
lat media gc --grace 1       # younger than one hour is left alone (default 24)
lat media gc --dry-run       # list, remove nothing
lat media verify             # re-hash every stored file against its record

A blob goes with the last record naming it. verify exits non-zero on a fault and prints one line per faulty record: missing, hash mismatch, or a size the row disagrees with. A record on a disk the command cannot reach (one the application registered itself) is skipped and counted. From code:

let report = laterite_media::gc::collect(&db, &disks, chrono::Duration::hours(24)).await?;
let report = laterite_media::verify::run(&db, &disks).await?;

Delete

if laterite_media::record::delete(&db, id).await? {
    disk.delete(&blob_path(&hash)).await?;
}

delete returns whether the blob is now unreferenced. Deleting a blob already gone succeeds.

Write a driver

#[async_trait]
impl StorageDriver for MyDisk {
    async fn put(&self, path: &str, data: ByteStream) -> Result<(), StorageError>;
    async fn get(&self, path: &str) -> Result<ByteStream, StorageError>;
    async fn delete(&self, path: &str) -> Result<(), StorageError>;
    async fn metadata(&self, path: &str) -> Result<Option<BlobMeta>, StorageError>;
    fn public_url(&self, path: &str) -> Option<String>;
    async fn signed_url(&self, path: &str, expires_in: Duration, disposition: Disposition)
        -> Result<Option<String>, StorageError>;
}

No list(): folders are database rows. Write to a temporary path and move into place. signed_url defaults to None: the framework then streams the file itself behind its own token. A store that presigns returns the URL.