ARCHITECTURE

Architecture

Three core containers, plus an mDNS service—no separate message broker or required cloud services. Photos are uploaded or imported from NAS folders, then processed for metadata, thumbnails, semantic embeddings, faces, and text. The result is a photo library searchable by natural language, person, and location.

3 + 1containers

Go backend, AI service and PostgreSQL, plus an mDNS service that announces the LAN domain

5job types

metadata, thumbnail, clip, face, ocr

512–1024dimensions

Chinese-CLIP image-text embeddings (512, 768 or 1024 depending on the model tier); ArcFace face embeddings are 512-d

HNSWvector index

pgvector cosine-distance index; search and face matching run inside the database

SYSTEM

System Architecture

Browsers talk only to the Go backend, which handles uploads, job scheduling, thumbnails and permissions, sends preview images to the AI service for inference, and writes every result to PostgreSQL. The AI service is stateless: it never touches the database or stores files, so it can run on a more powerful machine.

The backend is a single Go binary

The web app is bundled into the binary, so deployment needs no Node. Database migrations run automatically at startup, and upgrading is just swapping the image.

Run the AI service elsewhere

The built-in AI service runs on Docker’s internal network and runs inference on the CPU. Enter a remote address in the web settings to switch to a GPU machine or a Mac at home.

One database for everything

Photo metadata, people, settings, the job queue and vectors all live in PostgreSQL, so application records and job state can be backed up together. Photo files require a separate backup.

PIPELINE

Photo Processing Pipeline

After import, each photo first gets its metadata extracted and thumbnails generated, then semantic embeddings, faces and text are computed in parallel. Each step is queued independently, so restarts or an offline AI service never lose work; processing resumes once things are back.

01ingest

Import

Web uploads use tus and resume after network drops
/photos folders are scanned every 15 minutes
SHA-1 checksum computed and written to assets

Backend

02metadata

Metadata

ExifTool reads dimensions and capture time
GPS, camera, lens, aperture and shutter
Timestamps without time zone information use the configured time zone

Backend

03thumbnail

Thumbnails

libvips generates thumbnails
Plus a 1440px preview image
FFmpeg extracts frames and transcodes video

Backend

04clip

Embeddings

Chinese-CLIP computes 1024-d vectors
Natural-language search, auto categories
Aesthetic scores, similar photos

AI · parallel

05face

Faces

SCRFD finds faces, aligns on 5 landmarks
ArcFace computes 512-d embeddings
Grouped into people incrementally

AI · parallel

06ocr

Text

PP-OCR reads the text in photos
Search photos by their text
Detects IDs, documents, screenshots

AI · parallel

SCHEDULING

Background Job Scheduling

There is no separate message broker: the job queue is a jobs table in PostgreSQL, written in the same transaction as the photo data, so a stored photo always has its jobs queued. Workers claim jobs with SKIP LOCKED, leases prevent stuck jobs, and failures retry with quadratic backoff.

jobs/worker.go · claim

-- Claim one job (server/internal/jobs/worker.go)
UPDATE jobs SET status = 'running', attempts = attempts + 1,
       locked_by = $1, locked_until = now() + $2::interval
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending' AND run_after <= now() AND type = ANY($3)
  ORDER BY priority DESC, run_after
  FOR UPDATE SKIP LOCKED   -- concurrent workers never grab the same job
  LIMIT 1
)
RETURNING id, type, asset_id, payload, attempts, max_attempts;

Scheduling rules

A job of the same type for the same photo is never queued twice while one is pending or running.
An empty queue is polled once per second; jobs are claimed by priority (highest first), then by due time (earliest first).
Each job’s lease (also its timeout) is 10 minutes; when a process restarts, it returns its own unfinished jobs to the queue.

Yielding and throttling

While someone is uploading, AI jobs stop being claimed so bandwidth and CPU go to the upload first.
When the AI service is unreachable, the whole AI worker pauses instead of claiming jobs one by one only to fail and waste retries.
The Auto tier first detects the AI server’s hardware and picks suitable models before claiming jobs, which helps avoid loading models that exceed the available memory of a Raspberry Pi or similar device.

AI SCHEDULING

AI Scheduling

The AI service only receives preview images and returns faces, vectors and text. The backend decides the service address, models and concurrency; the AI service picks CUDA, CoreML / MPS, OpenVINO or CPU based on its hardware.

Three model tiers

With Auto, the tier is chosen from the AI server’s hardware: Quality with a usable GPU; on CPU only, Lite under 6 GB of memory and Balanced at 6 GB or more.

Tier	Suitable hardware	Faces	Semantic search
quality · Quality	Apple Silicon Mac, NVIDIA GPU	SCRFD-10G + ArcFace R100 (antelopev2)	Chinese-CLIP ViT-H/14 · 1024-d
balanced · Balanced	Intel i5-class NAS	SCRFD-10G + ArcFace R100 (antelopev2)	Chinese-CLIP ViT-L/14 · 768-d
lite · Lite	N100, Raspberry Pi 5	SCRFD + ArcFace R50 (buffalo_l)	Chinese-CLIP ViT-B/16 · 512-d

AI service API · /predict

POST /predict   (multipart: image + tasks)
Authorization: Bearer <token>

tasks = {
  "face": {"model": "antelopev2", "min_score": 0.5},
  "clip": {"model": "chinese-clip-vit-h-14"},
  "ocr":  {}
}

→ {
  "face": [{"box": [x, y, w, h], "score": 0.93,
            "embedding": [512 floats], "yaw": 0.12, "blur": 340}],
  "clip": {"embedding": [1024 floats]},
  "ocr":  {"text": "...", "lines": [...]},
  "model_ids": {"face": "antelopev2", "clip": "chinese-clip-vit-h-14"}
}

ALGORITHMS

Core Algorithms

The models, algorithms and thresholds behind search, people, bursts and AI picks. Some rely on AI models; others are vector math and rules.

Natural-language search

Chinese-CLIP places photos and text in the same vector space. Image vectors are computed when a photo is imported; at search time, a phrase such as "baby’s third birthday" becomes a text vector, and the nearest photos are found by cosine distance in pgvector’s HNSW index.

sim(q, p) = q · p / (‖q‖ ‖p‖)

Three result sources are merged: photos containing the searched words come first, then photos taken at a matching place name, then the closest semantic matches.
Only clear matches are returned: a result must be nearly as close as the best match and clearly closer than an average photo, which helps filter out weak matches instead of returning loosely related photos.
When the query is exactly a category name (scenery, food, screenshots…), that category is returned directly.

Face grouping

SCRFD finds each face and its 5 landmarks, aligns and crops it to 112×112, and ArcFace computes a 512-d embedding (after normalization, the dot product is the cosine similarity). Grouping runs incrementally as each photo finishes, without waiting for the whole library.

d(a, b) = 1 − a · b good face d < 0.55 · blurry/profile d < 0.45

Sharp, frontal, large-enough "good faces" look for their closest matches: if some already belong to a person, the face joins by majority vote; if none do but at least 3 are similar, a new person is created.
Blurry, profile or tiny faces never create a new person; they only join someone they closely match, so passers-by who appear once are not added.
Concurrent grouping jobs share one database advisory lock, so a person is never created twice. Covers prefer sharp (Laplacian variance), frontal (yaw), large single faces.

Bursts and similar photos

same camera Δt ≤ 2s or Δt ≤ 60s and sim ≥ 0.93

Consecutive shots of the same scene have a CLIP similarity of 0.94–0.98, while a new scene or pose typically falls at 0.7–0.89; a similarity threshold of 0.93 helps identify likely matches.
The recommended keeper: with faces, the one where faces are sharp with open eyes looking at the camera; without faces, the larger file (with the same camera and settings, blurry shots have less detail and compress smaller).
Multiple copies of the same file go to Duplicates, with nothing to choose. Keep moves the rest to Recently Deleted, where they can be restored until permanently deleted.

Aesthetic score for AI picks

No extra model is run; the CLIP vectors are reused. Averaging the vectors of a set of descriptions of "photographic work" and a set describing "ordinary snapshots", then subtracting, gives a "looks good" direction. A photo’s projection onto that direction is its score.

score(p) = p · (μ_artwork − μ_snapshot)

Used only for ranking, never as an absolute score; the highlight carousel on the home view picks photos by it.
A separate set of descriptions identifies failed shots (blurry, overexposed and the like), scored independently of how good a photo looks.

TECH STACK

Tech Stack

All mature open-source components with no proprietary services; every piece has an alternative.

BACKEND

Go
API, job queue, thumbnail scheduling
libvips
Thumbnails and previews
FFmpeg
Video frame extraction, transcoding
ExifTool
Reads and writes photo metadata
tus
Resumable uploads

AI SERVICE

Python + FastAPI
Stateless HTTP service
ONNX Runtime
Face and text recognition inference
PyTorch
Chinese-CLIP
InsightFace
SCRFD + ArcFace
RapidOCR
PP-OCR text recognition

DATA

PostgreSQL 17
App data and job queue
pgvector
Vector storage
HNSW
Cosine-distance index
ip2region
IP location for the sign-in log (offline)
Nominatim
Reverse geocoding (can be disabled)

FRONTEND & DEPLOYMENT

React + TypeScript
Web app
Vite + Tailwind CSS
Build and styling
MapLibre GL
Satellite globe
Docker Compose
amd64 / arm64 images
mDNS
wanmory.local LAN domain

PRIVACY

Privacy & Security

By default, photos and AI results stay within your private network. If you configure a remote AI service, preview images are sent to that server for processing.

Data minimization

The AI service only receives 1440-pixel previews, stores no files, and never touches the database, names or originals.

Authentication

The built-in AI service lives only on Docker’s internal network, reachable by the backend alone; once a token is set for a remote AI service, every request must carry a Bearer token.

Permissions

Members can delete only the photos they uploaded, while admins manage everything; hidden photos are visible only to their owner; the sign-in log shows an approximate location by IP.

Read-only originals

NAS folders can be mounted read-only with :ro, so the app cannot modify or delete the files in them.

Works offline

Once models are downloaded, recognition and search run locally; features that need the internet, such as reverse geocoding and satellite imagery, can be turned off.

Model licenses

InsightFace face models are for non-commercial use only; models are not bundled in the image, and each model’s license is shown in Settings before download.