Evaluation

Performance on a held-out test set of 139 images that never appear in the reference index. Scoring parameters were fit by leave-one-out on the reference split only, so nothing here was tuned on the test set. This page is the working draft of the Task 5 Internal Testing and Validation Report.

Top-1 accuracy

85.6%

95% CI 79%–90%, n=139

Top-3 accuracy

95.0%

True category among the first three

Source country correct

90.6%

127 designated-list images

Missed detections

0.0%

Designated objects ranked as control

False alarms

16.7%

Control objects flagged, n=12

Calibration error

0.035

Expected calibration error (lower is better)

Read these numbers as a baseline, not a field claim

The test images come from the same museum and photographic sources as the reference set: clean studio photographs of catalogued objects. Field photographs of seized or in-transit objects will score lower. The Task 6 simulated field test with CPEOC experts is where concordance on realistic inputs gets measured. With only 12 control images, the false-alarm rate has a wide interval.

Per-category results

Precision: when the tool says this category, how often it is right. Recall: how often it finds this category.

Category	n	Precision	Recall
Scythian gold and bronze ornamentUkraine	8	89%	100%
Cucuteni–Trypillia painted ceramicsUkraine	13	100%	92%
Eastern-rite icons and ecclesiastical objectsUkraine	6	100%	100%
Faience shabtis and amuletsEgypt	11	73%	100%
Coffins, cartonnage and funerary masksEgypt	12	80%	67%
Stone stelae and relief fragmentsEgypt	11	100%	82%
Moche modelled and stirrup-spout ceramicsPeru	9	58%	78%
Nasca polychrome ceramicsPeru	10	88%	70%
Andean textilesPeru	10	100%	90%
Black- and red-figure painted potteryGreece	9	80%	89%
Shang and Zhou bronzes (vessels and weapons)China	13	93%	100%
Han–Tang tomb ceramics and figuresChina	5	57%	80%
Khmer stone and bronze sculptureCambodia	10	88%	70%
Out-of-scope controlControl	12	100%	83%

Confusion matrix

Rows: true category. Columns: top prediction. Hover a cell for its count.

	UA-SCY	UA-TRY	UA-ICO	EG-SHA	EG-FUN	EG-REL	PE-MOC	PE-NAS	PE-TEX	GR-PAI	CN-RIT	CN-TAN	KH-KHM	CTL
UA-SCY	8
UA-TRY		12					1
UA-ICO			6
EG-SHA				11
EG-FUN				3	8		1
EG-REL	1				1	9
PE-MOC				1			7	1
PE-NAS							1	7		1			1
PE-TEX									9			1
GR-PAI							1			8
CN-RIT											13
CN-TAN										1		4
KH-KHM							1				1	1	7
CTL					1							1		10

UA-SCY: Scythian gold and bronze ornament
UA-TRY: Cucuteni–Trypillia painted ceramics
UA-ICO: Eastern-rite icons and ecclesiastical objects
EG-SHA: Faience shabtis and amulets
EG-FUN: Coffins, cartonnage and funerary masks
EG-REL: Stone stelae and relief fragments
PE-MOC: Moche modelled and stirrup-spout ceramics
PE-NAS: Nasca polychrome ceramics
PE-TEX: Andean textiles
GR-PAI: Black- and red-figure painted pottery
CN-RIT: Shang and Zhou bronzes (vessels and weapons)
CN-TAN: Han–Tang tomb ceramics and figures
KH-KHM: Khmer stone and bronze sculpture
CTL: Out-of-scope control

Calibration

Does a stated confidence of 70% mean the tool is right about 70% of the time? These are grouped held-out predictions.

Stated confidence	Predictions	Mean confidence	Observed accuracy
0%–20%	0	—	—
20%–40%	8	36%	25%
40%–60%	14	51%	57%
60%–80%	21	70%	71%
80%–100%	96	95%	98%

Ablation: what each signal contributes

Fused score = 0.6 × exemplar similarity + 0.4 × description similarity, softmax temperature 0.01, k = 1.

Configuration	Top-1	Top-3	False alarms	ECE
Fused (shipped)	85.6%	95.0%	16.7%	0.035
Exemplars only	86.3%	94.2%	16.7%	0.043
Descriptions only (zero-shot)	53.2%	80.6%	8.3%	0.083

Descriptions alone are weak. Their value is in explaining a result in words and in lowering calibration error, not in raw accuracy.

Model selection

Candidate on-device encoders, evaluated on the same corpus, split and procedure. Only permissively licensed weights were eligible.

Model	Licence	Size	Top-1	Top-3	False alarms	ECE	CPU p50
DINOv2 ViT-S/14 (Meta), 8-bit quantised	Apache-2.0	24.5 MB	89.3%	95.7%	25.0%	0.114	353 ms
CLIP ViT-B/32 (OpenAI), 8-bit quantised Shipped	MIT	89.1 MB	85.6%	95.0%	16.7%	0.035	168 ms

DINOv2 is smaller and slightly more accurate on top-1. CLIP was chosen for three reasons: its calibration error is about three times lower, it raised fewer false alarms, and it scores images against text descriptions, which supplies the plain-language rationale for each result. The top-1 gap (about five test images) is within the confidence interval. Apple MobileCLIP was excluded before testing because of its research-only licence.

Benchmark generated 2026-09-25 · corpus f2de7bac7cf69146 · model sha256 583fd1110a514667… · build-time latency p50 168 ms / p95 273 ms (Node v24.11.0, win32, CPU (onnxruntime)). Reproduce with npm run data:index.