Run seed-reference-bakeoffdone
1 Sept 2026, 12:00 pm · 10 frames × 8 model arms = 80 cells. Every frame was tiled once and the identical tile set was sent to every arm.
Configuration
Chosen once, applied to every arm. This is the fairness contract the whole comparison rests on.
- Tile size
- 512 px
- Overlap
- 20%
- Merge (IoS)
- 0.3
- Green prefilter
- off
- RF-DETR Small/Medium (+SAHI)SIMULATED PROFILEApache-2.0
- SAM 3.1 (exemplar + concept)SIMULATED PROFILESAM License (commercial use permitted, carve-outs + patent clause)
- YOLO26-s/m + P2 head (1280–1536, SAHI)SIMULATED PROFILEAGPL-3.0 (or paid Enterprise)
- GeCo2SIMULATED PROFILEUnverified — check repo LICENSE before shipping
- CountGDSIMULATED PROFILEMIT
- T-Rex2 (interactive visual prompt)SIMULATED PROFILEIDEA License 1.0 (API-gated)
- Gemini Flash (tiled)SIMULATED PROFILEClosed commercial API
- OWLv2 (image-conditioned one-shot)SIMULATED PROFILEApache-2.0
Totals
Results grid
Counts, not scores. A count next to its truth is the number Habitat NZ's consent condition actually asks for; accuracy is the Evidence page's job.
| Frame | Truth | RF-DETR Small/Medium (+SAHI) SIMULATED PROFILE | SAM 3.1 (exemplar + concept) SIMULATED PROFILE | YOLO26-s/m + P2 head (1280–1536, SAHI) SIMULATED PROFILE | GeCo2 SIMULATED PROFILE | CountGD SIMULATED PROFILE | T-Rex2 (interactive visual prompt) SIMULATED PROFILE | Gemini Flash (tiled) SIMULATED PROFILE | OWLv2 (image-conditioned one-shot) SIMULATED PROFILE |
|---|---|---|---|---|---|---|---|---|---|
| Synthetic ladder — 0.5 mm/px — scattered 0.5 mm/px | 32 | 135+103 — · 2.1 s 9/9 tiles | 159+127 — · 1.7 s 9/9 tiles | 171+139 — · 1.8 s 9/9 tiles | 197+165 — · 2.4 s 9/9 tiles | 222+190 — · 2.1 s 9/9 tiles | 182+150 — · 2.0 s 9/9 tiles | 362+330 — · 1.7 s 9/9 tiles | 313+281 — · 1.9 s 9/9 tiles |
| Synthetic ladder — 0.5 mm/px — dense 0.5 mm/px | 181 | 232+51 — · 1.4 s 9/9 tiles | 276+95 — · 2.0 s 9/9 tiles | 276+95 — · 1.6 s 9/9 tiles | 321+140 — · 1.9 s 9/9 tiles | 356+175 — · 2.1 s 9/9 tiles | 311+130 — · 1.7 s 9/9 tiles | 491+310 — · 2.4 s 9/9 tiles | 465+284 — · 2.0 s 9/9 tiles |
| Synthetic ladder — 1.0 mm/px — scattered 1 mm/px | 49 | 156+107 — · 1.9 s 9/9 tiles | 175+126 — · 2.0 s 9/9 tiles | 191+142 — · 2.0 s 9/9 tiles | 219+170 — · 1.6 s 9/9 tiles | 236+187 — · 2.5 s 9/9 tiles | 201+152 — · 2.3 s 9/9 tiles | 367+318 — · 1.8 s 9/9 tiles | 331+282 — · 2.0 s 9/9 tiles |
| Synthetic ladder — 1.0 mm/px — dense 1 mm/px | 378 | 388+10 — · 1.9 s 9/9 tiles | 456+78 — · 1.5 s 9/9 tiles | 428+50 — · 1.9 s 9/9 tiles | 509+131 — · 2.2 s 9/9 tiles | 535+157 — · 1.9 s 9/9 tiles | 470+92 — · 2.2 s 9/9 tiles | 568+190 — · 1.6 s 9/9 tiles | 595+217 — · 2.6 s 9/9 tiles |
| Synthetic ladder — 2.0 mm/px — scattered 2 mm/px | 62 | 161+99 — · 2.0 s 9/9 tiles | 180+118 — · 1.4 s 9/9 tiles | 193+131 — · 2.0 s 9/9 tiles | 225+163 — · 2.2 s 9/9 tiles | 239+177 — · 1.9 s 9/9 tiles | 207+145 — · 2.2 s 9/9 tiles | 359+297 — · 2.2 s 9/9 tiles | 319+257 — · 2.0 s 9/9 tiles |
| Synthetic ladder — 2.0 mm/px — dense 2 mm/px | 400 | 416+16 — · 1.8 s 9/9 tiles | 457+57 — · 1.5 s 9/9 tiles | 448+48 — · 2.0 s 9/9 tiles | 532+132 — · 2.2 s 9/9 tiles | 545+145 — · 1.9 s 9/9 tiles | 488+88 — · 1.3 s 9/9 tiles | 509+109 — · 1.6 s 9/9 tiles | 555+155 — · 2.3 s 9/9 tiles |
| Synthetic ladder — 4.0 mm/px — scattered 4 mm/px | 27 | 116+89 — · 1.9 s 9/9 tiles | 140+113 — · 2.1 s 9/9 tiles | 151+124 — · 2.2 s 9/9 tiles | 173+146 — · 2.0 s 9/9 tiles | 194+167 — · 2.3 s 9/9 tiles | 163+136 — · 1.7 s 9/9 tiles | 322+295 — · 1.7 s 9/9 tiles | 284+257 — · 1.9 s 9/9 tiles |
| Synthetic ladder — 4.0 mm/px — dense 4 mm/px | 237 | 224-13 — · 2.0 s 9/9 tiles | 278+41 — · 1.7 s 9/9 tiles | 257+20 — · 1.9 s 9/9 tiles | 333+96 — · 2.0 s 9/9 tiles | 339+102 — · 1.8 s 9/9 tiles | 290+53 — · 1.7 s 9/9 tiles | 436+199 — · 2.4 s 9/9 tiles | 404+167 — · 1.9 s 9/9 tiles |
| Synthetic ladder — 8.0 mm/px — scattered 8 mm/px | 36 | 112+76 — · 2.1 s 9/9 tiles | 135+99 — · 1.8 s 9/9 tiles | 141+105 — · 1.9 s 9/9 tiles | 169+133 — · 1.8 s 9/9 tiles | 191+155 — · 2.3 s 9/9 tiles | 152+116 — · 2.0 s 9/9 tiles | 310+274 — · 2.2 s 9/9 tiles | 271+235 — · 2.0 s 9/9 tiles |
| Synthetic ladder — 8.0 mm/px — dense 8 mm/px | 1207 | 295-912 — · 1.8 s 9/9 tiles | 426-781 — · 1.9 s 9/9 tiles | 343-864 — · 2.1 s 9/9 tiles | 498-709 — · 2.1 s 9/9 tiles | 499-708 — · 2.3 s 9/9 tiles | 405-802 — · 2.1 s 9/9 tiles | 449-758 — · 2.0 s 9/9 tiles | 440-767 — · 2.3 s 9/9 tiles |
The small figure beside each count is the signed count error against that row’s truth. It is kept signed on purpose: a model that misses half the plants and hallucinates the same number again has a perfect count and a useless map, and only the sign of the error tells you which of those you are looking at.
These are raw counts — every detection the arm returned, at no confidence threshold. They will run high, and they are not the figure the Evidence page reports: there each arm is scored at its own best-F1 operating point, because raw confidences are not comparable between providers. Use the slider in the inspector below to see what a threshold does to both the marks and the number.
Inspect a cell
Put the detections on the pixels. Toggle arms on and off, scroll to zoom to 1:1, and see whether the count is plants or schist. Layer visibility, pan and zoom are in the viewer itself.
CC0 (generated) — Synthetic scene — EmerTech generator
Applies to the marks and the numbers together, so the table always describes what you can see.
| Model | Access | Count | TP | FP | FN | Precision | Recall | F1 | Count error |
|---|---|---|---|---|---|---|---|---|---|
| RF-DETR Small/Medium (+SAHI) | SIMULATED PROFILE | 135 | 29 | 106 | 3 | 21.5% | 90.6% | 34.7% | +103 |
| SAM 3.1 (exemplar + concept) | SIMULATED PROFILE | 159 | 32 | 127 | 0 | 20.1% | 100.0% | 33.5% | +127 |
Scored against 32 plants and 24 planted hard negatives by greedy centre-distance matching, highest score first, one detection per plant, within 32.8 px (max 3 px, 1.5 × the median ground-truth edge). Never IoU: at 5–15 px per plant a one-pixel shift moves IoU more than the gap between two models. A detection landing on a planted negative is a false positive — that is what the decoys are for.
These figures are at the threshold on the slider (0.00). The leaderboard reports each arm at its own best-F1 operating point instead, because raw confidences are not comparable across providers — so the two will differ, and both are stated rather than one quietly replacing the other.
Run manifest
Written when the run finished, and attached to every export. A count is worth what the trail behind it is worth.
- App version
- 0.1.0
- Run id
- seed-reference-bakeoff
- Created
- 1 Sept 2026, 12:00 pm
greedy nearest-centroid within max(3px, 1.5 × median ground-truth box edge); highest-confidence first, one detection per plant; point annotations counted as 8px across; planted hard negatives are unmatchable, so a detection on one scores as a false positive (centre distance, never IoU: at 5–15 px per plant a one-pixel shift moves IoU more than the gap between two models)
- SIMULATED ARMS IN THIS RUN (8 of 8): RF-DETR Small/Medium (+SAHI) (rf-detr), SAM 3.1 (exemplar + concept) (sam3), YOLO26-s/m + P2 head (1280–1536, SAHI) (yolo26-p2), GeCo2 (geco2), CountGD (countgd), T-Rex2 (interactive visual prompt) (t-rex2), Gemini Flash (tiled) (gemini-flash), OWLv2 (image-conditioned one-shot) (owlv2). Their numbers come from a simulated profile, not from live inference.
- SIMULATED means no inference ran on this imagery. A simulated arm draws detections from a deterministic, seeded error model whose recall/false-positive/jitter parameters are read off published benchmarks for that model (see docs/RESEARCH.md), applied to the known ground truth. It is a PRIOR describing what that model would plausibly do, not a measurement of what it did, and it must never be quoted as this model's accuracy on Central Otago imagery. Every simulated figure is replaced by a real one the moment that arm runs live.
- IMAGE LICENCES IN THIS RUN: CC0 (generated) (10 images).
- Attribution travels with the pixels: every CSV row and GeoJSON feature carries the licence and attribution of its source image, so a count lifted out of this app into a spreadsheet keeps its credit. CC BY requires that credit wherever the derived count is published; CC BY-SA additionally share-alikes anything derived from it.
- Georeference quality: pixel-space (demo) — approximate; real flights get EXIF/RTK georeferencing. Export coordinates are image pixels (x = column, y = row, top-left origin) in original full-resolution space. Nothing here is a map projection.