Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar
Abstract
The Zanzibar State Trading Corporation (ZSTC) prices cloves (Syzygium aromaticum) by four commercial grades assigned through non-destructive visual inspection. Using an image dataset aligned with these grades (4,603 single-clove and 695 batch images), we evaluated grading models under a protocol that controls near-duplicate leakage and training differences. Sixteen of seventeen pretrained networks reached 98.1% to 99.3% test macro-F1. Because the grades were photographed in separate sittings on six capture days, we tested what the networks learned. Trained on images taken on some days and tested on days never seen in training, mean macro-F1 fell from 99% to between 29% and 47%, with recall ranging from 1.5% to 95% across held-out grade-days. Images with the bud removed by inpainting were still graded at 98.5%, whereas the bud silhouette alone reached 68%. When a clove of one grade was pasted onto the background of another, networks followed the background in 53% to 57% of cases and the clove in 30% to 33%, while pasting each clove back onto its own background left accuracy unchanged (97% to 98%). The bud region alone supported about 93% macro-F1, so grade information is present in the clove; trained networks combine it with a capture-specific cue that dominates when the two conflict and causes some unseen sittings to be misgraded as a whole. We describe a low-cost probe protocol for testing the measurement validity of image datasets collected one class at a time, and the collection design needed to measure clove grading accuracy without this confound.
Key Findings & Contributions
- Demonstrates across 17 pretrained architectures (16 reaching 98.1%-99.3% standard F1) that single-session class acquisition creates catastrophic hidden confounding masked by standard random or dHash-grouped splits.
- Quantifies extreme cross-day collapse (29% to 47% mean macro-F1, per-grade recall 1.5% to 95%) when models are tested on held-out photography days.
- Proves shortcut dominance directly via bud-inpainting (98.5% accuracy with empty background) and foreground-background swaps (53%-57% background override vs. 93% isolated-clove ceiling).
- Delivers a practical low-cost validity probe protocol and randomized lot collection design eliminating session confounding in agricultural vision datasets.
Key Empirical Figure
Benchmark Results
Summary of Validity Probes Exposed on Duplicate-Controlled Clove Grading (17 Networks, n = 4,603)
| Evaluation Probe | Measured Macro-F1 / Accuracy | Diagnostic Interpretation |
|---|---|---|
| Standard dHash-Grouped Split (16/17 CNN/ViT) | 98.1% - 99.3% macro-F1 | Top models statistically indistinguishable (DenseNet-121 / Xception / ViT 99.25%) |
| Held-Out Unseen Capture Day Split | 29% - 47% macro-F1 | Catastrophic cross-session failure despite zero duplicate overlap |
| Empty Background (Bud Removed via Inpainting) | 98.5% accuracy | Tray/illumination session artifacts carry almost full class signal |
| Conflict Swap (Grade A Bud on Grade B BG) | 53% - 57% follow BG · 30% - 33% follow bud | Session background cue overrides biological bud features under conflict |
| Isolated Clove Bud Foreground Alone | ~93% macro-F1 | True morphological grading signal exists once background confounds are removed |
BibTeX
@misc{ndowo2026capture_confounding,
title={Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar},
author={Ndowo, Patrick Vincent and Nyalala, Innocent},
year={2026},
note={Manuscript under peer review}
} Ndowo, P. V., & Nyalala, I. (2026). Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar [Manuscript under peer review].