SAAIL Lab SAAIL Lab
Under Peer Review · 2026 Under Peer Review

Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar

1School of Engineering and Science, Indian Institute of Technology Madras Zanzibar, Tanzania
2Wadhwani School of Data Science and AI (WSDSAI), Indian Institute of Technology Madras, Chennai, India
Manuscript Under Peer Review (2026)
Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar

Summary: Three-part experimental protocol testing capture-session confounding across 4,603 single-clove and 695 batch images, duplicate-aware 64-bit dHash grouping (3,455 groups), input mask ablation, and foreground/background swap probes.

Abstract

The Zanzibar State Trading Corporation (ZSTC) prices cloves (Syzygium aromaticum) by four commercial grades assigned through non-destructive visual inspection. Using an image dataset aligned with these grades (4,603 single-clove and 695 batch images), we evaluated grading models under a protocol that controls near-duplicate leakage and training differences. Sixteen of seventeen pretrained networks reached 98.1% to 99.3% test macro-F1. Because the grades were photographed in separate sittings on six capture days, we tested what the networks learned. Trained on images taken on some days and tested on days never seen in training, mean macro-F1 fell from 99% to between 29% and 47%, with recall ranging from 1.5% to 95% across held-out grade-days. Images with the bud removed by inpainting were still graded at 98.5%, whereas the bud silhouette alone reached 68%. When a clove of one grade was pasted onto the background of another, networks followed the background in 53% to 57% of cases and the clove in 30% to 33%, while pasting each clove back onto its own background left accuracy unchanged (97% to 98%). The bud region alone supported about 93% macro-F1, so grade information is present in the clove; trained networks combine it with a capture-specific cue that dominates when the two conflict and causes some unseen sittings to be misgraded as a whole. We describe a low-cost probe protocol for testing the measurement validity of image datasets collected one class at a time, and the collection design needed to measure clove grading accuracy without this confound.

Keywords: clove gradingcapture-session confoundingshortcut learningmeasurement validitydata leakageZanzibar

Key Findings & Contributions

99% to 29-47%
Cross-Day F1 Drop
Unseen capture sittings collapse
98.5%
Empty BG Grading
Accuracy after bud inpainting
53-57%
BG Override Rate
Background beats clove in swaps
17 Models
Duplicate-Controlled
3,455 dHash groups + 695 batches
  1. Demonstrates across 17 pretrained architectures (16 reaching 98.1%-99.3% standard F1) that single-session class acquisition creates catastrophic hidden confounding masked by standard random or dHash-grouped splits.
  2. Quantifies extreme cross-day collapse (29% to 47% mean macro-F1, per-grade recall 1.5% to 95%) when models are tested on held-out photography days.
  3. Proves shortcut dominance directly via bud-inpainting (98.5% accuracy with empty background) and foreground-background swaps (53%-57% background override vs. 93% isolated-clove ceiling).
  4. Delivers a practical low-cost validity probe protocol and randomized lot collection design eliminating session confounding in agricultural vision datasets.

Key Empirical Figure

Inpainting and background-swap diagnostic probes proving networks follow capture-session background cues (98.5% accuracy without buds; 53%-57% swap override) over isolated clove buds.
Inpainting and background-swap diagnostic probes proving networks follow capture-session background cues (98.5% accuracy without buds; 53%-57% swap override) over isolated clove buds.

Benchmark Results

Summary of Validity Probes Exposed on Duplicate-Controlled Clove Grading (17 Networks, n = 4,603)

Evaluation ProbeMeasured Macro-F1 / AccuracyDiagnostic Interpretation
Standard dHash-Grouped Split (16/17 CNN/ViT)98.1% - 99.3% macro-F1Top models statistically indistinguishable (DenseNet-121 / Xception / ViT 99.25%)
Held-Out Unseen Capture Day Split29% - 47% macro-F1Catastrophic cross-session failure despite zero duplicate overlap
Empty Background (Bud Removed via Inpainting)98.5% accuracyTray/illumination session artifacts carry almost full class signal
Conflict Swap (Grade A Bud on Grade B BG)53% - 57% follow BG · 30% - 33% follow budSession background cue overrides biological bud features under conflict
Isolated Clove Bud Foreground Alone~93% macro-F1True morphological grading signal exists once background confounds are removed

BibTeX

@misc{ndowo2026capture_confounding,
  title={Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar},
  author={Ndowo, Patrick Vincent and Nyalala, Innocent},
  year={2026},
  note={Manuscript under peer review}
}
APA Citation

Ndowo, P. V., & Nyalala, I. (2026). Capture-session confounding undermines the measurement validity of deep learning clove grading: evidence from Zanzibar [Manuscript under peer review].