DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics: A Cassava Case Study
Abstract
Aggregate classification accuracy on imbalanced agricultural datasets routinely hides silent failures on minority but economically devastating plant diseases. We introduce DGS-Bench, a Deployment Gap and Sensitivity Benchmark tailored for African agricultural AI, grounded in a multi-condition cassava pathology case study. Evaluating models across controlled versus real smallholder field acquisitions reveals that models boasting high top-line accuracy frequently suffer sharp drops in macro-averaged recall and per-class F1 on rare pathologies. DGS-Bench establishes a standardized evaluation protocol combining balanced per-class metrics, deployment gap quantification, and calibration checks, earning the Best Poster Award at Deep Learning Indaba 2026 (one of 18 papers accepted out of 114 submissions, 15.82% acceptance rate).
Key Findings & Contributions
- Defines the Deployment Gap Score (DGS) across raw accuracy, balanced accuracy, and macro-F1 to quantify lab-to-field distribution shift on imbalanced smallholder datasets.
- Demonstrates an evaluation mirage across four frozen linear-probe architectures (ResNet-50, EfficientNet-B3, MobileNetV3-S, ViT-Small/16) where 65.8% CMD prevalence inflates field accuracy despite minority-class collapse (Healthy recall drops from 52.5% to 47.7%).
- Establishes through seven controlled perturbations that background texture reliance (-13.6 to -21.3 pp) rather than brightness variation (<1.7 pp) dominates deployment fragility across CNNs and Vision Transformers.
- Reproduces the evaluation gap across three crops and class cardinalities (Cassava 5 classes, PlantVillage Tomato 10 classes, Rice Leaf Disease 4 classes) under a reproducible 11-minute free T4 GPU benchmark.
Key Empirical Figure
Benchmark Results
DGS-Bench Primary Benchmark: Lab (Balanced n=400) vs. Natural Field (n=19,397, 65.8% CMD)
Raw field accuracy artificially exceeds lab accuracy across all models (negative DGS_Acc), whereas Macro-F1 exposes +6.23 to +9.67 pp degradation.
| Architecture | Lab Acc (%) | Field Acc (%) | DGS Acc (pp) | Field BAcc (%) | Lab F1 (%) | Field F1 (%) | DGS F1 (pp) |
|---|---|---|---|---|---|---|---|
| ResNet-50 | 54.75 | 62.80 | -8.05 | 56.36 | 54.71 | 48.47 | +6.23 |
| EfficientNet-B3 | 56.25 | 61.93 | -5.68 | 54.66 | 56.10 | 46.78 | +9.32 |
| MobileNetV3-S | 54.00 | 56.97 | -2.97 | 53.26 | 54.05 | 44.37 | +9.67 |
| ViT-Small/16 | 57.00 | 64.14 | -7.14 | 57.51 | 56.88 | 49.37 | +7.51 |
Controlled Perturbation Drop (pp) on Field Features at Severity 1.0
| Architecture | Base Field Acc (%) | Background Δ (pp) | Gaussian Noise Δ (pp) | Brightness Δ (pp) |
|---|---|---|---|---|
| ResNet-50 | 62.8 | 14.3 | 9.9 | -0.9 |
| EfficientNet-B3 | 61.9 | 13.6 | 9.0 | -0.5 |
| MobileNetV3-S | 57.0 | 18.8 | 14.8 | -1.6 |
| ViT-Small/16 | 64.1 | 21.3 | 16.0 | -0.6 |
BibTeX
@inproceedings{nyalala2026dgsbench,
title={DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics -- A Cassava Case Study},
author={Nyalala, Innocent and Gudla, Surabhi and Vincent, Patrick},
booktitle={Deep Learning Indaba 2026},
address={Lagos, Nigeria},
year={2026},
month={August},
doi={10.13140/rg.2.2.12603.17448},
note={Best Poster Award; 18 of 114 accepted}
} Nyalala, I., Gudla, S., & Vincent, P. (2026, August). DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics , A Cassava Case Study. Deep Learning Indaba 2026, Lagos, Nigeria. https://doi.org/10.13140/rg.2.2.12603.17448