SAAIL Lab SAAIL Lab
Deep Learning Indaba 2026 Best Poster Award, Deep Learning Indaba 2026 (Lagos, Nigeria) Published · 18/114 Accepted (15.82%)

DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics: A Cassava Case Study

Innocent Nyalala 1,2 , Surabhi Gudla 1 , Patrick Vincent Ndowo 1
1School of Engineering and Science, Indian Institute of Technology Madras Zanzibar, Tanzania
2Wadhwani School of Data Science and AI (WSDSAI), Indian Institute of Technology Madras, Chennai, India
Deep Learning Indaba 2026, Pan-Atlantic University, Lagos, Nigeria (Aug 2026)
DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics: A Cassava Case Study

Summary: Camera-ready poster awarded Best Poster at Deep Learning Indaba 2026 (Lagos, Nigeria; 18 accepted of 114 submissions). Shows how class-imbalanced field splits induce a -5.96 pp raw-accuracy evaluation mirage while macro-F1 exposes +8.19 pp true deployment degradation.

Abstract

Aggregate classification accuracy on imbalanced agricultural datasets routinely hides silent failures on minority but economically devastating plant diseases. We introduce DGS-Bench, a Deployment Gap and Sensitivity Benchmark tailored for African agricultural AI, grounded in a multi-condition cassava pathology case study. Evaluating models across controlled versus real smallholder field acquisitions reveals that models boasting high top-line accuracy frequently suffer sharp drops in macro-averaged recall and per-class F1 on rare pathologies. DGS-Bench establishes a standardized evaluation protocol combining balanced per-class metrics, deployment gap quantification, and calibration checks, earning the Best Poster Award at Deep Learning Indaba 2026 (one of 18 papers accepted out of 114 submissions, 15.82% acceptance rate).

Keywords: DGS-BenchAfrican agricultural AIcassava diseasebalanced metricsdeployment gapDeep Learning Indaba

Key Findings & Contributions

+8.19 pp
Mean Macro-F1 Gap
Paired t = 10.2, p = 0.002
-5.96 pp
Raw Acc Mirage
Field Acc hides minority failure
-17.0 pp
BG Perturbation Drop
ViT-Small/16 loses 21.3 pp
Best Poster
DLI 2026 Winner
18 / 114 accepted (15.82%)
  1. Defines the Deployment Gap Score (DGS) across raw accuracy, balanced accuracy, and macro-F1 to quantify lab-to-field distribution shift on imbalanced smallholder datasets.
  2. Demonstrates an evaluation mirage across four frozen linear-probe architectures (ResNet-50, EfficientNet-B3, MobileNetV3-S, ViT-Small/16) where 65.8% CMD prevalence inflates field accuracy despite minority-class collapse (Healthy recall drops from 52.5% to 47.7%).
  3. Establishes through seven controlled perturbations that background texture reliance (-13.6 to -21.3 pp) rather than brightness variation (<1.7 pp) dominates deployment fragility across CNNs and Vision Transformers.
  4. Reproduces the evaluation gap across three crops and class cardinalities (Cassava 5 classes, PlantVillage Tomato 10 classes, Rice Leaf Disease 4 classes) under a reproducible 11-minute free T4 GPU benchmark.

Key Empirical Figure

Lab vs. natural field accuracy across four architectures revealing negative raw-accuracy DGS mirage while macro-F1 exposes +8.19 pp mean deployment degradation.
Lab vs. natural field accuracy across four architectures revealing negative raw-accuracy DGS mirage while macro-F1 exposes +8.19 pp mean deployment degradation.

Benchmark Results

DGS-Bench Primary Benchmark: Lab (Balanced n=400) vs. Natural Field (n=19,397, 65.8% CMD)

Raw field accuracy artificially exceeds lab accuracy across all models (negative DGS_Acc), whereas Macro-F1 exposes +6.23 to +9.67 pp degradation.

ArchitectureLab Acc (%)Field Acc (%)DGS Acc (pp)Field BAcc (%)Lab F1 (%)Field F1 (%)DGS F1 (pp)
ResNet-5054.7562.80-8.0556.3654.7148.47+6.23
EfficientNet-B356.2561.93-5.6854.6656.1046.78+9.32
MobileNetV3-S54.0056.97-2.9753.2654.0544.37+9.67
ViT-Small/1657.0064.14-7.1457.5156.8849.37+7.51

Controlled Perturbation Drop (pp) on Field Features at Severity 1.0

ArchitectureBase Field Acc (%)Background Δ (pp)Gaussian Noise Δ (pp)Brightness Δ (pp)
ResNet-5062.814.39.9-0.9
EfficientNet-B361.913.69.0-0.5
MobileNetV3-S57.018.814.8-1.6
ViT-Small/1664.121.316.0-0.6

BibTeX

@inproceedings{nyalala2026dgsbench,
  title={DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics -- A Cassava Case Study},
  author={Nyalala, Innocent and Gudla, Surabhi and Vincent, Patrick},
  booktitle={Deep Learning Indaba 2026},
  address={Lagos, Nigeria},
  year={2026},
  month={August},
  doi={10.13140/rg.2.2.12603.17448},
  note={Best Poster Award; 18 of 114 accepted}
}
APA Citation

Nyalala, I., Gudla, S., & Vincent, P. (2026, August). DGS-Bench: Measuring the Deployment Gap in African Agricultural AI Requires Balanced Metrics , A Cassava Case Study. Deep Learning Indaba 2026, Lagos, Nigeria. https://doi.org/10.13140/rg.2.2.12603.17448