Part of the DGEval benchmark. DGEval: all results and updates
Since the original study, a new wave of frontier and open-source models has been released. We re-ran the same 1,678 DGEval questions against 21 model families in 62 configurations to see whether the gaps the first study found have closed.
Gemini 3.1 Pro still leads, but the field has closed in
Gemini 3.1 Pro remains the strongest model overall at 77.0%, just short of the corrected human practitioner baseline of 83.8%. Several newer, cheaper models now land within a few points of it on everyday recall.
View data table
| Item | Score |
|---|---|
| Gemini 3.1 Pro (High) | 77.0% |
| Gemini 3.8 Flash (Medium) | 74.2% |
| Gemini 3.7 Flash (Medium) | 74.1% |
| GPT-5.6 Sol (High) | 72.2% |
| GPT-5.5 (High) | 70.7% |
| Claude Opus 5 (High) | 69.8% |
| Claude Opus 4.7 (Max) | 63.7% |
| Gemini 3.1 Flash-Lite (High) | 61.8% |
| GPT-5.6 Terra (High) | 57.1% |
| GPT-5.6 Luna (High) | 56.0% |
What has not changed is the shape of the problem. Regulatory recall (Section 4) is still the weakest task for every model tested, commercial or open-source, with a best score of 36.8%.
Stowage and segregation are still the hardest area
Gemini 3.1 Pro falls from 99% on straightforward classification questions to 88% on stowage and segregation questions. Across the commercial models on average the gap is wider: 93% on classification, but 58% on stowage and segregation. On Dangerous Goods List multiple-choice questions alone, Claude Opus 5 scored 69% and Claude Opus 4.7 scored 56% on stowage and segregation.
Retrieval narrows the citation gap more than a new model does
Giving Gemini 3.1 Pro retrieval access to dangerous goods reference data, rather than answering from what it learned in training, lifted its regulatory recall from 39.9% to 49.8% and its Dangerous Goods List lookup from 89.3% to 96.5%, at a cost of about five points of multiple-choice accuracy (96.2% to 91.5%). Gemini 3.8 Flash gained even more on regulatory recall, from 27.8% to 40.0%, and improved on multiple choice too.
View data table
| Measure | Without retrieval | With retrieval |
|---|---|---|
| Gemini 3.1 Pro: multiple choice | 96.2% | 91.5% |
| Gemini 3.1 Pro: DGL lookup | 89.3% | 96.5% |
| Gemini 3.1 Pro: regulatory recall | 39.9% | 49.8% |
| Gemini 3.8 Flash: multiple choice | 95.0% | 96.9% |
| Gemini 3.8 Flash: regulatory recall | 27.8% | 40.0% |
Costs vary widely
Gemini 3.7 Flash reaches 74.1% overall, within three points of the leader, for $2.04 and 1.77 hours per 1,000 questions. Gemini 3.1 Pro costs $10.80 and takes 4.3 hours, and GPT-5.5 costs $101.54.
Limits
These are single-run point estimates on IMDG Code Amendment 42-24, and several models give different answers from run to run. Section scores use the paper’s expert weighting, and overall scores are the equal-weighted mean of the four sections.