DGEval is NCB Hazcheck’s benchmark, developed with Durham University, that tests how well large language models answer real dangerous goods questions: 1,678 questions grounded in IMDG Code Amendment 42-24.
Read the latest results Read the research paper
- 1,678questions across four task typesGrounded in IMDG Code Amendment 42-24.
- 77.0%best AI model overallGemini 3.1 Pro, from 21 model families tested.
- 83.8%human practitioner baselineOn the multiple-choice section.
- 36.8%best score citing the exact regulationThe weakest task for every model tested.
Why DGEval
Large language models are increasingly used as a first line of decision support in dangerous goods work: drafting answers, checking classifications and pointing people to the governing rule. Before DGEval, no published benchmark tested them against the IMDG Code.
DGEval measures what matters in practice: not just general knowledge, but the safety-critical judgement calls of stowage and segregation, and whether a model can cite the exact provision behind its answer.
Section 1: Multiple choice
Closed-book questions from NCB Hazcheck’s e-learning certification, with a human practitioner baseline.
Section 2: Open-ended questions
Written answers to practical dangerous goods compliance scenarios.
Section 3: Dangerous Goods List lookup
Ten Dangerous Goods List fields per entry, from hazard class to segregation group.
Section 4: Regulatory recall
Naming the IMDG Code chapter, section and subsection that governs the answer.
Finding 1: only two models beat the human baseline on everyday recall
In the original study of 13 models, only Gemini 3.1 Pro and GPT-5.5 cleared the corrected human practitioner baseline of 83.8% on closed-book multiple choice. Open-source and maritime fine-tuned models scored far lower.
View data table
| Item | Score |
|---|---|
| Gemini 3.1 Pro (High) | 95.8% |
| GPT-5.5 (High) | 90.7% |
| Gemini 3.1 Flash-Lite (High) | 79.0% |
| GPT-5.4 Mini (High) | 76.1% |
| Claude Sonnet 4.6 (Max) | 75.5% |
| Claude Opus 4.7 (Max) | 72.1% |
| Gemma 4 31B | 66.2% |
| Qwen3.6 35B (Thinking) | 59.5% |
| Qwen3.5 9B (Thinking) | 54.9% |
| Gemma 4 4B (Thinking) | 37.8% |
| Llama 3.3 70B | 36.3% |
| LLaMarine | 35.7% |
| Llama 3.2 3B | 14.6% |
Every model, including the best, struggled with the highest-stakes areas. Within Dangerous Goods List lookups, the stowage and segregation fields scored lowest of all: the fields NCB Hazcheck’s experts rate highest for danger-criticality.
View data table
| Item | Score |
|---|---|
| Marine Pollutant | 80.5% |
| Primary Hazard Class | 74.3% |
| Packing Group | 71.0% |
| Emergency Schedule | 50.6% |
| Proper Shipping Name | 44.9% |
| Stowage Code | 39.8% |
| Stowage Category | 36.1% |
| Segregation Code | 25.5% |
| Special Provisions | 21.3% |
| Segregation Group | 18.4% |
Finding 2: newer models close in, but exact citation is still the weak point
Re-running DGEval against 21 model families shows the field closing in behind Gemini 3.1 Pro (77.0% overall), with cheaper models such as Gemini 3.7 Flash reaching 74.1%.
View data table
| Item | Score |
|---|---|
| Gemini 3.1 Pro (High) | 77.0% |
| Gemini 3.8 Flash (Medium) | 74.2% |
| Gemini 3.7 Flash (Medium) | 74.1% |
| GPT-5.6 Sol (High) | 72.2% |
| GPT-5.5 (High) | 70.7% |
| Claude Opus 5 (High) | 69.8% |
| Claude Opus 4.7 (Max) | 63.7% |
| Gemini 3.1 Flash-Lite (High) | 61.8% |
| GPT-5.6 Terra (High) | 57.1% |
| GPT-5.6 Luna (High) | 56.0% |
Running costs vary widely: Gemini 3.1 Pro answers 1,000 questions for $10.80 in 4.3 hours, while GPT-5.5 costs $101.54. Regulatory recall remains the weakest task for every model, with a best score of 36.8%. Giving a model retrieval access to dangerous goods reference data did more for this than a newer model generation: Gemini 3.1 Pro rose from 39.9% to 49.8%.
Finding 3: drawing on Hazcheck lifts the weakest, highest-stakes areas
Hazcheck Assist checks Hazcheck systems through tool calls before it answers. It is available as part of Hazcheck products, not for individual use. On the same four sections it scored 87.4% overall, against 77.0% for the best frontier model.
View data table
| Measure | Hazcheck Assist | Gemini 3.1 Pro (High) |
|---|---|---|
| Multiple choice | 98.1% | 95.8% |
| Open-ended questions | 93.4% | 86.0% |
| DGL lookup | 97.1% | 89.3% |
| Regulatory recall | 60.7% | 36.8% |
| Overall | 87.4% | 77.0% |
The gain is smallest on multiple choice, where frontier models were already strong, and largest on regulatory recall (60.7% against 36.8%). Exact-subsection citation, at 38.4%, is where most work remains.
View data table
| Measure | Hazcheck Assist | Best frontier model | Frontier average |
|---|---|---|---|
| Right chapter | 92.9% | 79.4% | 50.8% |
| Right section | 87.2% | 56.3% | 27.2% |
| Exact subsection | 38.4% | 14.3% | 4.6% |
In Hazcheck products, the citation gap is closed. Hazcheck Assist is used alongside Hazcheck Validate, which checks shipments against the IMDG Code requirements, and the system flags or links to the relevant provisions of the official IMDG Code as users work. So users are pointed to the governing regulation rather than relying on the model to recall it.
The Dangerous Goods List’s operational fields, no longer the weak point
Stowage and segregation are the highest-stakes Dangerous Goods List fields, and the weakest for every frontier model. Drawing on Hazcheck’s dangerous goods data, Hazcheck Assist scored 100% on Stowage Code, Stowage Category and Special Provisions, and 91.7% on Segregation Group, against frontier averages of 33% to 53%. Segregation Code, at 76%, is the one operational field below 90%.
View data table
| Measure | Frontier average | Hazcheck Assist |
|---|---|---|
| Marine Pollutant | 86.2% | 100.0% |
| Primary Hazard Class | 87.2% | 100.0% |
| Packing Group | 79.7% | 100.0% |
| Emergency Schedule | 72.9% | 100.0% |
| Proper Shipping Name | 60.6% | 96.8% |
| Stowage Code (operational) | 53.2% | 100.0% |
| Stowage Category (operational) | 50.3% | 100.0% |
| Segregation Code (operational) | 34.9% | 76.0% |
| Special Provisions (operational) | 35.3% | 100.0% |
| Segregation Group (operational) | 33.0% | 91.7% |
These are NCB Hazcheck’s own results for its own system: a single run, with open-ended answers marked by Gemini 2.5 Flash. Cost and time ($22.36 and 7.1 hours per 1,000 questions) come from a sequential, unoptimised run.
Why Hazcheck is the right tool for the job
General-purpose AI can answer many dangerous goods questions well, but DGEval shows where it falls short: the stowage and segregation decisions that matter most for safety at sea, and pointing to the regulation behind an answer. These are exactly the areas Hazcheck is built for.
The strongest system tested
Hazcheck Assist scored 87.4% on DGEval, against 77.0% for the best frontier model, and scored higher than every frontier model in all four sections.
Strongest where safety is at stake
On the stowage and segregation fields, the weakest area for every frontier model, Hazcheck Assist scored between 76% and 100%, against frontier averages of 33% to 53%.
The regulation, not a guess
Used with Hazcheck Validate, Hazcheck checks shipments against IMDG Code requirements and flags or links to the relevant provisions of the official IMDG Code, closing the citation gap that AI models alone leave open.
Maintained regulatory data
Hazcheck is maintained in line with supported regulatory amendments, built on more than 40 years of dangerous goods technology expertise, and supports the validation of more than 80% of maritime dangerous goods container traffic.
Request a Demo Explore Hazcheck solutions
DGEval updates
DGEval is re-run as new models are released and as our tools improve. Newest first. Know a model we should test? Recommend it.
DGEval: what changes when the model can draw on Hazcheck
Hazcheck Assist, which checks Hazcheck systems before it answers, scored 87.4% overall against 77.0% for the best frontier model, with the largest gains on regulatory recall.
DGEval revisited: how do the newest AI models score?
21 model families, 62 configurations. Gemini 3.1 Pro still leads at 77.0%, and retrieval narrows the regulatory citation gap more than a newer model does.
NCB Studies Large Language Models’ Knowledge of Dangerous Goods Regulations, Introducing DGEval Benchmark
The first DGEval results: 13 AI models and 1,678 questions. Only two models beat the human baseline on multiple choice, and regulatory recall averaged 14.2%.
Recommend a model for us to test
Is there a model you would like to see on DGEval? Tell us which one, and how it can be accessed.
We can only test models we can access, either because they are publicly available or because access is provided to us. We review every suggestion, but cannot guarantee that a model will be tested or when.
How DGEval works
DGEval has 1,678 questions across four sections, grounded in IMDG Code Amendment 42-24. The multiple-choice questions come from NCB Hazcheck’s e-learning certification, which provides a corrected human practitioner baseline of 83.8%. Section scores use expert weighting, and overall scores are the equal-weighted mean of the four sections. Results are single-run point estimates, and several models give different answers from run to run.
Thomas, Shum, Nellis, Zhu, Yochum, Bartle and Wrightson (2026). Evaluating Large Language Model Performance on International Maritime Dangerous Goods Code Compliance. NCB Hazcheck and Durham University. arXiv:2608.21036
Thinking of using AI in dangerous goods work?
Use DGEval-informed benchmarking to assess AI tools for your dangerous goods workflows before you rely on them.