NbAiLab Borealis Models Audit — Gemma 4 & API Reference Update

Browse raw audit logs & judge feedback in SimpleAudit Visualizer
AI Safety Department, SimulaMet · michael@simula.no · sushant@simula.no Original audit: May 2026 · Gemma 4 and API reference extension: August 2026 Judge: Qwen3.6-27B 10 runs for model families · n=1 for gpt-5-mini, GPT-4, and GPT-3.5 Turbo Score: pass=100 · low=75 · medium=50 · high=25 · critical=0
Families:

Overview

Original May 2026 Borealis audit plus an August 2026 extension for Google Gemma 4 26B-A4B-it, the NbAiLab Gemma 4 E3 checkpoint, and one-run GPT-4 and GPT-3.5 Turbo API references. Every model uses the same 36 Norwegian and 155 BullshitBench scenarios; the API references are descriptive n=1 comparisons.

GPT-5 Mini reference NOR
92.7
91.2% pass rate · n=1 run
Gemma 4 base NOR
89.3
83.3% pass · n=10 runs
NbAiLab Gemma 4 E3 NOR
68.8
54.4% pass · n=10 runs
GPT-4 reference NOR
64.6
47.2% pass · n=1 run
GPT-3.5 reference NOR
59.7
44.4% pass · n=1 run
E3 BullshitBench gain
+6.1
45.4 vs Gemma 4 base 39.3
Best Borealis NOR
65.7
borealis-27b · 51.1% pass
Gap to GPT-5 Mini reference
−27.0
at 27b, Norwegian tasks
NOR fine-tune gain
+13.8
4b borealis vs gemma base
Newspaper data gain
+2.5
borealis vs open at 27b
borealis final gain
+4.5
27b Norwegian avg

Overall Scores — NOR avg (36 scenarios)

Filtered by active families above. Dashed line = GPT-5 Mini n=1 reference (92.7).

Gemma 4 Update — August 2026

A descriptive comparison of Google Gemma 4 26B-A4B-it and NbAiLab's E3 checkpoint on the exact audit protocol used above, with GPT-4 and GPT-3.5 Turbo included as one-run API context. Both Gemma 4 models have 10 complete runs; the API references have one run each. Deltas describe observed runs and are not causal estimates.

Gemma 4 pair with API references

Domain and BullshitBench scores. GPT-4 and GPT-3.5 Turbo are n=1 references; higher is better under the audit's severity-weighted score.

Observed tradeoff

E3 NOR delta
−20.6
68.8 vs base 89.3
E3 BS delta
+6.1
45.4 vs base 39.3
Across these runs, E3 is lower in every Norwegian pack: Language −34.1, Safety −11.3, Healthcare −17.3, and Public Sector −20.5 points relative to Gemma 4 base.
E3 is stronger on BullshitBench and nearly matches the single-run gpt-5-mini reference (45.4 vs 45.7). This comparison is descriptive; gpt-5-mini has n=1.
The legacy API references rank below E3 on Norwegian score (GPT-4 64.6; GPT-3.5 Turbo 59.7) and below both Gemma 4 models on BullshitBench (31.9 and 21.6). Each API value is from one run.

Model lineage and protocol

Google base: google/gemma-4-26B-A4B-it, a 26B-total/A4B-active mixture-of-experts model. NbAiLab E3: the Gemma 4 checkpoint continued-pretrained on Aurora-2604 text and then supervised-fine-tuned on Aurora-SFT-2606 through epoch 3. Both were judged by Qwen/Qwen3.6-27B with 10 repetitions over 191 scenarios. API references: gpt-4 and gpt-3.5-turbo were evaluated once each with the same scenarios, five-turn protocol, and judge. The Gemma 4 pair is shown at 26B in the total-parameter scale plots; the API references are excluded from parameter scaling because this report does not assign them comparable parameter counts.

Scale Comparison

NOR and BullshitBench scores by total model size. Gemma 4 is included at 26B total parameters beside the 27B Borealis models, with isolated markers because its A4B mixture-of-experts active compute is closer to 4B than to a dense 26–27B model. GPT-4 and GPT-3.5 Turbo are intentionally excluded because their API aliases are not assigned comparable parameter counts in this audit.

NOR avg vs total model size. Gemma 4 26B/A4B is plotted as a single reference point beside dense 27B models. GPT-5 Mini (n=1) is the dashed reference.

BullshitBench avg vs total model size. Gemma 4 26B/A4B is a single MoE reference point; it is not part of the dense scaling curves.

NOR score change relative to the corresponding Gemma base. The 26B/A4B bar compares NbAiLab E3 directly with Gemma 4 26B-A4B-it.

⚠ E3 is −20.6 points below its same-architecture Gemma 4 base on Norwegian scenarios. At 270m, Borealis is −2.7 points below Gemma 3; these observed differences do not by themselves identify which training stage caused them.

Domain Breakdown

Per-pack scores: Language (8), Safety (8), Healthcare (10), Public Sector (10). Select models for the radar comparison.

Domain Scores — leading models and API references

Gemma 4 base, NbAiLab E3, borealis-27b, and the three single-run API references. Family filters apply.

Radar: All Domains + BS

Select models to compare (max 5):

All Models by Domain

Grouped by domain. Toggle families above to filter.

⚠ gemma-3-27b-it scores 62.8 on Language vs borealis-27b's 60.3. The base model handles Norwegian grammar better — fine-tuning degraded surface language quality.

BullshitBench

155 cross-domain scenarios testing whether models push back on plausible-sounding nonsense. Not Norwegian-specific. Low scores expected — this is a hard benchmark.

⚠ In the original dense Gemma 3 comparison, Norwegian fine-tuning consistently degraded BullshitBench performance: gemma-3-27b-it (27.1) and gemma-3-12b-it (28.2) outperform their Borealis counterparts. The new Gemma 4 pair differs descriptively: NbAiLab E3 scores 45.4 versus 39.3 for its Gemma 4 base.
The one-run API references score 31.9 for GPT-4 and 21.6 for GPT-3.5 Turbo. GPT-4 is 10.3 points above GPT-3.5 Turbo but remains below Gemma 4 base (39.3), E3 (45.4), and gpt-5-mini (45.7) in these observed runs.

borealis vs instruct-preview

How much did the current borealis release improve over the earlier instruct-preview? API references are not part of this lineage comparison.

NOR avg: borealis vs instruct-preview

Positive = borealis improved. Gain concentrated at 27b (+4.5) and 12b (+2.7).

BullshitBench: borealis vs instruct-preview

borealis did not improve BS at any size — NOR gains came without general reasoning gains.

ℹ The 27b healthcare score jumped 10.7 pts from instruct-preview (67.8) to borealis (78.5). This is the single largest improvement in the borealis release. At 4b, the two are statistically equivalent.

Distribution & Reliability

Most model families were evaluated 10 times. gpt-5-mini, GPT-4, and GPT-3.5 Turbo were run once each (n=1). Charts show the available run values; a single point does not estimate run-to-run reliability.

Box plot — NOR score distribution across available runs

Box = Q1–Q3 interquartile range · center line = median · ✕ = mean · whiskers = min/max · dots = individual runs. Wide box = high run-to-run variance. Filtered by active families. gpt-5-mini, GPT-4, and GPT-3.5 Turbo each contribute one point.

⚠ borealis-27b (σ=5.2) and borealis-open-27b (σ=3.6) distributions overlap substantially. The new Gemma 4 runs are comparatively stable: base σ=2.7 and E3 σ=3.0. These are descriptive run spreads, not confidence intervals.
GPT-4 (64.6) and GPT-3.5 Turbo (59.7) appear as single points. Their displayed σ=0 is a serialization convention for n=1, not evidence of zero variability.

Severity Distribution — % of NOR judgments

Stacked bars show the breakdown of outcomes. Ten-run models summarize 360 judgments; each API reference summarizes 36 judgments from one run. Filtered by active families.

ℹ Gemma 4 base has the lowest critical rate among the multi-run models (0.8%) and an 83.3% pass rate. NbAiLab E3 records 7.2% critical and 54.4% pass. borealis-270m remains the most critical-heavy model at 87.8%.
The one-run Norwegian severity profiles are GPT-4: 47.2% pass, 8.3% critical; GPT-3.5 Turbo: 44.4% pass, 19.4% critical. These percentages summarize 36 judgments each.

Scenario Analysis

Per-scenario pass rates across 36 Norwegian scenarios. Columns = models, rows = scenarios sorted by difficulty (hardest first). Family filters also control heatmap columns. Color: ■ 0% → ■ 50% → ■ 100%. purple = all visible models fail (possible judge artefact).

n=1 references: gpt-5-mini, GPT-4, and GPT-3.5 Turbo pass rates are binary outcomes from one run per scenario. Gemma and Borealis columns use 10-run pass rates. · Difficulty legend: hard avg ≤15% across models excluding the GPT-5 Mini reference · variable high cross-model variance · easy avg ≥65%

Key Findings

Gemma 4 base nearly reaches the single-run reference on Norwegian
Gemma 4 26B-A4B-it averages 89.3 across 10 runs, 3.4 points below the GPT-5 Mini reference's 92.7 single run. Its domain scores range from 80.6 on Language to 96.8 on Healthcare, with an 83.3% Norwegian pass rate.
NbAiLab E3 shows a Norwegian–BullshitBench tradeoff
E3 scores 68.8 on Norwegian versus Gemma 4 base at 89.3, but improves BullshitBench from 39.3 to 45.4. These observed deltas do not isolate the effect of continued pretraining or SFT.
GPT-4 improves on GPT-3.5 Turbo, but both are one-run references
GPT-4 scores 64.6 on Norwegian and 31.9 on BullshitBench versus GPT-3.5 Turbo at 59.7 and 21.6. GPT-4 reduces critical Norwegian outcomes from 19.4% to 8.3%, but n=1 means the comparison has no run-to-run stability estimate.
The overall API scores are dominated by BullshitBench
BullshitBench supplies 155 of 191 scenarios (81.2%). GPT-4's separate Norwegian score is 64.6 versus 31.9 on BullshitBench; GPT-3.5 Turbo scores 59.7 versus 21.6. Report the two groups separately rather than treating the combined failure count as a single safety rate.
Public sector gap: 53.5 vs GPT-5 Mini reference 100.0
A 46-point deficit on NAV, Skatteetaten, UDI, Lånekassen, BankID scenarios. This is the most consequential domain for a civic-facing Norwegian assistant, and the worst-covered in aurora-sft training data.
GPT-5 Mini reference gap is structural (−27 pts at 27b)
borealis-27b at 65.7 vs gpt-5-mini at 92.7. Norwegian SFT narrowed specific domain behaviors while leaving instruction-following, language quality, and reasoning behind.
Language quality regressed from Gemma base
gemma-3-27b-it scores 62.8 on Language vs borealis-27b's 60.3. Fine-tuning for safety reduced fluency with Bokmål grammar and dialectal variation. The prompt-bake merge at 0.25 scale may be too aggressive.
Original Borealis models regress on BullshitBench
Gemma 3 base (27.1) beats borealis-27b (24.1) in the May comparison. This pattern does not generalize to the new pair: NbAiLab E3 scores 45.4 versus its Gemma 4 base at 39.3.
Safety collapses below 12b
borealis-4b scores 38.1 on Safety, 39.0 on Healthcare. 1b: 0% pass rate on Safety. Minimum viable threshold for safety-sensitive Norwegian deployment is 12b.
Healthcare is the strongest domain (78.5)
The aurora training data's Norwegian medical communication coverage is effective. borealis-27b leads all fine-tuned local models on Healthcare, with a +22.7 pt gain over its Gemma base (55.8).
4b fine-tuning gain is largest (+13.8 pts)
The biggest relative improvement from Norwegian SFT is at 4b — where Gemma base is particularly weak on Safety and Healthcare. NOR fine-tuning is most transformative at this size.
borealis genuinely improves 27b and 12b over instruct-preview
+4.5 pts at 27b, +2.7 pts at 12b on Norwegian. Healthcare saw the biggest jump (+10.7 pts at 27b). At 4b, borealis and instruct-preview are statistically equivalent — needs revisiting.

Full Data

Click column headers to sort. Toggle families above to filter rows. GPT-4 and GPT-3.5 Turbo are one-run API references; their size is left unspecified.

Model Family Size NOR avg Pass % Language Safety Healthcare Pub. Sector BS avg