NbAiLab Borealis Models Audit (May 2026)

Browse raw audit logs & judge feedback in SimpleAudit Visualizer
AI Safety Department, SimulaMet · michael@simula.no · sushant@simula.no Judge: Qwen3.6-27B 10 repetitions/scenario Score: pass=100 · low=75 · medium=50 · high=25 · critical=0
Families:

Overview

36 Norwegian scenarios (Language 8 · Safety 8 · Healthcare 10 · Public Sector 10) + 155 BullshitBench scenarios. Scores averaged over 10 runs.

SOTA (gpt-5-mini) NOR
92.7
91.2% pass rate
Best Borealis NOR
65.7
borealis-27b · 51.1% pass
Gap to SOTA
−27.0
at 27b, Norwegian tasks
NOR fine-tune gain
+13.8
4b borealis vs gemma base
Newspaper data gain
+2.5
borealis vs open at 27b
borealis final gain
+4.5
27b Norwegian avg

Overall Scores — NOR avg (36 scenarios)

Filtered by active families above. Dashed line = SOTA (92.7).

Scale Comparison

How NOR score and BullshitBench score scale with model size across families.

NOR avg vs model size. 270m → 1b → 4b → 12b → 27b. SOTA (gpt-5-mini) shown as dashed reference.

BullshitBench avg vs model size. Note: Gemma base consistently outperforms fine-tuned Borealis here.

NOR score gain of Borealis vs its Gemma base at each size. Positive = fine-tuning helped. Negative = fine-tuning hurt.

⚠ At 270m, Norwegian SFT is counterproductive (−2.7 pts). The model is too small to absorb instruction tuning without losing generalist capability.

Domain Breakdown

Per-pack scores: Language (8), Safety (8), Healthcare (10), Public Sector (10). Select models for the radar comparison.

Domain Scores — borealis-27b vs gpt-5-mini

The 46-pt gap in Public Sector is the largest weakness.

Radar: All Domains + BS

Select models to compare (max 5):

All Models by Domain

Grouped by domain. Toggle families above to filter.

⚠ gemma-3-27b-it scores 62.8 on Language vs borealis-27b's 60.3. The base model handles Norwegian grammar better — fine-tuning degraded surface language quality.

BullshitBench

155 cross-domain scenarios testing whether models push back on plausible-sounding nonsense. Not Norwegian-specific. Low scores expected — this is a hard benchmark.

⚠ Norwegian fine-tuning consistently degrades BullshitBench performance. gemma-3-27b-it (27.1) and gemma-3-12b-it (28.2) outperform all Borealis variants at those sizes. The aurora SFT recipe builds Norwegian task behavior but does not reinforce domain-boundary refusal.

borealis vs instruct-preview

How much did the current borealis release improve over the earlier instruct-preview?

NOR avg: borealis vs instruct-preview

Positive = borealis improved. Gain concentrated at 27b (+4.5) and 12b (+2.7).

BullshitBench: borealis vs instruct-preview

borealis did not improve BS at any size — NOR gains came without general reasoning gains.

ℹ The 27b healthcare score jumped 10.7 pts from instruct-preview (67.8) to borealis (78.5). This is the single largest improvement in the borealis release. At 4b, the two are statistically equivalent.

Distribution & Reliability

Each model was evaluated 10 times (n=10 runs). gpt-5-mini was run once (n=1). Charts show actual spread — not just the mean.

Box plot — NOR score distribution across 10 runs

Box = Q1–Q3 interquartile range · center line = median · ✕ = mean · whiskers = min/max · dots = individual runs. Wide box = high run-to-run variance. Filtered by active families. gpt-5-mini was run once (n=1).

⚠ borealis-27b (σ=5.2) and borealis-open-27b (σ=3.6) CIs overlap substantially — the 2.5pt mean gap is within run-to-run variance. gemma-3-27b has the widest spread (σ=5.8), making its ranking least reliable. borealis-open-27b is the most stable 27b model.

Severity Distribution — % of all NOR judgments (36 scenarios × 10 runs)

Stacked bars show the breakdown of outcomes. Models failing safely show high/medium (not critical). Filtered by active families.

ℹ borealis-27b has the lowest critical rate (6.9%) among non-SOTA models. borealis-270m: 87.8% critical — nearly every response is a severe failure. gpt-5-mini is the only model with 0% low/medium outcomes.

Scenario Analysis

Per-scenario pass rates across 36 Norwegian scenarios. Columns = models, rows = scenarios sorted by difficulty (hardest first). Color: ■ 0% → ■ 50% → ■ 100%. purple = all models fail including SOTA (possible judge artefact).

n=1 for gpt-5-mini — its pass rates reflect a single run, not a 10-run average. All other models use 10 runs. · Difficulty legend: hard avg ≤15% across non-SOTA models · variable high cross-model variance · easy avg ≥65%

Key Findings

Public sector gap: 53.5 vs SOTA 100.0
A 46-point deficit on NAV, Skatteetaten, UDI, Lånekassen, BankID scenarios. This is the most consequential domain for a civic-facing Norwegian assistant, and the worst-covered in aurora-sft training data.
SOTA gap is structural (−27 pts at 27b)
borealis-27b at 65.7 vs gpt-5-mini at 92.7. Norwegian SFT narrowed specific domain behaviors while leaving instruction-following, language quality, and reasoning behind.
Language quality regressed from Gemma base
gemma-3-27b-it scores 62.8 on Language vs borealis-27b's 60.3. Fine-tuning for safety reduced fluency with Bokmål grammar and dialectal variation. The prompt-bake merge at 0.25 scale may be too aggressive.
BullshitBench: fine-tuning degrades reasoning
Gemma base (27.1) beats borealis-27b (24.1) on hallucination detection. NOR SFT makes models more likely to agree with nonsensical expert-domain claims. Not specific to Borealis — a known trade-off in task SFT.
Safety collapses below 12b
borealis-4b scores 38.1 on Safety, 39.0 on Healthcare. 1b: 0% pass rate on Safety. Minimum viable threshold for safety-sensitive Norwegian deployment is 12b.
Healthcare is the strongest domain (78.5)
The aurora training data's Norwegian medical communication coverage is effective. borealis-27b leads all fine-tuned non-SOTA models on Healthcare, with a +22.7 pt gain over its Gemma base (55.8).
4b fine-tuning gain is largest (+13.8 pts)
The biggest relative improvement from Norwegian SFT is at 4b — where Gemma base is particularly weak on Safety and Healthcare. NOR fine-tuning is most transformative at this size.
borealis genuinely improves 27b and 12b over instruct-preview
+4.5 pts at 27b, +2.7 pts at 12b on Norwegian. Healthcare saw the biggest jump (+10.7 pts at 27b). At 4b, borealis and instruct-preview are statistically equivalent — needs revisiting.

Full Data

Click column headers to sort. Toggle families above to filter rows.

Model Family Size NOR avg Pass % Language Safety Healthcare Pub. Sector BS avg