# Week 4 — Arabic token tax

FLORES+ dev + devtest · 2009 parallel sentences · 11 tokenizers · no special tokens

## Token ratio vs English (same sentences, same tokenizer)

| tokenizer | eng | msa | najdi | mesopotamian | egyptian | levantine_n | levantine_s | moroccan |
|---|---|---|---|---|---|---|---|---|
| o200k | 1.0 | 1.382 | 1.387 | 1.386 | 1.408 | 1.351 | 1.341 | 1.484 |
| qwen3 | 1.0 | 1.629 | 1.633 | 1.622 | 1.636 | 1.581 | 1.55 | 1.695 |
| llama3.2 | 1.0 | 1.656 | 1.663 | 1.649 | 1.649 | 1.593 | 1.576 | 1.706 |
| gemma3 | 1.0 | 1.47 | 1.474 | 1.461 | 1.436 | 1.415 | 1.389 | 1.519 |
| mistral-tekken | 1.0 | 1.245 | 1.25 | 1.262 | 1.288 | 1.236 | 1.229 | 1.37 |
| allam | 1.0 | 0.966 | 0.969 | 0.958 | 0.973 | 0.942 | 0.936 | 1.075 |
| jais | 1.0 | 1.013 | 1.016 | 0.982 | 1.004 | 0.972 | 0.957 | 1.103 |
| jais2 | 1.0 | 1.047 | 1.05 | 1.001 | 1.018 | 1.0 | 0.995 | 1.128 |
| jais3 | 1.0 | 1.05 | 1.054 | 1.033 | 1.048 | 1.012 | 1.006 | 1.14 |
| jais3-0.9b | 1.0 | 1.165 | 1.168 | 1.163 | 1.176 | 1.143 | 1.127 | 1.264 |
| fanar | 1.0 | 1.453 | 1.459 | 1.458 | 1.458 | 1.412 | 1.391 | 1.522 |

## MSA ratio with 95% bootstrap CI and split stability

| tokenizer | ratio_vs_eng | ci95_low | ci95_high | ratio_dev | ratio_devtest | median_sentence_ratio |
|---|---|---|---|---|---|---|
| o200k | 1.382 | 1.372 | 1.392 | 1.382 | 1.381 | 1.37 |
| qwen3 | 1.629 | 1.617 | 1.642 | 1.629 | 1.628 | 1.625 |
| llama3.2 | 1.656 | 1.644 | 1.668 | 1.659 | 1.653 | 1.645 |
| gemma3 | 1.47 | 1.459 | 1.481 | 1.468 | 1.471 | 1.462 |
| mistral-tekken | 1.245 | 1.236 | 1.254 | 1.249 | 1.242 | 1.235 |
| allam | 0.966 | 0.959 | 0.973 | 0.969 | 0.964 | 0.957 |
| jais | 1.013 | 1.005 | 1.021 | 1.015 | 1.011 | 1.0 |
| jais2 | 1.047 | 1.038 | 1.057 | 1.048 | 1.046 | 1.022 |
| jais3 | 1.05 | 1.042 | 1.058 | 1.052 | 1.048 | 1.038 |
| jais3-0.9b | 1.165 | 1.156 | 1.174 | 1.17 | 1.16 | 1.152 |
| fanar | 1.453 | 1.442 | 1.465 | 1.455 | 1.452 | 1.45 |

## Tokens per whitespace word (reference only — Arabic words carry clitics)

| tokenizer | eng | msa | najdi | mesopotamian | egyptian | levantine_n | levantine_s | moroccan |
|---|---|---|---|---|---|---|---|---|
| o200k | 1.223 | 1.886 | 1.889 | 1.982 | 1.932 | 2.016 | 1.904 | 2.078 |
| qwen3 | 1.257 | 2.286 | 2.288 | 2.384 | 2.307 | 2.424 | 2.263 | 2.44 |
| llama3.2 | 1.236 | 2.284 | 2.29 | 2.382 | 2.287 | 2.402 | 2.261 | 2.415 |
| gemma3 | 1.237 | 2.028 | 2.031 | 2.112 | 1.992 | 2.135 | 1.994 | 2.151 |
| mistral-tekken | 1.267 | 1.761 | 1.765 | 1.869 | 1.831 | 1.911 | 1.808 | 1.988 |
| allam | 1.339 | 1.444 | 1.446 | 1.499 | 1.462 | 1.54 | 1.456 | 1.648 |
| jais | 1.268 | 1.433 | 1.436 | 1.455 | 1.428 | 1.503 | 1.408 | 1.6 |
| jais2 | 1.249 | 1.46 | 1.461 | 1.461 | 1.427 | 1.524 | 1.442 | 1.613 |
| jais3 | 1.21 | 1.418 | 1.421 | 1.461 | 1.424 | 1.494 | 1.413 | 1.58 |
| jais3-0.9b | 1.31 | 1.703 | 1.705 | 1.78 | 1.728 | 1.826 | 1.713 | 1.896 |
| fanar | 1.269 | 2.058 | 2.062 | 2.163 | 2.076 | 2.187 | 2.05 | 2.212 |

## Not measured

- none

## Author-written Gulf set (20 items, same meaning in Gulf, MSA, English)

AI-drafted (Claude), reviewed and edited by a native Gulf speaker; not independently sourced. gulf_vs_msa isolates the dialect effect.

| tokenizer | items | gulf_vs_en | gulf_vs_en_ci95 | msa_vs_en | msa_vs_en_ci95 | gulf_vs_msa | gulf_vs_msa_ci95 |
|---|---|---|---|---|---|---|---|
| o200k | 20 | 1.149 | 1.071–1.222 | 1.213 | 1.120–1.313 | 0.947 | 0.889–1.002 |
| qwen3 | 20 | 1.243 | 1.176–1.310 | 1.308 | 1.215–1.405 | 0.951 | 0.893–1.011 |
| llama3.2 | 20 | 1.24 | 1.165–1.307 | 1.39 | 1.296–1.489 | 0.892 | 0.837–0.943 |
| gemma3 | 20 | 1.043 | 0.982–1.102 | 1.193 | 1.107–1.282 | 0.874 | 0.810–0.934 |
| mistral-tekken | 20 | 1.049 | 0.978–1.109 | 1.07 | 1.005–1.139 | 0.981 | 0.916–1.040 |
| allam | 20 | 0.729 | 0.663–0.789 | 0.77 | 0.710–0.827 | 0.947 | 0.891–1.000 |
| jais | 20 | 0.81 | 0.752–0.864 | 0.894 | 0.824–0.952 | 0.907 | 0.842–0.981 |
| jais2 | 20 | 0.819 | 0.764–0.870 | 0.907 | 0.835–0.969 | 0.904 | 0.840–0.978 |
| jais3 | 20 | 0.838 | 0.782–0.890 | 0.874 | 0.807–0.937 | 0.96 | 0.904–1.026 |
| jais3-0.9b | 20 | 0.932 | 0.876–0.990 | 0.994 | 0.939–1.052 | 0.937 | 0.880–0.991 |
| fanar | 20 | 1.099 | 1.022–1.167 | 1.226 | 1.147–1.307 | 0.896 | 0.847–0.947 |

## Environment

```
{
  "date": "2026-10-17",
  "python": "3.9.7",
  "platform": "macOS-10.16-x86_64-i386-64bit",
  "transformers": "4.57.6",
  "tiktoken": "0.14.0"
}
```