**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 21, 2026, 6:06am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlm-r-vs-llama-7b-tokenization%2F172649%2F2 "2026-01-21T06:06:14Z")
hmm…?
* * *
They do fall cleanly into the two main SentencePiece categories.
- **`xlm-roberta-base` (XLM-R)** uses **SentencePiece Unigram (Unigram LM)**.
- **`huggyllama/llama-7b` (LLaMA 7B family)** uses **SentencePiece BPE (Byte-Pair Encoding)**, with **byte fallback** behavior in common implementations. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
No “third unnamed SentencePiece-y algorithm” is needed to explain either model.
* * *
## 1) Background: “SentencePiece” is a library, not one algorithm
**SentencePiece** is a tokenizer toolkit that can _train_ and _apply_ different subword model types. In particular, it implements:
- **BPE** (merge-based)
- **Unigram LM** (probabilistic vocab + best segmentation)
The SentencePiece project states it “implements subword units” including **BPE** and **unigram language model**. ([GitHub](https://github.com/google/sentencepiece "google/sentencepiece: Unsupervised text tokenizer for ... - GitHub"))
The official options list shows `--model_type` can be `unigram`, `bpe`, `word`, or `char`. ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/options.md "sentencepiece/doc/options.md at master"))
So the correct question is exactly what you asked: “is this SentencePiece model Unigram or BPE?”
* * *
## 2) The two algorithms in plain English
### SentencePiece Unigram (Unigram LM)
Think “ **pick the best segmentation** from a fixed vocabulary.”
- Training builds a vocabulary and assigns each token a probability.
- Tokenization finds the **most likely sequence of tokens** that composes the input (often via a Viterbi-style dynamic program).
- Practical effect: it can choose among multiple valid segmentations by likelihood.
SentencePiece explicitly names this “Unigram language model” and exposes it as `--model_type=unigram`. ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/experiments.md "sentencepiece/doc/experiments.md at master"))
### SentencePiece BPE
Think “ **start with characters then repeatedly merge the most useful pairs**.”
- Training starts from smaller symbols and learns merge rules.
- Tokenization applies those merges (greedily, in practice).
SentencePiece exposes this as `--model_type=bpe`. ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/experiments.md "sentencepiece/doc/experiments.md at master"))
### Important nuance for LLaMA-style BPE
Many LLaMA-family tokenizers are described as **BPE with byte fallback**. That means “if a Unicode character is out of vocabulary coverage, fall back to encoding it as bytes so you do not emit ``.” Hugging Face’s LLaMA tokenizer docs call out **ByteFallback** explicitly. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
* * *
## 3) XLM-R (`xlm-roberta-base`) tokenization type
The XLM-R paper is explicit:
- “We apply subword tokenization directly on raw text data using \*\*SentencePiece … with a **unigram language model**.”
- It also notes a **250K vocabulary size** for XLM-R (a key design choice for multilingual coverage).
So XLM-R is not “SentencePiece maybe-BPE.” It is **SentencePiece Unigram**.
### Why it still confuses people
Two common sources of confusion:
1. **Misleading filenames / artifacts**
People run into files named like `sentencepiece.bpe.model` for XLM-R checkpoints and assume “must be BPE,” then hit odd training or loading issues. This exact confusion shows up in community threads. ([PyTorch Forums](https://discuss.pytorch.org/t/train-a-tokenizer-like-xlm-roberta-tokenizer/155641 "Train a tokenizer like XLM Roberta tokenizer - PyTorch Forums"))
2. **“Fast tokenizer” documentation blur**
Hugging Face’s XLM-R docs describe the slow tokenizer as SentencePiece-based, and the fast tokenizer section uses wording like “Based on BPE,” which can be read as “the model is BPE.” That wording is a known source of misunderstanding because the underlying SentencePiece model for XLM-R is Unigram per the paper. ([Hugging Face](https://huggingface.co/docs/transformers/v4.57.2/model_doc/xlm-roberta "XLM-RoBERTa"))
Bottom line: treat the **paper** and/or the **SentencePiece model’s internal `model_type`** as authoritative.
* * *
## 4) LLaMA 7B (`huggyllama/llama-7b`) tokenization type
For LLaMA-family models, the standard tokenizer is **SentencePiece BPE**.
Two strong anchors:
1. **Hugging Face model docs (implementation-facing)**
- “The tokenizer is a **byte-pair encoding model based on SentencePiece**.” ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
- The tokenizer class documentation also states: “Based on **byte-level Byte-Pair-Encoding**. This uses notably **ByteFallback** and **no normalization**.” ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
1. **LLaMA paper (training-facing)**
- The LLaMA paper states they tokenize with the **BPE algorithm** , using the **SentencePiece** implementation. ([arXiv](https://arxiv.org/pdf/2302.13971 "arXiv:2302.13971v1 [cs.CL] 27 Feb 2023"))
So for your specific model ID (`huggyllama/llama-7b`): it is in the **LLaMA 1** lineage and uses the same tokenizer design: **SentencePiece BPE** (with byte fallback behavior commonly referenced in implementations). ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
* * *
## 5) The most reliable way to verify: read `model_type` from the `.model` itself
If you want an answer that is immune to:
- confusing filenames,
- wrapper docs,
- “fast vs slow” implementation details,
then inspect the SentencePiece ModelProto inside the tokenizer model file.
SentencePiece stores an enum like:
- `UNIGRAM = 1`
- `BPE = 2`
in `sentencepiece_model.proto`. ([GitHub](https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto "sentencepiece/src/sentencepiece\_model.proto at master"))
And the options doc confirms the conceptual mapping (`model_type: unigram, bpe, word, char`). ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/options.md "sentencepiece/doc/options.md at master"))
### Practical approach (Python)
Conceptually:
1. download the tokenizer model file (often `sentencepiece.bpe.model` or `tokenizer.model`)
2. parse it as a SentencePiece `ModelProto`
3. read `trainer_spec.model_type`
You can do this with the protobuf definition and a small script. The key is: the proto field is the truth, regardless of filenames. ([GitHub](https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto "sentencepiece/src/sentencepiece\_model.proto at master"))
* * *
## 6) What “another SentencePiece-y algorithm” could mean (and why it does not apply here)
SentencePiece can also operate in:
- `word` mode
- `char` mode
But those are explicitly named, and neither XLM-R nor LLaMA are trained that way in the canonical descriptions. ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/options.md "sentencepiece/doc/options.md at master"))
So your choices for these two models really are the two you listed:
- XLM-R: **Unigram**
- LLaMA: **BPE**
* * *
## 7) Curated high-quality links (primary + practical)
### Primary sources
- XLM-R paper PDF (states SentencePiece + unigram LM; 250K vocab): [https://arxiv.org/pdf/1911.02116](https://arxiv.org/pdf/1911.02116)
- SentencePiece repo (states it implements BPE + unigram LM): [https://github.com/google/sentencepiece](https://github.com/google/sentencepiece) ([GitHub](https://github.com/google/sentencepiece "google/sentencepiece: Unsupervised text tokenizer for ... - GitHub"))
- SentencePiece options (`model_type=unigram|bpe|word|char`): [https://github.com/google/sentencepiece/blob/master/doc/options.md](https://github.com/google/sentencepiece/blob/master/doc/options.md) ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/options.md "sentencepiece/doc/options.md at master"))
- SentencePiece experiments doc (explicitly contrasts `--model_type=bpe` vs `--model_type=unigram`): [https://github.com/google/sentencepiece/blob/master/doc/experiments.md](https://github.com/google/sentencepiece/blob/master/doc/experiments.md) ([GitHub](https://github.com/google/sentencepiece/blob/master/doc/experiments.md "sentencepiece/doc/experiments.md at master"))
- SentencePiece proto enum (`UNIGRAM`, `BPE`): [https://github.com/google/sentencepiece/blob/master/src/sentencepiece\_model.proto](https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto) ([GitHub](https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto "sentencepiece/src/sentencepiece\_model.proto at master"))
### LLaMA implementation-facing docs
- Hugging Face LLaMA docs (states SentencePiece-based BPE; mentions ByteFallback in tokenizer docs): [https://huggingface.co/docs/transformers/en/model\_doc/llama](https://huggingface.co/docs/transformers/en/model_doc/llama) ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
### “Why is XLM-R’s file called `sentencepiece.bpe.model`?” confusion thread
- PyTorch forum thread showing the exact naming confusion and related training pitfalls: [https://discuss.pytorch.org/t/train-a-tokenizer-like-xlm-roberta-tokenizer/155641](https://discuss.pytorch.org/t/train-a-tokenizer-like-xlm-roberta-tokenizer/155641) ([PyTorch Forums](https://discuss.pytorch.org/t/train-a-tokenizer-like-xlm-roberta-tokenizer/155641 "Train a tokenizer like XLM Roberta tokenizer - PyTorch Forums"))
* * *
## Bullet summary
- **XLM-R (`xlm-roberta-base`)**: **SentencePiece Unigram (Unigram LM)**. Paper states it directly.
- **LLaMA 7B (`huggyllama/llama-7b`)**: **SentencePiece BPE** , commonly with **byte fallback** behavior. HF docs state BPE and ByteFallback; paper states BPE + SentencePiece. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
- If you want “cannot be argued with” verification, parse the SentencePiece model file and read `trainer_spec.model_type` from the proto. ([GitHub](https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto "sentencepiece/src/sentencepiece\_model.proto at master"))
---
**Author:**  [@JuliaGecko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJuliaGecko)%5C
**Post date:** [January 31, 2026, 11:12pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlm-r-vs-llama-7b-tokenization%2F172649%2F3 "2026-01-31T23:12:20Z")
ok thank you, but do you know this for sure or only prompted to chatgpt? Thanks in advance!
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [January 31, 2026, 11:43pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlm-r-vs-llama-7b-tokenization%2F172649%2F4 "2026-01-31T23:43:53Z")
This is information I asked ChatGPT for, based on the information I had on hand and the search results I provided. So it’s somewhere in between…? I didn’t try it myself.
---
**Author:**  [@JuliaGecko](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJuliaGecko)%5C
**Post date:** [February 5, 2026, 3:38pm UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlm-r-vs-llama-7b-tokenization%2F172649%2F5 "2026-02-05T15:38:31Z")
ok thank you, I need to dig a bit deeper though as this is quite important for me.
---
**Author:**  [@John6666](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Fu%2FJohn6666)%5C
**Post date:** [February 6, 2026, 4:26am UTC](/static-proxy?url=https%3A%2F%2Fdiscuss.huggingface.co%2Ft%2Fxlm-r-vs-llama-7b-tokenization%2F172649%2F6 "2026-02-06T04:26:05Z")
I tested the actual model’s behavior with code for now. If you need to dig deeper for some reason, I think it’s better to contact the author directly rather than using forums…
* * *
- **`FacebookAI/xlm-roberta-base` uses SentencePiece _Unigram LM_** (not BPE).
- **`huggyllama/llama-7b` uses SentencePiece _BPE_** , with **byte fallback enabled** (and “no normalization/identity” behavior in practice).
There is no “third, unnamed SentencePiece-y algorithm” involved here—these two models fall cleanly into **Unigram** vs **BPE** as defined by SentencePiece itself.
* * *
## Why this is definitive
A SentencePiece tokenizer is stored as a serialized protobuf (`*.model`). That file contains:
- `trainer_spec.model_type` with an enum:
- `trainer_spec.byte_fallback` (whether unknown UTF-8 characters can be decomposed into bytes) ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
- `normalizer_spec.name` (e.g., `nmt_nfkc`, `identity`) ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
- piece types such as `NORMAL`, `CONTROL`, and `BYTE` (the `BYTE` pieces appear when `byte_fallback` is true) ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
So reading `trainer_spec.model_type` from the `.model` file is the ground-truth way to distinguish **SentencePiece Unigram** vs **SentencePiece BPE**. ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
* * *
## Evidence for XLM-R (`xlm-roberta-base`) → **Unigram**
### Paper evidence
The XLM-R paper explicitly says they tokenize raw text using **SentencePiece with a unigram language model** , and they use a **250K vocabulary**.
### What your verifier run shows
Your run printed:
- `trainer_spec.model_type: 1 -> unigram`
- `vocab_size: 250000`
- `byte_fallback: False`
- `normalizer_spec.name: 'nmt_nfkc'`
Those fields map directly to the SentencePiece schema (Unigram = 1). ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
**Note on the confusing filename** : the file is named `sentencepiece.bpe.model` in that repo, but filenames are not authoritative; the embedded `trainer_spec.model_type` is. ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
* * *
## Evidence for LLaMA (`huggyllama/llama-7b`) → **BPE + byte fallback**
### Paper evidence
The LLaMA paper states they tokenize using **the byte-pair encoding (BPE) algorithm**, using **SentencePiece** , and they “fallback to bytes to decompose unknown UTF-8 characters.”
### HF docs evidence (implementation-facing)
Hugging Face’s LLaMA tokenizer docs describe it as **Byte-Pair-Encoding** and note it uses **ByteFallback** and **no normalization**. ([Hugging Face](https://huggingface.co/docs/transformers/en/model_doc/llama "Llama"))
Older HF docs also explicitly say: “The LLaMA tokenizer is a **BPE model based on sentencepiece**.” ([Hugging Face](https://huggingface.co/docs/transformers/v4.34.1/model_doc/llama "LLaMA"))
### What your verifier run shows
Your run printed:
- `trainer_spec.model_type: 2 -> bpe`
- `trainer_spec.byte_fallback: True`
- `piece.type distribution` included `BYTE: 256`
This matches the SentencePiece schema: `BPE = 2`, and `byte_fallback` causes `BYTE` pieces (typically 256 of them, `<0x00>` … `<0xFF>`) to appear. ([GitHub](https://raw.githubusercontent.com/google/sentencepiece/master/src/sentencepiece_model.proto "raw.githubusercontent.com"))
* * *
## Context: what “SentencePiece Unigram” vs “SentencePiece BPE” means
SentencePiece supports both **BPE** and **Unigram LM** tokenization. ([GitHub](https://github.com/google/sentencepiece "GitHub - google/sentencepiece: Unsupervised text tokenizer for Neural Network-based text generation."))
- **Unigram LM** : chooses a segmentation by optimizing a probabilistic model over a vocabulary.
- **BPE** : uses merge rules learned during training to build subword units.
For these two models:
- **XLM-R** : Unigram + large 250K vocab + NFKC-style normalization (`nmt_nfkc`).
- **LLaMA 7B** : BPE + byte fallback + “identity/no normalization” behavior.
* * *
## Bottom line
- **XLM-R (`xlm-roberta-base`)**: **SentencePiece Unigram LM** (`model_type=1`).
- **LLaMA (`huggyllama/llama-7b`)**: **SentencePiece BPE** (`model_type=2`) with **byte fallback**.
* * *
```python
#!/usr/bin/env python3
"""
SentencePiece model-type verifier (Unigram vs BPE) for Hugging Face repos.
CPU/GPU safety:
- Does NOT load any model weights (no torch import), only downloads the tokenizer *.model (typically KB–MB).
Deps:
pip install -U "huggingface_hub>=0.20" "sentencepiece>=0.1.99" "protobuf>=4.21"
Reference URLs:
- SentencePiece ModelProto schema (defines trainer_spec.model_type enum UNIGRAM=1, BPE=2, etc.):
https://github.com/google/sentencepiece/blob/master/src/sentencepiece_model.proto
- hf_hub_download docs (download a single file from a repo):
https://huggingface.co/docs/huggingface_hub/en/package_reference/file_download
"""
from __future__ import annotations
from collections import Counter
from pathlib import Path
from huggingface_hub import hf_hub_download, list_repo_files
from huggingface_hub.utils import EntryNotFoundError, GatedRepoError, RepositoryNotFoundError
# sentencepiece ships the protobuf bindings used to parse *.model into ModelProto
try:
import sentencepiece.sentencepiece_model_pb2 as sp_pb2 # type: ignore
except Exception:
# fallback (some environments expose it differently)
from sentencepiece import sentencepiece_model_pb2 as sp_pb2 # type: ignore
# Model types per sentencepiece_model.proto
MODEL_TYPE = {1: "unigram", 2: "bpe", 3: "word", 4: "char"}
# Piece types per sentencepiece_model.proto
PIECE_TYPE = {1: "NORMAL", 2: "UNKNOWN", 3: "CONTROL", 4: "USER_DEFINED", 5: "UNUSED", 6: "BYTE"}
def download_spm(repo_id: str, revision: str | None = None) -> Path:
"""
Download the SentencePiece *.model file from a repo (tries common filenames first,
otherwise finds any *.model in repo listing). Returns local cached path.
"""
candidates = (
"tokenizer.model",
"sentencepiece.model",
"spiece.model",
"sentencepiece.bpe.model",
)
for name in candidates:
try:
return Path(hf_hub_download(repo_id=repo_id, filename=name, revision=revision))
except EntryNotFoundError:
pass
# Fallback: list files and pick a *.model
files = list_repo_files(repo_id=repo_id, revision=revision)
model_files = [f for f in files if f.endswith(".model")]
if not model_files:
raise FileNotFoundError(f"No *.model tokenizer file found in repo {repo_id}")
# Prefer likely names if present
for pref in candidates:
if pref in model_files:
return Path(hf_hub_download(repo_id=repo_id, filename=pref, revision=revision))
# Otherwise just take the first *.model
return Path(hf_hub_download(repo_id=repo_id, filename=model_files[0], revision=revision))
def inspect_spm(model_path: Path) -> dict:
"""Parse ModelProto and return a compact summary."""
mp = sp_pb2.ModelProto()
mp.ParseFromString(model_path.read_bytes())
model_type_id = int(getattr(mp.trainer_spec, "model_type", 0))
model_type_name = MODEL_TYPE.get(model_type_id, f"unknown({model_type_id})")
byte_fallback = bool(getattr(mp.trainer_spec, "byte_fallback", False))
vocab_size = len(mp.pieces)
# Count piece.type values (NORMAL, CONTROL, BYTE, ...)
type_counts = Counter(int(p.type) for p in mp.pieces)
type_counts_named = {PIECE_TYPE.get(k, str(k)): v for k, v in sorted(type_counts.items())}
normalizer_name = getattr(mp.normalizer_spec, "name", "")
return {
"model_path": str(model_path),
"model_type_id": model_type_id,
"model_type": model_type_name,
"byte_fallback": byte_fallback,
"vocab_size": vocab_size,
"piece_type_counts": type_counts_named,
"normalizer": normalizer_name,
}
def verify_repo(repo_id: str, revision: str | None = None) -> None:
"""Download tokenizer *.model, parse it, and print key fields."""
print(f"Repo: {repo_id}")
try:
model_path = download_spm(repo_id, revision=revision)
info = inspect_spm(model_path)
print(f" tokenizer file: {info['model_path']}")
print(f" trainer_spec.model_type: {info['model_type_id']} -> {info['model_type']}")
print(f" trainer_spec.byte_fallback: {info['byte_fallback']}")
print(f" vocab_size (len(pieces)): {info['vocab_size']}")
print(f" normalizer_spec.name: {info['normalizer']!r}")
print(f" piece.type distribution: {info['piece_type_counts']}")
except GatedRepoError:
print(" ERROR: repo is gated. Set HF_TOKEN or accept the repo terms on Hugging Face.")
except RepositoryNotFoundError:
print(" ERROR: repo not found (or private without access).")
except Exception as e:
print(f" ERROR: {type(e). __name__ }: {e}")
print()
def main() -> None:
# Public examples discussed:
verify_repo("FacebookAI/xlm-roberta-base")
verify_repo("huggyllama/llama-7b")
# Optional: verify any local SentencePiece model file:
# local = Path("/path/to/tokenizer.model")
# print(inspect_spm(local))
if __name__ == " __main__":
main()
"""
Repo: FacebookAI/xlm-roberta-base
tokenizer file: /root/.cache/huggingface/hub/models--FacebookAI--xlm-roberta-base/snapshots/e73636d4f797dec63c3081bb6ed5c7b0bb3f2089/sentencepiece.bpe.model
trainer_spec.model_type: 1 -> unigram
trainer_spec.byte_fallback: False
vocab_size (len(pieces)): 250000
normalizer_spec.name: 'nmt_nfkc'
piece.type distribution: {'NORMAL': 249997, 'UNKNOWN': 1, 'CONTROL': 2}
Repo: huggyllama/llama-7b
tokenizer file: /root/.cache/huggingface/hub/models--huggyllama--llama-7b/snapshots/4782ad278652c7c71b72204d462d6d01eaaf7549/tokenizer.model
trainer_spec.model_type: 2 -> bpe
trainer_spec.byte_fallback: True
vocab_size (len(pieces)): 32000
normalizer_spec.name: 'identity'
piece.type distribution: {'NORMAL': 31741, 'UNKNOWN': 1, 'CONTROL': 2, 'BYTE': 256}
"""
```