OpenVoiceOS/ovos-m2v-intents-en

OVOS intent classifier, English

Static-embedding intent classifiers for OpenVoiceOS, trained on English only. Each maps an utterance to one of 217 intent labels of the form <skill_id>:<intent_name>, the names the OVOS m2v pipeline registers at runtime.

The repository holds one model for each size of the potion English base. The root holds the most accurate one, on potion-base-32M. Inference is a vector lookup, an average and one linear layer, so every size runs on a CPU.

Which model

Folder	Base	Format	Size	Gold, all (1,500 rows)	Gold, not needs_manual (472 rows)
root	potion-base-32M	model2vec, float32	126 MB	0.8767	0.8581
fp16	potion-base-32M	model2vec, float16	65 MB	0.8767	0.8581
int8	potion-base-32M	model2vec, int8	34 MB	0.8753	0.8538
potion-base-8M	potion-base-8M	model2vec, float32	31 MB	0.8680	0.8475
potion-base-4M	potion-base-4M	model2vec, float32	16 MB	0.8473	0.8305
potion-base-2M	potion-base-2M	model2vec, float32	8 MB	0.7940	0.7691

For comparison, the multilingual model ovos-m2v-intents-multilingual scores 0.8400 on all 1,506 English gold rows at 532 MB.

The int8 folder is the best choice when size matters: it loses little against the root model at a quarter of its size, and beats potion-base-8M at about the same size. The root model has an ONNX copy in onnx/, and each smaller base one in <folder>/onnx/.

Use

The OVOS m2v pipeline loads the multilingual model by default. To use this one for English, set models (or model for a device that only speaks English), and subfolder for any folder other than the root:

{"intents": {"ovos-m2v-pipeline": {
  "models": {"en": "OpenVoiceOS/ovos-m2v-intents-en"},
  "subfolder": "int8"}}}

subfolder applies to every model the pipeline loads, so set it only when all of them have that folder.

In Python, download one folder and load it:

from huggingface_hub import snapshot_download
from model2vec.inference import StaticModelPipeline

path = snapshot_download("OpenVoiceOS/ovos-m2v-intents-en", allow_patterns=["int8/*"])
model = StaticModelPipeline.from_pretrained(f"{path}/int8")
model.predict(["set a timer for ten minutes"])

For the root model, StaticModelPipeline.from_pretrained("OpenVoiceOS/ovos-m2v-intents-en").

ONNX

onnx/model.onnx runs without model2vec. It takes input_ids and attention_mask (int64, batch by tokens) and returns probabilities (float32, batch by 217), in the order of head_config.classes in onnx/config.json. Tokenize with onnx/tokenizer.json without special tokens, drop the unknown token, and keep at most 512 tokens:

import json
import numpy as np
import onnxruntime as ort
from tokenizers import Tokenizer

tok = Tokenizer.from_file("onnx/tokenizer.json")
classes = json.load(open("onnx/config.json"))["head_config"]["classes"]
unk = tok.token_to_id("[UNK]")
ids = [i for i in tok.encode("set a timer for ten minutes", add_special_tokens=False).ids if i != unk][:512]
sess = ort.InferenceSession("onnx/model.onnx")
probs = sess.run(None, {"input_ids": np.array([ids]), "attention_mask": np.ones((1, len(ids)), np.int64)})[0]
print(classes[probs[0].argmax()])

On 1,513 sentences each ONNX copy gives the same top label as its model2vec model.

The previous model

The previous English model, on potion-base-8M with 209 labels in the older label names, is revision f67cb4bcc411e10658a3cf00db8fdaf203791341. Set "revision" to that value to load it. On the 637 English gold rows whose label both models know (83 labels), the root model here scores 0.8823 and the previous model 0.8226.

Training

Training data	OpenVoiceOS/ovos-intents, revision v7-preview2 (commit e03197a), en-US rows only
Rows	737,362 English rows, reduced to 291,585: at most 10,000 rows per label, drawn evenly across the label's templates, and labels under 100 rows repeated up to 100
Base models	potion-base-32M 1e5a03f8, potion-base-8M bf8b0566, potion-base-4M 9b3cff41, potion-base-2M 389b9f64
Classifier	model2vec StaticModelForClassification, embeddings trained with the head
Optimiser	AdamW, learning rate 0.001, cross-entropy loss
Batch and epochs	batch 512, 8 epochs, the whole training set shuffled in every epoch, seed 13
Held-out split	10% of each label's templates, chosen by a hash of the template text; every row of a held-out template is left out of training, and a label keeps at least one training template

Training used 8,020 templates, 265,979 rows; 825 templates, 25,578 rows, were held out. The held-out accuracy is 0.9985 for potion-base-32M, 0.9925 for 8M, 0.9905 for 4M and 0.9895 for 2M.

Results

The gold sentences are the skills' own golden utterances, not generated from the templates the models train on. 1,506 are English; 6 have a label with no English training row and are not scored, which leaves 1,500. 1,028 of them are flagged needs_manual and are scored like the others.

The settings (row cap, minimum, epochs, base) were chosen on one half of the English gold rows, so the gold figures above are slightly optimistic.

Limits

• The gold sentences are model-written. Every gold sentence was written by a language model. needs_manual marks rows meant for a human check, and whether that check happened is not recorded, so the gold figures measure agreement with model-written sentences, not with recorded users.
• Classifier mode only. These models are measured as classifiers. Prototype mode in the OVOS pipeline was not measured with them.

License

Apache-2.0. The base models are MIT.