Stage verified Inflect v2 release candidate and evaluation evidence

.gitattributes

@@ -33,11 +33,9 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+assets/inflect-v2-repository-hero.png filter=lfs diff=lfs merge=lfs -text
 samples/male/conversational.wav filter=lfs diff=lfs merge=lfs -text
 samples/male/names_places.wav filter=lfs diff=lfs merge=lfs -text
 samples/male/numbers.wav filter=lfs diff=lfs merge=lfs -text
 samples/male/punctuation.wav filter=lfs diff=lfs merge=lfs -text
 samples/male/technical.wav filter=lfs diff=lfs merge=lfs -text
-smoke/release_smoke.wav filter=lfs diff=lfs merge=lfs -text
-smoke/standalone_cpu.wav filter=lfs diff=lfs merge=lfs -text
-assets/inflect-v2-repository-hero.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -30,8 +30,8 @@ inference: false
   <a href="https://huggingface.co/spaces/owensong/Inflect-v2"><img alt="Live playground" src="https://img.shields.io/badge/Playground-0B2A54?style=for-the-badge&logo=huggingface&logoColor=white"></a>
   <a href="https://github.com/owenawsong/Inflect"><img alt="GitHub" src="https://img.shields.io/badge/GitHub-1769E0?style=for-the-badge&logo=github&logoColor=white"></a>
   <a href="https://huggingface.co/owensong/Inflect-Nano-v2"><img alt="Inflect Nano v2" src="https://img.shields.io/badge/Nano_3.97M-FFFFFF?style=for-the-badge&logo=huggingface&logoColor=1769E0"></a>
-  <a href="docs/FINETUNING.md"><img alt="Fine-tuning guide" src="https://img.shields.io/badge/Fine--tuning_Guide-1769E0?style=for-the-badge&logo=readthedocs&logoColor=white"></a>
-  <a href="https://discord.gg/CVJYedvzvp"><img alt="Inflect Discord" src="https://img.shields.io/badge/Discord-0B2A54?style=for-the-badge&logo=discord&logoColor=white"></a>
+  <a href="https://discord.gg/CVJYedvzvp"><img alt="Inflect Discord" src="https://img.shields.io/badge/Discord-1769E0?style=for-the-badge&logo=discord&logoColor=white"></a>
+  <a href="docs/EVALUATION.md"><img alt="Benchmarks" src="https://img.shields.io/badge/Benchmarks-0B2A54?style=for-the-badge&logo=chartdotjs&logoColor=white"></a>
 </p>
 
 <p align="center"><strong>9.36M complete parameters</strong> · <strong>37.53 MB FP32 weights</strong> · <strong>one fixed English voice</strong> · <strong>no external vocoder</strong></p>

docs/EVALUATION.md

@@ -1,47 +1,234 @@
-# Final evaluation protocol
+# Inflect v2 evaluation and benchmark report
 
-## Matched corpus
+This page is the detailed evidence record for
+[Inflect-Micro-v2](https://huggingface.co/owensong/Inflect-Micro-v2) and
+[Inflect-Nano-v2](https://huggingface.co/owensong/Inflect-Nano-v2). The model
+cards show the headline results; this document defines the datasets, systems,
+normalization, uncertainty, exclusions, runtime boundaries, and raw artifacts
+behind those results.
 
-Modern400 intelligibility results use 400 identical unseen English prompts per system. The set combines 200 fixed modern/stress prompts with 200 deterministic Google FLEURS `en_us` test prompts. Exact normalized text was excluded against 87,362 training transcripts. The frozen prompt manifest, corpus hash, exclusion report, compact summary, and compressed row-level reports are included under `evaluation/final/modern400/`. UTMOS22 remains a separate matched quality evaluation and is not merged with Modern400.
+> **Interpret results by axis, not as one universal score.** Intelligibility,
+> predicted naturalness, human preference, model footprint, and runtime measure
+> different properties. Inflect does not combine them into a proprietary
+> aggregate.
 
-## Systems
+## Evidence at a glance
 
-- Inflect-Micro-v2 and Inflect-Nano-v2, fixed release checkpoints.
-- KittenTTS Nano, Bruno and Hugo reported separately.
-- Piper low, Ryan and Danny reported separately.
-- Supertonic 3, James (`M2` preset) at 3 and 8 steps, reported separately.
+| Release | Complete parameters | FP32 weights | UTMOS22 ↑ | Two-ASR semantic WER ↓ | Community preference ↑ |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| **Inflect-Micro-v2** | **9,356,513** | **37.53 MB** | **4.395** | **3.99%** | **66.2%** |
+| **Inflect-Nano-v2** | **3,966,721** | **15.97 MB** | **4.386** | **4.21%** | **63.9%** |
 
-Kitten and Piper voices are reported separately. The two Supertonic step counts are never pooled.
+![Community blind listening](../assets/evidence/human-preference.svg)
 
-## Intelligibility
+![Predicted quality versus footprint](../assets/evidence/quality-vs-footprint.svg)
 
-Semantic WER is run independently with Qwen3-ASR-1.7B, Nemotron 3.5 ASR 0.6B, and Whisper large-v3. Every input is explicitly resampled to each recognizer's required rate before inference. References and hypotheses are normalized into equivalent spoken English forms so formatting differences such as `12.6` versus `twelve point six` are not automatically counted as speech errors. Reports include every hypothesis, per-clip errors, prompt-category breakdowns, and 10,000-sample bootstrap intervals.
-
-`ASR avg.` is the unweighted arithmetic mean of the three corpus-level semantic WERs. Its interval uses matched-prompt bootstrap resampling. This average is descriptive rather than a replacement for individual ASR results because recognizers can share biases or fail catastrophically on isolated clips.
+![Two-ASR semantic WER](../assets/evidence/asr-consensus.svg)
 
-## Predicted quality
+## Comparator policy
 
-UTMOS22 is run over every generated clip with 10,000-sample bootstrap intervals. It is a learned quality predictor, not human MOS, and is reported separately from WER and human preference.
+The comparison set intentionally uses serious compact or on-device systems:
+KittenTTS Nano, Piper Low, and Supertonic 3. It does not use deliberately weak
+baselines. Every comparator in this package-level study has a larger deployable
+weight footprint than both Inflect releases. That makes the comparison
+demanding, but it does not mean one benchmark proves universal superiority.
 
-## Runtime
-
-The release CPU snapshot uses an AMD Ryzen 9 3900X (12 physical cores, 24
-logical processors), a fixed 12-thread policy, the same 48-prompt subset, three
-warmups, one isolated process per system, and separate model-load timing.
-`CUDA_VISIBLE_DEVICES` is empty; active providers are captured in every report.
-The timed boundary includes each public text frontend and complete waveform
-generation. RTF, audio-seconds per wall-second, median latency, p95 latency, and
-per-prompt rows are retained.
-
-This remains **host-specific engineering evidence**, not a universal speed
-ranking. Frameworks use different execution engines and output sample rates.
-The raw reports disclose those boundaries so results can be reproduced rather
-than silently generalized to unrelated hardware.
+Voice variants are separate rows when voice identity can affect quality or
+intelligibility. Kitten and Piper may be shown as equal-weight two-voice means
+in family-level figures. Supertonic 3-step and 8-step are never pooled.
 
-## Human evidence
+## Complete automated result matrix
 
-The community study was blind, randomized left/right order, and stored pairwise preferences rather than user identity. Preference rate is `(wins + 0.5 × ties) / appearances`. It is descriptive community evidence, not a formal laboratory MOS study.
+All systems received matched text within each benchmark. Lower WER is better;
+higher UTMOS22 is better.
 
-## Reporting rules
+| System / voice | Complete weights ↓ | UTMOS22 ↑ | Qwen3-ASR WER ↓ | Nemotron 3.5 WER ↓ | Whisper large-v3 WER ↓ | Two-ASR consensus ↓ |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| Inflect-Micro-v2 | 37.53 MB | **4.395** | 2.52% | 5.45% | 2.73% | **3.99%** |
+| Inflect-Nano-v2 | 15.97 MB | **4.386** | 2.78% | 5.63% | 2.65% | **4.21%** |
+| KittenTTS Nano · Bruno | 56.77 MB | **4.217** | 2.15% | 3.96% | 2.17% | **3.06%** |
+| KittenTTS Nano · Hugo | 56.77 MB | **4.191** | 2.39% | 3.80% | 2.11% | **3.09%** |
+| Piper Low · Danny | 63.10 MB | **4.195** | 2.62% | 5.60% | 2.55% | **4.11%** |
+| Piper Low · Ryan | 63.10 MB | **4.289** | 2.81% | 5.51% | 2.87% | **4.16%** |
+| Supertonic 3 · M2 · 8-step | 398.08 MB | **4.295** | 2.05% | 3.56% | 8.08% | **2.81%** |
+| Supertonic 3 · M2 · 3-step | 398.08 MB | **2.471** | 3.03% | 6.04% | 3.22% | **4.54%** |
 
-No aggregate score combines WER, UTMOS22, speed, size, and human preference. Confidence intervals, raw reports, model weight footprint, and known limitations accompany headline values.
+The two-ASR consensus is the unweighted mean of Qwen3-ASR and Nemotron 3.5
+corpus-level semantic WER. Whisper remains visible but is excluded from the
+headline consensus for every system because its Supertonic 8-step result
+contains insertion-heavy outliers that conflict with the other recognizers and
+human listening. This rule is applied globally, not selectively to Inflect.
+
+![Three-ASR intelligibility audit](../assets/evidence/modern400-three-asr.svg)
+
+## Uncertainty for the Inflect releases
+
+Intervals use 10,000 matched-prompt bootstrap samples.
+
+| Release | UTMOS22 mean (95% CI) | Qwen3-ASR WER (95% CI) | Nemotron WER (95% CI) | Whisper WER (95% CI) |
+| --- | ---: | ---: | ---: | ---: |
+| Inflect-Micro-v2 | 4.395 (4.381–4.408) | 2.52% (1.88%–3.24%) | 5.45% (4.50%–6.50%) | 2.73% (2.05%–3.49%) |
+| Inflect-Nano-v2 | 4.386 (4.372–4.399) | 2.78% (2.13%–3.52%) | 5.63% (4.72%–6.62%) | 2.65% (2.03%–3.35%) |
+
+## Modern400 intelligibility protocol
+
+- **400 identical unseen English prompts per system**
+- **3,200 generated clips** across eight system/voice configurations
+- 200 manually constructed modern and stress prompts
+- 200 deterministic Google FLEURS `en_us` test prompts
+- exact normalized-text exclusion against 87,362 training transcripts
+- deterministic frontend and inference settings within each system
+- explicit resampling to every recognizer's required sample rate
+- corpus-level WER plus matched-prompt bootstrap intervals
+
+The three independent recognizers are:
+
+| Evaluator | Model |
+| --- | --- |
+| Qwen3-ASR | `Qwen/Qwen3-ASR-1.7B` |
+| Nemotron 3.5 ASR | `nvidia/nemotron-3.5-asr-streaming-0.6b` |
+| Whisper | `openai/whisper-large-v3` |
+
+### Semantic normalization
+
+Raw orthographic WER is not sufficient for TTS. A recognizer may transcribe
+“twelve point six” as `12.6` or “July twenty-first” as `July 21st` without the
+audio being wrong. The semantic normalizer canonicalizes equivalent number,
+currency, date, time, unit, abbreviation, and punctuation forms before scoring.
+The release retains both raw and semantic hypotheses so normalization cannot
+silently hide errors.
+
+### Where Inflect errors remain
+
+The following diagnostic uses the same two-ASR consensus as the headline and
+does not introduce a new score.
+
+| Prompt category | Micro WER ↓ | Nano WER ↓ |
+| --- | ---: | ---: |
+| Names Places | 16.67% | 17.54% |
+| Short | 4.29% | 9.05% |
+| Rare Words | 7.89% | 7.89% |
+| Numbers | 6.25% | 4.69% |
+| Homographs | 3.57% | 5.71% |
+| External Heldout | 5.39% | 5.48% |
+| Technical | 4.10% | 3.12% |
+| Narrative | 0.57% | 1.56% |
+| Punctuation | 0.00% | 1.54% |
+| Long Range | 0.83% | 1.24% |
+| Conversational | 0.49% | 1.13% |
+| Descriptive | 0.44% | 0.88% |
+| Long | 0.81% | 0.58% |
+| Emotional | 0.15% | 0.30% |
+| Question | 0.00% | 0.00% |
+
+![Prompt-category semantic WER](../assets/evidence/category-semantic-wer.svg)
+
+## UTMOS22 predicted-naturalness protocol
+
+- **500 identical unseen prompts per system**
+- `tarepan/SpeechMOS` v1.2.0
+- one score per generated clip
+- mean, median, standard deviation, percentiles, extrema, and 95% bootstrap CI
+- Kitten and Piper family figures use equal-weight means across their two voices
+
+UTMOS22 is a learned non-intrusive quality predictor. It is useful for
+large-scale screening but is **not human MOS**, can be sensitive to voice and
+recording characteristics, and must be interpreted beside intelligibility and
+listening evidence.
+
+## Community blind listening
+
+The final community study stored anonymous pairwise choices rather than user
+identity. Model names were hidden during voting, left/right order was
+randomized, and a tie counts as half a win. The dataset contains **147 pairwise
+decisions**; it does not claim a laboratory-controlled listener panel.
+
+| System / voice | Wins | Losses | Ties | Appearances | Preference rate ↑ |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| KittenTTS Nano · Hugo | 22 | 8 | 3 | 33 | **71.2%** |
+| Inflect-Micro-v2 | 21 | 10 | 3 | 34 | **66.2%** |
+| Inflect-Nano-v2 | 22 | 12 | 2 | 36 | **63.9%** |
+| Supertonic 3 · James · 8-step | 16 | 12 | 3 | 31 | **56.5%** |
+| KittenTTS Nano · Bruno | 16 | 13 | 7 | 36 | **54.2%** |
+| Piper Low · Danny | 10 | 8 | 6 | 24 | **54.2%** |
+| Piper Low · Ryan | 14 | 13 | 6 | 33 | **51.5%** |
+| Inflect-Nano-v1 | 7 | 22 | 3 | 32 | **26.6%** |
+| Supertonic 3 · James · 3-step | 1 | 29 | 2 | 32 | **6.2%** |
+
+Preference rate is `(wins + 0.5 × ties) / appearances`. Unequal appearances
+mean this table is descriptive; it should not be treated as a perfectly
+balanced tournament or formal MOS.
+
+## Runtime and throughput
+
+### Current publication status
+
+The previous Ryzen 9 3900X comparison is withdrawn. The host was under sustained
+background saturation and the harness did not enforce the same thread policy
+inside every PyTorch and ONNX runtime. Those numbers are not release evidence.
+
+A replacement CPU benchmark must pass all of these gates before publication:
+
+1. named, idle host with sustained preflight load below the declared threshold;
+2. one isolated process per system;
+3. identical frozen prompt order;
+4. fixed warmup count and fixed thread budget;
+5. explicit PyTorch and ONNX intra/inter-op thread configuration;
+6. end-to-end timing including frontend and waveform generation;
+7. raw per-utterance latency, audio duration, RTF, provider, memory, and host load;
+8. repeated runs with median and dispersion, not one lucky pass.
+
+The public runtime also collapses training-time weight normalization once,
+immediately after loading the checkpoint. A fixed-seed regression test produced
+bit-for-bit identical waveforms before and after this reparameterization. The
+downloaded checkpoints retain **9,356,513** and **3,966,721** complete
+parameters; the optimized live inference graphs contain **9,344,753** and
+**3,958,801** parameters respectively. This is a deployment optimization, not
+a smaller checkpoint or a changed acoustic model.
+
+### Retained CUDA engineering snapshot
+
+The package retains a 50-prompt CUDA snapshot for the two Inflect releases. It
+is useful for regression checking but is not a cross-system leaderboard because
+the archived report does not identify the GPU model.
+
+| Release | Prompts | RTF ↓ | Audio / wall time ↑ | Median latency | p95 latency |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Inflect-Micro-v2 | 50 | 0.0174 | 57.49× | 78.6 ms | 82.7 ms |
+| Inflect-Nano-v2 | 50 | 0.0171 | 58.41× | 80.2 ms | 85.3 ms |
+
+## Complete model footprint
+
+![Complete model weight footprint](../assets/evidence/model-footprint.svg)
+
+“Complete” means the downloadable weight file required for text-to-waveform
+generation, including Inflect's integrated waveform decoder. Voice variants
+that share identical weights are counted once in family-level footprint
+figures. Package size is not the same as peak runtime memory.
+
+## Frozen artifacts and reproducibility
+
+| Artifact | Purpose |
+| --- | --- |
+| [`evaluation/final/modern400/prompts.jsonl`](../evaluation/final/modern400/prompts.jsonl) | Frozen Modern400 prompt manifest |
+| [`evaluation/final/modern400/summary.json`](../evaluation/final/modern400/summary.json) | Three-ASR summaries, intervals, and category results |
+| [`evaluation/final/modern400/exclusion_report.json`](../evaluation/final/modern400/exclusion_report.json) | Exact-text training exclusion audit |
+| [`evaluation/final/modern400/`](../evaluation/final/modern400/) | Compressed row-level hypotheses and scoring reports |
+| [`evaluation/final/utmos22/report.json`](../evaluation/final/utmos22/report.json) | UTMOS22 clip scores and distributions |
+| [`evaluation/final/runtime_snapshot/`](../evaluation/final/runtime_snapshot/) | Retained engineering snapshots |
+| [`release_manifest.json`](../release_manifest.json) | File sizes and SHA-256 integrity hashes |
+
+- Modern400 prompt SHA-256: `b7504ce2dce44a2da82770a6a5dfd2a034fe17e2113980f8a69663ade417a34c`
+- UTMOS22 prompt SHA-256: `5b7aa33852e6c738eb0e1dbf9fcfcc90772b97ad219167e44aaf3917d063b056`
+- Modern400 bootstrap samples: `10,000`
+- UTMOS22 bootstrap samples: `10,000`
+
+## Interpretation limits
+
+- ASR WER measures recognizer agreement with intended text, not naturalness.
+- UTMOS22 predicts perceived quality; it is not a substitute for human MOS.
+- Community preference is valuable human evidence but is not a controlled lab study.
+- One fixed voice per release limits conclusions about speaker diversity.
+- Runtime depends on hardware, framework, provider, compilation, thread policy, and host load.
+- Results apply to these frozen checkpoints, frontends, prompts, and inference settings.
+- No benchmark result should be presented as proof that one model is best for every deployment.

inference.py

@@ -1,9 +1,12 @@
 from __future__ import annotations
 
 import argparse
+import contextlib
+import io
 import logging
 import re
 import sys
+import warnings
 from pathlib import Path
 
 import numpy as np
@@ -73,17 +76,33 @@ def edge_fade(waveform: np.ndarray, sample_rate: int, milliseconds: float = 5.0)
     return output
 
 
+def optimize_for_inference(model: SynthesizerTrn) -> None:
+    """Collapse training-time weight normalization without changing outputs."""
+    with contextlib.redirect_stdout(io.StringIO()):
+        model.dec.remove_weight_norm()
+    for flow in model.flow.flows:
+        encoder = getattr(flow, "enc", None)
+        if encoder is not None and hasattr(encoder, "remove_weight_norm"):
+            encoder.remove_weight_norm()
+
+
 class InflectTTS:
     def __init__(self, model_dir: str | Path = PACKAGE_ROOT, device: str = "cpu") -> None:
         self.root = Path(model_dir).resolve()
         self.device = torch.device(device)
         self.hps = utils.get_hparams_from_file(str(self.root / "config.json"))
-        self.model = SynthesizerTrn(
-            len(symbols),
-            self.hps.data.filter_length // 2 + 1,
-            self.hps.train.segment_size // self.hps.data.hop_length,
-            **self.hps.model,
-        ).to(self.device).eval()
+        with warnings.catch_warnings():
+            warnings.filterwarnings(
+                "ignore",
+                message="`torch.nn.utils.weight_norm` is deprecated",
+                category=FutureWarning,
+            )
+            self.model = SynthesizerTrn(
+                len(symbols),
+                self.hps.data.filter_length // 2 + 1,
+                self.hps.train.segment_size // self.hps.data.hop_length,
+                **self.hps.model,
+            ).to(self.device).eval()
         root_logger = logging.getLogger()
         previous_level = root_logger.level
         try:
@@ -91,6 +110,9 @@ class InflectTTS:
             utils.load_checkpoint(str(self.root / "model.pth"), self.model, None)
         finally:
             root_logger.setLevel(previous_level)
+        self.checkpoint_parameters = sum(parameter.numel() for parameter in self.model.parameters())
+        optimize_for_inference(self.model)
+        self.deployed_parameters = sum(parameter.numel() for parameter in self.model.parameters())
         self.sample_rate = int(self.hps.data.sampling_rate)
 
     def _tokens(self, text: str) -> tuple[torch.Tensor, torch.Tensor]:

release_manifest.json

@@ -89,8 +89,8 @@
     },
     {
       "path": "docs/EVALUATION.md",
-      "bytes": 3246,
-      "sha256": "4041e56250564366ff80ef9278d66bac5139d8833d368264931ba28ffbad7991"
+      "bytes": 11854,
+      "sha256": "b2aaf6f467b60d7414f4a4303e09213c382c3e73063deddb76268068dae62730"
     },
     {
       "path": "docs/EXPORTS.md",
@@ -264,8 +264,8 @@
     },
     {
       "path": "inference.py",
-      "bytes": 6346,
-      "sha256": "bb8b5302b16c35ee04d6729fd73caa95d1e72b2fc50420b1cdec120327f8b672"
+      "bytes": 7311,
+      "sha256": "9bccc990aafa8e363a34269367a034fc69d8216f7317650be0e389bdd545eae3"
     },
     {
       "path": "inflect_nano_v2_frontend.py",
@@ -294,8 +294,8 @@
     },
     {
       "path": "README.md",
-      "bytes": 17214,
-      "sha256": "52ffe98120d0f014f498378941524fb00a57a12de028cc587c6bfa90bc6a10a1"
+      "bytes": 17198,
+      "sha256": "006177225cb135e5f376f7ac7383a93569d30639d3e555c2dfb500d84a04f7d5"
     },
     {
       "path": "requirements-tested.txt",