145 lines
12 KiB
Markdown
145 lines
12 KiB
Markdown
# Teddy voice experiment
|
|
|
|
This experiment is separate from the live `.speech` Piper installation. The first GPU fine-tuning experiment has now been launched in Ubuntu. The Piper experiment is retained for comparison; the live lab now defaults to Chatterbox Turbo (see below).
|
|
|
|
## Source recording
|
|
|
|
The original files remain at `D:/Projects/escape-gallery/TeddyFull.wav` and `teddy-transcript.txt`. Working copies and checksums are in `.training/teddy-v1/`. The source is 387.898 seconds, mono, 44.1 kHz, 16-bit PCM, with 1,107 whitespace-separated transcript words. `.training/` is ignored by Git; back it up separately if needed.
|
|
|
|
## Prepare the dataset
|
|
|
|
From this lab directory in PowerShell:
|
|
|
|
```powershell
|
|
.training/venv/Scripts/python.exe -X utf8 prepare_teddy_dataset.py
|
|
```
|
|
|
|
The preparation environment contains faster-whisper 1.2.1 and scipy. It downloads the small.en speech recognition model and processes the audio locally on CPU. It does not upload recordings. It caches word timestamps, compares normalized recognized words with the supplied transcript, and exports only fully matched sentences lasting 0.7 to 15 seconds as mono 22.05 kHz PCM.
|
|
|
|
Outputs in `.training/teddy-v1/`:
|
|
|
|
- `asr-words.json`: recognition timestamps.
|
|
- `audio/`: candidate sentence clips.
|
|
- `metadata.draft.csv`: candidate filename/text pairs for Piper.
|
|
- `alignment-report.json`: accepted boundaries, excluded sentences, and source clipping statistics.
|
|
|
|
These are draft ASR-derived boundaries, not verified forced alignment. Listen to the clips, especially beginnings and endings, and reconcile exclusions against the actual recording. Recognition mismatch does not prove that the actor made a mistake. Do not replace the supplied transcript with recognition guesses without listening. Once reviewed, produce a separate approved metadata file and keep several sentences out of training for evaluation.
|
|
|
|
## Training environment prerequisite
|
|
|
|
The attempted WSL installer returned “Windows Subsystem for Linux is not installed” and exited without installing. Training has not started. In an administrator PowerShell window, use Microsoft's installation procedure:
|
|
|
|
```powershell
|
|
wsl --install -d Ubuntu
|
|
```
|
|
|
|
Restart Windows if prompted (save work and stop the demo first), launch Ubuntu, and complete its initial Linux username/password setup. Then check:
|
|
|
|
```powershell
|
|
wsl --list --verbose
|
|
wsl -d Ubuntu -- nvidia-smi
|
|
```
|
|
|
|
Do not replace the Windows NVIDIA driver with a Linux display driver inside WSL. Next, build an isolated Linux Python environment following Piper's official training guide. Verify CUDA support for the RTX 5070 with a real tensor operation before training; detecting the GPU name alone is insufficient.
|
|
|
|
## First experiment plan
|
|
|
|
1. Review clips and transcript matches; reserve validation sentences.
|
|
2. Select and document a compatible medium-quality training checkpoint and its provenance. The stock ONNX inference file is not a training checkpoint.
|
|
3. Install and pin Piper training code/dependencies in a separate Linux environment; retain code revision and package versions.
|
|
4. Fine-tune with a small GPU batch and a bounded run, retaining checkpoints. Do not assume more epochs improve this small dataset.
|
|
5. Export ONNX plus the JSON configuration created during training.
|
|
6. Synthesize unseen hints and compare intelligibility, voice consistency, and pronunciation with the source and stock voice.
|
|
7. Add explicit custom-model selection to the lab only after validation. Keep the stock model as a fallback.
|
|
|
|
No training checkpoint, validated CUDA environment, approved dataset, or new model is claimed by this setup document.
|
|
|
|
## References
|
|
|
|
- Piper training: https://github.com/OHF-Voice/piper1-gpl/blob/main/docs/TRAINING.md
|
|
- Windows WSL installation: https://learn.microsoft.com/en-us/windows/wsl/install
|
|
- Local timestamp generation: https://github.com/SYSTRAN/faster-whisper
|
|
|
|
## First preparation result
|
|
|
|
89 draft clips, 294.96 seconds retained. Ten sentences require reconciliation; normalized word match was 90.79 percent. This measures ASR/transcript agreement, not verified accuracy. No full-scale clipped samples were found. All exported clips were checked as mono, 22.05 kHz, 16-bit PCM. Open `.training/teddy-v1/review.html` to listen alongside the proposed labels. This was the initial preparation status; Ubuntu setup is now complete and an experimental run is active as described below.
|
|
|
|
## October 18: Ubuntu training experiment launched
|
|
|
|
The earlier WSL blocker is resolved. Ubuntu is running as `jeff` under WSL 2. The isolated workspace is `/home/jeff/teddy-training`; the live Windows inference environment remains separate.
|
|
|
|
Verified environment: Python 3.12.15, PyTorch 2.14.1+cu130, NVIDIA RTX 5070 (12 GB). A real CUDA matrix multiplication completed successfully. Piper source revision: `619f513580c04dff5ef9a680f6f3f6979ed94d2e`. The source was cloned from the official OHF-Voice repository. Build extensions compiled successfully. Exact packages are recorded in `~/teddy-training/requirements-lock.txt`.
|
|
|
|
Starting weights: `rhasspy/piper-checkpoints/en/en_US/ljspeech/medium/lj-med_1000.ckpt`. The accompanying model card identifies the LJ Speech dataset as public domain. Its card, configuration and checksum are saved under `~/teddy-training/checkpoints`. This is the starting voice, not a claim about the final trained voice.
|
|
|
|
Dataset: 89 ASR-matched draft clips. Eight are explicitly withheld in `data/holdout.csv`; 81 enter Piper's seeded train/validation/test split (71 training, 8 validation, 2 test examples). The ten transcript-mismatch sentences remain excluded. This experiment uses automatically checked clips, not a manually auditioned and approved dataset. Listening review is still required before deployment.
|
|
|
|
Launch from PowerShell (only when no existing run is active):
|
|
|
|
```powershell
|
|
wsl -d Ubuntu -- bash /mnt/d/Projects/escape-gallery/room_glitch/labs/glitch_odin/train_teddy_wsl.sh
|
|
```
|
|
|
|
The first run writes `~/teddy-training/train.log`. It warm-starts matching generator/discriminator weights with fresh optimizers. The log confirmed 784 parameters copied and zero skipped. Batch size is 4, with at most 60 epochs or 1,000 Lightning optimizer steps. This is an initial bounded experiment, not an asserted optimal duration. Checkpoints are under `~/teddy-training/runs/teddy-v1/lightning_logs/`.
|
|
|
|
Check progress from PowerShell:
|
|
|
|
```powershell
|
|
wsl -d Ubuntu -- tail -n 25 /home/jeff/teddy-training/train.log
|
|
wsl -d Ubuntu -- nvidia-smi
|
|
```
|
|
|
|
Do not restart Windows or shut down WSL during training. Checkpoints can support a later explicit resume; rerunning the launch script starts another fresh experiment. Export and listening evaluation have not yet been completed. Do not switch the live hint voice until those checks pass.
|
|
|
|
### Resume after optional MOS callback issue
|
|
|
|
Epoch zero saved successfully, then upstream's optional `val_mos` checkpoint callback raised because MOS scoring was disabled. `train_teddy_entry.py` removes only that callback, retaining mel-loss and last-checkpoint saving. The active run resumed from `version_0/checkpoints/last.ckpt` into `version_1`; all training states were restored. Epochs one and two subsequently saved successfully. Version one's checkpoints are the ongoing experiment. The downloaded upstream source remains unchanged; our wrapper is in the project.
|
|
|
|
To resume the latest version-one checkpoint after a deliberate stop:
|
|
|
|
```powershell
|
|
wsl -d Ubuntu -- bash /mnt/d/Projects/escape-gallery/room_glitch/labs/glitch_odin/train_teddy_wsl.sh --ckpt_path /home/jeff/teddy-training/runs/teddy-v1/lightning_logs/version_1/checkpoints/last.ckpt
|
|
```
|
|
|
|
Do not launch that command while training is already running.
|
|
|
|
## Export and first listening comparison
|
|
|
|
The bounded run completed with `max_steps=1000`. Exported `version_1/checkpoints/last.ckpt` and `epoch=14-val_mel=0.4643.ckpt` to `.training/teddy-v1/exports/`, each with the training-generated `.onnx.json` configuration. `export_teddy.py` selects the legacy dynamic-axis exporter required by Piper's opset-15 export code on this PyTorch version. The upstream source remains unchanged.
|
|
|
|
Both exports loaded and synthesized successfully in the lab's Windows Piper inference environment. Comparison audio is in `.training/teddy-v1/samples/`: `teddy-final.wav`, `teddy-best-mel.wav`, and `stock-reference.wav`. All read the same unseen passage from `comparison-transcript.txt`. File/model hashes and durations are in `report.json`.
|
|
|
|
A local small.en ASR sanity check recovered most words, with word-sequence similarity approximately 0.919 for the final model, 0.968 for best-mel, and 1.0 for stock. These are rough recognition comparisons, not subjective voice-quality scores or formal WER measurements. Number formatting and recognition mistakes affect them. Listen before deciding which checkpoint to develop further. Neither trained voice has been selected in the live hint system.
|
|
|
|
## Chatterbox Turbo comparison
|
|
|
|
An isolated zero-shot comparison now lives in `/home/jeff/teddy-chatterbox` (separate venv, source, model cache and logs). The comparison did not change Piper; Chatterbox was subsequently selected for the live lab. Official Chatterbox source revision: `5de7a54aa4e5e2baadb0182dde554908b48b85c2`. GPU dependency overrides were necessary because upstream pins PyTorch 2.6 for Python 3.12; resolved versions are recorded in `requirements-lock.txt` (torch 2.14.1, torchaudio 2.11.0). The successful inference test validates this specific path, not every Torchaudio feature.
|
|
|
|
`compare_teddy_chatterbox.py` uses the first 13.12 seconds of the actor recording as a reference; no new training is performed and audio stays local. The model is downloaded from `ResembleAI/chatterbox-turbo`. Output is `.training/teddy-v1/samples/teddy-chatterbox-turbo.wav`, with three individual hint WAVs and `chatterbox-report.json`. The model's default audio watermark remains enabled.
|
|
|
|
Measured loaded-model generation times on RTX 5070: 2.20 seconds for 7.48 seconds of speech; 1.16 seconds for 6.20 seconds; 0.95 seconds for 5.20 seconds. These are whole-utterance synthesis times, not streaming time-to-first-audio or end-to-end browser latency. Model download/load and reference preparation are separate fields in the report. Human listening comparison remains necessary.
|
|
|
|
To reproduce with downloaded weights:
|
|
|
|
```powershell
|
|
wsl -d Ubuntu -- bash -lc 'export HF_HOME=~/teddy-chatterbox/models; export HF_HUB_DISABLE_XET=1; ~/teddy-chatterbox/venv/bin/python /mnt/d/Projects/escape-gallery/room_glitch/labs/glitch_odin/compare_teddy_chatterbox.py'
|
|
```
|
|
|
|
|
|
## Chatterbox live lab backend
|
|
|
|
The lab now defaults to Chatterbox Turbo when launched with `--speech`. The existing Ubuntu installation at `~/teddy-chatterbox` supplies its Python environment, official source and model cache. `chatterbox_client.py` starts one resident worker, and `chatterbox_worker.py` loads the GPU model and actor reference once. The Windows follower continues to provide the existing audio/health routes, cancellation and browser playback. Odin and the production cabinet bridge are unchanged.
|
|
|
|
Required local assets for another development host:
|
|
|
|
- Ubuntu/WSL and an NVIDIA GPU with a working CUDA-enabled PyTorch installation.
|
|
- The Chatterbox source revision and dependency overrides recorded in the comparison section above. Preserve `~/teddy-chatterbox/requirements-lock.txt` when reproducing this tested environment.
|
|
- Cached model files under `~/teddy-chatterbox/models`, or network access for their initial download.
|
|
- The approved 13.12-second actor reference copied to `.speech/voices/teddy-chatterbox-reference.wav`. The original comparison copy is `.training/teddy-v1/samples/chatterbox-reference.wav`.
|
|
|
|
`TEDDY_CHATTERBOX_HOME` can override the Linux installation directory. Windows uses the Ubuntu WSL distribution; native Linux uses the same worker directly through bash. This is not a CPU/Pi inference installer.
|
|
|
|
Launch: `py -3 lab.py start-room --speech --lan`. Startup waits for model readiness. Logs: `.speech/chatterbox.log`. The reference hash and backend version are included in cache keys, so old Piper WAVs are not reused. Hints are split into sentences (and bounded word groups for long sentences) to avoid long-utterance truncation. No actor recording is uploaded; synthesis and caching stay local.
|
|
|
|
Fallback: `py -3 lab.py start-room --speech --speech-backend piper --lan`. The fallback uses the original stock voice and requires `setup-speech` first. Chatterbox errors do not silently substitute Piper.
|