whatmodel.sh / tts / VITS (LJ Speech)

VITS (LJ Speech)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

VITS (LJ Speech) is an open-weights text-to-speech model from Kakao / KAIST, released June 2021.

Catalogue identity notes

Exact ljspeech system in VITS Table 1; HiFi-GAN fine-tuning is a distinct vocoder configuration. Published implementation/checkpoint links in §3.4; no generic Tacotron or Glow-TTS score substitution.

67.4±24.0#1 of 1 text-to-speech modelsfrom 1 benchmark result across 1 source* thinly tested, treat the rank as provisional

Score within the VITS 2021 LJ Speech human MOS comparison set only.

$ how it compares

Scored text-to-speech models released closest to VITS (LJ Speech), out of 148 with a score.

#modelscorewhereRAMreleased
81Azure Ameha Neural Microsoft50±16comparison only2021-10
82Azure Mekdes Neural Microsoft50±16comparison only2021-10
124VITS (VCTK multi-speaker) Kakao Enterprise (Jaehyeon Kim et al.)34±5local~0.2GB2021-06
135Glow-TTS + HiFi-GAN (VCTK, VITS study) VITS study (public implementations)28±6comparison only2021-06
142Tacotron 2 + HiFi-GAN (VCTK, VITS study) VITS study (public implementations)19±6comparison only2021-06

see all text-to-speech models · find the ones that fit your machine

$ what it's good at

other tests1 not in a task · 53% of its score
testresultvs the others in its rundifficultycountswhy
VITS 2021 LJ Speech human MOS lab4.43 / 5 ✎█····█·█·█·█ beats 4/4easy · avg 4.22 / 5▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 53%typical

$ where to run it

Other ways to run it

  1. Run it as published · needs ~0.15 GB, NVIDIA only · from github.com/jaywalnut310
    git clone https://github.com/jaywalnut310/vits && cd vits && pip install -r requirements.txt && apt-get install espeak && cd monotonic_align && python setup.py build_ext --inplace && cd .. && python -c "import torch, commons, utils; from models import SynthesizerTrn; from text.symbols import symbols; from text import text_to_sequence; from scipy.io.wavfile import write; hps = utils.get_hparams_from_file('./configs/ljs_base.json'); net_g = SynthesizerTrn(len(symbols), hps.data.filter_length // 2 + 1, hps.train.segment_size // hps.data.hop_length, **hps.model).cuda(); _ = net_g.eval(); _ = utils.load_checkpoint('/path/to/pretrained_ljs.pth', net_g, None); text_norm = text_to_sequence('VITS is Awesome!', hps.data.text_cleaners); text_norm = commons.intersperse(text_norm, 0) if hps.data.add_blank else text_norm; stn_tst = torch.LongTensor(text_norm); torch.set_grad_enabled(False); x_tst = stn_tst.cuda().unsqueeze(0); x_tst_lengths = torch.LongTensor([stn_tst.size(0)]).cuda(); audio = net_g.infer(x_tst, x_tst_lengths, noise_scale=.667, noise_scale_w=0.8, length_scale=1)[0][0,0].data.cpu().float().numpy(); write('ljs.wav', hps.data.sampling_rate, audio)"

Weights and documentation: github.com/jaywalnut310 ↗

$ how the score was worked out · VITS: ICML 2021 original list… 53%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
VITS: ICML 2021 original listening studies · lab-reported1▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 53%CC-BY-4.0
prior · pull toward the middle▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 47%
weighted average= 67.4

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json