whatmodel.sh / video / LVDM (text-to-video)

LVDM (text-to-video)

+ contribute
Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

LVDM (text-to-video) is an open-weights video model from HKUST / Tencent AI Lab, released May 2023. Licence: mit.

Catalogue identity notes

Yingqing/LVDM lvdm_short/t2v.ckpt, the text-to-video checkpoint released with the LVDM code (He et al., arXiv 2211.13221); the other LVDM checkpoints are unconditional sky/taichi/UCF models. Code MIT; the HF weights repo states no separate licence. Not VideoCrafter.

24.6±3.4#90 of 93 video modelsfrom 2 benchmark results across 1 source* thinly tested, treat the rank as provisional

$ how it compares

The video models ranked either side of LVDM (text-to-video), out of 139 with a score.

#modelscorewhereRAMreleased
133VideoCrafter1 T2V Tencent AI Lab / HKUST26±2local~8.1GB2023-10
134Show-1 Show Lab / NUS25±2local~83GB2023-10
135LVDM (text-to-video) HKUST / Tencent AI Lab25±3local~6.1GB2023-05
136ModelScope T2V 1.7B Alibaba DAMO Academy25±2local~4.6GB2023-03
137Floor33 Pictures (August 2023) Floor33 Pictures24±2comparison only2023-08

see all video models · find the ones that fit your machine

$ what it's good at

text to video▆▄▃▄▆▅█▃▂▂▂▄ #66 of 68 * · 95% of its score · 2 tests

Trails ModelScope T2V 1.7B here, though they rank lower overall.

testresultvs the others in its rundifficultycountswhy
AIGVE-60K human perception MOS (LOVE)33.8% ✎▄▄·█··█▄▄▄▄▄ beats 0/10easy · avg 73%▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 60%at its level
AIGVE-60K human text-video correspondence MOS (LOVE)42.2% ✎▄▄·█▄█▄··▄▄▄ beats 0/10easy · avg 70%▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 35%at its level

$ where to run it

Other ways to run it

  1. Run it as published · needs ~6.1 GB, NVIDIA only · from github.com/YingqingHe
    git clone https://github.com/YingqingHe/LVDM && cd LVDM && conda create -y -n lvdm python=3.8.5 && conda run -n lvdm pip install -r requirements.txt && mkdir -p models/t2v && wget -O models/t2v/model.ckpt https://huggingface.co/Yingqing/LVDM/resolve/main/lvdm_short/t2v.ckpt && conda run -n lvdm bash shellscripts/sample_lvdm_text2video.sh

Weights and documentation: github.com/YingqingHe ↗

$ how the score was worked out · LOVE: Benchmarking and Evalua… 95%

Each result implies a score on its own, after adjusting for how hard its test is and who ran it. The score is their weighted average, with a small pull toward the middle for thinly tested models. Each test's weight is its counts column above; here it's added up by who ran the tests. How it works

sourceresultsshare of the scorelicence
LOVE: Benchmarking and Evaluating Text-to-Video Generation and Video-to-Text Interpretation (AIGVE-60K)2▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇ 95%CC-BY-4.0
prior · pull toward the middle▇ 5%
weighted average= 24.6

$ improve this page

Update model informationAdd a place to run itAdd a test resultPropose a benchmarkCorrect a benchmark scoreEdit data on GitHub

Use a short GitHub form or edit the data through a pull request. Every number needs a source; here's which ones we can use.

data updated · sources and licences · json