mochi 1 video ai notes

2024-10-26T21:44:30

I’ve been playing with mochi1 a bit since it came out, these are some scrambled notes of what i’ve noticed

wtf is mochi

mochi is a 480p sota video model that you can run locally, it’s great at realistic cinematic footage (feels like it was trained on movies)

these folks made it: https://www.genmo.ai/blog

and it’s apache, nice https://github.com/genmoai/models

try it: https://fal.ai/models/fal-ai/mochi-v1

i don’t have 4 h100s

yeah me neither, it runs on a 3090 and probably will work on less

you can run it at home with a fairly powerful gpu using comfyui and this:
https://github.com/kijai/ComfyUI-MochiWrapper

performance

out of the box I was getting 20-40 minutes for a short 480p clip and oom for anything over 30ish frames. some necessary and easy improvements:

use vae tiling, it looks fine and is pretty necessary. if you hit an error you might need to update pytorch or some other packages
set up https://github.com/woct0rdho/triton-windows/
- if you’re using comfy ui portable:
- make sure you have msvc / vc runtime and cuda 12.x
- find your python_embeded path
- ../path/to/python_embeded/python.exe --version to get your version
- download matching release from that github (ie 311 for 3.11.9)
- ./python_embeded/python.exe -m pip install {the .whl for your python version}
- … pip install sageattention
- restart comfy
320×576 works great
50 steps is enough, more doesn’t seem that valuable

with all those applied I’m getting about 4 minutes for a 320×576 55 frame video render on a 3090

settings

resolution:

best: 848×480
fast: 576×320
square: 480×480 (havent tried 512×512 but i expect some issues)
vertical 480×840 seems bad, maybe works with enough steps? someone else got it working

frames: 50-120 seems like a good range but pushing longer seems to work, i haven’t found a limit yet

cfg: 4.5 is default, need to do more testing to compare

attention: sage seems best, see setup notes under performance section

steps: 50 is good enough for some prompts, further can help somewhat but i haven’t seen huge gains yet. needs side by side testing. low step count gives a good preview of the motion

checkpoints

bf16, fp8, gguf – needs testing and notes. i’ve been mostly using fp8

additional tooling

there’s little to nothing available yet but a lot of concepts applied to existing video models should work here i assume, and will likely be ported soon.

upscaling and frame interpolation can be done using normal methods (ie esrgan, rife)

I’ve experimented a little bit with using animatediff as a vid2vid 2nd pass after upscaling, I think this could be a viable technique but I haven’t honed in a workflow that’s worth the runtime yet.

topics

tracking how well it can create certain people or topics

scoring:
5 exact likeness
4 facsimile
3 sort of similar
2 loose correlation
1 bad, no correlation
no number means it’s not tested yet

celebrities and actors

*5 donald trump – for some reason it does trump better than almost anything
3 joe biden
3 kanye west
3 michael jackson
2 jackie chan (asian lady)
3 shaq (basketball player, bad likeness)
3 jim carrey
5 dawyne the rock johnson
1 arnold schwarzenegger
3 kim kardashian
2 brad pitt
3 johnny depp (looks like keanu reaves more?)
5 tom hanks
2 bruce willis (some guy)
1 keanu reaves (basketball? larry bird?)
3 macauley culkin
michael jordan
larry bird

famous characters

1 edna mode, incredibles (80s lookin lady)
4 optimus prime
5 minions, despicable me
3 spongebob (weird yellow cartoonish thing)
3 the joker (weird white face paint guy)
2 the dude big lebowski (some dude)
2 patrick bateman (guy in a suit)
4 gandalf
1 yoda
1 gollum
2 neo matrix (green vertical stripes on black lol)
2 george costanza (guy in a suit)

sports

basketball
soccer
futbol
football
tennis
kung fu
boxing
mma
pole vault
high jump
bobsled
hurdles
100m sprint
long jump
triple jump

animals

5 cat (duh)
dog
corgi
pug
chimpanzee
bonobo
gecko
frog
5 butterflies
shark, underwater
blue whale, underwater
octopus, underwater
cuttlefish, underwater
deep-sea anglerfish
jumping salmon

locations/settings

forest
oval office
5 nba court
space battle
office
graveyard

unsorted

ufffd.com