Spaces:
Running
on
Zero
Apply for community grant: Academic project (gpu)
Hi, we are building a video-to-audio synthesis project (https://github.com/hkchengrex/MMAudio) that can add sounds to user-generated videos. We would be grateful if we could receive a community grant for the community to try this model out!
Hi
@hkchengrex
, we assigned L4 to this Space for now, but it would be nice if you could migrate the Space to use ZeroGPU. As you are subscribing to PRO, you can assign ZeroGPU yourself. It seems that your code has @spaces.GPU decorators, so you tried ZeroGPU, but it didn't work?
Thanks! Yes, we have tried it but it didn't work. It complains that the returned video has a frame rate of 0. The same error does not occur in other configurations so I am not sure how to debug that.
Hmm, it sounds like a very weird error. Will look into it a bit, but not sure if I can find anything.
BTW, looks like the models are hosted on a university server and GitHub, but would it be possible to host them on HF Hub?
I also uploaded the weights to HF earlier today https://huggingface.co/hkchengrex/MMAudio/tree/main
Are you referring to that we should also download models from HF in the script?
Ah, cool! I simply missed it because I was looking at the model links in the GitHub. But, yeah, it would be nice to update the code to download models from HF Hub because it's usually faster.
Got it.
I have the ZeroGPU problem reproduced here: https://huggingface.co/spaces/hkchengrex/test
(I tried using tempfile instead of saving in ./output, but that does not solve the problem)
Yeah, I can reproduce the issue on ZeroGPU as well. Actually, https://huggingface.co/spaces/hkchengrex/MMAudio/blob/03013d2151270a01c15244182e98a64fdc3c31ae/mmaudio/eval_utils.py#L223 doesn't work even in my local environment. Probably it's due to some library version discrepancy. Maybe you can get fps of the input video using other libraries like ffmpeg-python or opencv-python.
Thanks. Switching to pyav fixed the problem! I think StreamingMediaDecoder internally uses ffmpegso I am not sure why it wouldn't work. The return frame rate is not always zero but is all over the place, sometimes >180. Anyway, I have updated the space to ZeroGPU. Thanks again!
Awesome! Thanks for migrating to ZeroGPU.