Can it runs on llama.cpp?
I noticed that it's based on Mistreal Nemo, it will work on llama.cpp if the tokenizer doesn't changed. Can you create a gguf version if convient?
You need the versions from here: https://huggingface.co/mradermacher/Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf-GGUF
Usage:
from huggingface_hub import hf_hub_download
from llama_cpp import Llama
model = hf_hub_download(
"mradermacher/Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf-GGUF",
filename="Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf.Q6_K.gguf",
)
llama = Llama(
model,
n_gpu_layers=-1,
n_ctx=2048,
verbose=False
)
completion = llama.create_chat_completion(
messages=[{"role": "user", "content": "Upsample the short caption to a long caption: A dog is playing with a ball."}],
max_tokens=512,
seed=12345
)
response = completion["choices"][0]["message"]["content"]
print(response)
Output:
Long Caption: In a sun-drenched backyard, a playful golden retriever bounds joyfully across the lush, green grass, its tail wagging with boundless enthusiasm. The dog, adorned with a vibrant blue collar, is the epitome of canine exuberance, its glossy coat glistening under the warm, golden-hour light. As it darts towards a bright yellow tennis ball, the camera captures its every movement with a steady, close-up shot, emphasizing the dog's agility and grace. The ball, a stark contrast against the verdant backdrop, rolls gently, inviting the eager pup to engage in a game of fetch. With a swift lunge, the dog snatches the ball, its eyes gleaming with excitement as it holds it firmly in its mouth, ready to share the joy of play with its unseen companion. The scene is bathed in a soft, natural light, enhancing the serene atmosphere, while the gentle rustling of leaves in the background adds a touch of tranquility to this heartwarming moment of unbridled canine joy.
Overall it takes about ~10Gb of VRAM, ~5 seconds to load and ~3 seconds to execute on an RTX 4080 Laptop with the Q6-K quantization.
You need the versions from here: https://huggingface.co/mradermacher/Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf-GGUF
Usage:
from huggingface_hub import hf_hub_download from llama_cpp import Llama model = hf_hub_download( "mradermacher/Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf-GGUF", filename="Cosmos-1.0-Prompt-Upsampler-12B-Text2World-hf.Q6_K.gguf", ) llama = Llama( model, n_gpu_layers=-1, n_ctx=2048, verbose=False ) completion = llama.create_chat_completion( messages=[{"role": "user", "content": "Upsample the short caption to a long caption: A dog is playing with a ball."}], max_tokens=512, seed=12345 ) response = completion["choices"][0]["message"]["content"] print(response)Output:
Long Caption: In a sun-drenched backyard, a playful golden retriever bounds joyfully across the lush, green grass, its tail wagging with boundless enthusiasm. The dog, adorned with a vibrant blue collar, is the epitome of canine exuberance, its glossy coat glistening under the warm, golden-hour light. As it darts towards a bright yellow tennis ball, the camera captures its every movement with a steady, close-up shot, emphasizing the dog's agility and grace. The ball, a stark contrast against the verdant backdrop, rolls gently, inviting the eager pup to engage in a game of fetch. With a swift lunge, the dog snatches the ball, its eyes gleaming with excitement as it holds it firmly in its mouth, ready to share the joy of play with its unseen companion. The scene is bathed in a soft, natural light, enhancing the serene atmosphere, while the gentle rustling of leaves in the background adds a touch of tranquility to this heartwarming moment of unbridled canine joy.
Overall it takes about ~10Gb of VRAM, ~5 seconds to load and ~3 seconds to execute on an RTX 4080 Laptop with the Q6-K quantization.
I have switched to Wan2.1 1.3b the day before yesterday,but thank you all the same.