Does this model work with the API for OpenAI compatibility of Ollama?
I'm using the OpenAI API as described here:
https://github.com/ollama/ollama/blob/main/docs/openai.md
But the model cannot describe the images, and gives meaningless answers.
Is this a limitation?
Hi, thanks for bringing this up! My guess is that this we're missing the system prompt. This model is very sensitive to having the correct system prompt to produce good results and we've seen this kind of garbage output when the system prompt is missing. The prompt should be baked into the ollama model here, but it's possible that going through the OpenAI compatibility layer in Ollama is not applying this correctly.
Using the same system prompt shown, the model starts to respond coherently.
But apparently the images sent through the OpenAI API do not work.
I will continue testing. It's something to keep in mind.
Edited chat in JSON
{
"messages": [
{
"role": "system",
"content": "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions."
},
{
"role": "user",
"content": [
{
"type": "text",
"text": "\n<img src=\"attachments/eefe8003-1ad5-4ff1-b5cb-ac1120ad6f91\">\n\n\n assets:\n [Screenshot 2025-03-05 191900.png](/attachments/eefe8003-1ad5-4ff1-b5cb-ac1120ad6f91)\n"
},
{
"type": "image_url",
"image_url": {
"url": "data:image/png;base64,REMOVED/TO/REDUCE/SIZE="
}
}
]
},
{
"role": "user",
"content": "Hi, can you see any images?"
},
{
"role": "assistant",
"content": "\nI'm sorry, but I cannot provide images or links due to the restrictions. You may try opening them through other means."
}
]
}
Interesting, thanks for investigating!