cognitivecomputations

/

dolphin-vision-7b

like 29

Any fix @ehartford

@catworld1212 you need to pass trust_remote_code = True into AutoModelForCausalLM.from_pretrained

@qnguyen3 I did still same error please check my code:

    import torch
        import transformers
        from transformers import AutoModelForCausalLM, AutoTokenizer
        from PIL import Image
        import warnings
        import requests
        # disable some warnings
        transformers.logging.set_verbosity_error()
        transformers.logging.disable_progress_bar()
        warnings.filterwarnings('ignore')

        # set device
        torch.set_default_device('cuda')  # or 'cpu'

        model_name = 'cognitivecomputations/dolphin-vision-7b'

        # create model
        model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.float16,
            device_map='auto',
            trust_remote_code=True)
        tokenizer = AutoTokenizer.from_pretrained(
            model_name,
            trust_remote_code=True)

        # text prompt
        prompt = 'Describe this image in detail'

        messages = [
            {"role": "user", "content": f'<image>\n{prompt}'}
        ]
        text = tokenizer.apply_chat_template(
            messages,
            tokenize=False,
            add_generation_prompt=True
        )

        print(text)

        text_chunks = [tokenizer(chunk).input_ids for chunk in text.split('<image>')]
        input_ids = torch.tensor(text_chunks[0] + [-200] + text_chunks[1], dtype=torch.long).unsqueeze(0)

        # image, sample images can be found in images folder
        image = Image.open(requests.get(
            "https://uploads4.wikiart.org/images/paul-klee/death-for-the-idea-1915.jpg!Large.jpg", stream=True).raw)
        image_tensor = model.process_images([image], model.config).to(dtype=model.dtype)

        # generate
        output_ids = model.generate(
            input_ids,
            images=image_tensor,
            max_new_tokens=2048,
            use_cache=True)[0]

        print(tokenizer.decode(output_ids[input_ids.shape[1]:], skip_special_tokens=True).strip())

:/

@qnguyen3 Any fix?

hi @qnguyen3 can you pls test the model?

yes i will try to test it tonight

@catworld1212 should be fixed, could you give it a check?

@qnguyen3 it works

Works for me too :3

The image portrays a contemporary living room bathed in natural light from a large window on the left. Dominating the center of the room is a beige sofa, comfortably flanked by a blue armchair and a wooden side table, each adorned with a plant, adding a touch of greenery to the space.

Above the sofa, four framed pictures of insects are neatly arranged on the white wall, creating a striking contrast against the plain background. To the right of the sofa stands a wooden bookshelf, its top decorated with an array of pottery and other objects, adding a personal touch to the room.

A wooden floor stretches out across the room, leading to a black and white striped rug that lies on the wood, adding a patterned element to the otherwise neutral palette. The room is a harmonious blend of modern design and personal style, with each piece of furniture and decor contributing to the overall aesthetic.

Can I add this to the model card? This is a killer example

@catworld1212 should be fixed, could you give it a check?

Thank you for fixing it!

Can I add this to the model card? This is a killer example

Sure!
It reminds me a lot of howlmms-lab/llava-next-interleave-qwen-7b describes an image

But this 7b dolphin model can't rate things on a scale of 1/10 it says as AI it can't do that even with a system prompt.