Add support for tgi multimodal models (#531)

* wip: add support for tgi multimodal models

* wip work on passing images to prompt

* working idefics config!

* rm allowed conv feature

* lint

* Add image resizing

* fix ssr

* add upload button

* add delete button

* misc formatting

* lint

* server file size check

* optimistic update of images

* retry with images

* fix websearch button

* lint

* better error handling & max one image at a time

* replace test image by blank one

* disable loading on page change

* Fix sharing of images

* fix comments

* Update filedropzone (#544)

* Update src/lib/buildPrompt.ts

Co-authored-by: Mishig <[email protected]>

* small tweaks

* Fix merge conflicts

* lint

* wildcard image mime type

* fix lint and comment

* added comments

* added comment about file size

* Readme update

---------

Co-authored-by: Mishig <[email protected]>
Co-authored-by: Victor Mustar <[email protected]>

.env.template

@@ -111,7 +111,7 @@ MODELS=`[
       },
       "promptExamples": [
         {
-            "title": "Write an email from bullet list",
+          "title": "Write an email from bullet list",
           "prompt": "As a restaurant owner, write a professional email to the supplier to get these products every week: \n\n- Wine (x10)\n- Eggs (x24)\n- Bread (x12)"
         }, {
           "title": "Code a snake game",

PROMPTS.md

@@ -31,3 +31,9 @@ System: {{preprompt}}\nUser:{{#each messages}}{{#ifUser}}{{content}}\nFalcon:{{/
 ```env
 <|system|>\n{{preprompt}}</s>\n{{#each messages}}{{#ifUser}}<|user|>\n{{content}}</s>\n<|assistant|>\n{{/ifUser}}{{#ifAssistant}}{{content}}</s>\n{{/ifAssistant}}{{/each}}
 ```
+
+## IDEFICS
+
+```env
+{{#each messages}}{{#ifUser}}User: {{content}}{{/ifUser}}<end_of_utterance>\nAssistant: {{#ifAssistant}}{{content}}\n{{/ifAssistant}}{{/each}}
+```

README.md

@@ -168,7 +168,65 @@ MODELS=`[
 
 You can change things like the parameters, or customize the preprompt to better suit your needs. You can also add more models by adding more objects to the array, with different preprompts for example.
 
-#### OpenAI API compatible models
+#### chatPromptTemplate
+
+When querying the model for a chat response, the `chatPromptTemplate` template is used. `messages` is an array of chat messages, it has the format `[{ content: string }, ...]`. To identify if a message is a user message or an assistant message the `ifUser` and `ifAssistant` block helpers can be used.
+
+The following is the default `chatPromptTemplate`, although newlines and indentiation have been added for readability. You can find the prompts used in production for HuggingChat [here](https://github.com/huggingface/chat-ui/blob/main/PROMPTS.md).
+
+```prompt
+{{preprompt}}
+{{#each messages}}
+  {{#ifUser}}{{@root.userMessageToken}}{{content}}{{@root.userMessageEndToken}}{{/ifUser}}
+  {{#ifAssistant}}{{@root.assistantMessageToken}}{{content}}{{@root.assistantMessageEndToken}}{{/ifAssistant}}
+{{/each}}
+{{assistantMessageToken}}
+```
+
+#### Multi modal model
+
+We currently only support IDEFICS as a multimodal model, hosted on TGI. You can enable it by using the followin config (if you have a PRO HF Api token):
+
+```env
+    {
+      "name": "HuggingFaceM4/idefics-80b-instruct",
+      "multimodal" : true,
+      "description": "IDEFICS is the new multimodal model by Hugging Face.",
+      "preprompt": "",
+      "chatPromptTemplate" : "{{#each messages}}{{#ifUser}}User: {{content}}{{/ifUser}}<end_of_utterance>\nAssistant: {{#ifAssistant}}{{content}}\n{{/ifAssistant}}{{/each}}",
+      "parameters": {
+        "temperature": 0.1,
+        "top_p": 0.95,
+        "repetition_penalty": 1.2,
+        "top_k": 12,
+        "truncate": 1000,
+        "max_new_tokens": 1024,
+        "stop": ["<end_of_utterance>", "User:", "\nUser:"]
+      }
+    }
+```
+
+#### Running your own models using a custom endpoint
+
+If you want to, instead of hitting models on the Hugging Face Inference API, you can run your own models locally.
+
+A good option is to hit a [text-generation-inference](https://github.com/huggingface/text-generation-inference) endpoint. This is what is done in the official [Chat UI Spaces Docker template](https://huggingface.co/new-space?template=huggingchat/chat-ui-template) for instance: both this app and a text-generation-inference server run inside the same container.
+
+To do this, you can add your own endpoints to the `MODELS` variable in `.env.local`, by adding an `"endpoints"` key for each model in `MODELS`.
+
+```env
+{
+// rest of the model config here
+"endpoints": [{
+  "type" : "tgi",
+  "url": "https://HOST:PORT",
+  }]
+}
+```
+
+If `endpoints` are left unspecified, ChatUI will look for the model on the hosted Hugging Face inference API using the model name.
+
+##### OpenAI API compatible models
 
 Chat UI can be used with any API server that supports OpenAI API compatibility, for example [text-generation-webui](https://github.com/oobabooga/text-generation-webui/tree/main/extensions/openai), [LocalAI](https://github.com/go-skynet/LocalAI), [FastChat](https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md), [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), and [ialacol](https://github.com/chenhunghan/ialacol).
 
@@ -217,7 +275,7 @@ MODELS=`[{
 }]`
 ```
 
-#### Llama.cpp API server
+##### Llama.cpp API server
 
 chat-ui also supports the llama.cpp API server directly without the need for an adapter. You can do this using the `llamacpp` endpoint type.
 
@@ -253,70 +311,29 @@ MODELS=[
 
 Start chat-ui with `npm run dev` and you should be able to chat with Zephyr locally.
 
-#### Custom prompt templates
-
-By default, the prompt is constructed using `userMessageToken`, `assistantMessageToken`, `userMessageEndToken`, `assistantMessageEndToken`, `preprompt` parameters and a series of default templates.
-
-However, these templates can be modified by setting the `chatPromptTemplate` and `webSearchQueryPromptTemplate` parameters. Note that if WebSearch is not enabled, only `chatPromptTemplate` needs to be set. The template language is <https://handlebarsjs.com>. The templates have access to the model's prompt parameters (`preprompt`, etc.). However, if the templates are specified it is recommended to inline the prompt parameters, as using the references (`{{preprompt}}`) is deprecated.
-
-For example:
-
-```prompt
-<System>You are an AI, called ChatAI.</System>
-{{#each messages}}
-  {{#ifUser}}<User>{{content}}</User>{{/ifUser}}
-  {{#ifAssistant}}<Assistant>{{content}}</Assistant>{{/ifAssistant}}
-{{/each}}
-<Assistant>
-```
-
-##### chatPromptTemplate
-
-When querying the model for a chat response, the `chatPromptTemplate` template is used. `messages` is an array of chat messages, it has the format `[{ content: string }, ...]`. To identify if a message is a user message or an assistant message the `ifUser` and `ifAssistant` block helpers can be used.
+#### Amazon
 
-The following is the default `chatPromptTemplate`, although newlines and indentiation have been added for readability. You can find the prompts used in production for HuggingChat [here](https://github.com/huggingface/chat-ui/blob/main/PROMPTS.md).
-
-```prompt
-{{preprompt}}
-{{#each messages}}
-  {{#ifUser}}{{@root.userMessageToken}}{{content}}{{@root.userMessageEndToken}}{{/ifUser}}
-  {{#ifAssistant}}{{@root.assistantMessageToken}}{{content}}{{@root.assistantMessageEndToken}}{{/ifAssistant}}
-{{/each}}
-{{assistantMessageToken}}
-```
-
-##### webSearchQueryPromptTemplate
-
-When performing a websearch, the search query is constructed using the `webSearchQueryPromptTemplate` template. It is recommended that the prompt instructs the chat model to only return a few keywords.
-
-The following is the default `webSearchQueryPromptTemplate`.
-
-```prompt
-{{userMessageToken}}
-  My question is: {{message.content}}.
+You can also specify your Amazon SageMaker instance as an endpoint for chat-ui. The config goes like this:
 
-Based on the conversation history (my previous questions are: {{previousMessages}}), give me an appropriate query to answer my question for web search. You should not say more than query. You should not say any words except the query. For the context, today is {{currentDate}}
+```env
+"endpoints": [
+    {
+      "type" : "aws",
+      "service" : "sagemaker"
+      "url": "",
+      "accessKey": "",
+      "secretKey" : "",
+      "sessionToken": "",
+      "region": "",
 
-{{userMessageEndToken}}
-{{assistantMessageToken}}
+      "weight": 1
+    }
+]
 ```
 
-#### Running your own models using a custom endpoint
-
-If you want to, instead of hitting models on the Hugging Face Inference API, you can run your own models locally.
-
-A good option is to hit a [text-generation-inference](https://github.com/huggingface/text-generation-inference) endpoint. This is what is done in the official [Chat UI Spaces Docker template](https://huggingface.co/new-space?template=huggingchat/chat-ui-template) for instance: both this app and a text-generation-inference server run inside the same container.
-
-To do this, you can add your own endpoints to the `MODELS` variable in `.env.local`, by adding an `"endpoints"` key for each model in `MODELS`.
+You can also set `"service" : "lambda"` to use a lambda instance.
 
-```env
-{
-// rest of the model config here
-"endpoints": [{"url": "https://HOST:PORT"}]
-}
-```
-
-If `endpoints` are left unspecified, ChatUI will look for the model on the hosted Hugging Face inference API using the model name.
+You can get the `accessKey` and `secretKey` from your AWS user, under programmatic access.
 
 ### Custom endpoint authorization
 
@@ -343,55 +360,6 @@ You can then add the generated information and the `authorization` parameter to
 ]
 ```
 
-### Amazon
-
-#### SageMaker
-
-You can also specify your Amazon SageMaker instance as an endpoint for chat-ui. The config goes like this:
-
-```env
-"endpoints": [
-    {
-      "type" : "aws",
-      "service" : "sagemaker"
-      "url": "",
-      "accessKey": "",
-      "secretKey" : "",
-      "sessionToken": "",
-      "weight": 1
-    }
-]
-```
-
-#### Lambda
-
-You can also specify your Amazon Lambda instance as an endpoint for chat-ui. The config goes like this:
-
-```env
-"endpoints" : [
-  {
-        "type": "aws",
-        "service": "lambda",
-        "url": "",
-        "accessKey": "",
-        "secretKey": "",
-        "sessionToken": "",
-        "region": "",
-        "weight": 1
- }
-]
-```
-
-You can get the `accessKey` and `secretKey` from your AWS user, under programmatic access.
-
-#### Client Certificate Authentication (mTLS)
-
-Custom endpoints may require client certificate authentication, depending on how you configure them. To enable mTLS between Chat UI and your custom endpoint, you will need to set the `USE_CLIENT_CERTIFICATE` to `true`, and add the `CERT_PATH` and `KEY_PATH` parameters to your `.env.local`. These parameters should point to the location of the certificate and key files on your local machine. The certificate and key files should be in PEM format. The key file can be encrypted with a passphrase, in which case you will also need to add the `CLIENT_KEY_PASSWORD` parameter to your `.env.local`.
-
-If you're using a certificate signed by a private CA, you will also need to add the `CA_PATH` parameter to your `.env.local`. This parameter should point to the location of the CA certificate file on your local machine.
-
-If you're using a self-signed certificate, e.g. for testing or development purposes, you can set the `REJECT_UNAUTHORIZED` parameter to `false` in your `.env.local`. This will disable certificate validation, and allow Chat UI to connect to your custom endpoint.
-
 #### Models hosted on multiple custom endpoints
 
 If the model being hosted will be available on multiple servers/instances add the `weight` parameter to your `.env.local`. The `weight` will be used to determine the probability of requesting a particular endpoint.
@@ -408,9 +376,16 @@ If the model being hosted will be available on multiple servers/instances add th
 }
 ...
 ]
-
 ```
 
+#### Client Certificate Authentication (mTLS)
+
+Custom endpoints may require client certificate authentication, depending on how you configure them. To enable mTLS between Chat UI and your custom endpoint, you will need to set the `USE_CLIENT_CERTIFICATE` to `true`, and add the `CERT_PATH` and `KEY_PATH` parameters to your `.env.local`. These parameters should point to the location of the certificate and key files on your local machine. The certificate and key files should be in PEM format. The key file can be encrypted with a passphrase, in which case you will also need to add the `CLIENT_KEY_PASSWORD` parameter to your `.env.local`.
+
+If you're using a certificate signed by a private CA, you will also need to add the `CA_PATH` parameter to your `.env.local`. This parameter should point to the location of the CA certificate file on your local machine.
+
+If you're using a self-signed certificate, e.g. for testing or development purposes, you can set the `REJECT_UNAUTHORIZED` parameter to `false` in your `.env.local`. This will disable certificate validation, and allow Chat UI to connect to your custom endpoint.
+
 ## Deploying to a HF Space
 
 Create a `DOTENV_LOCAL` secret to your HF space with the content of your .env.local, and they will be picked up automatically when you run.

package-lock.json

@@ -12,10 +12,12 @@
 				"@huggingface/inference": "^2.6.3",
 				"@xenova/transformers": "^2.6.0",
 				"autoprefixer": "^10.4.14",
+				"browser-image-resizer": "^2.4.1",
 				"date-fns": "^2.29.3",
 				"dotenv": "^16.0.3",
 				"handlebars": "^4.7.8",
 				"highlight.js": "^11.7.0",
+				"image-size": "^1.0.2",
 				"jsdom": "^22.0.0",
 				"marked": "^4.3.0",
 				"mongodb": "^5.8.0",
@@ -1796,6 +1798,11 @@
 				"base64-js": "^1.1.2"
 			}
 		},
+		"node_modules/browser-image-resizer": {
+			"version": "2.4.1",
+			"resolved": "https://registry.npmjs.org/browser-image-resizer/-/browser-image-resizer-2.4.1.tgz",
+			"integrity": "sha512-gqrmr7+NTI9FgZVVyw/GIqwJE3MhNWaBn1R5ptu75r+/M5ncyntSMQYuYhOPonm44qQNnkGN9cnghlpd9h1Hug=="
+		},
 		"node_modules/browserslist": {
 			"version": "4.21.5",
 			"resolved": "https://registry.npmjs.org/browserslist/-/browserslist-4.21.5.tgz",
@@ -3266,6 +3273,20 @@
 				"node": ">= 4"
 			}
 		},
+		"node_modules/image-size": {
+			"version": "1.0.2",
+			"resolved": "https://registry.npmjs.org/image-size/-/image-size-1.0.2.tgz",
+			"integrity": "sha512-xfOoWjceHntRb3qFCrh5ZFORYH8XCdYpASltMhZ/Q0KZiOwjdE/Yl2QCiWdwD+lygV5bMCvauzgu5PxBX/Yerg==",
+			"dependencies": {
+				"queue": "6.0.2"
+			},
+			"bin": {
+				"image-size": "bin/image-size.js"
+			},
+			"engines": {
+				"node": ">=14.0.0"
+			}
+		},
 		"node_modules/import-fresh": {
 			"version": "3.3.0",
 			"resolved": "https://registry.npmjs.org/import-fresh/-/import-fresh-3.3.0.tgz",
@@ -4986,6 +5007,14 @@
 			"resolved": "https://registry.npmjs.org/querystringify/-/querystringify-2.2.0.tgz",
 			"integrity": "sha512-FIqgj2EUvTa7R50u0rGsyTftzjYmv/a3hO345bZNrqabNqjtgiDMgmo4mkUjd+nzU5oF3dClKqFIPUKybUyqoQ=="
 		},
+		"node_modules/queue": {
+			"version": "6.0.2",
+			"resolved": "https://registry.npmjs.org/queue/-/queue-6.0.2.tgz",
+			"integrity": "sha512-iHZWu+q3IdFZFX36ro/lKBkSvfkztY5Y7HMiPlOUjhupPcG2JMfst2KKEpu5XndviX/3UhFbRngUPNKtgvtZiA==",
+			"dependencies": {
+				"inherits": "~2.0.3"
+			}
+		},
 		"node_modules/queue-microtask": {
 			"version": "1.2.3",
 			"resolved": "https://registry.npmjs.org/queue-microtask/-/queue-microtask-1.2.3.tgz",

package.json

@@ -48,10 +48,12 @@
 		"@huggingface/inference": "^2.6.3",
 		"@xenova/transformers": "^2.6.0",
 		"autoprefixer": "^10.4.14",
+		"browser-image-resizer": "^2.4.1",
 		"date-fns": "^2.29.3",
 		"dotenv": "^16.0.3",
 		"handlebars": "^4.7.8",
 		"highlight.js": "^11.7.0",
+		"image-size": "^1.0.2",
 		"jsdom": "^22.0.0",
 		"marked": "^4.3.0",
 		"mongodb": "^5.8.0",

src/lib/buildPrompt.ts

@@ -2,18 +2,17 @@ import type { BackendModel } from "./server/models";
 import type { Message } from "./types/Message";
 import { format } from "date-fns";
 import type { WebSearch } from "./types/WebSearch";
-/**
- * Convert [{user: "assistant", content: "hi"}, {user: "user", content: "hello"}] to:
- *
- * <|assistant|>hi<|endoftext|><|prompter|>hello<|endoftext|><|assistant|>
- */
+import { downloadFile } from "./server/files/downloadFile";
+import type { Conversation } from "./types/Conversation";
 
 interface buildPromptOptions {
-	messages: Pick<Message, "from" | "content">[];
+	messages: Pick<Message, "from" | "content" | "files">[];
+	id?: Conversation["_id"];
 	model: BackendModel;
 	locals?: App.Locals;
 	webSearch?: WebSearch;
 	preprompt?: string;
+	files?: File[];
 }
 
 export async function buildPrompt({
@@ -21,6 +20,7 @@ export async function buildPrompt({
 	model,
 	webSearch,
 	preprompt,
+	id,
 }: buildPromptOptions): Promise<string> {
 	if (webSearch && webSearch.context) {
 		const lastMsg = messages.slice(-1)[0];
@@ -49,6 +49,38 @@ export async function buildPrompt({
 		];
 	}
 
+	// section to handle potential files input
+	if (model.multimodal) {
+		messages = await Promise.all(
+			messages.map(async (el) => {
+				let content = el.content;
+
+				if (el.from === "user") {
+					if (el?.files && el.files.length > 0 && id) {
+						const markdowns = await Promise.all(
+							el.files.map(async (hash) => {
+								try {
+									const { content: image, mime } = await downloadFile(hash, id);
+									const b64 = image.toString("base64");
+									return `![](data:${mime};base64,${b64})})`;
+								} catch (e) {
+									console.error(e);
+								}
+							})
+						);
+						content += markdowns.join("\n ");
+					} else {
+						// if no image, append an empty white image
+						content +=
+							"\n![](data:image/png;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/2wBDAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQH/wAARCAAQABADAREAAhEBAxEB/8QAHwAAAQUBAQEBAQEAAAAAAAAAAAECAwQFBgcICQoL/8QAtRAAAgEDAwIEAwUFBAQAAAF9AQIDAAQRBRIhMUEGE1FhByJxFDKBkaEII0KxwRVS0fAkM2JyggkKFhcYGRolJicoKSo0NTY3ODk6Q0RFRkdISUpTVFVWV1hZWmNkZWZnaGlqc3R1dnd4eXqDhIWGh4iJipKTlJWWl5iZmqKjpKWmp6ipqrKztLW2t7i5usLDxMXGx8jJytLT1NXW19jZ2uHi4+Tl5ufo6erx8vP09fb3+Pn6/8QAHwEAAwEBAQEBAQEBAQAAAAAAAAECAwQFBgcICQoL/8QAtREAAgECBAQDBAcFBAQAAQJ3AAECAxEEBSExBhJBUQdhcRMiMoEIFEKRobHBCSMzUvAVYnLRChYkNOEl8RcYGRomJygpKjU2Nzg5OkNERUZHSElKU1RVVldYWVpjZGVmZ2hpanN0dXZ3eHl6goOEhYaHiImKkpOUlZaXmJmaoqOkpaanqKmqsrO0tba3uLm6wsPExcbHyMnK0tPU1dbX2Nna4uPk5ebn6Onq8vP09fb3+Pn6/9oADAMBAAIRAxEAPwD+/igAoAKACgD/2Q==)";
+					}
+				}
+
+				return { ...el, content };
+			})
+		);
+	}
+
 	return (
 		model
 			.chatPromptRender({ messages, preprompt })

src/lib/components/UploadBtn.svelte

@@ -0,0 +1,23 @@
+<script lang="ts">
+	import CarbonUpload from "~icons/carbon/upload";
+
+	export let classNames = "";
+	export let files: File[];
+	let filelist: FileList;
+
+	$: if (filelist) {
+		files = Array.from(filelist);
+	}
+</script>
+
+<button
+	class="btn relative h-8 rounded-lg border bg-white px-3 py-1 text-sm text-gray-500 shadow-sm transition-all hover:bg-gray-100 dark:border-gray-600 dark:bg-gray-700 dark:text-gray-300 dark:hover:bg-gray-600 {classNames}"
+>
+	<input
+		bind:files={filelist}
+		class="absolute w-full cursor-pointer opacity-0"
+		type="file"
+		accept="image/*"
+	/>
+	<CarbonUpload class="mr-2 text-xs " /> Upload image
+</button>

src/lib/components/chat/ChatInput.svelte

@@ -6,7 +6,6 @@
 	export let maxRows: null | number = null;
 	export let placeholder = "";
 	export let disabled = false;
-
 	// Approximate width from which we disable autofocus
 	const TABLET_VIEWPORT_WIDTH = 768;
 

src/lib/components/chat/ChatMessage.svelte

@@ -234,36 +234,59 @@
 {/if}
 {#if message.from === "user"}
 	<div class="group relative flex items-start justify-start gap-4 max-sm:text-sm">
-		<div class="mt-5 h-3 w-3 flex-none rounded-full" />
-		<div
-			class="max-w-full whitespace-break-spaces break-words rounded-2xl px-5 py-3.5 text-gray-500 dark:text-gray-400"
-		>
-			{message.content.trim()}
-		</div>
-		{#if !loading}
-			<div class="absolute right-0 top-3.5 flex gap-2 lg:-right-2">
-				{#if downloadLink}
-					<a
-						class="rounded-lg border border-gray-100 p-1 text-xs text-gray-400 group-hover:block hover:text-gray-500 dark:border-gray-800 dark:text-gray-400 dark:hover:text-gray-300 md:hidden"
-						title="Download prompt and parameters"
-						type="button"
-						target="_blank"
-						href={downloadLink}
-					>
-						<CarbonDownload />
-					</a>
-				{/if}
-				{#if !readOnly}
-					<button
-						class="cursor-pointer rounded-lg border border-gray-100 p-1 text-xs text-gray-400 group-hover:block hover:text-gray-500 dark:border-gray-800 dark:text-gray-400 dark:hover:text-gray-300 md:hidden lg:-right-2"
-						title="Retry"
-						type="button"
-						on:click={() => dispatch("retry", { content: message.content, id: message.id })}
-					>
-						<CarbonRotate360 />
-					</button>
-				{/if}
+		<div class="flex flex-col">
+			{#if message.files && message.files.length > 0}
+				<div class="mx-auto grid w-fit grid-cols-2 gap-5 px-5">
+					{#each message.files as file}
+						<!-- handle the case where this is a hash that points to an image in the db, hash is always 64 char long -->
+						{#if file.length === 64}
+							<img
+								src={$page.url.pathname + "/output/" + file}
+								alt="input from user"
+								class="my-2 aspect-auto max-h-48 rounded-lg shadow-lg"
+							/>
+						{:else}
+							<!-- handle the case where this is a base64 encoded image -->
+							<img
+								src={"data:image/*;base64," + file}
+								alt="input from user"
+								class="my-2 aspect-auto max-h-48 rounded-lg shadow-lg"
+							/>
+						{/if}
+					{/each}
+				</div>
+			{/if}
+
+			<div
+				class="max-w-full whitespace-break-spaces break-words rounded-2xl px-5 py-3.5 text-gray-500 dark:text-gray-400"
+			>
+				{message.content.trim()}
 			</div>
-		{/if}
+			{#if !loading}
+				<div class="absolute right-0 top-3.5 flex gap-2 lg:-right-2">
+					{#if downloadLink}
+						<a
+							class="rounded-lg border border-gray-100 p-1 text-xs text-gray-400 group-hover:block hover:text-gray-500 dark:border-gray-800 dark:text-gray-400 dark:hover:text-gray-300 md:hidden"
+							title="Download prompt and parameters"
+							type="button"
+							target="_blank"
+							href={downloadLink}
+						>
+							<CarbonDownload />
+						</a>
+					{/if}
+					{#if !readOnly}
+						<button
+							class="cursor-pointer rounded-lg border border-gray-100 p-1 text-xs text-gray-400 group-hover:block hover:text-gray-500 dark:border-gray-800 dark:text-gray-400 dark:hover:text-gray-300 md:hidden lg:-right-2"
+							title="Retry"
+							type="button"
+							on:click={() => dispatch("retry", { content: message.content, id: message.id })}
+						>
+							<CarbonRotate360 />
+						</button>
+					{/if}
+				</div>
+			{/if}
+		</div>
 	</div>
 {/if}

src/lib/components/chat/ChatWindow.svelte

@@ -5,6 +5,8 @@
 	import CarbonSendAltFilled from "~icons/carbon/send-alt-filled";
 	import CarbonExport from "~icons/carbon/export";
 	import CarbonStopFilledAlt from "~icons/carbon/stop-filled-alt";
+	import CarbonClose from "~icons/carbon/close";
+
 	import EosIconsLoading from "~icons/eos-icons/loading";
 
 	import ChatMessages from "./ChatMessages.svelte";
@@ -17,7 +19,10 @@
 	import type { WebSearchUpdate } from "$lib/types/MessageUpdate";
 	import { page } from "$app/stores";
 	import DisclaimerModal from "../DisclaimerModal.svelte";
+	import FileDropzone from "./FileDropzone.svelte";
 	import RetryBtn from "../RetryBtn.svelte";
+	import UploadBtn from "../UploadBtn.svelte";
+	import file2base64 from "$lib/utils/file2base64";
 
 	export let messages: Message[] = [];
 	export let loading = false;
@@ -28,6 +33,7 @@
 	export let settings: LayoutData["settings"];
 	export let webSearchMessages: WebSearchUpdate[] = [];
 	export let preprompt: string | undefined = undefined;
+	export let files: File[] = [];
 
 	$: isReadOnly = !models.some((model) => model.id === currentModel.id);
 
@@ -47,7 +53,25 @@
 		message = "";
 	};
 
+	let lastTarget: EventTarget | null = null;
+
+	let onDrag = false;
+
+	const onDragEnter = (e: DragEvent) => {
+		lastTarget = e.target;
+		onDrag = true;
+	};
+	const onDragLeave = (e: DragEvent) => {
+		if (e.target === lastTarget) {
+			onDrag = false;
+		}
+	};
+	const onDragOver = (e: DragEvent) => {
+		e.preventDefault();
+	};
 	$: lastIsError = messages[messages.length - 1]?.from === "user" && !loading;
+
+	$: sources = files.map((file) => file2base64(file));
 </script>
 
 <div class="relative min-h-0 min-w-0">
@@ -84,94 +108,134 @@
 			if (!loading) dispatch("retry", ev.detail);
 		}}
 	/>
+
 	<div
-		class="dark:via-gray-80 pointer-events-none absolute inset-x-0 bottom-0 z-0 mx-auto flex w-full max-w-3xl flex-col items-center justify-center bg-gradient-to-t from-white via-white/80 to-white/0 px-3.5 py-4 dark:border-gray-800 dark:from-gray-900 dark:to-gray-900/0 max-md:border-t max-md:bg-white max-md:dark:bg-gray-900 sm:px-5 md:py-8 xl:max-w-4xl [&>*]:pointer-events-auto"
+		class="pointer-events-none absolute inset-x-0 bottom-0 z-0 mx-auto flex w-full max-w-3xl flex-col items-center justify-center md:px-5 md:py-8 xl:max-w-4xl [&>*]:pointer-events-auto"
 	>
-		<div class="flex w-full pb-3">
-			{#if settings?.searchEnabled}
-				<WebSearchToggle />
-			{/if}
-			{#if loading}
-				<StopGeneratingBtn classNames="ml-auto" on:click={() => dispatch("stop")} />
-			{/if}
-			{#if lastIsError}
-				<RetryBtn
-					classNames="ml-auto"
-					on:click={() =>
-						dispatch("retry", {
-							id: messages[messages.length - 1].id,
-							content: messages[messages.length - 1].content,
-						})}
-				/>
-			{/if}
+		<div class="flex flex-row flex-wrap justify-center gap-2.5 max-md:pb-3">
+			{#each sources as source, index}
+				{#await source then src}
+					<div class="relative h-24 w-24 overflow-hidden rounded-lg shadow-lg">
+						<img
+							src={`data:image/*;base64,${src}`}
+							alt="input content"
+							class="h-full w-full rounded-lg bg-gray-400 object-cover dark:bg-gray-900"
+						/>
+						<!-- add a button on top that deletes this image from sources -->
+						<button
+							class="absolute left-1 top-1"
+							on:click={() => {
+								files = files.filter((_, i) => i !== index);
+							}}
+						>
+							<CarbonClose class="text-md font-black text-gray-300  hover:text-gray-100" />
+						</button>
+					</div>
+				{/await}
+			{/each}
 		</div>
-		<form
-			on:submit|preventDefault={handleSubmit}
-			class="relative flex w-full max-w-4xl flex-1 items-center rounded-xl border bg-gray-100 focus-within:border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:focus-within:border-gray-500
-			{isReadOnly ? 'opacity-30' : ''}"
+
+		<div
+			class="dark:via-gray-80 w-full bg-gradient-to-t from-white via-white/80 to-white/0 dark:border-gray-800 dark:from-gray-900 dark:to-gray-900/0 max-md:border-t max-md:bg-white max-md:px-4 max-md:dark:bg-gray-900"
 		>
-			<div class="flex w-full flex-1 border-none bg-transparent">
-				{#if lastIsError}
-					<ChatInput value="Sorry, something went wrong. Please try again." disabled={true} />
-				{:else}
-					<ChatInput
-						placeholder="Ask anything"
-						bind:value={message}
-						on:submit={handleSubmit}
-						on:keypress={(ev) => {
-							if ($page.data.loginRequired) {
-								ev.preventDefault();
-								loginModalOpen = true;
-							}
-						}}
-						maxRows={4}
-						disabled={isReadOnly || lastIsError}
+			<div class="flex w-full pb-3 max-md:pt-3">
+				{#if settings?.searchEnabled}
+					<WebSearchToggle />
+				{/if}
+				{#if loading}
+					<StopGeneratingBtn classNames="ml-auto" on:click={() => dispatch("stop")} />
+				{:else if lastIsError}
+					<RetryBtn
+						classNames="ml-auto"
+						on:click={() =>
+							dispatch("retry", {
+								id: messages[messages.length - 1].id,
+								content: messages[messages.length - 1].content,
+							})}
 					/>
+				{:else if currentModel.multimodal}
+					<UploadBtn bind:files classNames="ml-auto" />
 				{/if}
+			</div>
+			<form
+				on:dragover={onDragOver}
+				on:dragenter={onDragEnter}
+				on:dragleave={onDragLeave}
+				tabindex="-1"
+				aria-label="file dropzone"
+				on:submit|preventDefault={handleSubmit}
+				class="relative flex w-full max-w-4xl flex-1 items-center rounded-xl border bg-gray-100 focus-within:border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:focus-within:border-gray-500
+			{isReadOnly ? 'opacity-30' : ''}"
+			>
+				{#if onDrag && currentModel.multimodal}
+					<FileDropzone bind:files bind:onDrag />
+				{:else}
+					<div class="flex w-full flex-1 border-none bg-transparent">
+						{#if lastIsError}
+							<ChatInput value="Sorry, something went wrong. Please try again." disabled={true} />
+						{:else}
+							<ChatInput
+								placeholder="Ask anything"
+								bind:value={message}
+								on:submit={handleSubmit}
+								on:keypress={(ev) => {
+									if ($page.data.loginRequired) {
+										ev.preventDefault();
+										loginModalOpen = true;
+									}
+								}}
+								maxRows={4}
+								disabled={isReadOnly || lastIsError}
+							/>
+						{/if}
 
-				{#if loading}
-					<button
-						class="btn mx-1 my-1 inline-block h-[2.4rem] self-end rounded-lg bg-transparent p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100 md:hidden"
-						on:click={() => dispatch("stop")}
-					>
-						<CarbonStopFilledAlt />
-					</button>
-					<div
-						class="mx-1 my-1 hidden h-[2.4rem] items-center p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100 md:flex"
-					>
-						<EosIconsLoading />
+						{#if loading}
+							<button
+								class="btn mx-1 my-1 inline-block h-[2.4rem] self-end rounded-lg bg-transparent p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100 md:hidden"
+								on:click={() => dispatch("stop")}
+							>
+								<CarbonStopFilledAlt />
+							</button>
+							<div
+								class="mx-1 my-1 hidden h-[2.4rem] items-center p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100 md:flex"
+							>
+								<EosIconsLoading />
+							</div>
+						{:else}
+							<button
+								class="btn mx-1 my-1 h-[2.4rem] self-end rounded-lg bg-transparent p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100"
+								disabled={!message || isReadOnly}
+								type="submit"
+							>
+								<CarbonSendAltFilled />
+							</button>
+						{/if}
 					</div>
-				{:else}
+				{/if}
+			</form>
+			<div
+				class="mt-2 flex justify-between self-stretch px-1 text-xs text-gray-400/90 max-md:mb-2 max-sm:gap-2"
+			>
+				<p>
+					Model: <a
+						href={currentModel.modelUrl || "https://huggingface.co/" + currentModel.name}
+						target="_blank"
+						rel="noreferrer"
+						class="hover:underline">{currentModel.displayName}</a
+					> <span class="max-sm:hidden">·</span><br class="sm:hidden" /> Generated content may be inaccurate
+					or false.
+				</p>
+				{#if messages.length}
 					<button
-						class="btn mx-1 my-1 h-[2.4rem] self-end rounded-lg bg-transparent p-1 px-[0.7rem] text-gray-400 disabled:opacity-60 enabled:hover:text-gray-700 dark:disabled:opacity-40 enabled:dark:hover:text-gray-100"
-						disabled={!message || isReadOnly}
-						type="submit"
+						class="flex flex-none items-center hover:text-gray-400 hover:underline max-sm:rounded-lg max-sm:bg-gray-50 max-sm:px-2.5 dark:max-sm:bg-gray-800"
+						type="button"
+						on:click={() => dispatch("share")}
 					>
-						<CarbonSendAltFilled />
+						<CarbonExport class="text-[.6rem] sm:mr-1.5 sm:text-primary-500" />
+						<div class="max-sm:hidden">Share this conversation</div>
 					</button>
 				{/if}
 			</div>
-		</form>
-		<div class="mt-2 flex justify-between self-stretch px-1 text-xs text-gray-400/90 max-sm:gap-2">
-			<p>
-				Model: <a
-					href={currentModel.modelUrl || "https://huggingface.co/" + currentModel.name}
-					target="_blank"
-					rel="noreferrer"
-					class="hover:underline">{currentModel.displayName}</a
-				> <span class="max-sm:hidden">·</span><br class="sm:hidden" /> Generated content may be inaccurate
-				or false.
-			</p>
-			{#if messages.length}
-				<button
-					class="flex flex-none items-center hover:text-gray-400 hover:underline max-sm:rounded-lg max-sm:bg-gray-50 max-sm:px-2.5 dark:max-sm:bg-gray-800"
-					type="button"
-					on:click={() => dispatch("share")}
-				>
-					<CarbonExport class="text-[.6rem] sm:mr-1.5 sm:text-primary-500" />
-					<div class="max-sm:hidden">Share this conversation</div>
-				</button>
-			{/if}
 		</div>
 	</div>
 </div>

src/lib/components/chat/FileDropzone.svelte

@@ -0,0 +1,110 @@
+<script lang="ts">
+	import { onDestroy } from "svelte";
+	import CarbonImage from "~icons/carbon/image";
+	// import EosIconsLoading from "~icons/eos-icons/loading";
+
+	export let files: File[];
+
+	let file_error_message = "";
+	let errorTimeout: ReturnType<typeof setTimeout>;
+
+	export let onDrag = false;
+
+	async function dropHandle(event: DragEvent) {
+		event.preventDefault();
+		if (event.dataTransfer && event.dataTransfer.items) {
+			// Use DataTransferItemList interface to access the file(s)
+			if (files.length > 0) {
+				files = [];
+			}
+			// get only the first file
+			// optionally: we need to handle multiple files, if we want to support document upload for example
+			// for multimodal we only support one image at a time but we could support multiple PDFs
+			if (event.dataTransfer.items[0].kind === "file") {
+				const file = event.dataTransfer.items[0].getAsFile();
+				if (file) {
+					if (!event.dataTransfer.items[0].type.startsWith("image")) {
+						setErrorMsg("Only images are supported");
+						files = [];
+						return;
+					}
+					// if image is bigger than 2MB abort
+					if (file.size > 2 * 1024 * 1024) {
+						setErrorMsg("Image is too big. (2MB max)");
+						files = [];
+						return;
+					}
+					files = [file];
+					onDrag = false;
+				}
+			}
+		}
+	}
+
+	function setErrorMsg(errorMsg: string) {
+		if (errorTimeout) {
+			clearTimeout(errorTimeout);
+		}
+		file_error_message = errorMsg;
+		errorTimeout = setTimeout(() => {
+			file_error_message = "";
+			onDrag = false;
+		}, 2000);
+	}
+
+	onDestroy(() => {
+		if (errorTimeout) {
+			clearTimeout(errorTimeout);
+		}
+	});
+</script>
+
+<div
+	id="dropzone"
+	role="form"
+	on:drop={dropHandle}
+	class="relative flex w-full max-w-4xl flex-col items-center rounded-xl border bg-gray-100 focus-within:border-gray-300 dark:border-gray-600 dark:bg-gray-700 dark:focus-within:border-gray-500"
+>
+	<div class="object-center">
+		{#if file_error_message}
+			<div
+				class="absolute bottom-0 left-0 right-0 top-0 flex flex-col items-center justify-center gap-2 rounded-xl bg-gray-100 bg-opacity-50 dark:bg-gray-700 dark:bg-opacity-50"
+			>
+				<p class="text-red-500 dark:text-red-400">{file_error_message}</p>
+				<div class="h-2.5 w-1/2 rounded-full bg-gray-200 dark:bg-gray-700">
+					<div
+						class="animate-progress-bar h-2.5
+						rounded-full bg-red-500
+						dark:text-red-400
+					"
+					/>
+				</div>
+			</div>
+		{/if}
+		<div class="mt-3 flex justify-center" class:opacity-0={file_error_message}>
+			<CarbonImage class="text-5xl text-gray-500 dark:text-gray-400" />
+		</div>
+		<p
+			class="mb-3 mt-3 text-sm text-gray-500 dark:text-gray-400"
+			class:opacity-0={file_error_message}
+		>
+			Drag and drop <span class="font-semibold">one image</span> here
+		</p>
+	</div>
+</div>
+
+<style>
+	@keyframes slideInFromLeft {
+		0% {
+			width: 0;
+		}
+		100% {
+			width: 100%;
+		}
+	}
+
+	.animate-progress-bar {
+		/* This section calls the slideInFromLeft animation we defined above */
+		animation: 2s linear 0s 1 slideInFromLeft;
+	}
+</style>

src/lib/server/database.ts

@@ -1,5 +1,5 @@
 import { MONGODB_URL, MONGODB_DB_NAME, MONGODB_DIRECT_CONNECTION } from "$env/static/private";
-import { MongoClient } from "mongodb";
+import { GridFSBucket, MongoClient } from "mongodb";
 import type { Conversation } from "$lib/types/Conversation";
 import type { SharedConversation } from "$lib/types/SharedConversation";
 import type { WebSearch } from "$lib/types/WebSearch";
@@ -29,6 +29,7 @@ const settings = db.collection<Settings>("settings");
 const users = db.collection<User>("users");
 const webSearches = db.collection<WebSearch>("webSearches");
 const messageEvents = db.collection<MessageEvent>("messageEvents");
+const bucket = new GridFSBucket(db, { bucketName: "files" });
 
 export { client, db };
 export const collections = {
@@ -39,6 +40,7 @@ export const collections = {
 	users,
 	webSearches,
 	messageEvents,
+	bucket,
 };
 
 client.on("open", () => {

src/lib/server/endpoints/endpoints.ts

@@ -11,6 +11,7 @@ interface EndpointParameters {
 	conversation: {
 		messages: Omit<Conversation["messages"][0], "id">[];
 		preprompt?: Conversation["preprompt"];
+		_id?: Conversation["_id"];
 	};
 }
 

src/lib/server/endpoints/tgi/endpointTgi.ts

@@ -23,6 +23,7 @@ export function endpointTgi({
 			webSearch: conversation.messages[conversation.messages.length - 1].webSearch,
 			preprompt: conversation.preprompt,
 			model,
+			id: conversation._id,
 		});
 
 		return textGenerationStream({

src/lib/server/files/downloadFile.ts

@@ -0,0 +1,36 @@
+import { error } from "@sveltejs/kit";
+import { collections } from "../database";
+import type { Conversation } from "$lib/types/Conversation";
+import type { SharedConversation } from "$lib/types/SharedConversation";
+
+export async function downloadFile(
+	sha256: string,
+	convId: Conversation["_id"] | SharedConversation["_id"]
+) {
+	const fileId = collections.bucket.find({ filename: `${convId.toString()}-${sha256}` });
+	let mime = "";
+
+	const content = await fileId.next().then(async (file) => {
+		if (!file) {
+			throw error(404, "File not found");
+		}
+		if (file.metadata?.conversation !== convId.toString()) {
+			throw error(403, "You don't have access to this file.");
+		}
+
+		mime = file.metadata?.mime;
+
+		const fileStream = collections.bucket.openDownloadStream(file._id);
+
+		const fileBuffer = await new Promise<Buffer>((resolve, reject) => {
+			const chunks: Uint8Array[] = [];
+			fileStream.on("data", (chunk) => chunks.push(chunk));
+			fileStream.on("error", reject);
+			fileStream.on("end", () => resolve(Buffer.concat(chunks)));
+		});
+
+		return fileBuffer;
+	});
+
+	return { content, mime };
+}

src/lib/server/files/uploadFile.ts

@@ -0,0 +1,21 @@
+import type { Conversation } from "$lib/types/Conversation";
+import { sha256 } from "$lib/utils/sha256";
+import { collections } from "../database";
+
+export async function uploadFile(file: Blob, conv: Conversation): Promise<string> {
+	const sha = await sha256(await file.text());
+
+	const upload = collections.bucket.openUploadStream(`${conv._id}-${sha}`, {
+		metadata: { conversation: conv._id.toString(), mime: "image/jpeg" },
+	});
+
+	upload.write((await file.arrayBuffer()) as unknown as Buffer);
+	upload.end();
+
+	// only return the filename when upload throws a finish event or a 10s time out occurs
+	return new Promise((resolve, reject) => {
+		upload.once("finish", () => resolve(sha));
+		upload.once("error", reject);
+		setTimeout(() => reject(new Error("Upload timed out")), 10000);
+	});
+}

src/lib/server/models.ts

@@ -57,6 +57,7 @@ const modelConfig = z.object({
 		})
 		.passthrough()
 		.optional(),
+	multimodal: z.boolean().default(false),
 });
 
 const modelsRaw = z.array(modelConfig).parse(JSON.parse(MODELS));
@@ -144,4 +145,4 @@ export const smallModel = TASK_MODEL
 	  defaultModel
 	: defaultModel;
 
-export type BackendModel = Optional<typeof defaultModel, "preprompt" | "parameters">;
+export type BackendModel = Optional<typeof defaultModel, "preprompt" | "parameters" | "multimodal">;

src/lib/stores/pendingMessage.ts

@@ -1,3 +1,9 @@
 import { writable } from "svelte/store";
 
-export const pendingMessage = writable<string>("");
+export const pendingMessage = writable<
+	| {
+			content: string;
+			files: File[];
+	  }
+	| undefined
+>();

src/lib/types/Message.ts

@@ -10,4 +10,5 @@ export type Message = Partial<Timestamps> & {
 	webSearchId?: WebSearch["_id"]; // legacy version
 	webSearch?: WebSearch;
 	score?: -1 | 0 | 1;
+	files?: string[]; // can contain either the hash of the file or the b64 encoded image data on the client side when uploading
 };

src/lib/types/MessageUpdate.ts

@@ -31,9 +31,16 @@ export type StatusUpdate = {
 	message?: string;
 };
 
+export type ErrorUpdate = {
+	type: "error";
+	message: string;
+	name: string;
+};
+
 export type MessageUpdate =
 	| FinalAnswer
 	| TextStreamUpdate
 	| AgentUpdate
 	| WebSearchUpdate
-	| StatusUpdate;
+	| StatusUpdate
+	| ErrorUpdate;

src/lib/types/Model.ts

@@ -13,4 +13,5 @@ export type Model = Pick<
 	| "modelUrl"
 	| "datasetUrl"
 	| "preprompt"
+	| "multimodal"
 >;

src/lib/utils/file2base64.ts

@@ -0,0 +1,14 @@
+const file2base64 = (file: File): Promise<string> => {
+	return new Promise<string>((resolve, reject) => {
+		const reader = new FileReader();
+		reader.readAsDataURL(file);
+		reader.onload = () => {
+			const dataUrl = reader.result as string;
+			const base64 = dataUrl.split(",")[1];
+			resolve(base64);
+		};
+		reader.onerror = (error) => reject(error);
+	});
+};
+
+export default file2base64;

src/lib/utils/models.ts

@@ -1,4 +1,4 @@
 import type { Model } from "$lib/types/Model";
 
-export const findCurrentModel = (models: Model[], id?: string) =>
+export const findCurrentModel = (models: Model[], id?: string): Model =>
 	models.find((m) => m.id === id) ?? models[0];

src/routes/+layout.server.ts

@@ -102,6 +102,7 @@ export const load: LayoutServerLoad = async ({ locals, depends, url }) => {
 			promptExamples: model.promptExamples,
 			parameters: model.parameters,
 			preprompt: model.preprompt,
+			multimodal: model.multimodal,
 		})),
 		oldModels,
 		user: locals.user && {

src/routes/+page.svelte

@@ -9,6 +9,7 @@
 
 	export let data;
 	let loading = false;
+	let files: File[] = [];
 
 	async function createConversation(message: string) {
 		try {
@@ -33,7 +34,10 @@
 			const { conversationId } = await res.json();
 
 			// Ugly hack to use a store as temp storage, feel free to improve ^^
-			pendingMessage.set(message);
+			pendingMessage.set({
+				content: message,
+				files,
+			});
 
 			// invalidateAll to update list of conversations
 			await goto(`${base}/conversation/${conversationId}`, { invalidateAll: true });
@@ -56,4 +60,5 @@
 	currentModel={findCurrentModel([...data.models, ...data.oldModels], data.settings.activeModel)}
 	models={data.models}
 	settings={data.settings}
+	bind:files
 />

src/routes/conversation/[id]/+page.svelte

@@ -14,7 +14,7 @@
 	import type { Message } from "$lib/types/Message";
 	import type { MessageUpdate, WebSearchUpdate } from "$lib/types/MessageUpdate";
 	import titleUpdate from "$lib/stores/titleUpdate";
-
+	import file2base64 from "$lib/utils/file2base64.js";
 	export let data;
 
 	let messages = data.messages;
@@ -32,6 +32,8 @@
 	let loading = false;
 	let pending = false;
 
+	let files: File[] = [];
+
 	async function convFromShared() {
 		try {
 			loading = true;
@@ -79,14 +81,37 @@
 				retryMessageIndex = messages.length;
 			}
 
+			const module = await import("browser-image-resizer");
+
+			// currently, only IDEFICS is supported by TGI
+			// the size of images is hardcoded to 224x224 in TGI
+			// this will need to be configurable when support for more models is added
+			const resizedImages = await Promise.all(
+				files.map(async (file) => {
+					return await module
+						.readAndCompressImage(file, {
+							maxHeight: 224,
+							maxWidth: 224,
+							quality: 1,
+						})
+						.then(async (el) => await file2base64(el as File));
+				})
+			);
+
 			// slice up to the point of the retry
 			messages = [
 				...messages.slice(0, retryMessageIndex),
-				{ from: "user", content: message, id: messageId },
+				{
+					from: "user",
+					content: message,
+					id: messageId,
+					files: isRetry ? messages[retryMessageIndex].files : resizedImages,
+				},
 			];
 
-			const responseId = randomUUID();
+			files = [];
 
+			const responseId = randomUUID();
 			const response = await fetch(`${base}/conversation/${$page.params.id}`, {
 				method: "POST",
 				headers: { "Content-Type": "application/json" },
@@ -96,9 +121,11 @@
 					response_id: responseId,
 					is_retry: isRetry,
 					web_search: $webSearchParameters.useSearch,
+					files: isRetry ? undefined : resizedImages,
 				}),
 			});
 
+			files = [];
 			if (!response.body) {
 				throw new Error("Body not defined");
 			}
@@ -107,6 +134,7 @@
 				error.set((await response.json())?.message);
 				return;
 			}
+
 			// eslint-disable-next-line no-undef
 			const encoder = new TextDecoderStream();
 			const reader = response?.body?.pipeThrough(encoder).getReader();
@@ -143,6 +171,8 @@
 							if (update.type === "finalAnswer") {
 								finalAnswer = update.text;
 								reader.cancel();
+								loading = false;
+								pending = false;
 								invalidate(UrlDependency.Conversation);
 							} else if (update.type === "stream") {
 								pending = false;
@@ -174,6 +204,9 @@
 								} else if (update.status === "error") {
 									$error = update.message ?? "An error has occurred";
 								}
+							} else if (update.type === "error") {
+								error.set(update.message);
+								reader.cancel();
 							}
 						} catch (parseError) {
 							// in case of parsing error we wait for the next message
@@ -233,8 +266,9 @@
 	onMount(async () => {
 		// only used in case of creating new conversations (from the parent POST endpoint)
 		if ($pendingMessage) {
-			await writeMessage($pendingMessage);
-			$pendingMessage = "";
+			files = $pendingMessage.files;
+			await writeMessage($pendingMessage.content);
+			$pendingMessage = undefined;
 		}
 	});
 
@@ -264,7 +298,7 @@
 		}
 	}
 
-	$: $page.params.id, (isAborted = true);
+	$: $page.params.id, ((isAborted = true), (loading = false));
 	$: title = data.conversations.find((conv) => conv.id === $page.params.id)?.title ?? data.title;
 </script>
 
@@ -285,6 +319,7 @@
 	shared={data.shared}
 	preprompt={data.preprompt}
 	bind:webSearchMessages
+	bind:files
 	on:message={onMessage}
 	on:retry={onRetry}
 	on:vote={(event) => voteMessage(event.detail.score, event.detail.id)}

src/routes/conversation/[id]/+server.ts

@@ -12,6 +12,8 @@ import { runWebSearch } from "$lib/server/websearch/runWebSearch";
 import type { WebSearch } from "$lib/types/WebSearch";
 import { abortedGenerations } from "$lib/server/abortedGenerations";
 import { summarize } from "$lib/server/summarize";
+import { uploadFile } from "$lib/server/files/uploadFile.js";
+import sizeof from "image-size";
 
 export async function POST({ request, locals, params, getClientAddress }) {
 	const id = z.string().parse(params.id);
@@ -92,6 +94,7 @@ export async function POST({ request, locals, params, getClientAddress }) {
 		id: messageId,
 		is_retry,
 		web_search: webSearch,
+		files: b64files,
 	} = z
 		.object({
 			inputs: z.string().trim().min(1),
@@ -99,9 +102,42 @@ export async function POST({ request, locals, params, getClientAddress }) {
 			response_id: z.optional(z.string().uuid()),
 			is_retry: z.optional(z.boolean()),
 			web_search: z.optional(z.boolean()),
+			files: z.optional(z.array(z.string())),
 		})
 		.parse(json);
 
+	// files is an array of base64 strings encoding Blob objects
+	// we need to convert this array to an array of File objects
+
+	const files = b64files?.map((file) => {
+		const blob = Buffer.from(file, "base64");
+		return new File([blob], "image.png");
+	});
+
+	// check sizes
+	if (files) {
+		const filechecks = await Promise.all(
+			files.map(async (file) => {
+				const dimensions = sizeof(Buffer.from(await file.arrayBuffer()));
+				return (
+					file.size > 2 * 1024 * 1024 ||
+					(dimensions.width ?? 0) > 224 ||
+					(dimensions.height ?? 0) > 224
+				);
+			})
+		);
+
+		if (filechecks.some((check) => check)) {
+			throw error(413, "File too large, should be <2MB and 224x224 max.");
+		}
+	}
+
+	let hashes: undefined | string[];
+
+	if (files) {
+		hashes = await Promise.all(files.map(async (file) => await uploadFile(file, conv)));
+	}
+
 	// get the list of messages
 	// while checking for retries
 	let messages = (() => {
@@ -113,7 +149,13 @@ export async function POST({ request, locals, params, getClientAddress }) {
 			}
 			return [
 				...conv.messages.slice(0, retryMessageIdx),
-				{ content: newPrompt, from: "user", id: messageId as Message["id"], updatedAt: new Date() },
+				{
+					content: newPrompt,
+					from: "user",
+					id: messageId as Message["id"],
+					updatedAt: new Date(),
+					files: conv.messages[retryMessageIdx]?.files,
+				},
 			];
 		} // else append the message at the bottom
 
@@ -125,6 +167,7 @@ export async function POST({ request, locals, params, getClientAddress }) {
 				id: (messageId as Message["id"]) || crypto.randomUUID(),
 				createdAt: new Date(),
 				updatedAt: new Date(),
+				files: hashes,
 			},
 		];
 	})() satisfies Message[];
@@ -268,6 +311,8 @@ export async function POST({ request, locals, params, getClientAddress }) {
 				type: "finalAnswer",
 				text: messages[messages.length - 1].content,
 			});
+
+			return;
 		},
 		async cancel() {
 			await collections.conversations.updateOne(

src/routes/conversation/[id]/output/[sha256]/+server.ts

@@ -0,0 +1,49 @@
+import { authCondition } from "$lib/server/auth";
+import { collections } from "$lib/server/database";
+import { error } from "@sveltejs/kit";
+import { ObjectId } from "mongodb";
+import { z } from "zod";
+import type { RequestHandler } from "./$types";
+import { downloadFile } from "$lib/server/files/downloadFile";
+
+export const GET: RequestHandler = async ({ locals, params }) => {
+	const sha256 = z.string().parse(params.sha256);
+
+	const userId = locals.user?._id ?? locals.sessionId;
+
+	// check user
+	if (!userId) {
+		throw error(401, "Unauthorized");
+	}
+
+	if (params.id.length !== 7) {
+		const convId = new ObjectId(z.string().parse(params.id));
+
+		// check if the user has access to the conversation
+		const conv = await collections.conversations.findOne({
+			_id: convId,
+			...authCondition(locals),
+		});
+
+		if (!conv) {
+			throw error(404, "Conversation not found");
+		}
+	} else {
+		// check if the user has access to the conversation
+		const conv = await collections.sharedConversations.findOne({
+			_id: params.id,
+		});
+
+		if (!conv) {
+			throw error(404, "Conversation not found");
+		}
+	}
+
+	const { content, mime } = await downloadFile(sha256, params.id);
+
+	return new Response(content, {
+		headers: {
+			"Content-Type": mime ?? "application/octet-stream",
+		},
+	});
+};

src/routes/conversation/[id]/share/+server.ts

@@ -43,6 +43,23 @@ export async function POST({ params, url, locals }) {
 
 	await collections.sharedConversations.insertOne(shared);
 
+	// copy files from `${conversation._id}-` to `${shared._id}-`
+	const files = await collections.bucket
+		.find({ filename: { $regex: `${conversation._id}-` } })
+		.toArray();
+
+	await Promise.all(
+		files.map(async (file) => {
+			const newFilename = file.filename.replace(`${conversation._id}-`, `${shared._id}-`);
+			// copy files from `${conversation._id}-` to `${shared._id}-` by downloading and reuploaidng
+			const downloadStream = collections.bucket.openDownloadStream(file._id);
+			const uploadStream = collections.bucket.openUploadStream(newFilename, {
+				metadata: { ...file.metadata, conversation: shared._id.toString() },
+			});
+			downloadStream.pipe(uploadStream);
+		})
+	);
+
 	return new Response(
 		JSON.stringify({
 			url: getShareUrl(url, shared._id),