Model resources usage
Hello, I tried to run this model on T4 and A10 GPUs and ran into datatypes and memory problems.
Is it only possible to run it on high-end GPU or I am doing something wrong?
Should not be that memory constraining, it runs on MacBooks or on local small GPUs. Are you loading it in bf16 ? Did you play with the batch size during indexing ?
Very weird... It OOMs on GPU ? Can queries be indexed ?
Wrong account on the phone :),
Yes. I run it successfully but slowly on my Mac so I switched to A10 on g5.xlarge on AWS and limited the batch size to 1 and the program gets killed after images are processed and the model part starts. I used bf16 as in the repo example
Sorry for the mistake but I went through my pipeline once again and I realised that it is infact processing images from one long pdf which crashes because I was iterating over files, not pdf pages. Then the model works without OOM. By bad :(