h2oai

/

h2ogpt-oasst1-falcon-40b

like 17

I added the file, but @arnocandel will verify why that is needed etc.

It doesn't run as:
python generate.py --base_model=h2oai/h2ogpt-oasst1-falcon-40b --load_8bit=True

works for me from a directory that doesn't contain any files

(env) arno@rippa:/nfs4/llm/h2ogpt(main)$ mkdir tmp
(env) arno@rippa:/nfs4/llm/h2ogpt(main)$ cd tmp
(env) arno@rippa:/nfs4/llm/h2ogpt/tmp(main)$ python ../generate.py --base_model=h2oai/h2ogpt-oasst1-falcon-40b --load_8bit=True
bin /nfs4/llm/h2o-llm/env/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
Using Model h2oai/h2ogpt-oasst1-falcon-40b
device_map: {'': 0}
Downloading shards: 100%|████████████████████████████████████████████████████████████████| 18/18 [00:02<00:00,  8.68it/s]
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████| 18/18 [06:27<00:00, 21.52s/it]
device_map: {'': 2}
Running on local URL:  http://0.0.0.0:7860
Running on public URL: https://14218264691331d5ff.gradio.live

This share link expires in 72 hours. For free permanent hosting and GPU upgrades (NEW!), check out Spaces: https://huggingface.co/spaces

I found that updating transformers solves the issue
pip install transformers==4.29.2

Thanks, I updated the README. Sorry about that.