Norallm vs Norwai
I followed the release of NorLLM (https://www.ntnu.no/norllm/lansering-av-neste-generasjon-norllm-modeller) where they refered to their versions of Mistral-models.
How are your models compared to NorwAis models, and do you work together or is this separarate efforts?
I’ll let the NoraLLM crew answer in detail, but from what I understand there is no direct collaboration in training or work on these models. They’re quite direct competitors.
Obviously they’re still part of the same academic field in a quite small country, so on a personal and academic level there’s bound to be some intermingling.
I’ll also mention that the NorwAI models (they renamed them to NorwAI to reduce confusion) have documented less about their training data, but it’s natural to assume that the base dataset is not too unlike the NorLLM models.
I'm also curious how do the models compare, but unfortunately, their models are not published anywhere, as far as I know. As their models essentially replicate our approach, I don't expect a big difference in performance or the overall behavior.
There has been a lot of activity in the space of large Norwegian language models recently, which is awesome! Just a few days ago, Bineric released NorskGPT-Llama-3-70b-adapter, taking a very different approach to us: they take a very large base model and then train a lightweight adapter on top of it with just 1 million subword tokens. On the other end of the "total training cost" spectrum, our friends from Finland have just released Viking-7b, a Nordic model trained from scratch on 2 trillion tokens.
My mistake, I thought the NorwAI models were already made public on HF during the release May 15 (I have private access so I could see them anyway). I should probably be a little cautious about sharing more details, then.
From what I understand publication on HF should happen quite soon, but I don't have a date. I would guess within a week or so, but I don't know specifically. The org URL is https://huggingface.co/NorwAI, so watch that space :)
i now see a mixtral 8x7B here https://huggingface.co/NorwAI/NorwAI-Mixtral-8x7B-instruct
Well would you look at that, they published it within a couple hours of my comment. Nice! There may be more coming, but great to see they're getting things public!
They have different licenses. This model has apache 2.0 which is a fully permissive and truly open source license.
The NorLLM models have custom NorLLM license and is not truly open source. Restrictions include geographical usage (only permitted in nordic countries for some reason),
usage for unethical stuff, and "resale" which i'm not fully sure how should be interpreted. (maybe selling products that incorporate the model?)
The NorLLM models have custom NorLLM license and is not truly open source. Restrictions include geographical usage (only permitted in nordic countries for some reason),
usage for unethical stuff, and "resale" which i'm not fully sure how should be interpreted. (maybe selling products that incorporate the model?)
It would be nice if NorwAI could comment on the licence and how it should be interpreted. My take - it is permissible to use the model for inference in commercial products. I will ask them :)
I see a discussion has been started on this over on one of NorwAI’s model pages: https://huggingface.co/NorwAI/NorwAI-Mistral-7B-instruct/discussions/2
(I see you did it Janus, just putting the link here for reference😊)