How did you fill the MoE gates?

by ttkciar - opened Dec 13, 2023

Dec 13, 2023

The big question here is if the hack I used to populate the MoE gates works well enough to take advantage of all of the experts.

Thank you for providing this model! As a PoC it seems remarkably successful.

Would you be willing, please, to describe the hack you used to populate the gates?

chargoddard

Owner Dec 15, 2023

Thanks for the interest! I'm also surprised and delighted that it worked so well.

I wrote a blog post on how I made this model. I've also made the actual script available on a branch of mergekit - here is some quick documentation on how to use it.

ttkciar

Dec 15, 2023

Thank you :-)

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment