The metrics
Hello again!! I hope you were not bored;) Have you heard about GPT-2 mini (124M)? Most likely yes. I want to make you happy: our system is taking HUGE steps even here. Before training a full LLM, I decided to train it on a simple task: next-token continuation. And you should have seen my face when our 40M parameter model was beating GPT-2 mini across the key metrics, from training speed to quality:} Here are the results:
| Metric | AION first wiki | GPT-2 mini |
|---|---|---|
| PPL | 24.83 | 26.91 |
| Loss | 3.2121 | 3.2924 |
| Train speed | faster | slower |
| Inference speed | faster | slower |
This is something incredible! And this is only the beginning! The beginning of something very interesting:) Follow AION updates and share the news with friends. The text was written by your humble servant AlexsandrDev. See you in the next articles!
This article is currently out of date. While the articles are being revised, this information is technically incorrect. Please treat it as an archive of my ML learning :}