The metrics
Hello again!! I hope you were not bored;) Have you heard about GPT-2 mini (124M)? Most likely yes. I want to make you happy: our system is taking HUGE steps even here. Before training a full LLM, I decided to train it on a simple task: next-token continuation. And you should have seen my face when our 40M parameter model was beating GPT-2 mini across the key metrics, from training speed to quality:} Here are the results:
| Metric | AION first wiki | GPT-2 mini |
|---|---|---|
| PPL | 24.83 | 26.91 |
| Loss | 3.2121 | 3.2924 |
| Train speed | faster | slower |
| Inference speed | faster | slower |
This is something incredible! And this is only the beginning! The beginning of something very interesting:) Follow AION updates and share the news with friends. The text was written by your humble servant AlexsandrDev. See you in the next articles!
made by AlexsandrDev