This article is mostly about current problems and system achievements. If you do not want to read about the exact line of logic where something broke — and how I did or did not fix it — you may want to skip it XD
Hello again! Development is moving at full speed, especially after I received a powerful server for my birthday :)
As you know, AION is not just another LLM or a LoRa-like GPT. It is a much more complex system internally (see AION First). That complexity creates training problems. With a standard LLM, the process is relatively simple: provide a huge dataset of text and answers, wait for PPL and loss to decrease, and make sure the model does not overfit.
My system requires much more work. AION has three main modules: Concept Space (CS), Language Space (LS), and Librarian (LB). Each module must be trained, but is there a large text-to-concept or token-to-graph dataset? Unfortunately, nothing like that exists. I have to design and build the training process for every module.
The main issue is what I call a “cyclical problem.” The system works like this:
Text Input → LS → LB → CS → LS → Text Output
You never know exactly what broke: CS, LS, or LB. LB fails most often because it is one of the first elements in the chain and creates the graph of new knowledge. Sometimes LS anonymizes the text incorrectly. For example, from “Egor likes playing Roblox and often walks the dog,” LB may create Egor → dog instead of Egor → Roblox. I suspect this comes from a lack of difficult training data; many datasets have to be created manually. As of 06.08.26, I am training LB on roughly 2,000 lines, which is not much.
I am considering using a small LLM to generate datasets. It is funny: I am using an LLM I strongly dislike to build a new system that, according to my predictions, will become much better and smarter than LLMs themselves XD
I have also started studying neuroscience and psychology to make the neural system more human-like. I have already created several psychology-inspired mechanisms that could give AION its own digital personality and psyche. It is still an early alpha-stage project, so I cannot claim that I created AGI or gave it self-awareness. But the direction is promising.
I used to write about AION as if it would be ready tomorrow. But active development keeps revealing architectural problems. Every time I think we are almost finished, I discover another mechanism that must be removed or replaced. So I am now afraid to say “almost ready” XD
What I can say for certain is that I plan to release AION alpha on August 15 and give early access to a limited number of people. AION First is planned for August 31 or September 1. Setting deadlines for myself seems more effective than developing without a boundary :)
Thank you for reading! More articles are coming soon :)