A research startup to add to introduce Dynamic Transformers architecture for Large Language Models.
Research startup idea is to create a Dynamic Transformers architecture where all weights can change in time depending on the input. So model can learn continuously, a bit similar to human brain. This approach allows to "Unlock" static frozen LLM weights and totally eliminate LLM context window issue.
Current issue with all modern Large Language Models is their lock in a static state which is not changed after initial model creation. Such models should be trained again from scratch to reflect any changes in its weights. Existing approaches such as LoRa which allow to change small amount of model layer cannot provide comprehensive model update, it's still the same model for most of internal layers.
Current issue with all modern Large Language Models is their lock in a static state which is not changed after initial model creation. Such models should be trained again from scratch to reflect any changes in its weights. Existing approaches such as LoRa which allow to change small amount of model layer cannot provide comprehensive model update, it's still the same model for most of internal layers.
What Would This Enable?
- True individuality: No two Fluid LLMs would be the same, even if they started from the same checkpoint.
- Unpredictable growth: Some models might become “experts” in a user’s niche interests, others become strange and idiosyncratic.
- Organic errors: The model can develop “quirks” or “biases”—again, like real humans.
Techinical Details
- Fluid LLM Vision: No “training vs inference” dichotomy. Every “inference” (model usage) naturally is a training step: the weights evolve based on the input and maybe outputs, even if the user isn’t explicitly “fine-tuning” or giving feedback.
- No external controls on drift or forgetting. The model is free to evolve as it will—like a person’s personality and memory change over years, with no “admin panel” to reset or constrain the process.
- No external evaluation necessary. The goal isn’t to benchmark or keep the model static; the whole point is to observe and experience the evolution.
- Implications & How This Might Look in Practice 1. Every Interaction Changes the ModelEven casual conversation tweaks weights—some connections strengthen, others weaken.
- Over time, the model becomes uniquely attuned to its environment, possibly idiosyncratic (just like people!).
- No Catastrophic Forgetting “Protection”If the user changes topics for a while, the model’s old expertise can fade, new patterns take over.
- It’s not a bug, it’s a feature — like how humans lose proficiency in unused languages or skills.
- Pure “Evolving Organism” ParadigmThe model is more like a living thing—shaped by its life, not “kept pure” for anyone else.
- Possible Implementation (Rough Ideas)On every token/step: Apply an online update to the weights, based on the current prediction error (loss), possibly with a very small learning rate.
- Data stream: The model never “sees” a batch or an epoch, just an endless stream of real user interactions.
- No replay, no regularization: Just forward motion.
- Organic errors: The model can develop “quirks” or “biases”—again, like real humans.
- Technical Openings: Model can use some more brain-inspired online update rule (Hebbian, Oja’s)
Meet the team
Get to know the dedicated individuals behind our AI solutions. Learn about their expertise and contributions.
-
Oleksii LiashukFounder, Senior AI Researcher, PhD -
Konstantin GlubokovSenior AI Researcher -
Liliia KorostylyovaFunctional analyst, mathematic reasoning
Html code will be here
