Reactive Transformer MVP model with 3B total params and 190M activated in decoder. Training in progress
Reactive AI
\nWe are working on our own ideas of Reactive Neural Networks (RxNN) and Event-Driven AI, advancing from language models to AGI awareness models.
\nReactive Neural Networks and Event-Driven AI
\nReactive Neural Networks (RxNN) are memory-augmented neural networks with higher levels of recurrence (inter-sequence vs. intra-sequence in RNNs),\nfocused on processing single interactions with access to previous interactions via memory layers. We call this event-driven real-time processing\nto distinguish it from classical data-driven processing of the full conversation history in each interaction. This difference is crucial in case\nof AGI and awareness - the key feature of humans awareness, is that we remember what we were doing 10 mins ago, without recalling the whole-day history - we\nare working in real-time - just like event-driven Reactive Neural Networks.
\nIn Event-Driven AI models are processing the data in reaction to environment or internal events, and are emitting other response events as a result.\nProcessing of input and output events by the model is called the interaction. Event or an interaction could occur in any point in continous time. Models\nhave to be stateful and remember the data between the interactions.
\nStrong Reactive Neural Networks like Reactor could emit and listen to its internal events, while the Weak Reactive Neural Networks are\nworking only on environment events.
\nReactor AGI
\n\n\nOur primary architecture - Reactor - is planned as the first awareness AGI model, that's modelling awareness as an Infinite Chain-of-Thoughts,\nconnected to Short-Term and Long-Term Memory (Attention-based Memory System) and Receptors/Effectors systems for real-time reactive processing.\nIt will be able to constantly and autonomously learn from interactions in Continouos Live Learning process.
\n\n\nReactor architecture details and mathematical model were analysed by 30 state-of-the-art LLM/Reasoning models that rated it's potential\nto reach the AGI as ~4.35/5
\n
Reactive Language Models (RxLM)
\nWhile the Reactor is the main goal, it's extremely hard to achieve, as it's definitely the most advanced neural network ensemble ever.
\nThat's why we designed simplified architectures, for incremental transformation from language/reasoning models to awareness model:
\n- \n
- Reactive Transformer is introducing Attention-based Memory System and adding Short-Term Memory to Transformer language models \n
- Preactor is adding Long-Term Memory and ability to learn from interactions \n
RxLM vs LLM advantages
\nProcessing single interactions in real-time by Reactive Language Models leads to revolutional improvements in inference speed/cost:
\n- \n
- LLM inference costs are increasing quadratically with conversation length (accumulated for each next message), because of full dialog history processing \n
- RxLM inference costs are linear, depending only on single interaction tokens (not accumulated) - each next interaction is
number of stepstimes cheaper than for LLM \n - same for inference speed - LLM has to process full history, while RxLM only single message (only first interaction could be slower because of encoder/memory attention overhead) \n
\n\nIn example, for a dialog with DeepSeek R1, that have overally ~90k tokens, I paid for about 1.5M tokens. With RxLM it will cost only that ~90k tokens, so it\nwill be about 15x cheaper
\n
RxNN Platform
\nAdditional Research
\n- \n
- Sparse Query Attention (SQA) - the most cost-effective GQA variant, even 2-3x faster for long sequences! \n
- Flex-SQA - combination of Flex Attention and (symmetric) Sparse Query Attention, enabling 4-8x longer sliding windows \n
- Flex Memory Attention/Memory Cross-Attention - connecting spatially sparse attention with memory layers to enable very long single interactions - smaller sliding window for input sequences attends to full memory, or the opposite \n
- Mixture-of-Experts for Grouped Attention - MoE Router dynamically selects GQA/SQA groups, instead of static selection. Abandoned, because results were worse than for GQA/SQA \n
AI & ML interests
AGI, ASI, Reactive Awareness Models, Real-Time Reactive Language Models, Memory Systems, Reactive Neural Networks & Event-Driven AI
Recent Activity
Papers
TensorBLEU: Vectorized GPU-based BLEU Score Implementation for Per-Sentence In-Training Evaluation
Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models
Organization Card
Reactive AI
We are working on our own ideas of Reactive Neural Networks (RxNN) and Event-Driven AI, advancing from language models to AGI awareness models.
Reactive Neural Networks and Event-Driven AI
Reactive Neural Networks (RxNN) are memory-augmented neural networks with higher levels of recurrence (inter-sequence vs. intra-sequence in RNNs), focused on processing single interactions with access to previous interactions via memory layers. We call this event-driven real-time processing to distinguish it from classical data-driven processing of the full conversation history in each interaction. This difference is crucial in case of AGI and awareness - the key feature of humans awareness, is that we remember what we were doing 10 mins ago, without recalling the whole-day history - we are working in real-time - just like event-driven Reactive Neural Networks.
In Event-Driven AI models are processing the data in reaction to environment or internal events, and are emitting other response events as a result. Processing of input and output events by the model is called the interaction. Event or an interaction could occur in any point in continous time. Models have to be stateful and remember the data between the interactions.
Strong Reactive Neural Networks like Reactor could emit and listen to its internal events, while the Weak Reactive Neural Networks are working only on environment events.
Reactor AGI
Our primary architecture - Reactor - is planned as the first awareness AGI model, that's modelling awareness as an Infinite Chain-of-Thoughts, connected to Short-Term and Long-Term Memory (Attention-based Memory System) and Receptors/Effectors systems for real-time reactive processing. It will be able to constantly and autonomously learn from interactions in Continouos Live Learning process.
Reactor architecture details and mathematical model were analysed by 30 state-of-the-art LLM/Reasoning models that rated it's potential to reach the AGI as ~4.35/5
Reactive Language Models (RxLM)
While the Reactor is the main goal, it's extremely hard to achieve, as it's definitely the most advanced neural network ensemble ever.
That's why we designed simplified architectures, for incremental transformation from language/reasoning models to awareness model:
- Reactive Transformer is introducing Attention-based Memory System and adding Short-Term Memory to Transformer language models
- Preactor is adding Long-Term Memory and ability to learn from interactions
RxLM vs LLM advantages
Processing single interactions in real-time by Reactive Language Models leads to revolutional improvements in inference speed/cost:
- LLM inference costs are increasing quadratically with conversation length (accumulated for each next message), because of full dialog history processing
- RxLM inference costs are linear, depending only on single interaction tokens (not accumulated) - each next interaction is
number of stepstimes cheaper than for LLM - same for inference speed - LLM has to process full history, while RxLM only single message (only first interaction could be slower because of encoder/memory attention overhead)
In example, for a dialog with DeepSeek R1, that have overally ~90k tokens, I paid for about 1.5M tokens. With RxLM it will cost only that ~90k tokens, so it will be about 15x cheaper
RxNN Platform
Additional Research
- Sparse Query Attention (SQA) - the most cost-effective GQA variant, even 2-3x faster for long sequences!
- Flex-SQA - combination of Flex Attention and (symmetric) Sparse Query Attention, enabling 4-8x longer sliding windows
- Flex Memory Attention/Memory Cross-Attention - connecting spatially sparse attention with memory layers to enable very long single interactions - smaller sliding window for input sequences attends to full memory, or the opposite
- Mixture-of-Experts for Grouped Attention - MoE Router dynamically selects GQA/SQA groups, instead of static selection. Abandoned, because results were worse than for GQA/SQA
models
28
ReactiveAI/RxT-Beta-MLM-iSFT
Updated
•
173
ReactiveAI/RxT-Beta-Encoder-iSFT
97.1M
•
Updated
•
192
ReactiveAI/RxT-Beta-Decoder-iSFT
3B
•
Updated
•
187
ReactiveAI/RxT-Beta-Mini-Encoder-Base
57.4M
•
Updated
•
251
ReactiveAI/RxT-Beta-Mini-Decoder-Base
1B
•
Updated
•
274
ReactiveAI/RxT-Beta-Mini-MLM-Base
Updated
•
248
ReactiveAI/RxT-Beta-Encoder-Base
Fill-Mask
•
97.1M
•
Updated
•
115
•
1
ReactiveAI/RxT-Beta-Decoder-Base
Text Generation
•
3B
•
Updated
•
116
ReactiveAI/RxT-Beta-MLM-Base
Fill-Mask
•
Updated
•
106
ReactiveAI/RxT-Beta
Text Generation
•
Updated
•
1
datasets
34
ReactiveAI/coqa-retrieval
Viewer
•
Updated
•
6.59k
•
40
ReactiveAI/beta-reasoning
Updated
•
1.57k
ReactiveAI/passkey-retrieval
Viewer
•
Updated
•
1.57k
•
45
ReactiveAI/Beta-Hybrid-Interaction-SFT
Viewer
•
Updated
•
55.7M
•
184
ReactiveAI/codeparrot-apps-reupload
Viewer
•
Updated
•
10k
•
13
ReactiveAI/BAAI-TACO-reupload
Viewer
•
Updated
•
26.4k
•
24
ReactiveAI/Beta-Pre-Train-Corpus
Viewer
•
Updated
•
333M
•
14.2k
ReactiveAI/algebraic-stack-fixed
Viewer
•
Updated
•
3.39M
•
685
ReactiveAI/Beta-Code
Viewer
•
Updated
•
6.15M
•
2.01k
ReactiveAI/finepdfs-edu-beta
Viewer
•
Updated
•
13.7M
•
235