Originally published on X ↗
Sparse circuit assembly, continual learning, and the path to compute-efficient embodied intelligence
microagi Research
microagi is solving the end-to-end data problem for embodied AI. We do frontier data research, collect at massive scale and we are building to deploy robots in the physical world to collect even more. Our operations and technology revolve around improving data quality and diversity. We serve our customers only what we believe is the best data, paired with the best forward-deployed engineering.
We believe something on the order of 100 billion hours of diverse, multimodal real-world data and interaction is needed to reach embodied AGI. Data volume is a major bottleneck, and we are building the infrastructure to break it. But there is a second bottleneck that the field has not yet solved: the memory architecture. Today's models cannot keep learning from diverse, continually arriving experiences without forgetting what they already know. Even if you collect all the data, you need somewhere scalable to put it.
The direction we care about: keep active compute small, let total capacity grow sparsely, reuse old circuits when an input is familiar, create new sparse circuits when the model genuinely needs new structure. In one sentence: real-world data needs a scalable place to go.
One example of what our research team is producing: a mechanism called Coactivation-Assembled Weight Layers (CAWL). CAWL is one piece of a broader research program. The architecture constraints we outline in this article are the research agenda. CAWL is one specific attack on one part of it. All of our research will be open sourced. Papers are pending. Below is the technical case.
Robotics has a scaling story that looks, on the surface, a lot like the language model story from 2019 to 2023. Collect more data, train bigger models, watch capabilities emerge. Recent generalist robot models already make one point hard to ignore: scale works in robotics too. RT-1 argued for large task-agnostic training on real robot data. Open X-Embodiment assembled data from 22 robots across 21 institutions. Octo trained on 800K trajectories from that corpus. OpenVLA pushed the open VLA line with a 7B model trained on 970K real-world demonstrations. DROID showed that broader in-the-wild collection improves robustness and generalization.
But the scale that has been demonstrated so far is nowhere near enough. Robotics is an open-world problem. A deployed robot does not see i.i.d. data. It sees an open-ended stream of homes, factories, tools, object layouts, camera changes, sensor drift, user corrections, and entirely new tasks. Inputs are multimodal, noisy, asynchronous, partially observed, and grounded in a changing physical environment. The same skill looks different under camera pose, lighting, object wear, material compliance, latency, human behavior, floor friction, clutter, and hundreds of other variables.
The important shifts are often orthogonal to each other. Lighting changes are different from texture changes. Camera latency is different from motion blur. Material compliance is different from shape. Human style is different from room layout. Tool wear is different from object appearance. Covering those orthogonal dimensions of covariate shift is not optional. It is the whole point. And covering them requires data at a scale the field has not yet reached.
microagi is built to solve this. We are a data research lab for embodied AI, doing frontier research on what high-quality data looks like across modalities, and collecting at a scale of over 100,000 hours per month of diverse real-world multimodal data. We will be deploying in the physical world, which will generate even more. The ideal dataset, roughly speaking, is every job and every person on earth recorded for about 10 hours, capturing the full diversity of human physical interaction with the world, and then robots deployed doing continual learning on top of that foundation. We believe something on the order of 100 billion hours of multimodal real-world interaction is what it takes.
That is the data problem. We are solving it. But there is a second problem sitting right behind it: even if you collect all that data, what architecture can actually absorb it?
Dense models are powerful, but their default learning rule is brutal: new experience pushes gradients through the same shared weight table that already encodes old behavior. That is acceptable when the distribution is stable and training is mostly one giant offline phase. It is much less acceptable when a robot keeps seeing new rooms, new users, new objects, new tools, and new action consequences.
The standard mitigations are replay buffers (store old data and mix it in), elastic weight consolidation (penalize changes to important weights), or progressive networks (freeze old parameters and add new ones). All of these work to varying degrees. None of them scale cleanly. Replay requires storing and retraining on old data indefinitely. EWC requires computing and storing a Fisher information matrix over all parameters. Progressive networks freeze old capacity that never benefits from new experience. The root cause is always the same: dense weight matrices store everything in the same place. Learning and forgetting are mechanically coupled.
Sparse mixture-of-expert models look like a step in the right direction. They route tokens to different expert subnetworks, so not all parameters are active for every input. MoE scales impressively, from the original adaptive mixtures framework through Switch Transformers to modern soft routing variants.
But MoE sparsifies dispatch, not necessarily learning. The router may choose only a few experts, but each expert is still a dense chunk of parameters. The experts are all created at initialization. None of them grow or change structure during training. New domains can still overlap on the same experts. That means the model can remain sparse at the routing level while still forgetting at the weight-update level.
MoE separates which computation to apply, but does not change what computation is available. The menu is fixed at initialization. Long context windows do not solve this either. Replaying a huge raw history is not the same as having memory. It is often just a more expensive form of retrieval.
In a dense model, all capacity lives in one shared parameter table, most of the layer is active on every forward pass, and new learning overwrites old weights. Standard MoE improves on this by routing to a bank of fixed dense experts so only a few run at a time, but expert overlap still causes overwrite, and the memory story is still static experts plus a context window. In a sparse pathway growth architecture, capacity lives in a latent circuit bank that can grow. Only K latent units are active per step, so compute stays bounded. New domains get new structure allocated rather than destructively reusing old structure. And the memory architecture separates into three layers: active state, fast writable memory, and slow sparse consolidated memory.
Bounded active compute. The amount of work per forward pass should stay roughly fixed even as the model learns more. You cannot scale a deployed robot system if every new domain makes inference more expensive.
Growing total capacity. The model must have a way to add new structure for genuinely new situations. A fixed parameter budget is a hard ceiling on what the system can represent.
Continual learning by allocation, not overwrite. Old skill circuits should survive because they are not constantly rewritten, not because an anti-forgetting penalty is applied on top of a fundamentally destructive update rule.
Multimodal grounding. Vision, language, proprioception, action, and world-state cues should land in a shared system that can route and remember across modalities.
Memory across timescales. Immediate state, short-term writable traces, and slow consolidated structure should be different things, not one giant context buffer.
This is not a claim about one layer or one training trick. It is a claim about the whole regime that embodied AI needs.
Our research team is working across sparse routing, adaptive capacity allocation, multi-timescale memory, and multimodal grounding. CAWL is the first published result from that program. Instead of storing a dense weight matrix W that gets applied identically regardless of input, a CAWL layer stores building blocks: a pool of latent neurons with learned embeddings, two projection matrices, a small set of shared pair modules, and a sparse persistent table. At inference time, the layer assembles a sparse circuit from these components, conditioned on the current input context.
The standard MoE picture can be written loosely as y = Σ ge(c) · Ee(x) summed over a top-K set of experts. That gives sparse dispatch, but the underlying expert blocks are still coarse and dense. CAWL does not dispatch tokens to prewritten experts. It selects nodes inside one shared latent graph and writes edges among them. The core equation:
Weff(c) = U A(c) Vᵀ
Here U and V are learned projections, M is the latent width, and A(c) ∈ ℝM × M is a context-conditioned sparse latent adjacency matrix assembled at inference time. The input does not just select coefficients in a fixed dense matrix. It selects a small active set of latent neurons, forms edges among them, and applies the projected sparse operator.
Each latent neuron has a learned embedding eᵢ. The context is projected into the same space as a query vector q(c). Cosine similarity between the query and each neuron embedding produces a score sᵢ(c). The maximum similarity across all neurons defines the raw familiarity signal:
F(c) = maxᵢ sᵢ(c)
A sigmoid with learnable threshold and temperature produces the familiarity gate:
γ(c) = σ( (F(c) − τF) / TF )
When γ(c) is close to 1, the input looks familiar. The layer relies heavily on its persistent table of consolidated connections. When γ(c) is close to 0, the input is novel. The layer activates underused neurons and leans on its dynamic pair modules to construct a new circuit. This is a much better fit for robotics than always paying to run the same coarse blocks, because familiar situations can become cheap while novelty gets a dedicated path.
The routing score for each neuron includes a novelty bias:
s′ᵢ(c) = sᵢ(c) + λnov (1 − γ(c))(1 − ρᵢ)
Here ρᵢ is the running activation frequency of neuron i. Novel inputs get a bonus for selecting neurons that have not been used much. Familiar inputs gravitate toward the neurons they have always used. This turns routing into a memory decision instead of a generic discriminative score.
Once the active set S(c) = TopK(s′(c), K) is selected, the layer computes connections between every ordered pair. For each pair (i, j), the shared pair feature concatenates their embeddings and routing scores:
ξᵢⱼ(c) = [eᵢ ; eⱼ ; s′ᵢ(c) ; s′ⱼ(c)]
A gate network produces a Bernoulli parameter pᵢⱼ(c) = σ(fθ(ξᵢⱼ)) that decides whether the edge exists, and a value network produces the dynamic edge weight αᵢⱼ(c) = gψ(ξᵢⱼ). The binary gate uses a straight-through estimator: hard threshold forward, continuous gradient backward.
For familiar inputs, the persistent table contributes stored edge weights in proportion to the familiarity gate:
α̃ᵢⱼ(c) = αᵢⱼ(c) + γ(c) Cᵢⱼ
The persistent table C consolidates frequently occurring edges over time. Edges that fire reliably get strengthened via a familiarity-modulated learning rate. Edges that stop being useful get pruned after consecutive low-utility updates. The table has a fixed budget; when it fills, the least useful entries are evicted.
The layer output never materializes the full effective matrix. It gathers the active columns of V, applies the sparse adjacency in latent coordinates, and scatters through the active columns of U:
y = U A(c) Vᵀx = Σᵢ Σⱼ zᵢⱼ(c) α̃ᵢⱼ(c) uᵢ(vⱼᵀx)
where uᵢ and vⱼ are columns of U and V respectively, and the sums run only over the active set S(c).
With K active neurons selected from a pool of M, and each ordered pair independently connected or not (including self-loops), the number of possible binary circuit topologies is:
NK = C(M, K) × 2K²
For M = 1024 and K = 64, this is approximately 5 × 101335. That is the number of binary support patterns before edge values and before projection through U and V. Not all of them produce functionally distinct effective matrices. But the point is that a compact latent basis can range over an enormous family of sparse circuits while storing significantly fewer parameters than a dense matrix of the same input/output dimension.
We are straightforward about this: in its current form, a CAWL layer is more expensive per forward pass than a dense layer of the same dimension. The pairwise edge computation over the active set is the dominant cost, and it needs efficient fused kernels to be competitive at scale. The familiarity routing path itself is lightweight. The pair modules are where the compute goes.
The storage picture is different. A CAWL layer stores substantially fewer parameters than the equivalent dense matrix, because the layer is never a single dense matrix. It is a set of building blocks that combine differently for each context.
The scalable version has to become recall-first. Familiar inputs should mostly read consolidated support from the persistent table and bypass edge synthesis. Novel inputs can afford to pay the construction cost because they are the reason the model needs plasticity in the first place. If this recall-first execution path works, familiar inference becomes cheaper than a dense baseline, not more expensive. That is the engineering target.
Fixed-basis CAWL is only the first step. It can prove that familiarity-gated routing is stable and that sparse circuit assembly works as a mechanism. It is not the full continual-learning answer. With fixed M, novelty bias can delay overlap between domains. It cannot eliminate it.
The real move is to let the latent basis grow. That is the point where learning stops meaning 'rewrite the same table' and starts meaning 'add new sparse structure.'
Mt+1 = Mt + mt, Ut+1 = [Ut | Unew], Vt+1 = [Vt | Vnew]
The allocation rule is what prevents capacity explosion. Low familiarity alone is not enough. Growth should happen only when novelty is paired with genuine residual error:
n(c) = α(1 − γ(c)) + β ε(c) + χ r⊥(c)
Here ε(c) is task error or surprise, and r⊥(c) measures how much of the desired update sits outside the currently recalled span. If n(c) crosses a threshold, the model allocates new latent directions. If not, it reuses or recombines what already exists.
Growth is structural. New rows are added to the latent embedding matrix E, new columns are appended to U and V, and the persistent table C receives new sparse support. K stays fixed. Per-example activation stays sparse even as total capacity grows. The total parameter count will be high, because it has to be, but activation remains thin.
Old circuits survive not because the model is regularized into barely changing, but because new domains get genuinely new structure. Once a pathway is reused often enough, it consolidates. Once consolidated, its associated columns can be protected with lower learning rates or projected gradients. Dormant old circuits become much harder to corrupt.
The decisive experiment we are running. Train on domain A, then B, then C, then evaluate A again. Without replay buffers and without EWC-style penalties. If A survives because its circuits went dormant rather than because we froze the world around it, that is the result that matters.
There is a second bottleneck that matters just as much as weight overwrite: short-term memory. Robotics systems often carry giant buffers, large histories, or long token windows because they do not have a better way to keep track of what just happened. That is expensive and fragile.
The target is not infinite exact memory with zero storage. That is impossible. The target is very large effective memory with bounded active compute. A better design separates three timescales:
Active state ht for what is currently in play.
Fast writable memory Mt for short-lived traces that can be updated online.
Slow sparse memory S for consolidated pathways and reusable long-term structure.
rt = Read(ht, Mt, S)
ht+1 = f(ht, xt, rt)
Mt+1 = λMt + Write(ht, xt, rt)
If a trace repeats or proves useful, it gets consolidated into S. The robot needs a compact state, a writable scratchpad, and a persistent circuit bank. It does not need every prior token to remain live forever. Eventually this could absorb part of what current architectures carry as KV cache, but that is not the immediate claim. The immediate milestone is continual learning that allocates new sparse pathways instead of repeatedly negotiating over the same dense table.
microagi exists to solve the whole end-to-end data problem for embodied AI. Our operations and technology revolve around improving data quality and diversity across modalities. We do research to test our data quality rigorously, and we serve our customers only what we believe is the best data paired with the best forward-deployed engineering.
But data research without architecture research is incomplete. If you scale data and the model cannot absorb it without forgetting, you have built a pipeline with no destination. If you build an architecture and do not understand what data it needs, at what quality, across which modalities, you have built a destination with no pipeline. The two problems are inseparable.
That is why we do both. As we scale collection and deploy in the physical world, the data volume will keep growing. Robots operating in the real world will generate even more data than human operators, and that data will be richer, more diverse, and more sequential than anything collected offline. The architecture has to be ready. Continual learning at this scale is not a nice-to-have. It is a prerequisite.
That is why we have a research team working on these problems. CAWL is one output of that effort. There will be more. The bottleneck we are working on is one the entire field faces, and we intend to open source everything we produce.
This is foundational model research for embodied AGI, but the standard should be concrete. We are not trying to win a beauty contest for sparse math. We are trying to build a system that can keep learning from real-world data without either exploding compute or silently forgetting.
If routing collapses, none of the rest matters.
Stability. The familiarity gate γ(c) should not collapse or flatline during training. The router should not degenerate to always selecting the same neurons. The persistent table should remain bounded without manual intervention.
Signal quality. Low γ should correlate with rare, shifted, or poorly fitted inputs. High γ should correlate with useful recall from the persistent table. If the gate collapses to a constant, the familiarity mechanism is dead.
Continual learning. Sequential domains should preserve prior performance without replay buffers becoming the whole story. The A-then-B-then-C-then-test-A experiment is the core proof.
Systems viability. Familiar inputs should become cheaper than novel ones once consolidated. If the cost remains dominated by construction even after consolidation, the design needs another iteration.
Papers are in preparation. We will share results and code as they mature.
Robotics will not be solved by a single trick. It will need better data, better simulation, better world models, better action representations, and better hardware. But the memory problem sits underneath all of that. If every new skill still means overwriting a shared dense table, deployment will remain fragile no matter how impressive pretraining looks.
Our thesis is that continual learning for embodied systems is a sparse capacity allocation problem. Familiar inputs should recall consolidated circuits. Novel inputs should build new ones. Unused structure should decay. Capacity should grow only when the current span cannot explain the input. The reason this does not explode compute is that activation is sparse. The total parameter count grows with experience, but the active slice at any given moment stays thin.
microagi is solving the end-to-end data problem for embodied AI. We do frontier data research, collect at massive scale, and will be deploying robots in the physical world to collect even more. In parallel, we are developing compute-efficient architectures that can absorb that data through sparse activation, structured memory, and continual learning by capacity allocation rather than destructive overwrite. The goal is not a prettier benchmark model. The goal is a scalable foundation for real-world robotics and embodied AGI.
All of our research will be open sourced. We believe the bottleneck we are working on is one the entire field faces, and the solutions should be shared.
We think the answer involves sparse circuit assembly, and we have a team that can generate ideas like this and deploy them. CAWL is the first. More soon.
Brohan, A., et al. (2022). RT-1: Robotics Transformer for Real-World Control at Scale.
Fedus, W., Zoph, B., and Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.
Ghosh, D., et al. (2024). Octo: An Open-Source Generalist Robot Policy.
Jacobs, R. A., Jordan, M. I., Nowlan, S. J., and Hinton, G. E. (1991). Adaptive Mixtures of Local Experts.
Khazatsky, A., et al. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset.
Kim, M. J., et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
Kontogianni, T., et al. (2024). Is Continual Learning Ready for Real-world Challenges?
Lesort, T., et al. (2020). Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges.
McClelland, J. L., McNaughton, B. L., and O'Reilly, R. C. (1995). Why There Are Complementary Learning Systems in the Hippocampus and Neocortex.
Open X-Embodiment Collaboration, et al. (2023). Open X-Embodiment: Robotic Learning Datasets and RT-X Models.
Powers, S., Gupta, A., and Paxton, C. (2023). Evaluating Continual Learning on a Home Robot.
Puigcerver, J., et al. (2023). From Sparse to Soft Mixtures of Experts.
Rusu, A. A., et al. (2016). Progressive Neural Networks.
Vander Mijnsbrugge, D., Ongenae, F., and Van Hoecke, S. (2023). Context-Aware Deep Learning with Dynamically Assembled Weight Matrices.
Yoon, J., et al. (2017). Lifelong Learning with Dynamically Expandable Networks.
microagi Research
microagi.ai