MSA (Memory Sparse Attention) — An Open-Source Answer to AI’s Memory Bottleneck
The Core Problem
Where AI memory stands today:
- The strongest models have an effective context of roughly 1M tokens
- The information a human can store in a lifetime is on the order of 200–300 million tokens
- That’s a gap of two orders of magnitude
Both of the industry’s well-worn paths have hit a wall:
- Stretching the context window — compute cost grows quadratically; that road ends here
- Bolting on RAG — retrieval and generation stay disconnected, and accuracy has a ceiling
What MSA Is
MSA (Memory Sparse Attention) comes from the EverMind team (under Shanda). It embeds memory directly into the attention mechanism itself — no longer context, no external retrieval.
In one sentence: traditional RAG gives a model an external hard drive; MSA gives it a native memory chip.
- Finding and using are no longer two separate steps — they’re fused into a single neural network and done end to end
- The model learns on its own what to remember, how to find it, and how to use it
- Plug and play: you only swap out the Self-Attention layer of a standard Transformer
Key Technical Details
- Compression mechanism: brings 100 million tokens of storage down to an acceptable footprint
- Tiered storage: routing indexes on the GPU, content details on the CPU — total capacity depends on memory, not VRAM
- Sparse routing: complexity drops from O(L²) to O(L)
- Positional encoding: each document is numbered independently, so training at 64K extrapolates to 100M
Performance
Built on Qwen3-4B, with 159B tokens of continued pre-training:
| Result | Data |
|---|---|
| Memory span | from ~10K tokens → 100 million tokens (nearly 4 orders of magnitude) |
| Quality degradation | answer quality drops only <9% |
| Standard QA benchmarks | a 4-billion-parameter model beats traditional RAG approaches by 16% |
| vs. top-tier retriever + 235-billion-parameter LLM | still wins on multiple tests, with a 60× parameter gap |
Hardware Bar
It runs directly on a machine with two A800 GPUs — no cluster needed. Small and mid-sized teams, even individual developers, can have long-term memory at the hundred-million-token scale.
Team Background
- EverMind (under Shanda)
- Previously built Omne, the multi-agent framework that reached SOTA on the GAIA leaderboard, and EverOS, an open-source memory platform
- More than nine months from project kickoff to completed paper
Key insight: a model needs different information when it’s “finding material” versus “writing the answer” — finding calls for macro-level judgment, writing calls for micro-level detail. Split them apart and handle each with a dedicated module, and performance changes qualitatively.
Application Prospects
- Genuinely personalized AI assistants: remembering dietary preferences, project progress, family members’ personalities
- AI education: truly personalized, teaching to the individual
- Medical assistants: tracking a complete medical history
- Enterprise knowledge bases: remembering a decade of accumulated project knowledge
- Memory as a service: the memory layer as an independent, pluggable module, so memory assets aren’t locked to a single model
Related Links
- MSA GitHub: https://github.com/EverMind-AI/MSA
- EverOS GitHub: https://github.com/EverMind-AI/EverOS