Fam Zheng
d923aa1e31
Hopfield + Hebbian hybrid memory system for LLMs. Two nights of experiments (16 iterations), validated on LongMemEval (ICLR 2025). Architecture: - Single-hop: Two-Stage Hopfield (NN top-20 → softmax settle) - Multi-hop: Hebbian W matrix with WTA pattern separation - 64% on LongMemEval (500 questions), retrieval-only, no LLM dependency - 4ms latency @ 20K memories, ~1GB VRAM Key findings: - Hopfield attention solved noise tolerance (20% → 100% vs flat Hebbian) - WTA pattern separation enables 20K+ capacity - Multi-hop associative chains (6 hops, CosSim=1.0) — RAG can't do this - MiniLM-L6 is optimal (discrimination gap > absolute similarity) - Paraphrase cue augmentation: 55% → 100% on synthetic, 36% → 64% on benchmark - SNN encoder viable (CosSim 0.99) but not needed for current architecture
3.1 KiB
3.1 KiB
NuoNuo: Hippocampal Memory Module for LLMs
通宵原型验证实验记录（2026-04-06 ~ 07）。
最终架构：Hopfield + Hebbian 混合记忆系统。
核心能力
| 能力 | 数值 |
|---|---|
| 跨会话召回 (12 memories) | 100% |
| Paraphrase recall (+ augmentation, 2K bg) | 95-100% |
| Multi-hop 联想 (3 hops, 500 bg) | 100% |
| Scale (20K memories, no augmentation) | 80% |
| Latency @ 20K | 4ms |
| VRAM | ~1 GB |
实验索引
Phase 1: 基础验证（exp01-06）
| # | 实验 | 结论 |
|---|---|---|
| 01 | SNN Encoder Roundtrip | ✅ CosSim 0.99 |
| 02 | Associative Recall | ✅ WTA+Hebbian 20K, multi-hop 完美 |
| 03 | Sleep Consolidation | ⚠️ 简化为权重重建 |
| 04 | Real Embeddings | ✅ 语义 embedding 可用 |
| 05 | Benchmark | ✅ 3ms E2E |
| 06 | BioHash | ⚠️ 改善编码但不解决 W 矩阵问题 |
Phase 2: 突破（exp07）
| # | 实验 | 结论 |
|---|---|---|
| 07 | Hopfield Attention | ⭐ 噪声容忍 + 多跳 = 完美 |
Phase 3: P0-P6 深入探索
| # | 问题 | 文档 | 结论 |
|---|---|---|---|
| P0 | LLM Integration | exp08 |
✅ Pipeline 可用，LLM Gateway 待验证 |
| P1 | Embedding Models | exp09 |
⭐ MiniLM 最优（gap 比 sim 重要） |
| P2 | Auto Paraphrase | exp10 |
✅ Heuristic +20pp, Oracle +45pp |
| P3 | Scale Ceiling | exp11 |
结论=P2（ceiling 来自 embedding 不是架构） |
| P4 | Lifecycle | exp12 |
✅ Dedup + importance scoring 可行 |
| P5 | SNN Hopfield | exp13 |
❌ 不可行，softmax 远优于 LIF dynamics |
| P6 | Multi-turn | exp14 |
✅ 12/12 跨会话召回 |
综合文档
核心模块
src/nuonuo/hippocampus.py— Hopfield-Hebbian 混合实现 (v2)llm.py— LLM 集成（提取/paraphrase/context injection）src/nuonuo/encoder.py— SNN spike encoder (备用)
Quick Start
from sentence_transformers import SentenceTransformer
from nuonuo.hippocampus import HippocampalMemory
model = SentenceTransformer('all-MiniLM-L6-v2', device='cuda')
memory = HippocampalMemory(embed_dim=384)
def emb(text):
return model.encode([text], convert_to_tensor=True,
normalize_embeddings=True, device='cuda')[0]
# Store with paraphrase augmentation
memory.store(emb("The database is slow"), emb("Check missing indexes"),
cue_variants=[emb("DB performance terrible"), emb("Database crawling")],
metadata={"target": "Check missing indexes"})
# Recall
results = memory.recall(emb("DB is really slow today"), top_k=3)
chain = memory.recall_chain(emb("DB is really slow today"), hops=3)
# Save/Load
memory.save("hippocampus.pt")