Fam Zheng
d923aa1e31
Hopfield + Hebbian hybrid memory system for LLMs. Two nights of experiments (16 iterations), validated on LongMemEval (ICLR 2025). Architecture: - Single-hop: Two-Stage Hopfield (NN top-20 → softmax settle) - Multi-hop: Hebbian W matrix with WTA pattern separation - 64% on LongMemEval (500 questions), retrieval-only, no LLM dependency - 4ms latency @ 20K memories, ~1GB VRAM Key findings: - Hopfield attention solved noise tolerance (20% → 100% vs flat Hebbian) - WTA pattern separation enables 20K+ capacity - Multi-hop associative chains (6 hops, CosSim=1.0) — RAG can't do this - MiniLM-L6 is optimal (discrimination gap > absolute similarity) - Paraphrase cue augmentation: 55% → 100% on synthetic, 36% → 64% on benchmark - SNN encoder viable (CosSim 0.99) but not needed for current architecture
85 lines
3.1 KiB
Markdown
85 lines
3.1 KiB
Markdown
# NuoNuo: Hippocampal Memory Module for LLMs
|
||
|
||
通宵原型验证实验记录（2026-04-06 ~ 07）。
|
||
|
||
最终架构：**Hopfield + Hebbian 混合记忆系统**。
|
||
|
||
## 核心能力
|
||
|
||
| 能力 | 数值 |
|
||
|------|------|
|
||
| 跨会话召回 (12 memories) | **100%** |
|
||
| Paraphrase recall (+ augmentation, 2K bg) | **95-100%** |
|
||
| Multi-hop 联想 (3 hops, 500 bg) | **100%** |
|
||
| Scale (20K memories, no augmentation) | **80%** |
|
||
| Latency @ 20K | **4ms** |
|
||
| VRAM | **~1 GB** |
|
||
|
||
## 实验索引
|
||
|
||
### Phase 1: 基础验证（exp01-06）
|
||
|
||
| # | 实验 | 结论 |
|
||
|---|------|------|
|
||
| 01 | [SNN Encoder Roundtrip](exp01_encoder_roundtrip.md) | ✅ CosSim 0.99 |
|
||
| 02 | [Associative Recall](exp02_associative_recall.md) | ✅ WTA+Hebbian 20K, multi-hop 完美 |
|
||
| 03 | [Sleep Consolidation](exp03_consolidation.md) | ⚠️ 简化为权重重建 |
|
||
| 04 | [Real Embeddings](exp04_real_embeddings.md) | ✅ 语义 embedding 可用 |
|
||
| 05 | [Benchmark](exp05_benchmark.md) | ✅ 3ms E2E |
|
||
| 06 | [BioHash](exp06_biohash.md) | ⚠️ 改善编码但不解决 W 矩阵问题 |
|
||
|
||
### Phase 2: 突破（exp07）
|
||
|
||
| # | 实验 | 结论 |
|
||
|---|------|------|
|
||
| 07 | [**Hopfield Attention**](exp07_hopfield.md) | ⭐ 噪声容忍 + 多跳 = 完美 |
|
||
|
||
### Phase 3: P0-P6 深入探索
|
||
|
||
| # | 问题 | 文档 | 结论 |
|
||
|---|------|------|------|
|
||
| P0 | [LLM Integration](p0_llm_integration.md) | `exp08` | ✅ Pipeline 可用，LLM Gateway 待验证 |
|
||
| P1 | [Embedding Models](p1_embedding_models.md) | `exp09` | ⭐ MiniLM 最优（gap 比 sim 重要） |
|
||
| P2 | [Auto Paraphrase](p2_auto_paraphrase.md) | `exp10` | ✅ Heuristic +20pp, Oracle +45pp |
|
||
| P3 | [Scale Ceiling](p3_scale_ceiling.md) | `exp11` | 结论=P2（ceiling 来自 embedding 不是架构）|
|
||
| P4 | [Lifecycle](p4_lifecycle.md) | `exp12` | ✅ Dedup + importance scoring 可行 |
|
||
| P5 | [SNN Hopfield](p5_snn_hopfield.md) | `exp13` | ❌ 不可行，softmax 远优于 LIF dynamics |
|
||
| P6 | [Multi-turn](p6_multiturn.md) | `exp14` | ✅ 12/12 跨会话召回 |
|
||
|
||
## 综合文档
|
||
|
||
- [**架构设计 v2**](architecture.md) — Hopfield + Hebbian 混合架构
|
||
- [核心发现](findings.md) — 什么有用、什么没用、反直觉结论
|
||
|
||
## 核心模块
|
||
|
||
- **`src/nuonuo/hippocampus.py`** — Hopfield-Hebbian 混合实现 (v2)
|
||
- `llm.py` — LLM 集成（提取/paraphrase/context injection）
|
||
- `src/nuonuo/encoder.py` — SNN spike encoder (备用)
|
||
|
||
## Quick Start
|
||
|
||
```python
|
||
from sentence_transformers import SentenceTransformer
|
||
from nuonuo.hippocampus import HippocampalMemory
|
||
|
||
model = SentenceTransformer('all-MiniLM-L6-v2', device='cuda')
|
||
memory = HippocampalMemory(embed_dim=384)
|
||
|
||
def emb(text):
|
||
return model.encode([text], convert_to_tensor=True,
|
||
normalize_embeddings=True, device='cuda')[0]
|
||
|
||
# Store with paraphrase augmentation
|
||
memory.store(emb("The database is slow"), emb("Check missing indexes"),
|
||
cue_variants=[emb("DB performance terrible"), emb("Database crawling")],
|
||
metadata={"target": "Check missing indexes"})
|
||
|
||
# Recall
|
||
results = memory.recall(emb("DB is really slow today"), top_k=3)
|
||
chain = memory.recall_chain(emb("DB is really slow today"), hops=3)
|
||
|
||
# Save/Load
|
||
memory.save("hippocampus.pt")
|
||
```
|