Fam Zheng
d923aa1e31
Hopfield + Hebbian hybrid memory system for LLMs. Two nights of experiments (16 iterations), validated on LongMemEval (ICLR 2025). Architecture: - Single-hop: Two-Stage Hopfield (NN top-20 → softmax settle) - Multi-hop: Hebbian W matrix with WTA pattern separation - 64% on LongMemEval (500 questions), retrieval-only, no LLM dependency - 4ms latency @ 20K memories, ~1GB VRAM Key findings: - Hopfield attention solved noise tolerance (20% → 100% vs flat Hebbian) - WTA pattern separation enables 20K+ capacity - Multi-hop associative chains (6 hops, CosSim=1.0) — RAG can't do this - MiniLM-L6 is optimal (discrimination gap > absolute similarity) - Paraphrase cue augmentation: 55% → 100% on synthetic, 36% → 64% on benchmark - SNN encoder viable (CosSim 0.99) but not needed for current architecture
89 lines
2.8 KiB
Markdown
89 lines
2.8 KiB
Markdown
# 实验7：Hopfield + 网络结构探索
|
||
|
||
## 背景
|
||
|
||
exp02-06 的核心问题：Hebbian W 矩阵做模糊单跳检索在大规模下失败（SNR 不够）。
|
||
Fam 指出这是**网络结构问题**，不是 hash 函数问题。
|
||
|
||
## 7a: 架构对比
|
||
|
||
| 架构 | bg=0 | bg=100 | bg=500 | bg=1000 |
|
||
|------|------|--------|--------|---------|
|
||
| Flat Hebbian | 80% | 60% | 30% | 20% |
|
||
| Attractor (auto+hetero) | 90% | 40% | 30% | 10% |
|
||
| **Hopfield (β=16)** | **100%** | **90%** | **90%** | **100%** |
|
||
| Recurrent+inhibition | 20% | 20% | 10% | 10% |
|
||
|
||
**Hopfield 完胜。** softmax attention 天然解决了归一化和锐化问题。
|
||
|
||
## 7b: Hopfield 深入测试
|
||
|
||
- **Multi-hop**: 3 跳 × 3 链 + 200 bg = 全部 sim=1.0 ✓
|
||
- **Scale (code space)**: 100+ bg 后不稳定（60-80%）
|
||
- **Hard distractors**: 高 β 下被语义相似的干扰项吸走
|
||
- **关键发现**: WTA code 空间的距离不忠实于语义距离
|
||
|
||
## 7c: Embedding-Space Hopfield
|
||
|
||
直接在 embedding 空间做 Hopfield attention（不经过 WTA）：
|
||
- 比 code-space 在中等规模（≤2K）更稳定
|
||
- Multi-hop 在 embedding 空间也完美（500 bg, sim=1.0）
|
||
- Hard distractors 在 β=8 时正确（attention 分散但正确）
|
||
|
||
## 7d: Two-Stage 检索
|
||
|
||
NN pre-filter (top-K) → Hopfield settle on candidates:
|
||
|
||
| N | K=20 | K=50 | 延迟 |
|
||
|---|------|------|------|
|
||
| 110 | 90% | 90% | 1ms |
|
||
| 1010 | 80% | 80% | 1ms |
|
||
| 5010 | 80% | 70% | 2ms |
|
||
| 10010 | 80% | 70% | 2ms |
|
||
| 20010 | **80%** | 70% | 4ms |
|
||
|
||
**K=20 最稳定**：20K 规模下 80%，4ms。
|
||
|
||
Diverse query test (20 对 + 2000 bg): 70% baseline → 分析 failure 发现是 embedding 模型质量问题。
|
||
|
||
## 7e: Cue Augmentation ⭐
|
||
|
||
| 方法 | 准确率 (20 对 + 2000 bg) |
|
||
|------|------------------------|
|
||
| 无 augmentation | 70% |
|
||
| Noise augmentation (各种参数) | 70% |
|
||
| **Paraphrase augmentation** | **95%** |
|
||
|
||
Noise 完全无效（高斯噪声 ≠ 真实 paraphrase 方向）。
|
||
Hand-crafted paraphrase 直接 70% → 95%。
|
||
|
||
实际系统中让 LLM 生成 3-5 个 paraphrase 一起存。
|
||
|
||
## 最终架构
|
||
|
||
```
|
||
Query → Two-Stage Hopfield (NN top-20 → softmax settle) → Target
|
||
↓
|
||
Hebbian W matrix (multi-hop chain from settled cue)
|
||
```
|
||
|
||
### 组件职责
|
||
|
||
| 组件 | 功能 | 容错 |
|
||
|------|------|------|
|
||
| Hopfield attention | 单跳检索 | 噪声/paraphrase 容忍 |
|
||
| Cue augmentation | 扩大记忆覆盖 | 弥补 embedding 模型不足 |
|
||
| NN pre-filter | 缩小候选集 | O(N) → O(K) |
|
||
| Hebbian W | 多跳联想 | 精确 cue 下完美 |
|
||
| WTA separation | 稀疏编码 | 20K+ 容量 |
|
||
|
||
### 性能指标
|
||
|
||
| 指标 | 数值 |
|
||
|------|------|
|
||
| Paraphrase recall (+ augmentation) | 95% |
|
||
| Multi-hop (3 hops, 500 bg) | 100% |
|
||
| Scale (20K memories) | 80% |
|
||
| Latency (20K) | 4ms |
|
||
| VRAM (W=16384²) | 1GB |
|