The short version
Agent memory does not need a big GPU. In our tests on an Orange Pi 5 Plus (RK3588, 16 GB), taOSmd kept the archive, the vector index and the knowledge graph on the board itself, ran the embedder on the CPU, and used the NPU only for the larger helper models. Bigger embedders did not make retrieval better in our tests, and binary vectors cut storage 32x with no measurable recall loss.
Pick a small embedder and run it on the CPU
The shipped default is snowflake-arctic-embed-s as ONNX on the CPU: 384 dimensions, about 130 MB. Older installs keep all-MiniLM-L6-v2 until they re-index.
What we learned:
- ONNX matters. PyTorch was about 200x slower than ONNX for a model this size.
- The NPU is slower than the CPU for small embedders. Save it for bigger models.
- Small embedders are much cheaper on a board. On the Orange Pi 5 Plus A76 CPU, embedding one document (p50) took 28 ms with MiniLM (225 MB RSS) against 413 ms with EmbeddingGemma-2 fp32 (1.06 GB RSS), measured by 2026-10-07.
- Bigger embedders did not help. On LoCoMo (200 questions, RTX 3060 12 GB, run 2026-05-06):
| Embedder | LoCoMo score |
|---|---|
| all-MiniLM-L6-v2 | 0.56 |
| Arctic-L v2 | 0.55 |
| EmbeddingGemma-300m | 0.54 |
| Nomic v1.5 | 0.54 |
The larger models traded single-hop accuracy for temporal and multi-hop accuracy; the total did not go up.
Retrieval alone tells the same story. On LongMemEval-S (500 questions, Recall@5, query expansion on, same harness, re-run 2026-10-06; retrieval scores do not depend on hardware):
| Embedder | Recall@5 |
|---|---|
| all-MiniLM-L6-v2 | 97.0% (485/500) |
| EmbeddingGemma-2 | 96.8% (484/500) |
| snowflake-arctic-embed-s (shipped default) | 95.6% (478/500) |
Shrink the vectors
Binary embedding quantization is opt-in (VectorMemory(binary_quant=True)). On the full 1540-question LoCoMo set it scored 0.520 against 0.519 for full vectors under a strict judge, with vectors 32x smaller. On a board with limited RAM and storage, this is the biggest footprint lever.
What the Orange Pi 5 Plus does
The tested stack on the Orange Pi 5 Plus 16 GB: MiniLM ONNX on the CPU for embeddings, with Qwen3-Reranker-0.6B and a 1.7B query-expansion model on the NPU through rkllama. The NPU path is optional: without it, embedding and search run on the CPU through ONNX Runtime, and the reranker is off by default.
| Measurement (Orange Pi 5 Plus 16 GB) | Result | Measured by |
|---|---|---|
| Optional Qwen3-4B generator on the NPU | about 17 s per turn | 2026-04-25 |
| LoCoMo, external qwen3:4b judge, baseline | 0.382 | 2026-05-01 |
| LoCoMo, external qwen3:4b judge, full stack | 0.490 | 2026-05-01 |
| Gain from the BGE-v2-m3 reranker | +0.074 | 2026-05-01 |
Both LoCoMo rows used the optional Qwen3-4B NPU generator, since a judged score needs one.
The reranker earns its place on the Pi: its gain there was about 12 times what it added on a 3060.
A fresh install on a Pi-class board defaults to retrieval only, with no local generator. Generation is better placed on a GPU or a faster machine in the cluster.
Splitting memory across a small cluster
The layout we recommend:
- On the single-board computer: the archive, the vector index and the knowledge graph.
- On any CPU node: embedding and reranking.
- On the GPU or NPU node: generation.
- Evaluation: never run the judge model on the same box as the generator.
A practical note
A 36+ hour flat-out benchmark (NPU generation, CPU embeddings, plus a camera service on the same board) drew more than the stock USB-C supply could hold. The board hard-powered-off and stayed down until power-cycled (2026-04-28). We moved to a higher-rated PSU and active cooling.