agent-memory

A persistent memory layer for Claude Code (and any MCP-compatible agent) that captures every user prompt and tool_call from your coding sessions, exposes them for recall via an MCP server + FastAPI, and feeds them back into a local fine-tune pipeline that produces project-specific tool-calling LoRAs. Built locally, runs locally — Postgres + pgvector, FastAPI on 127.0.0.1:3377, GGUFs in LM Studio. The hooks are the recorder; the mem_* tables are the journal; the fine-tune pipeline is what turns that journal into a model that actually knows your codebases.

System overview

Claude session ──► hooks (UserPromptSubmit, PreToolUse, PostToolUse, SessionStart/End)
                          │
                          ▼
                   FastAPI ingest (port 3377, Bearer-token auth)
                          │
                          ▼
                   Postgres (mem_user_prompts, mem_tool_calls, mem_sessions,
                             mem_projects, mem_observations, mem_lessons)
                          │
            ┌─────────────┴──────────────┐
            ▼                            ▼
   MCP recall surface          fine-tune pipeline
   (search, timeline,          (build_v3_dataset.py)
    get_observations,                  │
    save_memory, lessons)              ▼
                              LoRA train → merge → GGUF
                                       │
                                       ▼
                                  LM Studio

The write path (hooks → FastAPI → queue → Postgres) is fire-and-forget and never blocks Claude. The read path (MCP tools) uses Reciprocal Rank Fusion across pgvector and Postgres FTS. The training path reads directly from mem_tool_calls joined to mem_user_prompts and emits Qwen-format chat datasets.

Repository structure

Database schema (overview)

All tables are mem_*-prefixed to avoid collisions in a shared Postgres.

Full schema lives in scripts/migrations/*.sql. The current head is 013-project-consolidation.sql. See docs/PRIMER.md for column-level details and docs/fine_tune/V2_DATA_PIPELINE_PLAN.md for the prompt↔tool_call linkage design introduced by migration 012.

Hooks — how data gets in

Five Node.js hooks live in hooks/. They are designed fire-and-forget (~30-40ms p99) and exit 0 on every error path so a misconfigured or down agent-memory never blocks Claude.

Hook auth shares hooks/auth-header.js which reads AGENT_MEMORY_TOKEN from the environment. Hooks also send X-Agent-Name: claude so the trusted-agents bypass applies on localhost.

To wire them into Claude Code, symlink each hooks/*.js file into ~/.claude/hooks/ and register the hook list in ~/.claude/settings.json. The exact commands are in HANDOFF.md under "Setup on New Machine".

Fine-tune pipeline status

Anchor docs:

• •docs/fine_tune/V3_PLAN.md — current training plan with multi-turn fixes baked in
• •docs/training_runs/v2-real-world-test.md — verbatim A/B transcripts that drove the retraction
• •docs/fine_tune/FAILURE_MODES.md — 12 operational gotchas (resolve()-into-Dropbox, llama-cli hangs, YaRN config, anti-loop, etc.)
• •docs/fine_tune/PIPELINE_RUNBOOK.md — phase-by-phase recipe
• •docs/fine_tune/V2_DATA_PIPELINE_PLAN.md — how the v2 dataset shape was built (still the v3 data shape too)

The training script (models/lora/qwen2.5-3b-toolcalls-lora/run_train_lora.py) is env-var-driven, not argparse — set MODEL_SLUG, DATASET_VERSION, DATASET_TIER, RUN_TAG, EPOCHS.

Setup / quickstart

This assumes the legacy installer is not desired. For a one-shot install, node install.js still works (sets up Docker or native Postgres, MCP registration, hook symlinks, daily backup, .env).

# 1. Clone
git clone https://github.com/metazen11/agent-memory.git ~/_CODING/agentMemory
cd ~/_CODING/agentMemory

# 2. Python venv (project targets 3.12+; current dev runs 3.14)
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt

# 3. Postgres (native Homebrew recommended on macOS)
brew install postgresql@16 pgvector
brew services start postgresql@16
createuser -s mz
createdb -O mz agent_memory
psql -d agent_memory -c "CREATE EXTENSION vector;"

# 4. Configure .env (copy from .env.example, set POSTGRES_* + REQUIRE_AUTH=true)
cp .env.example .env

# 5. Start the API (migrations run on startup)
.venv/bin/uvicorn app.main:app --port 3377 --host 127.0.0.1

# 6. In another shell: generate tokens for trusted agents
.venv/bin/python -m app.cli setup
echo 'export AGENT_MEMORY_TOKEN="<claude-token-from-step-6>"' >> ~/.zshenv

# 7. Symlink hooks into ~/.claude/hooks/ and register them in settings.json
#    Full commands: see HANDOFF.md "Setup on New Machine"

# 8. Install the daily Postgres backup schedule (idempotent)
bash scripts/install_backup_schedule.sh
bash scripts/install_backup_schedule.sh --check

Verify the install:

curl http://localhost:3377/api/health
.venv/bin/python -m app.cli list-tokens

The wizard (operator tool for fine-tunes)

scripts/fine_tune/wizard.py is a Textual TUI that sequences the v2/v3 ad-hoc playbook into one command. It runs phase by phase with gates so a bad dataset or a failed smoke can't silently turn into a 36-hour training run.

.venv-finetune/bin/python scripts/fine_tune/wizard.py

# Or, replay a saved config non-interactively
.venv-finetune/bin/python scripts/fine_tune/wizard.py \
    --config train_config.yaml --no-tui

Stages: verify env (Dropbox quit, MPS available, disk free) → build dataset (build_v2_dataset.py / build_v3_dataset.py) → audit gate (token counts, tool histogram, drop-reason MANIFEST) → tiny smoke (200 rows, 1 epoch, ~25-40 min) → tiny validator (≥3% parse rate) → full train (~3-4h MPS for 3B, ~36-40h for 8B) → full validator (≥85% on merged HF + GGUF) → GGUF convert + LM Studio install → chat-loop verification on llama-server.

Full reference: docs/fine_tune/WIZARD.md.

Security model

Auth and isolation are configured via .env and app/config.py. Defaults err on the safe side; production install (this machine) has all of these on.

• •Bearer token auth — REQUIRE_AUTH=true enables AuthMiddleware on

every endpoint. Tokens are generated by python -m app.cli setup and scoped per-agent (anvil, claude, codex, gemini, python-httpx).

• •Trusted-agent bypass — TRUSTED_AGENTS allows a known agent name

via the X-Agent-Name header on localhost only. Hooks use this so the recorder never has to ship a token to ~/.claude/.

• •Host bound to `127.0.0.1` — no external interface ever.
• •CORS locked to localhost origins.
• •Rate limits — 100/min writes, 500/min reads (RateLimitMiddleware).
• •Secret redaction — REDACT_SECRETS=true by default. Strips API

keys, tokens, and password-shaped strings from tool_input before persistence. See app/redact.py.

• •Audit logging — audit_log_level=writes_only, 30-day retention.

See AuditMiddleware.

• •PG trust-auth warning — startup logs CRITICAL if POSTGRES_PASSWORD

is empty and ALLOW_TRUST_AUTH=true is not explicitly set.

There is no docs/security.md yet — read app/middleware.py, app/auth.py, app/redact.py, and the security block in app/config.py for current behavior.

Daily backups

launchd runs pg_dump at 03:14 local time and writes data/backups/daily_YYYYMMDD_HHMMSS.sql.gz. Retention keeps the 3 most recent daily_*.sql.gz; manually-created snapshots (pre_v2_backfill_*.sql.gz, etc.) are never auto-deleted.

# Install (idempotent — safe to re-run, called from session-start hook too)
bash scripts/install_backup_schedule.sh

# Verify
bash scripts/install_backup_schedule.sh --check
ls -lht data/backups/daily_*.sql.gz | head -3

# Manual snapshot
bash scripts/backup.sh

The plist installed is ~/Library/LaunchAgents/com.metazen.agent-memory-backup.plist. Operator details: docs/backups.md.

Where to read next

• •HANDOFF.md — current state (v2 retraction status, v3 plan refs, setup-on-new-machine, resume commands)
• •AGENTS.md — file-map operating guide for agents working in this repo
• •docs/fine_tune/V3_PLAN.md — current training plan with multi-turn fixes
• •docs/fine_tune/FAILURE_MODES.md — 12 known failure modes + fixes (start here when something breaks)
• •docs/fine_tune/WIZARD.md — wizard reference
• •docs/fine_tune/PIPELINE_RUNBOOK.md — phase-by-phase training recipe
• •docs/fine_tune/V2_DATA_PIPELINE_PLAN.md — full data-pipeline design (prompt↔tool_call linkage, project consolidation, build_v2_dataset)
• •docs/training_runs/ — per-run reports including the v2 real-world A/B
• •docs/backups.md — daily backup operator reference
• •docs/PRIMER.md — multi-agent integration guide (Cursor, Windsurf, Cline, Codex, Zed, custom)

License + contact

No LICENSE file is committed. Treat the repo as private until one is added. Maintainer: [email protected] (see ~/.claude/CLAUDE.md).