Hermes Agent Phase 4: The Master Level

"Phase 3 made it a platform. Phase 4 lets you reshape the platform into anything."

---

What You'll Learn in Phase 4

You've come a long way. In Phase 1 you installed and ran Hermes. Phase 2 gave it memory and skills. Phase 3 turned it into an automation platform. Now it's time for the final frontier: shaping the agent itself.

This is the most technical phase. It's designed for developers, ML engineers, and power users who want full control over their agent's behavior.

By the end of this guide, you'll be able to:

• 🎯 Record and analyze your agent's decision-making trajectories
• 🤖 Train a reinforcement learning model from your own agent usage data
• 🧪 Generate synthetic training data for custom agent behaviors
• 🔧 Fine-tune a custom Hermes model for your specific domain
• 🏗️ Build a complete "train-deploy-loop" for continuous agent improvement

---

Part 1: Trajectory Recording — Watching Your Agent Think

Before you can train a better agent, you need to understand how your current agent behaves. Hermes records trajectories — complete logs of every decision, tool call, and thought process.

Enabling Trajectory Recording

# Enable trajectory recording (disabled by default)
hermes config set trajectory-recording true
hermes config set trajectory-directory ~/.hermes/trajectories/
hermes config set trajectory-retention-days 30

What a Trajectory Contains

Each trajectory is a JSON file capturing the complete flow of an agent session:

{
  "session_id": "sess_abc123",
  "timestamp": "2026-07-10T08:00:00Z",
  "turns": [
    {
      "turn": 1,
      "user_message": "Search for the latest AI news and summarize",
      "agent_thinking": "The user wants a news summary. I should use the web_search tool first to find current AI news, then synthesize the results.",
      "tool_calls": [
        {
          "tool": "web_search",
          "input": {"query": "AI news July 10 2026"},
          "output": {"results": [...], "status": "success"},
          "latency_ms": 1200
        }
      ],
      "response": "Here are today's top AI stories...",
      "reward": 0.95
    }
  ],
  "summary": {
    "total_turns": 5,
    "total_tool_calls": 8,
    "avg_latency_ms": 850,
    "user_satisfaction_score": 0.92
  }
}

Analyzing Trajectories

hermes trajectory list                    # List all recorded trajectories
hermes trajectory view --id "sess_abc123"  # View specific trajectory
hermes trajectory stats                   # Aggregate statistics
hermes trajectory export --format "rl"    # Export for RL training
hermes trajectory prune --before "2026-06-10"  # Clean old trajectories

Trajectory Statistics Dashboard

hermes trajectory stats

# Output:
# Trajectory Statistics (last 30 days)
# ─────────────────────────────────────
# Total sessions:       847
# Total turns:         12,394
# Total tool calls:    18,572
# Avg session length:  14.6 turns
# Avg response time:   920ms
# Most used tools:     web_search (42%), code (23%), file (15%)
# Success rate:        87.3%
# Satisfaction score:  0.88

Key insight: Trajectories are the raw material for all the techniques in this guide. The quality of your training data depends entirely on the quality and diversity of your recorded trajectories. Run Hermes for at least a week with trajectory recording enabled before attempting training.

---

Part 2: RL from Agent Feedback — Training on Your Own Usage

RL (Reinforcement Learning) from agent feedback is the technique of training an agent to improve its behavior based on rewards derived from actual usage. Hermes implements a simplified version called RLfAF (Reinforcement Learning from Agent Feedback) .

The RLfAF Pipeline

┌──────────┐    ┌───────────┐    ┌──────────┐    ┌──────────┐
│ Record   │───▶│ Score &   │───▶│ Train RL │───▶│ Deploy   │
│ Traject. │    │ Reward    │    │ Policy   │    │ New Agent│
└──────────┘    └───────────┘    └──────────┘    └──────────┘

Step 1: Collect Your Training Data

# Export trajectories in RL training format
hermes trajectory export \
  --format "rl" \
  --min-score 0.7 \
  --output "~/hermes-training/trajectories.jsonl"

Step 2: Score and Reward

Hermes can automatically score each turn based on:

• Did the user approve or reject the response? (explicit feedback)
• Did the agent complete the requested task? (tool success rate)
• How long did it take? (efficiency reward)
• Did the agent need multiple attempts? (wasted effort penalty)

# Score trajectories automatically
hermes trajectory score --method "auto" --output "~/hermes-training/scored.jsonl"

# Manual scoring (review specific trajectories)
hermes trajectory score --id "sess_abc123" --interactive

Step 3: Run RL Training

# Start RL training from collected trajectories
hermes train rl \
  --data "~/hermes-training/scored.jsonl" \
  --base-model "hermes-3-8b" \
  --output-dir "~/hermes-training/rl-model" \
  --learning-rate 1e-5 \
  --epochs 3 \
  --batch-size 8

This runs PPO (Proximal Policy Optimization) — the same algorithm behind RLHF — using your own trajectory data as the reward signal.

Step 4: Evaluate and Deploy

# Test the trained model
hermes train evaluate \
  --model "~/hermes-training/rl-model" \
  --test-data "~/hermes-training/test-set.jsonl"

# Deploy the model as a local provider
hermes model add "my-personal-agent" \
  --provider local \
  --path "~/hermes-training/rl-model"

Practical Example: Training a "Better Writing" Agent

# 1. Record trajectories from writing sessions
hermes config set trajectory-recording true

# 2. After a week of writing tasks, export
hermes trajectory export --format "rl" \
  --filter-tools "code,file" \
  --output "writing-trajectories.jsonl"

# 3. Train a writing-optimized model
hermes train rl \
  --data "writing-trajectories.jsonl" \
  --base-model "hermes-3-12b" \
  --output-dir "~/hermes-training/writing-model" \
  --reward-weight-style 0.3 \
  --reward-weight-efficiency 0.2 \
  --reward-weight-accuracy 0.5

---

Part 3: Synthetic Data Generation — Creating Training Data at Scale

Real trajectories are the gold standard, but you often need more data than you can collect manually. Hermes can generate synthetic training data by simulating agent-user interactions.

Why Synthetic Data?

Limitation of Real Data	Synthetic Data Solution
Takes weeks to collect enough	Generate thousands of examples in hours
Covers only your usage patterns	Explore edge cases and rare scenarios
May contain noise or errors	Clean, controlled generation
Privacy-sensitive	No real user data in training

Generating Synthetic Conversations

hermes train generate \
  --output "~/hermes-training/synthetic.jsonl" \
  --num-examples 5000 \
  --topics "code_review,web_research,data_analysis" \
  --difficulty "mixed" \
  --format "rl_trajectory"

Domain-Specific Generation

# Generate examples for a code review specialist agent
hermes train generate \
  --output "code-review-synthetic.jsonl" \
  --num-examples 2000 \
  --topics "code_review" \
  --template "Review this {{language}} code for bugs, style issues, and security vulnerabilities" \
  --languages "python,javascript,typescript,rust,go" \
  --difficulty "hard"

Quality Filtering

Not all synthetic data is good data. Hermes includes a quality filter:

hermes train filter \
  --input "~/hermes-training/synthetic.jsonl" \
  --output "~/hermes-training/filtered.jsonl" \
  --min-quality 0.8 \
  --deduplicate true \
  --max-length 4096

---

Part 4: Fine-Tuning Custom Agent Models

For maximum control, you can fine-tune an entire model — not just the RL policy, but the base model itself. This requires more compute but gives you the deepest level of customization.

Prerequisites

Requirement	Minimum	Recommended
GPU VRAM	24GB (LoRA)	48GB+ (full fine-tune)
Training data	500 examples	2000+ examples
Time	2-4 hours (LoRA)	8-24 hours (full)
Storage	10GB	50GB+

LoRA Fine-Tuning (Low Resource)

LoRA (Low-Rank Adaptation) is the most accessible fine-tuning method. It trains a small set of adapter parameters while keeping the base model frozen.

# Prepare training data
hermes train prepare \
  --trajectories "~/hermes-training/filtered.jsonl" \
  --format "lora" \
  --output "~/hermes-training/lora-data/"

# Run LoRA fine-tuning
hermes train lora \
  --base-model "hermes-3-8b" \
  --data "~/hermes-training/lora-data/" \
  --output-dir "~/hermes-finetuned/lora-adapter" \
  --rank 16 \
  --alpha 32 \
  --epochs 5 \
  --batch-size 4

Full Fine-Tuning (Maximum Customization)

# Full fine-tune requires a GPU with sufficient VRAM
hermes train full \
  --base-model "hermes-3-8b" \
  --data "~/hermes-training/lora-data/" \
  --output-dir "~/hermes-finetuned/full-model" \
  --learning-rate 2e-5 \
  --epochs 3 \
  --fp16 true

Merging and Deploying

# Merge LoRA adapter with base model
hermes train merge \
  --base-model "hermes-3-8b" \
  --adapter "~/hermes-finetuned/lora-adapter" \
  --output "~/hermes-finetuned/merged-model"

# Register as a local provider in Hermes
hermes model add "my-finetuned-agent" \
  --provider local \
  --path "~/hermes-finetuned/merged-model"

# Test it
hermes model switch "my-finetuned-agent"
hermes --model "my-finetuned-agent"

---

Part 5: Building the Train-Deploy Loop

The final step is automating the entire pipeline — so your agent continuously improves without manual intervention.

The Continuous Improvement Pipeline

# ~/.hermes/training-pipeline.yaml
pipeline:
  name: "continuous-improvement"
  schedule: "0 3 * * 0"  # Every Sunday at 3 AM
  
  stages:
    - stage: collect
      action: export trajectories
      filter:
        min-score: 0.7
        last-days: 7
  
    - stage: filter
      action: quality filter
      min-quality: 0.8
      deduplicate: true
  
    - stage: augment
      action: generate synthetic
      count: 1000
      topics: ["code", "research", "writing"]
  
    - stage: train
      action: lora fine-tune
      base-model: "hermes-3-8b"
      epochs: 3
  
    - stage: evaluate
      action: run evaluation
      test-set: "~/hermes-training/test-set.jsonl"
      min-pass-rate: 0.85
  
    - stage: deploy
      action: deploy model
      name: "weekly-upgrade-{{date}}"
      only-if: "evaluation passed"

Running the Pipeline

# Run the full pipeline
hermes pipeline run --config "~/hermes-training-pipeline.yaml"

# Check pipeline status
hermes pipeline status

# View pipeline history
hermes pipeline history

---

Phase 4 Quick Reference

Trajectory Commands

hermes config set trajectory-recording true
hermes trajectory list
hermes trajectory view --id <id>
hermes trajectory stats
hermes trajectory export --format "rl"

RL Training

hermes trajectory score --method "auto"
hermes train rl --data <file> --base-model <model>
hermes train evaluate --model <path>

Synthetic Data

hermes train generate --num-examples <n> --topics <list>
hermes train filter --input <file> --min-quality <score>

Fine-Tuning

hermes train lora --base-model <model> --data <dir>
hermes train full --base-model <model> --data <dir>
hermes train merge --base-model <model> --adapter <path>
hermes model add <name> --provider local --path <dir>

Pipeline

hermes pipeline run --config <file>
hermes pipeline status
hermes pipeline history

---

Troubleshooting Phase 4

Out of Memory During Training

# Reduce batch size
hermes train lora --batch-size 2

# Use gradient accumulation
hermes train lora --gradient-accumulation-steps 4

# Try a smaller base model
hermes train lora --base-model "hermes-3-8b"

# Enable CPU offloading
hermes train lora --cpu-offload true

Poor Trained Model Performance

# Check training data quality
hermes train analyze-data --data "trajectories.jsonl"

# Common issues:
# - Too few examples (< 500)
# - Low-quality trajectories (score < 0.7)
# - Too much synthetic vs real data
# - Wrong base model for the task

# Try with more data or better filtering
hermes train filter --min-quality 0.85

Training Pipeline Fails

# Run stages individually to isolate the failure
hermes pipeline run --stage "collect"
hermes pipeline run --stage "filter"

# Check logs
hermes pipeline logs --last-run

---

The Complete Hermes Agent Series

You've completed the entire series. Here's what you've built:

Phase	Skills Acquired	Real-World Impact
🟢 Phase 1	Install, configure, first conversation	You have a working AI agent
🟡 Phase 2	Memory, skills, multi-platform gateway	Your agent remembers and reaches you anywhere
🟠 Phase 3	Custom models, cron, sub-agents, custom tools	Your agent works autonomously on a schedule
🔴 Phase 4	RL training, synthetic data, fine-tuning	Your agent learns and improves itself

The true power of Hermes Agent isn't any single feature — it's the compounding effect. Phase 1 gave you an agent. Phase 2 gave it memory. Phase 3 gave it autonomy. Phase 4 gave it the ability to improve.

Each level builds on the last. A Phase 4 agent that has been running for months with trajectory recording, periodic RL training, and custom fine-tuned models is in a completely different league from a fresh install.

Your final mission:

1. Enable trajectory recording and let it run for a week
2. Generate your first synthetic dataset
3. Run an RL training session with your own data
4. Deploy your first fine-tuned model as a custom provider
5. Set up the continuous improvement pipeline

---

Series accuracy: This guide reflects Hermes Agent latest stable release as of July 10, 2026. RL training requires compatible hardware — check hermes hardware check before starting.