Hilo · How it works

The short version, with diagrams. Deeper detail is collapsed — click to expand.

In one sentence: Hilo is a FastAPI chat app that wraps the Claude Agent SDK. Every user message spawns a fresh agent subprocess that connects to a handful of HTTP MCP servers (Excel, Trustpilot, Freshchat), streams tokens back to the browser over Server-Sent Events, and persists the result to Postgres.

1 · Overview

One EC2 instance. One Postgres database. One FastAPI server. Three long-running MCP sidecar processes. One ephemeral agent subprocess per chat turn. That's the whole production stack.

2 · System architecture

The stack is intentionally boring. Everything runs on a single EC2 host and is wired together with systemd. No microservices, no queue, no worker pool.

3 · How a prompt is assembled

Before the agent ever runs, the orchestrator builds the prompt that the model will see. It's a layering process — start from a fixed foundation, then add context tailored to this user, this conversation, and the tools they have access to. Think of it as a sandwich: the bottom is generic, the top is the user's actual question, and the middle is everything we've stitched in to make the model competent.

The five layers, top to bottom

1. The base system prompt — fixed text shared by every conversation. It establishes the agent's identity (who it is, today's date, the company it works for), the house style for responses, and the list of built-in tools it can use (Read, Write, Bash, Grep, etc).

2. A usage guide per attached MCP server — for each MCP the user has access to, a short paragraph is appended explaining when to reach for it. For Trustpilot it says something like "use get_summary for instant star ratings, query_reviews when you need to run SQL against the review table". This is what stops the model from blindly guessing tool names.

3. The user's identity — their email gets interpolated into the prompt so the agent knows who it's talking to.

4. A skill preamble — Hilo runs a semantic search (pgvector) against a library of Markdown "skill cards" using the last few messages of context. The top matches are listed as "here are skills relevant to this task" — the agent reads each one on demand if it decides it applies.

5. The user's actual message — appended last. This is what the user typed.

The whole stack is serialized into a single JSON config file. The orchestrator then spawns the agent subprocess and points it at that file. The model reads layers 1–4 once at the start of the turn and treats layer 5 as the prompt to respond to.

4 · A single chat turn

The browser sends a POST to kick off the turn and immediately opens an EventSource to stream the response. Both share an in-memory queue keyed by session ID.

5 · MCP servers and their tools

The agent's "hands" are three Model Context Protocol servers — each a small HTTP service that exposes a handful of tools to the model. The SDK discovers them at session start and surfaces every tool as mcp__<server>__<tool> in the model's tool catalogue.

Excel MCP — spreadsheet over HTTP

Reads (and can write) a single xlsx workbook sitting on disk. The agent uses it to look up data the team has dropped into a spreadsheet — orders, customer lists, pricing, whatever.

Trustpilot MCP — reviews from the Trustpilot database

Backs onto a Postgres database holding ingested Trustpilot reviews. The agent reaches for this whenever the user asks about reputation, sentiment, churn risk, or specific reviewer complaints.

Freshchat MCP — live customer support history

Hits the Freshchat REST API to pull a specific customer's chat history. Used when the agent is helping with a support question and needs to know what's already been said.

{
  "trustpilot": {
    "type": "http",
    "url": "http://127.0.0.1:57401/mcp",
    "headers": { ... }
  }
}