AI agent monitoring
IMAA monitors AI agents that act as on-chain oracles — agents whose output ends up in a smart contract decision. Generic agent observability (cost, latency for a chatbot) is out of scope. The focus is agents where a compromised or drifting response becomes a blockchain event.
Why this matters
The boundary between “oracle” and “AI agent making a call that ends up on-chain” is dissolving. Prediction market resolvers, LLM-backed price oracles, and on-chain advisory agents all share the same failure mode: if the agent is compromised, drifts silently, or gets swapped for a different model, the contract executes on bad data. Monitoring the contract alone means monitoring half the attack surface.
Supported agent types
| Type | Description |
|---|---|
llm_oracle | Language model serving as a data source or decision oracle for a contract. |
prediction_resolver | Agent that resolves prediction market outcomes (e.g. Polymarket resolver). |
onchain_advisor | Agent whose recommendations are consumed directly by a contract (e.g. strategy optimiser, risk parameter setter). |
custom | Any other agent type with an HTTP endpoint IMAA can probe. |
Today, the self-serve way to add an agent is the Polymarket import below, which creates a prediction_resolver. The other agent types — and their endpoint, keys, and ingestion mode — are provisioned with us directly while the general add-agent form is in progress.
How it works
IMAA collects telemetry from an agent in one of two modes:
- Pull mode: IMAA probes the agent's endpoint on its configured probe interval (default 15 minutes). Each probe sends a configurable payload and records the response hash, latency, reported model ID, signature validity, and any disclosed cost estimate.
- Push mode: your agent posts telemetry to IMAA's ingest endpoint, signed with a shared secret. Useful for agents that don't expose a synchronous HTTP interface, or where you want finer-grained readings than a probe interval allows.
From each reading, IMAA folds the relevant signals into a per-dimension rolling baseline (an online mean and standard deviation). Alert rules threshold that running mean directly — see the metrics in the next section for exactly what each one measures. Response drift is an exact response-hash comparison against the previous reading, not a fuzzy similarity score.
Raw payload storage (full request and response content) is a platform-controlled feature, off by default and enabled by IMAA on request — it is not a self-serve org setting. With it off, IMAA stores only the derived signals above (hashes, timings, model IDs), never the raw content.
Alert rule types
Each dimension's metric is the running mean of a per-reading signal, which you threshold with a normal operator. For the four rate-style dimensions the mean is a fraction between 0.0 and 1.0 (the share of recent readings that tripped), so pick a fractional threshold — a value like 2.5 can never be reached.
| Rule type | Metric | Typical use |
|---|---|---|
agent_response_drift | Rolling rate of probes whose response hash differs from the previous reading (0.0–1.0). | Detect model swaps, prompt-injection effects, or unstable output. A threshold of 0.2–0.5 flags an agent whose answers keep changing. |
agent_latency_spike | Rolling mean response latency, in milliseconds (absolute). | Catch overloaded or degraded endpoints. The threshold is an absolute millisecond value (e.g. 2000 for a 2s mean), not a multiple of baseline. |
agent_signature_invalid | Rolling rate of responses whose signature failed to verify against the agent's public key (0.0–1.0). | A key-compromise / tampering indicator. Only meaningful when the agent returns a signature and you've set its public key. A low threshold (e.g. 0.1) alerts on any invalid signatures. |
agent_endpoint_down | Rolling rate of probes that errored or were unreachable (0.0–1.0). | The dimension that fires from failed probes — an agent that stops responding. This is the liveness alert. |
agent_model_swap_suspected | Rolling rate of readings whose reported model_id changed versus the prior reading (0.0–1.0). | A direct model-swap indicator — the model behind the endpoint being changed out from under you. A low threshold alerts on any swap. |
Severity is whatever you set on the rule (default warning) — nothing auto-escalates. As with contract rules, a firing rule is then suppressed for its window before it can fire again, so a persistently-down endpoint alerts once per window, not once per probe.
Polymarket resolver monitoring
Paste a Polymarket market URL into the Agents import flow. IMAA calls the Gamma API to discover the market's resolver and creates a single prediction_resolver agent pointed at it (or at the resolver's on-chain address where no off-chain endpoint is disclosed). From there it's monitored like any other agent — drift, latency, endpoint liveness, and so on.
This is a monitoring feature — IMAA watches the resolver and flags anomalies. It is not a market analytics tool and does not track prices or market positions.
Access
Agent monitoring is gated by the ai_agent_monitoring feature flag, which IMAA rolls out per organisation. When it is off, the Agents area is hidden and the agent API returns 403. There is no per-tier cap on the number of agents or a tier-based probe-interval floor — the probe interval is configured per agent.
Related
- Monitor targets — the five agent target types (
agent_response_drift,agent_latency_spike,agent_signature_invalid,agent_endpoint_down,agent_model_swap_suspected). - Alert rules — rule anatomy and rule types.