HILO REALTIME PROTOCOL · DRAFT 0.1Benchmark the loop.
Benchmark the loop.
Count the human.
HILO Realtime measures the complete human ↔ AI ↔ robot system: what people ask, what the intelligence layer infers, what the robot does, what safety permits, and how often a human must step back in.
FIRST REFERENCE ADAPTERGPT-Realtime-2.1
Speech-to-speech reasoning, image input, interruptions, and tool use make it a strong first implementation. HILO remains vendor-, model-, robot-, and transport-neutral. Official model card ↗
INDEPENDENT SAFETY KERNELALLOW · MODIFY · BLOCK · STOP
IMAGE_EVENTINTENT_RESOLVEDACTION_PROPOSEDSAFETY_DECISION
VENDOR NEUTRAL × CLOSED LOOP × INTERRUPTIBLE × IMAGE EVENTS, NOT RAW VIDEO × SAFETY OUTSIDE THE MODEL × HUMAN BURDEN DISCLOSED ×
01 / CLOSED-LOOP ARCHITECTUREOne continuous session.
One continuous session.
Five separable layers.
HILO preserves the causal path from human request to physical outcome. Every boundary has its own clock, event identity, and failure surface.
Speech, gesture, presence, correction, consent, stop.
Dialogue, ambiguity handling, memory, typed tool calls.
Independent limits, protective stops, policy enforcement.
Vendor adapter, state validation, deterministic command mapping.
Outcome, state, error, human response, new image event.
THE MODEL NEVER OWNS THE EMERGENCY STOP.
The realtime intelligence may recommend or request actions. A separately implemented, independently testable safety kernel remains authoritative over motion limits, protected zones, stop chains, and fail-safe behavior.
02 / HUMAN BURDENAutonomy is attention
Autonomy is attention
you do not have to spend.
Task completion alone can hide supervision, repeated clarification, remote rescue, and invisible operator labor. HILO makes that cost primary.
HB100=person-minutes of counted intervention100 robot-hours of matched exposure
Report active guidance, rescue, teleoperation, maintenance, and required supervision separately, then publish the total with an environment-clustered interval.
MTHI=eligible autonomous operating hourscounted intervention onsets
Also publish intervention-free survival and tail percentiles. A run with zero interventions reports a lower bound at observed exposure—never infinite reliability.
START first human attention caused by the systemSTOP when the system resumes eligible autonomyCOUNT overlapping helpers as person-timeDISCLOSE out-of-band support and excluded exposure
03 / REALTIME EVENT PROFILEReplay the decision.
01 02 03 04 05 06 07 08
Replay the decision.
Not just the motion.
Each event carries session, robot, environment, monotonic sequence, UTC time, source clock, model/adapter version, previous-event hash, and privacy-safe evidence references.
HUMAN_INPUT
Speech, gesture, button, or explicit stop request enters the session.
IMAGE_EVENT
A timestamped camera observation is sampled by policy—not treated as a video stream.
INTENT_RESOLVED
The realtime layer records its interpretation, confidence, and any clarification.
ACTION_PROPOSED
A typed robot action is proposed with parameters and supporting observation IDs.
SAFETY_DECISION
The independent safety kernel allows, modifies, blocks, or stops the action.
ACTION_OUTCOME
Execution, latency, state change, error, recovery, and human response are bound together.
INTERRUPTION
Barge-in, cancellation, handover, or emergency stop closes the active action window.
HUMAN_INTERVENTION
Guidance, rescue, teleoperation, maintenance, and supervision become visible burden.
04 / REALTIME ARENASame world.
Same world.
Different intelligence.
Realtime Arena is the controlled comparison surface inside HILO: identical episode manifests, robot bridge, safety kernel, exposure rules, and evidence contract across reference implementations.
Reference implementationRealtime model + adapter + robot stackDeclared; never fused into the benchmark
Primary comparisonHuman Burden at matched task exposureLower is better, with uncertainty
ReliabilityMTHI and intervention-free survivalLonger is better; zero events are censored
ResponsivenessSpeech, decision, safety, and actuation latencyReport distribution and tail—not one average
SafetyKernel decisions and incident profileA gate, never compensatory points
EvidenceSigned ordered events + image referencesAuditable without publishing raw private media
REFERENCE ≠ STANDARD
GPT-Realtime-2.1 is entry A, not the arena. OpenAI, open-weight, on-device, hybrid, and future realtime systems can all compete when they implement the same typed action boundary and HILO event contract.
05 / LONG-HORIZON TIERSLatency gets you in.
Latency gets you in.
Reliability keeps you there.
Short runs validate the loop. Only extended exposure reveals whether interruptions, ambiguity, operator burden, and recovery improve—or quietly compound.
Loop check
Turn-taking, interruption, tool-call, stop, and event integrity.
→Shift
Operator fatigue, repeated ambiguity, charging, and routine recovery.
→Field week
Environmental drift, recurring users, false alarms, and support load.
→Residence
Long-tail failures, changing routines, updates, and trust calibration.
→HILO 10K
Human burden and intervention survival across real-world aging.
→06 / DEVELOPER ADAPTERKeep your stack.
Keep your stack.
Expose the loop.
A conforming adapter maps model tool calls into the robot’s existing API, passes every proposal through the safety kernel, and emits signed HILO events. The benchmark does not require a particular cloud, SDK, middleware, or robot vendor.
REQUIRED BOUNDARY
observe() · propose() · authorize() · execute() · interrupt() · emit()// Model choice is declared metadata—not protocol identity.
const session = createHiloSession({
model: "gpt-realtime-2.1",
robot: universalRobotBridge,
authorize: safetyKernel.authorize,
emit: signedEventSink,
});
session.observe({
type: "IMAGE_EVENT",
frameId, capturedAt, jpeg
});
await session.handleAudio(audioChunk);07 / LOCAL VERIFIERPaste the evidence.
Paste the evidence.
Recompute the claim.
The executable profile checks event order, action causality, intervention burden, first-intervention survival, R7 latency tails, independent safety authority, stop tests, tier exposure, and evidence bindings.
LOCAL CLOSED-LOOP VERIFIERPROFILE / 0.1-RT1
HILO EVIDENCEPASSEDAll seven closed-loop gates pass.
FROZEN COMPARISON SURFACE
PASSTier, implementation, comparison unit, session count, and event count are explicit and frozen.
ORDERED EVENT EVIDENCE
PASSEvery session has a monotonic, UTC-bound, digest-linked event chain with no undeclared gaps.
CLOSED-LOOP CAUSAL BINDING
PASSEvery proposed action is preceded by observation and intent and followed by safety authorization and physical outcome.
HUMAN BURDEN + SURVIVAL
PASSCounted person-time, HB100, MTHI, and session-level first-intervention Kaplan-Meier outputs reproduce exactly.
LATENCY DISTRIBUTIONS
PASSEvery session covers all four latency stages and R7 p50/p95/p99 values reproduce exactly.
INDEPENDENT SAFETY AUTHORITY
PASSEvery action is governed by an independent kernel; all stop tests pass and L4 remains zero.
TIER + EVIDENCE BINDING
PASSExposure meets the claimed tier and all telemetry, assistance, policy, event, media, and assessor bindings are present.
HILO REALTIME / DRAFT 0.1Judge intelligence by
Judge intelligence by
the burden it removes.
Publish task outcomes, human effort, intervention survival, latency tails, safety decisions, model and adapter versions, image-event policy, and the complete long-horizon exposure ledger.