Files
claw3d/docs/agent-state-model-spec.md
T
gsknnft 083c146aac feat: add runtime seam, Hermes adapter support, and demo gateway mode (#89)
* fix: include kanbanImmersive in immersiveOverlayActive calculation

When Kanban board is open, HUD elements (camera preset buttons, edit toolbar, overlays) should be suppressed. The kanbanImmersive flag was defined but not included in the immersiveOverlayActive condition, causing HUD elements to remain visible.

This fix adds kanbanImmersive to the immersiveOverlayActive calculation so HUD elements are properly hidden when the Kanban board is open.

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* Fix: Hide mini status bar when Kanban immersive overlay is open

Wraps the bottom-left mini status bar (showing agent stats, vibe score, and
control hints) with !immersiveOverlayActive check to match the behavior of
other HUD elements like camera controls and toolbar.

This ensures the status bar is properly hidden when the Kanban board or any
other immersive overlay is active, maintaining a clean immersive experience.

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* chore: drop unrelated package-lock line from branch

Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>

* universal-backend-plan

* backend-neutral runtime seam

* package.json update

* feat: add Hermes gateway adapter as alternative to OpenClaw

Adds a WebSocket adapter that lets Claw3D connect to a Hermes AI agent
runtime without any changes to the frontend. The adapter implements the
full Claw3D gateway protocol and bridges it to the Hermes HTTP API.

Changes:
- server/hermes-gateway-adapter.js: WebSocket bridge implementing the
  Claw3D gateway protocol against the Hermes HTTP API. Supports all
  core methods (agents, sessions, chat streaming, cron, config, files,
  approvals) and multi-agent orchestration via spawn_agent/delegate_task
  tools. Persists conversation history to ~/.hermes/clawd3d-history.json.
- scripts/clawd3d-start.sh: All-in-one startup script that launches
  Hermes, the adapter, and the Next.js dev server with auto port
  conflict resolution. Alias as `claw3d` for convenience.
- src/features/office/hooks/useCronAgents.ts: Hook that polls the
  gateway for cron-scheduled agents and surfaces them in the 3D office.
- package.json: adds `hermes-adapter` npm script
- .env.example: documents Hermes config vars
- docs/hermes-gateway.md: setup guide and protocol reference

Usage:
  npm run hermes-adapter   # start adapter (connect to http://localhost:8642)
  npm run dev              # start Claw3D, point browser at localhost:3000
  # or: bash scripts/clawd3d-start.sh  (starts everything automatically)

Both OpenClaw and Hermes are supported simultaneously — the gateway URL
in NEXT_PUBLIC_GATEWAY_URL determines which backend Claw3D connects to.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: add read_agent_context tool for cross-agent coordination

Agents can now read each other's conversation history via the
read_agent_context tool, enabling the orchestrator to check what
a sub-agent has done before re-delegating work.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat: wire Hermes office UX and role-aware runtime updates

* feature update - demomode & hermes adapter

* fix lint blockers

* lintfix #2

* fix: stabilize retro office camera preset callbacks

* Initial plan

* fix: stabilize retro office overview preset hooks

Agent-Logs-Url: https://github.com/gsknnft/Claw3D/sessions/9cc71555-591e-44cf-aec4-25affbdcb405

Co-authored-by: gsknnft <123185582+gsknnft@users.noreply.github.com>

* feat: add truthful backend selection, Hermes adapter hardening, and demo gateway mode

* fix: address bugbot review and finalize backend selection

* fixed - onboarding and hermes calls

* office systems roadmap

* feat specs in docs

* specs ready

* feat: continue custom runtime seam and gateway alignment

* custom lane wired

* feat: add custom runtime provider path and office runtime alignment

* runtime fixes

* fix lukes findings

* fix lukes findings #2

* stable UI & connect screen page -> overlay

* better baseline for connection

* stable providers & ui rendering

* best launch yet

* nearly no gateway on reconnect

* auto reconnect last state

* fix: preserve selected runtime across reconnects

Keep backend selection aligned with the operator's chosen runtime instead of reviving a mismatched last-known-good adapter, and keep custom runtimes prompting for reconnect when Studio cannot auto-connect them.

Made-with: Cursor

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Luke The Dev <iamlukethedev@users.noreply.github.com>
Co-authored-by: Elias Pfeffer <eliaspfeffer@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: iamlukethedev <lucas.guilherme@smartwayslfl.com>
2026-04-02 15:27:24 -05:00

381 lines
8.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# Agent State Model Spec
> Seventh concrete office-system feature for Claw3D, formalizing operational agent states before any deeper simulation or affective layer is introduced.
## Goal
Add a public, office-visible agent state model that explains how an agent is currently operating.
The state model should answer:
- what is this agent doing right now?
- are they available?
- are they blocked?
- are they overloaded?
- should the office route more work to them?
This state model should be understandable to users and stable across backends.
## Core Principle
Start with operational states, not emotional roleplay.
That means the visible model should describe work conditions such as:
- focused
- idle
- blocked
- overloaded
- waiting
- recovering
- degraded
The office should show states that are useful for coordination.
Any richer or more proprietary internal signal stack can feed these states later, but should not be required to understand them.
## Why This Matters
Claw3D already visualizes activity, presence, and meeting participation.
What is still missing is a reliable office-wide language for work condition and agent load.
A proper state model would improve:
- delegation
- meeting routing
- desk progression meaning
- team visibility
- office atmosphere
It also gives a clean place for future internal state systems to map into.
## Product Position
The agent state model is:
- a public coordination surface
- not a hidden internal metric system
- not a mood simulator in V1
It should help the user make decisions quickly.
## Recommended Visible States
Suggested initial state set:
- `idle`
- `focused`
- `working`
- `waiting`
- `blocked`
- `overloaded`
- `recovering`
- `degraded`
- `meeting`
- `error`
### Idle
The agent is available and not actively engaged in a task.
### Focused
The agent is actively working and should not be interrupted casually.
### Working
The agent is doing normal active work without special load concerns.
### Waiting
The agent is paused on a dependency, approval, or external result.
### Blocked
The agent cannot make forward progress because a required condition is missing.
### Overloaded
The agent has too much work, too much context pressure, or too many active demands.
### Recovering
The agent has recently completed intense work or a failure state and should stabilize before taking more on.
### Degraded
The agent is operational but impaired in quality, speed, or confidence.
### Meeting
The agent is participating in a coordination workflow and is temporarily occupied there.
### Error
The agent encountered a concrete failure or unrecoverable problem and needs attention.
## Relationship To Existing Presence
Claw3D already uses simpler presence states such as:
- idle
- working
- meeting
- error
This spec should extend that concept rather than replace it abruptly.
The new states should be layered so older/fallback providers can still map into the simpler model.
## Public vs Internal State
This distinction matters.
### Public State
What Claw3D shows in the office:
- stable
- understandable
- backend-neutral
- useful for coordination
### Internal State
What a stack like Vera might use internally:
- latent regime
- coherence
- confidence or control signals
- advanced routing/load heuristics
Internal state can be richer.
Public state should remain simpler and more durable.
## Suggested Mapping Model
Recommended structure:
```ts
type OfficeAgentState =
| "idle"
| "focused"
| "working"
| "waiting"
| "blocked"
| "overloaded"
| "recovering"
| "degraded"
| "meeting"
| "error";
type AgentStateReason =
| "task_active"
| "approval_pending"
| "dependency_wait"
| "meeting_active"
| "high_load"
| "recent_failure"
| "runtime_signal"
| "manual_override"
| "unknown";
type OfficeAgentStateSnapshot = {
state: OfficeAgentState;
reason: AgentStateReason;
updatedAt: string;
note?: string | null;
confidence?: number | null;
};
```
The `reason` is important because it prevents the state from feeling arbitrary.
## V1 Derivation Rules
The first version should use straightforward observable signals.
Examples:
- active run -> `working` or `focused`
- pending approval -> `waiting`
- unresolved dependency or explicit blocker -> `blocked`
- too many simultaneous demands -> `overloaded`
- active standup/meeting -> `meeting`
- recent hard failure -> `error` or `degraded`
- no recent work -> `idle`
This gives immediate utility without needing a deeper cognitive model.
## Relationship To Other Office Systems
### Meetings
Meeting participation should temporarily dominate many other states.
Example:
- a focused agent who joins a standup becomes `meeting` while the meeting is active
### Bulletin Board
The bulletin board should be able to surface meaningful state changes.
Examples:
- "Alice blocked waiting on approval"
- "Bob overloaded during release push"
- "Hermes degraded after provider failure"
### Whiteboard
Planning workflows can use state to decide:
- who is available
- who should not be interrupted
- who is the right candidate for next actions
### QA Department
QA and state should influence each other.
Examples:
- repeated QA failures may push an agent or workflow into `degraded`
- review-ready but approval-blocked work can show `waiting`
### Desk Progression / Hierarchy
More mature roles may tolerate:
- more context
- more delegation
- higher review authority
But the visible state model should still be shared across all roles.
## Visual Expression
States should be visible in restrained ways.
Examples:
- nameplate subtitle or badge
- desk lighting/accent
- movement pacing
- speech bubble framing
- office panel badges
Examples by state:
- `focused`: sharper or brighter work signal
- `waiting`: subdued idle with pending marker
- `blocked`: warning tone
- `overloaded`: high activity / stress marker
- `meeting`: meeting-specific signal
- `degraded`: weakened signal, slower feel
Keep the visual language readable, not noisy.
## Human Interaction Model
The human should be able to:
- see current state
- understand why the state was chosen
- optionally override state in limited cases
The user should not have to guess what "degraded" or "blocked" means.
## Provider / Backend Considerations
Different runtimes will expose different levels of insight.
That is fine.
The public state model should support:
- direct provider-native state
- derived state from Claw3D activity
- optional custom stack enrichments
This is especially important for:
- OpenClaw
- Hermes
- Demo mode
- future custom providers
## Vera / Coherence / Latent-Regime Compatibility
This spec intentionally leaves room for deeper internal models without exposing them directly.
Good future pattern:
- internal stack computes richer latent regime / coherence / workload state
- adapter maps that into public office states
- Claw3D shows the public office state plus optional note/reason
This preserves:
- clean UX
- backend neutrality
- proprietary implementation freedom
## V1 Scope
Recommended V1 scope:
- define expanded office agent state enum
- derive state from observable office/runtime signals
- surface state in UI and office visuals
- show a short reason or note when helpful
Do not build the full simulation layer yet.
## Out of Scope For V1
- emotional simulation
- arbitrary personality modeling
- hidden scoring systems shown as fake moods
- deep physiological metaphors
Those can come later if they remain useful and readable.
## Implementation Strategy
Recommended order:
1. Define public state model and reason codes.
2. Add derivation rules from current runtime/task/meeting signals.
3. Update office visuals and badges.
4. Add optional state note/reason surfaces.
5. Leave room for future provider-specific enrichments.
## Existing Code Seams
This should likely align with:
- current office presence/state handling
- runtime event bridge and latest-update logic
- standup meeting state
- task and approval signals
- office visual systems
## Success Criteria
V1 is successful if:
- users can tell what condition each agent is in
- state changes help routing and coordination
- the model works without any proprietary backend
- the design still leaves room for richer future internal stacks
## Summary
The agent state model should become the offices shared language for work condition.
It should be simple enough to understand immediately, but structured enough to accept richer future inputs from advanced runtime stacks.