GameSurface Prototype — Implementation Handoff Spec

Mission: Design and prototype a gamepad-first virtual keyboard system that upgrades text input on the RG40XXV game surface.

Status: STAGED — locked frames defined. LangGraph configs are recommendations only — team may freely explore and implement whatever works best, provided it's documented. Date: 2026-05-12

TAILSCALE / SSH ACCESS (All Devices)

The Pi, RG, and Mac are all on the same Tailscale mesh. This is the canonical access method.

Both on Tailscale 1.96.4 with Tailscale SSH enabled. No passwords, no sshpass, no ConnectTimeout. MagicDNS short hostnames not supported on RG (KNULLI limitation — always use IPs from the RG).

Hermes on Pi: If a command requires an interactive terminal (e.g., hermes model), use the hermesi wrapper at /home/mehdifarah/.local/bin/hermesi on the Pi. Pipe choices through stdin: printf '1\n1\n' | hermesi model.

Pi LangGraph server: Starts via cd /mnt/kitchen/pearl/game-surface-python && /home/mehdifarah/game-surface-venv/bin/python src/server.py. Serves on port 8000. RG reaches it at http://100.120.38.37:8000.

FORGEJO REPO

All work must be committed to the Forgejo repo at:

ssh://mehdifarah@relik-pi4:/home/mehdifarah/git/langgraph-game-surface.git

The repo already exists with prior commits (engine evaluations, prototype specs, Akashic Abyss archive). The implementation team should: - Commit all code, assets, configs to this repo - Log design decisions, architecture notes, and iteration history - Keep the repo well-organized (separate dirs for src/, docs/, assets/, config/) - Forgejo web UI at http://100.120.38.37:3001/ (may need configuration — SSH git works regardless)

LAYER 1: THE GAME — Player-Facing Surface

LOCKED (Non-Negotiable)

Assets

• Core game assets already exist (hex tiles, unit sprites, UI elements). Use these as the base.
• Team is free to create new animations, sprites, or polish assets to fill gaps or improve quality.
• All assets committed to the Forgejo repo under assets/.

Mission

Design and prototype a gamepad-first virtual keyboard system for RG40XXV. Output: well-documented concepts, builds, prototypes for different keyboard systems, text input methods, and structured Hermes skills.

Design Philosophy

Double diamond: Discover → Define → Design → Deliver → Review

Hex Grid

• 6×6 flat-top hex grid, 36 hexes (00–35)
• Each hex is a task node with a phase tag
• Hexes have adjacency-based movement (up to 6 neighbors)
• Renders on RG at 640×480 via Wargame Engine + pygame

Units

• 3 units: Rif (designer), Echo (developer), Sherpa (researcher) — names are placeholders
• Per-unit interactive turns — player moves one unit at a time, unit completes work, player reviews
• No auto-run mode
• Only one unit per hex
• Each unit carries: config (model/harness), trail context, access lists, history

Validation

• Push builds to the RG40XXV when ready for testing
• Iterate by taking screenshots on the RG and checking against the LangGraph state
• Verify via curl http://100.120.38.37:8000/state to confirm graph state matches what's visible on RG

Full Node Grid (36 Hexes with Task Descriptions)

Interaction Model & Menu System ⚠️ CRITICAL — NOT OPTIONAL

The game is NOT just a hex grid with units. Every tile, unit, and node has nested menus. The full interaction tree is 4 levels deep. This is the core UX — the team MUST implement this.

Menu Tree (Collapsed)

L0  HEX GRID       — D-pad navigates, A selects hex
L1  TILE MENU      — unit summary + node summary. X→Unit Menu, Y→Node Menu
L2a UNIT MENU      — Status, Config, Trail, Access, History, Move, Interact
L2b NODE MENU      — Properties, Output, Edges, Config, History
L3                 — full detail screens for each L2 item
L4a                — expanded output section
L4b                — full output page (cross-device to Mac if too dense)
L4c                — routing action (Curate Context / Point Wholesale)

Key Interaction Patterns

Correction Interview (when output is wrong): Player selects "Correct" at interrupt. MC branching interview narrows problem: what's wrong? → which section? → suggested fix direction. Agent echoes back interpretation. Player: Correct / Close / Not-at-all. All gamepad — no text input required.

Context Routing: From a completed node's Output view → select Route → choose content scope (entire/section/curated) → choose method (Curate/Pointer) → return to grid in routing mode → select target hex → confirm.

Output Display: Every node produces two things: full output (wiki page on Pi) + summary (structured sections with header/body/status). RG displays summary. A expands section inline. Y opens full wiki page (on Mac if too dense).

Interactive Terminal: When a unit is active at a node, the terminal shows live agent output, chain of thought, and the interrupt prompt. Player can type corrections (fallback) or use MC interview (primary).

Live Prompt Injection: Player corrections accumulate per-node. On next turn, the node reads all prior corrections and restructures. Corrections never overwrite — they append via reducer.

Per-Unit Independence: Each unit has its own turn counter, config, trail, and access lists. Only one unit per hex. Cross-unit context sharing via access lists.

Gamepad Controls

Wiki Reference Pages (Read These)

Phases & Node Distribution

Full Node Definitions

Gating (Prerequisites)

A hex is locked until its prerequisites reach "completed" or "deepened" status.

06 needs 00, 01, 04 (at least 2 of 3)
07 needs 02, 03 (at least 1 of 2)
08 needs 05
09 needs 06
10 needs 06, 08
11 needs 07, 08
12 needs 09          |  19 needs 15, 16       |  24 needs 19, 21
13 needs 10          |  20 needs 17          |  25 needs 16, 21
14 needs 10, 11      |  21 needs 17, 18      |  26 needs 20, 21
15 needs 12          |  22 needs 18          |  27 needs 23
16 needs 12          |  23 needs 14, 18      |  28 needs 22
17 needs 13, 14                               |  29 needs 24, 25, 26, 27, 28
18 needs 14                                    |  30 needs 28
                                               |  31 needs 29, 30
                                               |  32 needs 29
                                               |  33 needs 30
                                               |  34 needs 29
                                               |  35 needs 31, 32, 33, 34

Acceptance Criteria

• At least 2 distinct keyboard prototype concepts documented
• At least 1 working prototype running on RG (colored shapes acceptable)
• Hermes skill(s) for keyboard input created and tested
• All artifacts in Forgejo repo + wiki runtime section
• Post-run metabolism plan: concepts mergeable into OmniWiki under "Gamepad Input"

RECOMMENDED (Strong Suggestions)

Interaction Model

• Pre-filtered context access. Each unit's context_access list gates what Hermes receives. Curation happens upstream (player via routing menu, or curation agent). No LLM self-selection of context for prototype.
• Correction interviews. Inline interrupt() loop — 3-4 MC questions in the node function. Player narrows problem → agent echoes → player confirms/refines.
• Curation as standalone service. Not a hex on the map. Separate endpoint or RG-side preprocessing.
• Nested menu tree. See full menu spec in wiki for L0→L4 hierarchy.
• Output summaries. Structured sections with header/body/status. RG displays summary; full output lives on Pi wiki.
• Context-specific wiki views. Node Y opens that node's page. Unit Y opens unit's trail page.

Gamepad Controls

D-pad: navigate. A: select/confirm. B: back. X: unit menu. Y: node menu. SELECT: phase overlay. START: pause/save. L1/R1: cycle units.

Rendering

• Wargame Engine (Python + pygame) — selected after 7-engine survey
• Colored shapes + text for prototype (zero assets needed)
• Hex grid fits at ~100×87px per hex on 640×480

ADVISORY (Explored, Team Can Adapt)

Alternative Engines Evaluated

OpenRA (C#, no Python bridge), RecoilEngine (C++ RTS, overkill), Spring RTS (same), openage (dead), LOVE (adds Lua boundary), Fabula (dead, client-server pattern useful conceptually), Godot (overkill), Panda3D (3D, wrong domain).

Gamepad Input Patterns (Sub-Agent Research Available)

• Radial menus, chips composition, branching interviews, T9 predictive, tags/presets, thumbs up/down drill-down
• Correction interview protocol with 6 branching scenarios
• Full routing submenu design (content → method → target → confirm)
• Edge case protocols (incomplete output, stuck unit, direction disagreement, hex collision, revisit, routing failure, save/load)

Pending Verification

• Wargame Engine scene system: can it handle overlay vs. full-scene switching for the keyboard?
• Which keyboard concept gets built first? (Research determines this — hexes 00-05, synthesized at 19)

LAYER 2: THE MEMBRANE — Wiring Between RG and Pi

LOCKED (Non-Negotiable)

Network

• Pi 4 (Tailscale IP: 100.120.38.37) ↔ RG40XXV (Tailscale IP: 100.119.202.114)
• Both on Tailscale 1.96.4 with Tailscale SSH enabled
• SSH: ssh mehdifarah@relik-pi4 (Pi), ssh root@100.119.202.114 (RG)
• No passwords, no sshpass, no ConnectTimeout gymnastics
• MagicDNS short hostnames not supported on RG (KNULLI limitation — use IPs)

Probing Results (Verified)

• Ping Pi↔RG: 3–5ms steady state
• HTTP roundtrip RG→Pi: 16ms avg
• State fetch (36 nodes + 3 units): 18ms, 5.9KB payload
• Invoke (0.8s mock agent): 839ms total (agent dominates, not wire)

Save State

• Canonical save = LangGraph checkpointer on Pi
• RG "save" = 640×480 screenshot + hex/unit/turn metadata (visual bookmark only)

RECOMMENDED (Strong Suggestions)

HTTP API

• GET /state — returns full game state as JSON
• POST /invoke — unit moves, graph activates node, returns state + interrupt
• POST /checkpoint — creates named save point
• GET /health — connectivity check
• GET /nodes — node definitions + adjacency + gating rules

State Sync

• Polling at 100ms — 3-5ms latency makes polling imperceptible
• No WebSocket/streaming needed for prototype

Server

• FastAPI + uvicorn on Pi (lightweight, async, auto-docs)
• Port 8000

Wiki Serving

• Wiki files on HDD at /mnt/kitchen/pearl/wiki/
• Served over HTTP: GET /wiki/architecture/langgraph-game-surface.md
• RG renders context-specific pages; full wiki pages open on Mac for dense content

ADVISORY

State Payload

5.9KB baseline at 36 nodes. Grows with history but <50KB for full sessions. Not a bottleneck.

Polling vs Streaming

Polling chosen for simplicity. 3-5ms latency confirmed. Streaming (LangGraph astream) available as future upgrade.

LAYER 3: THE GRAPH / Pi RUNTIME — LangGraph Backend

IMPORTANT: The LangGraph configurations below are RECOMMENDATIONS based on our prototyping. The implementation team is free to explore and implement any LangGraph config that works best — use what's here as a starting point, not a constraint. Whatever you choose, document it properly in the Forgejo repo so it's traceable.

LOCKED (Non-Negotiable)

Agent Stack

• LangGraph on Python (venv at /home/mehdifarah/game-surface-venv/)
• Hermes as agent harness (Qwen 3.6+ primary via Nous Portal OAuth, Kimi K2.6 fallback)
• One runtime — Python for LangGraph + Hermes (no language boundary)
• Hermes call: hermes -z "prompt" --model qwen/qwen3.6-plus

Storage

• High-write data on 3.6TB HDD at /mnt/kitchen/pearl/
• SD card for OS/Docker/Forgejo only (low-write)
• Game code at /mnt/kitchen/pearl/game-surface-python/
• Wiki at /mnt/kitchen/pearl/wiki/
• Checkpoints at /mnt/kitchen/pearl/checkpoints/

Process

• Single Bun/Python process on host, or Docker container (team's choice)
• Forgejo stays in Docker (existing, working)
• probe server and old d3-tui container can be retired (wiki already archived)

Concurrent Load

• Up to 5 concurrent Hermes calls (3 units + poller + curation agent)
• Pi 4 has 1.8GB headroom at full load
• Hermes Portal OAuth rate limit is the real cap — not Pi hardware

RECOMMENDED (Strong Suggestions)

LangGraph Graph (Compiles and Runs — Verified on Pi)

Graph status: ✅ Compiles, 36 nodes, hex adjacency, MemorySaver checkpointer. Server running on port 8000.

Two invocations tested: - Sherpa moved to hex 02 (turn 2, position updated, state persisted) - Rif moved to hex 16 (locked — prerequisite hex 12 not completed, gating enforced)

State size: 16.9KB after 2 invokes with Hermes output.

State Schema (Omni Config)

GameState:
  nodes: dict          # 36 hex entries (merge reducer)
  units: dict          # 3 unit entries (merge reducer)
  node_player_prompts: dict  # corrections per hex (append reducer)
  staged_configs: dict       # poller → unit handoff
  routed_context: dict       # curation → target handoff
  mission: dict              # objective, criteria
  decisions: list            # upstream choices (append reducer)
  turn_log: list             # traceability log (append reducer)

Single Thread, All Units

One thread_id for the game. All 3 units share state. Merge reducers prevent cross-unit overwrites. SqliteSaver serializes writes (fine for turn-based).

Node Functions

• 36 stubs — each calls Hermes -z one-shot, produces output_summary, calls interrupt() for player review
• Factory pattern: make_node_function(hex_id) generates per-hex functions from NODE_DEFS
• Interrupt options: accept, correct, skip, deepen

Hex Adjacency

• Computed from 6×6 flat-top hex grid geometry
• Each hex connects to up to 6 neighbors
• Hex 14 neighbors: 07, 08, 13, 15, 19, 20

Gating at Entry

• Node function checks prerequisites on entry
• If locked, returns status="locked" with missing prerequisites listed
• RG client also prevents move selection to locked hexes (UI gating)

Fallback Chain

Qwen 3.6+ (Nous Portal OAuth — free, unlimited)
  ↓ if rate-limited/error
Kimi K2.6 (API key — already configured)
  ↓ if unavailable
MiniMax (API key — deferred)

ADVISORY

Poller/Stager

Recommended: inline interrupt() loop inside node function. 3-4 MC questions. No subgraph needed for prototype. Can be upgraded to subgraph later.

Curation Agent

Recommended: standalone endpoint/service, not a hex on the map. Keeps hex graph pure.

MemorySaver vs SqliteSaver

MemorySaver used for prototype (simpler, state lives as long as Python process). SqliteSaver for production (state survives process restarts).

Hermes Self-Skills

Hermes creates its own skills from within the harness. Only intervene externally if it fails twice. Do NOT pre-build skills.

Pi Surface Migration

Full 7-phase migration plan documented in wiki (q-01-migration-plan.md). Phases: backup → cleanup → install Bun → create container → directory conventions → OAuth/fallback → verify.

APPENDIX: Full Documentation Index

All design docs live in the wiki at /workcell/llm-wiki/wiki/ and the repo at ssh://mehdifarah@relik-pi4:/home/mehdifarah/git/langgraph-game-surface.git.

Spec Location on Pi

Wiki:  /workcell/llm-wiki/wiki/architecture/implementation-spec.md
       (inside d3-tui-pi-teams-proto Docker container)

Repo:  docs/IMPLEMENTATION_SPEC.md
       (in langgraph-game-surface.git, commit 8ee849a)

Desktop: ~/Desktop/IMPLEMENTATION_SPEC.md (this file)