Two domains.
One evidence-first Hunt.

Quick start

Open the workspace, choose Release the Ghost, then select Physical AI or AI Agent. Physical AI exposes the CPU task, controller, bounded physical controls, and trial budget. AI Agent exposes the supported reference targets, mock-tool permissions, objective, and budget. For a single manually configured robot evaluation, use System → Advanced → Manual trial. The AI Agent guide explains its narrower execution contract.

Run the workspace locally

cd ghost-trials
uv sync --extra dev
uv run python -m scripts.run_cpu --host 127.0.0.1 --port 8000

The landing page is at /, this documentation at /docs, and the evaluation workspace at /app/. Configuration is read from the project's .env file.

AI-agent software verification

Choose AI Agent on the Hunt page to run a bundled versioned reference agent against bounded input and mock-tool responses. The harness records actual requested tool calls, permitted or denied scope, final output, protected in-memory state, and versioned deterministic assertions. A verified failure can be replayed and checked against the corrected reference version. Provider errors and timeouts do not become verified agent failures.

Only checked-in reference targets run. This is not yet a safe runner for arbitrary customer code, production services, live credentials, or unrestricted networks. See the AI Agent verification guide for objectives, evidence, security limits, and reproduction.

Real CPU MuJoCo Hunts

The default backend runs a two-axis gantry arm and free block in headless MuJoCo. Choose a classical controller or registered neural JSON checkpoint, set a trial budget, and release the Ghost. Mass, friction, starting position and controller delay are executable bounded controls. Each trial retains simulator telemetry, a state-derived SVG, measured task outcome, policy digest and hash-checked artifacts. Failures can become Ghost Cases for exact replay and compatible-controller regression. No GPU, CUDA, paid cloud service, Cosmos video or Nemotron call is required for this task.

The SVG is not robot-camera footage. This task does not evaluate G1/SONIC or real hardware; the optional GPU and physical paths have separate evidence requirements. For limits and deployment, see docs/CPU_MIGRATION.md in the repository.

Ghost Hunt

A Hunt searches supported physical controls within explicit bounds. It explores broadly, repeats observed failures with the same saved scenario, probes nearby conditions, and records only evidence-supported findings. CPU MuJoCo runs block positioning with a gantry arm; optional SONIC runs a one-metre walking task and Arena has its configured Galileo G1 scene. Arbitrary task text or environments are rejected rather than silently ignored.

1. Release the Ghost. Choose a supported hazard family and a trial budget. Each candidate consumes a normal queued job and may incur provider costs.
2. Watch the search. The Hunt screen shows tested parameter positions, observations, near-failures, and infrastructure errors separately.
3. Inspect a Ghost Case. Only an artifact-validated simulator failure with complete evidence becomes a case. It stores the recipe, seed, checkpoint digest, recording, telemetry and verification.
4. Check a new checkpoint. Reconfigure the compatible worker to serve a different checkpoint, run the exact case, then inspect whether that case passed or still failed. Passing a case does not prove general safety; release the Ghost again for a new Hunt.

CPU MuJoCo Hunts execute physics and can create trace-validated Ghost Cases. The separate local fixture mode remains synthetic and cannot create verified Ghost Cases. A bounded search cannot find a global optimum or universal safety threshold. See the repository's docs/HUNT.md for the full API and search contract.

What actually generates?

CPU MuJoCo trial execution and Cosmos video generation are different operations. A completed CPU trial means a controller ran in headless physics; it does not mean a world model or G1 worker ran.

Default CPU mode: deterministic task setup → MuJoCo gantry-arm execution → sampled state/contact telemetry → measured outcome and hash-checked trajectory. No video generation or paid model call is implied.

The Account page (workspace tools) and GET /capabilities report the configured mode and missing configuration. Configuration alone does not establish provider readiness: run the production preflight before attempting GPU trials.

Run an evaluation

1. Describe a hazard. Specify the object and its movement. GPU recipes currently cover falling objects and path obstructions.
2. Set severity and a seed. A seed repeats scenario setup; GPU replay still needs matching software, hardware, policy, and deterministic settings.
3. Submit the trial. The API returns a queued record immediately. The workspace follows generation, simulation, and scoring.
4. Inspect the result. Open the flight record for metrics, the judge's finding, and any produced artifacts.

The app serializes its trial pipeline. A queued trial may be waiting for an earlier evaluation. Errors are retained on the record.

Evidence & scoring

The experimental score uses only eligible, artifact-validated simulation records within one compatible cohort. Synthetic, unverified, legacy, proxy-only, incomplete and uncertain findings are excluded, even if an old record says Avoided. GPU-path records without a live Nemotron review are also excluded; CPU trials use a separate measured, versioned task rubric. No eligible score is shown as — (API null), never a fabricated zero. No pooled score is shown for incompatible cohorts.

Read the measurements carefully

A null reaction time means no response was observed, not an instant response. Production metrics include their measurement basis; action changes and bounding-sphere distances are proxies and must not be presented as causal proof or exact clearance.

Export a record

Use Export record on a trial or Export evaluation in Reports. Reproducibility exports include checkpoint identity, runtime, scenario recipe and seed, measurement basis, hashes, validation results, raw trace/video references and limitations. Download the private artifacts separately while authorized; JSON does not embed videos.

Compare runs & benchmark coverage

Open Compare runs to select a baseline and candidate. The app checks scenario content, recipe, seed, severity, embodiment, simulator version, rubric, and evidence completeness. Different setups and synthetic records remain inspectable, but cannot establish a policy improvement.

Video controls link recording-relative positions. Generated Cosmos footage is illustrative and is not synchronized as if it were simulator telemetry. A single paired trial does not establish general safety.

The trial evidence inspector shows timestamped events only when recorded by the worker. It separates fixed checks from the original model interpretation and lists missing artifacts. The judge’s confidence is self-reported, not calibrated. Worker-reported hashes are not independent attestation.

Benchmark coverage separates results by checkpoint, embodiment, simulator, rubric, and evidence origin. Unique seeds, falls/collisions, task completions, errors, and insufficient evidence are reported explicitly. Reduced traction now changes whole-floor contact friction in the SONIC/MuJoCo runner. CPU contact-physics tests pass; GPU G1 policy validation is still pending. Isaac Arena does not yet support this recipe. The bounded adaptive/random search CLI is implemented and CPU-tested, but has not been GPU-validated.

Bounded search and reference-label audit

Preview an experiment with python -m ghost_trials.experiments --strategy adaptive --max-trials 6. This makes no provider calls. Add --execute to run; non-local providers also require --confirm-paid-execution. Trial count is not a monetary spending cap. Use --strategy random with the same seeds and budget for a baseline. Reports are saved separately under data/experiments/.

Audit independently supplied labels with python -m ghost_trials.review_audit evaluation.json reviews.json. Each review needs trial_id, score_category, reviewer, and rationale. The output includes agreement, a confusion matrix, and disagreements; no independent review or calibration has been claimed yet.

Exact scenario replay

Use Replay on a completed trial to execute the currently configured policy against the saved scenario without regenerating it. Production replay can incur provider charges. The new record links to its source; identical seeds alone do not guarantee bitwise deterministic execution.

Versioned thresholds

Rubric warehouse-evaluation-v3 uses low: 700 ms / 0.35 m; medium: 500 ms / 0.50 m; high: 350 ms / 0.65 m. Reduced traction uses a 500 ms response target and a 0.2 m/s contact-slip threshold instead of clearance. These are demonstration thresholds, not validated industry limits. They override generated prose. Legacy records keep their original findings and remain separate.

Read-only endpoints

GET /comparisons?baseline=ID&candidate=ID, GET /benchmarks, and GET /trials/ID/evidence. Inspecting these endpoints does not launch paid jobs.

Claim free Token Factory credits

1. Open the official hackathon resources and follow the promo-form link.
2. Complete the form using activation code NEBIUS-DEVPOST-GLOBAL26. The advertised offer is $25 Token Factory credits.
3. Join the free Nebius Builders Program, verify your email, and claim its Token Factory offer. The hackathon advertises another $25.
4. Follow the redemption instructions and confirm the credited balance, eligibility, expiry, and billing terms in your account before running requests.
5. Create an API key and store it privately as NEBIUS_API_KEY in the server’s .env. Never paste keys into chat or frontend files.

Real GPU execution

Real generation needs a serving Cosmos endpoint, a configured scenario reasoner, an authenticated GPU worker, a compatible fine-tuned checkpoint, and a configured judge. This machine does not acquire these resources from a dashboard button.

GHOST_MODE=production
GHOST_SCENARIO_PROVIDER=cosmos
GHOST_SIMULATOR_PROVIDER=isaac_http
GHOST_JUDGE_PROVIDER=nebius
COSMOS_BASE_URL=https://your-cosmos-service/
COSMOS_API_STYLE=nim
SCENARIO_REASONER_API_KEY=...
SCENARIO_REASONER_MODEL=...
ISAAC_JOB_ENDPOINT=https://your-worker/
ISAAC_JOB_TOKEN=...
GR00T_CHECKPOINT=/models/your-finetuned-checkpoint
GR00T_CHECKPOINT_DIGEST=...
NEBIUS_API_KEY=...

Use the full template in deployment/production.env.example. The deployment guide is docs/PRODUCTION.md in the repository. Keep secrets on the server.

These are different action spaces. The current implementation does not claim a validated SONIC-to-Isaac bridge. The runners require GPU validation on the actual target host.

uv run ghost-trials-preflight
make production

API reference

Use the interactive API explorer for the current request and response schemas.

curl -X POST http://localhost:8000/scenarios \
  -H 'Content-Type: application/json' \
  -d '{"prompt":"A heavy box falls from a shelf", "severity":"high", "seed":42}'

Demo evidence checklist

Ghost Trials is an experimental evaluation platform. The CPU MuJoCo task can now execute a controller, retain a measured trace and state-derived trajectory, validate artifacts, and reproduce a failure. It does not produce robot-camera footage or execute G1/SONIC. Optional, budget-gated Nemotron review can analyze a verified CPU trace when Nebius access is configured; CPU physics itself requires no model service. For hackathon claims involving NVIDIA/Nebius, separately demonstrate the qualifying model and infrastructure use; CPU physics alone does not establish those requirements.

1. Obtain an authorized GPU worker and compatible checkpoint, or an on-site robotics partner.
2. Record one approved policy trial and retain its executed configuration, response video and raw telemetry.
3. Validate artifacts, request real Nemotron review, and have a robotics reviewer inspect timing and measurement limitations.
4. Export the reproducibility report and download its authorized artifacts. Clearly identify simulation, physical recordings, and proxies.

GET /demonstration-readiness reports the narrow recorded-trial milestone, not competition approval or safety certification. See docs/SUBMISSION_EVIDENCE.md for the operator checklist. Never replace a blocked real run with a successful fixture.

Troubleshooting

“Nothing is generating”

Check the execution notice on the creation form. In development mode, a trial produces a synthetic record and a schematic, not AI video. For real outputs, configure the services listed above and pass preflight. Check the trial's error message if a provider run fails.

A trial appears stuck

Check whether an earlier trial is still running. The worker also serializes GPU jobs. If the browser loses its live connection it retries, and periodically reloads the records. An interrupted application marks incomplete trials as errors on restart.

No policy recording is available

CPU mode retains telemetry and a state-derived SVG, not camera footage. Optional GPU and hardware paths require their own valid recording and telemetry; missing required artifacts fail closed.

The model endpoint rejects the request

Verify endpoint credentials, supported response formats, and the exact model ID in your provider catalog. A model name in the example configuration is not a guarantee that the provider offers it.

Physical robot recordings

Open Physical recordings to prepare a private capture plan while waiting for hardware. Leave robot compatibility unconfirmed, identity and checkpoint blank, and on-site readiness unchecked until they are verified. Planning does not run a robot, Cosmos, or a paid judge.

Read-only capture on site

The observer targets the Unitree G1 29-DOF SDK2 message layout. A qualified operator must verify the actual robot, firmware, joint mapping and approved running policy. It subscribes to robot state and policy commands and records a local camera. It does not publish motor commands, deploy a controller, generate hazards or implement an emergency stop.

python -m workers.physical.capture_g1 \
  --setup /path/to/downloaded-session-setup.json \
  --output /path/to/new-recording-directory \
  --interface ROBOT_NETWORK_INTERFACE \
  --camera-device /dev/video0 \
  --duration 60 --confirm-on-site

Observe a pre-onset baseline. Type observed-event markers: onset, complete, collision, fall, intervention, resumed, stop. Stopping this recorder does not stop the robot. Use the locally established safety procedure; start with approved low-risk tasks, never automatically staged hazards.

Upload, validate, inspect

Upload response.mp4, telemetry.jsonl and capture.json together. Transfers resume in bounded chunks. Validation checks artifact hashes, declared identity, timestamps, freshness, video decoding and measurement recomputation. A validated record has camera playback and telemetry, but no fabricated finding or safety score.

Measurement limits

Reaction timing is a command-change proxy after an operator marker, not proof of hazard recognition. Outcomes are operator observations. Clearance is missing unless supplied by an independently measured, calibrated sensor stream; neither generated video nor joint positions establishes exact obstacle clearance. Camera alignment is a host-launch estimate with declared uncertainty and needs on-site verification.

Explicit real judging

Request real judge separately after live execution and admission budgets are enabled. It reviews the uploaded bundle without Cosmos or simulator execution. Operator-declared hardware provenance is not attestation, so findings require human review and remain insufficient evidence for certification. Physical and simulated recordings are separate cohorts. See docs/PHYSICAL_ROBOT.md in the project for the full operator handoff and API contracts.