Plugin route / Eval

Measure what Codex actually loaded.

Gloamere Eval binds the plugin manifest, Skill, and agent configuration to exact paths and SHAs before observing native Codex activation. Evidence gaps stay visible instead of becoming false accuracy.

Install Gloamere Eval Browse source

Native trace ledgerschema v3 / record 001

01 / promptprompt_sha25696f3…d221Raw prompt omitted
02 / target lockmanifest / Skill / agent SHAgloamere-workflowsThree path-bound files
03 / observationevidence_statusverifiedverdict pass

Specimen record: field anatomy only—not a published native-admission result.

Three explicit actions

Lock, validate, observe.

The runner uses Python 3.10+ standard library code and local Codex CLI execution. Each action has a separate responsibility.

inspect
Lock the installed identity.
Record plugin ID and version, installed and enabled state, plus path-bound SHA-256 values for the manifest, Skill, and agent configuration.
lint
Reject drift before execution.
Validate the suite, target lock, or report, then reread the target to catch content or identity changes.
native
Observe Codex’s event stream.
Run normal activation through codex exec --json and verify the actual SKILL.md path. Unknown or incomplete events fail closed.

Evidence contract

Coverage and accuracy stay separate.

Only verified observations receive a routing verdict. Every other state explains why scoring is unavailable and returns verdict = null.

evidence_status	What it means	verdict
`verified`	The recognized Codex event stream is complete, agrees with the model statement, and resolves to the locked Skill path.	`pass or fail`
`unobservable`	The event stream is malformed, truncated, or contains an unknown event.	`null`
`unavailable`	Codex, the plugin directory, or an enabled target cannot currently be observed.	`null`
`identity_conflict`	The model statement, duplicate Skill name, installed identity, or target lock disagree.	`null`
`execution_error`	Codex or the event adapter did not complete successfully.	`null`

Report privacy

Hash the prompt. Keep the path local.

Reports store the prompt SHA-256 by default, not the raw prompt.
Raw prompts require an explicit --include-prompts choice after reviewing the privacy boundary.
Reports do not copy absolute paths from the target lock.
JSON goes to stdout unless the user supplies --output.

Designed to do

Inspect routing evidence.

Evaluate native Codex Skill activation.
Surface duplicate names and identity drift.
Produce reproducible schema v3 reports.
Read compatible historical schema v2 reports.

Designed not to do

Change the system under test.

Does not execute the task written inside a test case.
Does not edit Skills or install dependencies.
Does not treat model self-report as loading evidence.
Does not bundle a business-specific benchmark suite.

Pinned installation

Install only after the immutable beta tag is published.

These commands become valid only after the release gate passes and the exact tag is published. Nothing is fetched from a moving branch.

Codex CLIv4.0.0-beta.1

codex plugin marketplace add gloamere/codex-plugins --ref v4.0.0-beta.1codex plugin add gloamere-eval@gloamerecodex plugin list --json

Default installer: installs Gloamere Eval.

Legacy safety: detects old 3.x selectors but never removes, disables, or modifies them.

PowerShell, macOS, and Linux installer commands