Methodology

What's here

Tools (reference implementations, deterministic, self-testing)

File	What it does
tools/goodhart_auditor.py	An epistemic linter: flags fields/metrics whose name claims a verified property (reviewed, verified) that nothing actually checks.
tools/knowledge_maturity.py	A deterministic evidence-maturity classifier: rates how much evidentiary work stands behind a claim (not whether it's true), with critical gates that quantity can't buy past.
patterns/containment_guard.py	A fail-closed guard that rejects any agent action which isn't human-authorized, reversible, bounded, and logged.
tools/optimal_timing.py	An optimal-stopping timing layer: from a cost structure (false-alarm vs miss) and an evidence model, it solves the Bayes-optimal act-or-wait boundary on the posterior — the decision layer that sits on top of any early-warning forecast and answers when to act.
tools/decoupling_monitor.py	A Goodhart-in-the-wild monitor: watches a reported proxy against an independent truth signal and alerts when the proxy keeps improving while the truth degrades — operational metric gaming, caught (in the demo) 17 steps before visible failure.
tools/ground_truth_auditor.py	A ground-truth independence auditor: checks whether the truth signal a decoupling monitor relies on is actually independent of the proxy (shadow detection without a reference; error-correlation with one), so you know whether a decoupling alarm means anything. Attacks the binding constraint every other tool depends on.
tools/eval_gaming_detector.py	A defensive eval-gaming / sandbagging detector (AI-native): flags when a model-evaluation score is decoupled from true capability — contamination (inflated → undermines a capability claim) or sandbagging (hidden → undermines an inability/safety claim). Per-eval signals (held-out gap, elicitation gap, non-monotone difficulty profile) plus a revision-stream monitor that reuses the decoupling monitor and ground-truth auditor. Abstract scores only; no techniques.

Every tool runs its own self-test: python <file>.py.

Two of these close the loop with the wider work in this project. decoupling_monitor is the operational complement to goodhart_auditor: the auditor catches a metric that overclaims by name at definition time; the monitor catches one that was honest but is being gamed in operation (the same proxy-vs-reality divergence as the money model's peg drifting from its backing, and "same price, worse product" hidden inflation). optimal_timing is the decision layer for early-warning systems: given a forecast (e.g. a collapse probability) and a cost asymmetry, it computes the act threshold — and it composes with the rest of the toolkit, since an "act now" recommendation still routes through non-self-approval and containment before anything executes.

The consolidation layer

File	What it does
soi/soi_pipeline.py	The Scientific Knowledge-Governance ("order") pipeline: one deterministic function that assigns and maintains a claim's epistemic-status order (PROVISIONAL → WORKING_BASIS → MULTI_DOMAIN_TESTED → VALIDATED → CANONICAL_CANDIDATE) by wiring the three tools below into five stages — maturity, disclosed open issues, metric hygiene, non-self-approval, adoption-vs-validation. It orders claims by status, never by truth; human authority is final. Self-testing: python soi/soi_pipeline.py.
soi/SOI-000_...Infrastructure.md	The spec: scope, the claim it does and doesn't make, object model, the ordering pipeline, authority model, limitations, and honest positioning.

This is the piece the rest of the toolkit points at: the individual tools are the enforceable components; soi_pipeline is the single ordering pipeline that composes them, and it ships two runnable demonstrations (a foundational-ontology derivation and a high-consequence "is it sentient?" claim) showing the order withhold what it must and promote only what earns each gate.

Patterns (design writeups)

File	What it covers
patterns/agent_containment_pattern.md	Composing agents so non-autonomy is structural, not behavioral — they propose; a gate + a human authorize; an external executor performs one reversible action.
patterns/federation_pattern.md	System-of-systems composition (router → specialists → synthesis) with artifact-only exchange and no shared memory, so capability scales without creating one opaque super-agent.
patterns/non_self_approving_derivation.md	Letting an AI do deep derivation (ontologies, option scoring, decision records) while making it structurally unable to certify its own conclusions — deterministic generation, self-reported open problems, non-averaged scoring, blank human-authority signatures.

Companion documents

• REF_Literature_Crosswalk.md — for the non_self_approving_derivation pattern's foundational-ontology worked example: maps the derived ontology onto the existing formal-ontology and metaphysics literature (Spencer-Brown, ontic structural realism, process philosophy, constraint-closure, BFO/DOLCE/UFO, OntoClean), separating genuine re-derivation from the method's real novelty, and listing what a submission would need to engage.
• CEM_SAI_Consciousness_Crosswalk.md — the same treatment for the pattern's high-consequence assessment worked example (machine consciousness, sentience, ASI candidacy): maps the in-house consciousness and sentience modules onto the science of consciousness (Butlin/Long et al. 2023 indicator-property approach, IIT, Global Workspace, Higher-Order, Recurrent Processing, Attention Schema, Block's access/phenomenal split) and the AI-welfare literature (Long/Sebo et al. 2024, Birch's precautionary sentience framework). Same verdict shape: the framework is an uncredited re-derivation of an active literature, and the real novelty is the governance wrapper — so publish the apparatus, engage the science as scholarship. Both crosswalks illustrate the same lesson the toolkit is built around: keep the disciplined method well away from any totalizing theory.

Companion framework

The broader QIERA epistemic-governance overlay (the L0–L6 reasoning pipeline these pieces slot into) is documented separately. The tools here are the enforceable components; QIERA is the pipeline that produces the artifacts they check.

Quick start

python tools/goodhart_auditor.py       # self-test + demo
python tools/knowledge_maturity.py     # self-test + demo
python patterns/containment_guard.py   # self-test + demo

No dependencies beyond the Python standard library. The compliance layer in compliance-toolkit/ additionally needs PyYAML and jsonschema.