Synthetic evaluation framework¶
TrustWeave includes a versioned, local-only synthetic corpus for checking whether the tool produces the documented review categories from supplied declarations and minimized metadata. The corpus is designed for reproducibility and evidence clarity; it is not a live-agent test, penetration test, or runtime security benchmark.
Run the corpus¶
From a local checkout with development dependencies installed, validate the checked-in corpus contract before executing any case:
python scripts/run_evaluation_corpus.py --check
Then run the corpus:
python scripts/run_evaluation_corpus.py --verify
The preflight validates the stable corpus identity, ordered case IDs, local path boundaries, and assertion shapes without running a case. The verification command runs twelve checked-in synthetic cases in a temporary local directory and returns non-zero only when a corpus expectation does not match. Review-required cases are expected controls when their declared exit state and finding category match.
What the corpus covers¶
| Area | Evidence produced | Important limit |
|---|---|---|
| Manifest validation | Fail-closed errors for missing declared trust or classification data. | It does not authenticate supplied declarations. |
| Policy review | Clear and review-required local policy artifacts. | It does not enforce policy at runtime. |
| Bundle diff | Review signals for declared capability and approval-control changes. | It does not discover undeclared deployed changes. |
| Trace review | Local comparison of minimized metadata with supplied flows and policy. | It does not inspect message content, tool arguments, or trace completeness. |
| MCP metadata profile review | Local mapping and action-class review from supplied metadata. | It does not connect to, discover, or authenticate an MCP server. |
| Declaration-consistency benchmark | Exact-label comparison of supplied framework metadata and a supplied manifest, with separately retained reviewer-declared local label pairings. | It does not inspect source code, infer semantic equivalence, establish either declaration's completeness, or observe runtime behavior. |
A prepared declaration-consistency benchmark provides fourteen reproducible, local-only controls across supplied OpenAI Agents-style, LangGraph-style, and CrewAI-style descriptors. The fixtures cover exact agreement, empty inventories, one- and multi-label differences in either direction, raw bidirectional differences, explicit reconciliation, and partial reconciliation; they retain raw labels even when a fixture declares a pairing. This remains a fixture-level consistency demonstration, not an external benchmark result or proof that any framework export is authentic or complete.
Evidence status¶
The corpus and future reviewer protocol are prepared foundations. Independent reviewer responses, pilots, comparative benchmark outcomes, adoption evidence, and a durable research archive are not yet collected. The project will report those only after they exist, are consented where required, and are reviewed against the stated evidence rules.
For the complete operational materials, see the repository’s evaluation charter, reviewer quickstart, prepared offline reviewer packet, corpus lifecycle, archive-readiness checklist, local artifact-manifest tool, data-minimization policy, and status ledger.
Passing this corpus demonstrates only conformance on supplied synthetic inputs. It does not demonstrate runtime enforcement, input authenticity, attack prevention, general security efficacy, user adoption, or productivity outcomes.