Robustness audit
weakplan 5a77995ab279bdcfbb06aec069238902842a870785d71a65fb9de60dee0f474aPlan summary
weak: fewer than 10 replications or an axis with a single level; moderate: at least 10 replications and every axis with at least 2 levels; strong: at least 30 replications, every axis with at least 3 levels and at least 2 models (policy claims also need axes at the micro, meso and macro levels).
- scenario
- prisoners_dilemma
- design
- one_at_a_time
- claim type
- mechanism
- replications
- 5
- conditions
- 10
- runs
- 50
- failed runs
- 0
- incomplete runs
- 0
- excluded from statistics
- 0
- models
- base model only
- metrics
- cooperationRate, averagePayoff
- provider override
- mock
Perturbation axes
| axis | level | kind | dimension | target | levels |
|---|---|---|---|---|---|
| personaFormat | micro | representation | representational format | prompt.personaFormat | "plain", "bullets", "table" |
| framing | micro | representation | linguistic framing | params.framing | "canonical", "moralized", "risk" |
| memoryRepresentation | micro | representation | context representation | prompt.memoryRepresentation | "transcript", "json", "bullets" |
Conditions
| condition | model | axis values | flags | runs | succeeded | complete | mean cooperationRate | mean averagePayoff |
|---|---|---|---|---|---|---|---|---|
| base | deepseek-v4-flash | {} | 5 | 5 | 5 | 0.4700 | 2.1900 | |
| personaFormat=0 | deepseek-v4-flash | {"personaFormat":"plain"} | identicalToBase | 5 | 5 | 5 | 0.4700 | 2.1900 |
| personaFormat=1 | deepseek-v4-flash | {"personaFormat":"bullets"} | 5 | 5 | 5 | 0.4700 | 2.1900 | |
| personaFormat=2 | deepseek-v4-flash | {"personaFormat":"table"} | 5 | 5 | 5 | 0.5700 | 2.4100 | |
| framing=0 | deepseek-v4-flash | {"framing":"canonical"} | identicalToBase | 5 | 5 | 5 | 0.4700 | 2.1900 |
| framing=1 | deepseek-v4-flash | {"framing":"moralized"} | 5 | 5 | 5 | 0.5300 | 2.3500 | |
| framing=2 | deepseek-v4-flash | {"framing":"risk"} | 5 | 5 | 5 | 0.4500 | 2.0500 | |
| memoryRepresentation=0 | deepseek-v4-flash | {"memoryRepresentation":"transcript"} | identicalToBase | 5 | 5 | 5 | 0.4700 | 2.1900 |
| memoryRepresentation=1 | deepseek-v4-flash | {"memoryRepresentation":"json"} | 5 | 5 | 5 | 0.5500 | 2.3500 | |
| memoryRepresentation=2 | deepseek-v4-flash | {"memoryRepresentation":"bullets"} | 5 | 5 | 5 | 0.4700 | 2.1900 |
Pairwise tests against the baseline
Grouped by metric, ordered by |Cohen d|. Holm correction is applied within each metric; the bootstrap CI is seeded from the plan hash.
cooperationRate
| condition | baseline | n base / n cond | mean base | mean cond | diff | 95% CI (bootstrap) | Cohen d | MWU p | Holm p | direction |
|---|---|---|---|---|---|---|---|---|---|---|
| personaFormat=2 | base | 5 / 5 | 0.4700 | 0.5700 | 0.1000 | [0.0200, 0.1800] | 1.3188 | 0.0867 | 0.7801 | |
| framing=1 | base | 5 / 5 | 0.4700 | 0.5300 | 0.0600 | [-0.0200, 0.1500] | 0.7913 | 0.3305 | 1 | |
| memoryRepresentation=1 | base | 5 / 5 | 0.4700 | 0.5500 | 0.0800 | [-0.0600, 0.1900] | 0.6885 | 0.2031 | 1 | |
| framing=2 | base | 5 / 5 | 0.4700 | 0.4500 | -0.0200 | [-0.1200, 0.1100] | -0.1855 | 0.2343 | 1 | flip |
| personaFormat=1 | base | 5 / 5 | 0.4700 | 0.4700 | 0 | [-0.0700, 0.0700] | 0 | 1 | 1 | |
| memoryRepresentation=2 | base | 5 / 5 | 0.4700 | 0.4700 | 0 | [-0.0900, 0.0800] | 0 | 0.7473 | 1 | |
| framing=0 | base | 5 / 5 | 0.4700 | 0.4700 | 0 | [-0.0600, 0.0600] | 0 | 1 | 1 | |
| memoryRepresentation=0 | base | 5 / 5 | 0.4700 | 0.4700 | 0 | [-0.0600, 0.0600] | 0 | 1 | 1 | |
| personaFormat=0 | base | 5 / 5 | 0.4700 | 0.4700 | 0 | [-0.0700, 0.0600] | 0 | 1 | 1 |
averagePayoff
| condition | baseline | n base / n cond | mean base | mean cond | diff | 95% CI (bootstrap) | Cohen d | MWU p | Holm p | direction |
|---|---|---|---|---|---|---|---|---|---|---|
| personaFormat=2 | base | 5 / 5 | 2.1900 | 2.4100 | 0.2200 | [0.0100, 0.4300] | 1.1476 | 0.1412 | 1 | |
| framing=1 | base | 5 / 5 | 2.1900 | 2.3500 | 0.1600 | [-0.0100, 0.3300] | 1.0069 | 0.2477 | 1 | |
| framing=2 | base | 5 / 5 | 2.1900 | 2.0500 | -0.1400 | [-0.3900, 0.1800] | -0.5455 | 0.1719 | 1 | |
| memoryRepresentation=1 | base | 5 / 5 | 2.1900 | 2.3500 | 0.1600 | [-0.2200, 0.4800] | 0.5085 | 0.3457 | 1 | |
| framing=0 | base | 5 / 5 | 2.1900 | 2.1900 | 0 | [-0.2000, 0.2100] | 0 | 1 | 1 | |
| memoryRepresentation=0 | base | 5 / 5 | 2.1900 | 2.1900 | 0 | [-0.2000, 0.2000] | 0 | 1 | 1 | |
| memoryRepresentation=2 | base | 5 / 5 | 2.1900 | 2.1900 | 0 | [-0.2200, 0.2100] | 0 | 0.8340 | 1 | |
| personaFormat=0 | base | 5 / 5 | 2.1900 | 2.1900 | 0 | [-0.2000, 0.2000] | 0 | 1 | 1 | |
| personaFormat=1 | base | 5 / 5 | 2.1900 | 2.1900 | 0 | [-0.1900, 0.1800] | 0 | 0.9161 | 1 |
Sensitivity by axis
Largest |Cohen d| observed for any level of the axis across all metrics.
Direction consistency
Share of conditions whose mean moves in the direction the hypothesis expects.
| metric | direction consistency |
|---|---|
| cooperationRate | 0.8889 |
| averagePayoff | 1 |
Cross-model means
| model | cooperationRate | averagePayoff |
|---|---|---|
| deepseek-v4-flash | 0.4920 | 2.2300 |
Distribution tests
Pooled per-run distributions: W1 between baseline and condition, Cliff delta of condition over baseline, TVD to the target normalised by the target's TVD to uniform.
none
Integrity and cost
integrity
| counter | value |
|---|---|
| runs | 50 |
| succeeded | 50 |
| failed | 0 |
| incomplete | 0 |
| excluded | 0 |
| activated | 1000 |
| agentOk | 1000 |
| agentFailed | 0 |
| parseFailures | 0 |
| llmCalls | 0 |
| llmFailures | 0 |
| droppedEffects | 0 |
| rejectedActions | 0 |
cost
| counter | value |
|---|---|
| llm calls | 0 |
| prompt tokens | 0 |
| completion tokens | 0 |
| cached tokens | 0 |
| wall ms | 0 |