Testing modes
Every dial on an endpoint test: what to generate, how hard to try, and how much access Kerno gets.
These dials set the ratio of test quality to speed you want, and how much access Kerno gets while doing it.
kerno_endpoint_test takes one required dial and several optional ones. The defaults are deliberately thorough, so you only need to reach for these when you want something faster, narrower, or more cautious.
Kerno stores none of these preferences. If you settle on a set you like, your agent writes them into your own rules file so they are diffable, reviewable in a pull request, and travel with the repo. See Saving your preferences.
type — what the run does
Required. No default.
generate
Analyzes the endpoint, plans scenarios, pauses for your approval, then implements and runs them. Use this the first time.
validate
Runs the scenarios already on disk. No planning, no implementation. Use this after a code change to see whether behaviour moved.
update
Conservatively repairs existing scenarios after an intentional change. Keeps what still applies and edits only what the change requires.
validate and update both need scenarios to exist. If none do, the run ends immediately with needs_generate and a message telling you to generate first.
Choosing between update and generate. Use update when your endpoint changed on purpose and the tests should follow: it defaults every existing scenario to "keep", makes surgical edits, and never deletes the happy path. Use generate when you want to start the endpoint's coverage over.
generate always pauses for plan approval. update only pauses if you set interactive: true.
effort — how hard Kerno tries
Optional. Defaults to high.
low
1
no
The fastest, cheapest signal. The scenario is written and proven to pass once.
medium
2 consecutive, both must be clean
no
Proof the scenario is genuinely re-runnable, plus a second sample of randomized test data.
high
2 consecutive
yes
Everything medium gives, plus an implement-critique-repair loop that rejects weakened, tautological, or swallowed assertions and incomplete coverage.
The second run is not noise suppression. If a scenario passes once and fails the second time, that is a real finding: either its own setup and cleanup are not isolated, or the fresh random data hit an edge case the first sample missed. Both are worth knowing.
type: validate runs existing scenarios rather than implementing new ones, so it performs a single run regardless of the effort you pass.
box_testing_strategy — how much access Kerno gets
Optional. Defaults to white_box.
black_box
white_box
Datastore drivers available to scenario code
no
yes
Database schema in the planning prompt
suppressed
included
Assertions about stored state
not planned
planned where useful
How preconditions and results are handled
through your HTTP API only
through the API, falling back to the datastore where the API cannot do it
Black box means Kerno only talks to your application over HTTP. Every precondition is established and every result verified through your own endpoints.
White box means Kerno may additionally read and write your datastore directly, for the cases your API cannot express: seeding a record that has no creation endpoint, or checking that a stored password is not the plaintext that was sent. This is what the datastore blocks in kerno_save_config enable. See Environment Setup for wiring them up.
Two things that surprise people:
Your datastore credentials still reach the sandbox under
black_box. The dial works by controlling planning and access. Drivers are withheld, the schema is suppressed, and preconditions and results run over HTTP. The environment still receives your credentials.black_boxdoes not hide your source code from Kerno's planner. It constrains how scenarios interact with your running system, not what Kerno reads while writing them.
Configuring datastore access does not lock you into white box. Any individual run can opt out by passing box_testing_strategy: black_box.
White box needs a schema it can derive from your source. To read or write your datastore directly, Kerno first works out its shape from your source code, for example schema files, migrations, or ORM models. If it cannot derive one, any scenario that needs direct datastore access is marked blocked rather than run, and Kerno may ask you once to point it at a schema. Scenarios that only exercise HTTP are unaffected.
tags — what kind of coverage
Optional. Defaults to validation.
tags
Happy path
Functional coverage
Security scenarios
omitted or ["validation"]
yes
yes
no
["security"]
yes
no
yes
["validation","security"]
yes
yes
yes
Note the middle row: security on its own narrows the run rather than adding to it. If you want your normal functional coverage plus security scenarios, pass both tags.
The happy path is always planned, whatever you pass.
See Security testing for what the security tag actually produces.
test_generation_context — guidance for this endpoint
Optional free text describing what Kerno should know or do differently for this endpoint.
Its meaning depends on type:
generate— complete guidance for planning. Passing a different value than last time discards the scenarios on disk and re-plans from scratch, so resend your previous guidance in full when you are adding to it.update— change intent: what changed and how the tests should adapt. It does not wipe existing scenarios.validate— ignored.
Relationship to workspace config. Durable rules belong in your Kerno config under test-generation, where they are version-controlled and apply automatically — see Custom rules. The two are combined for each run, with this per-call argument taking precedence where they conflict.
Only the per-call argument triggers a re-plan. Editing test-generation rules in .kerno/config.yaml changes the guidance Kerno receives on the next generate, but existing scenarios are kept rather than re-planned. To force config changes through, either pass a changed test_generation_context or remove the endpoint's scenarios first.
interactive — step through the run
Optional, defaults to false. When true, Kerno pauses before implementing each scenario to ask whether to proceed, skip, or take feedback, and pauses again if automatic retries are exhausted. For type: update, it also restores the plan-approval gate.
These pauses are gates, not failures. They surface through kerno_feedback_pending, not through kerno_job.
scenario_ids — target specific scenarios
Optional. When provided, only the named scenarios are implemented or run; the rest are skipped. Omit it to target every scenario.
Note that this filters implementation and execution but not planning: a generate run still plans the full set. If none of the ids match, the call fails rather than quietly doing nothing.
Saving your preferences
Kerno is stateless about your habits. It stores no testing preferences, and it does not learn them silently over time.
Instead, every endpoint-test response echoes a resolved_intent object showing what was used and which values had to be defaulted:
If you have a standing preference, your coding agent writes it into your own rules file — CLAUDE.md, .cursor/rules, or AGENTS.md — as a pattern-matched table. That keeps it diffable, reviewable in a pull request, and travelling with the repository.
Ask your agent for kerno_guide with topic rules_template to get the table format. The rule is that the most specific endpoint pattern wins, and you should always keep one * fallback row describing what a bare call should resolve to for your codebase.
Last updated