> For the complete documentation index, see [llms.txt](https://kerno.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kerno.gitbook.io/docs/core-concepts/scenarios-and-baselines.md). # Baseline Tests Understand how Kerno generates test scenarios for your endpoints, what a baseline is, and how both stay in sync as your code evolves. A baseline test reflects how your app currently behaves. Kerno captures your endpoint's current behavior as a test, then assesses every future code change against it. If a change alters how your endpoint behaves, the baseline catches it. ### Test scenarios Scenarios are the test cases Kerno generates for your endpoints. Each one describes what state to set up, what request to send, what response to expect, and how to verify side effects. They live under `.kerno/scenarios/endpoints/` in your repository and are meant to be committed alongside your code. Every scenario is written twice, as a plain-English `.scenario.md` for you and an executable `.scenario.ts` for the sandbox. See [Anatomy of a Kerno test](/docs/references/anatomy-of-a-kerno-test.md) for how to read them. {% hint style="info" %} Kerno always writes scenarios in TypeScript, whatever language your application is written in. They reach your app over HTTP, so the language of your service does not matter. {% endhint %} A test scenario runs in four phases. * **Arrange.** Set up any state the test needs, such as seeding a user, obtaining a token, or inserting a row. * **Act.** Make the request to the endpoint under test. * **Assert.** Check the response and any observable side effects. * **Clean up.** Tear down test data and close connections. ### What baseline test record A baseline records what a correct response looks like. It captures the status code, every field in the response body, and any state the endpoint stored. Kerno builds the baseline from your endpoint's own data contract, the response types defined in your code. So it checks that every field holds the type and value your handler produces, deciding each expected value from its source: * A value your handler always produces the same way, like a fixed status code, a constant string, or a field copied straight from the request, is recorded exactly. Any change to it shows up as a diff. * A value generated fresh on every run, like an id, a token, or a timestamp, is recorded by its format. A new id on each run never counts as a diff. The baseline also fixes the exact set of fields a response returns. A field the endpoint did not return before shows up as a diff, which is how an accidentally exposed field, like an internal flag or a leaked password hash, gets caught. ### What Kerno tests Kerno generates two kinds of tests. **Validation tests** check that your endpoint behaves correctly, and **security tests** check that it resists the vulnerabilities that apply to it. You choose which of them to generate with the `tags` argument when you baseline an endpoint. {% hint style="info" %} Kerno asks which you want, validation, security, or both, unless you have already set that preference in your rules. {% endhint %} #### **Validation tests** For each endpoint, Kerno generates validation scenarios across several categories of behavior, including:
CategoryDescription
Functional API WorkflowsCovers endpoint behaviour, multi step request sequences, coordinated service interactions, and integration patterns across services.
Contract & Schema ValidationChecks request and response structures, data types, required fields, serialization rules, and version compatibility for the API.
Error Handling & ResilienceValidates status codes, error body formats, retry behaviour, backoff procedures, timeout handling, and controlled fallback behaviour.
Authorization & AuthenticationReviews token validation, role based access rules, permission scopes, session handling, and all credential related flows.
Boundary & Edge CasesExamines payload size limits, pagination behaviour, null or empty values, malformed inputs, and constraint based validation.
Data Integrity & Persistence

Confirms data consistency, transaction behaviour, idempotent operations,

state handling, and enforcement of database rules.

### **Security tests** Security tests baseline your endpoints' security posture. Kerno captures how the endpoint stands up to a specific class of attack today, so if a later change opens that vulnerability, the test starts failing and the regression is caught. Kerno assesses which **OWASP API and Web Top 10 categories** plausibly apply to the endpoint, given its inputs, its auth model, and what the surrounding code does, then writes tests only for the categories that fit. Common ones include **broken object-level and function-level authorization**, **injection**, **excessive data exposure**, **mass assignment**, and **server-side request forgery**. ### Testing MCP servers Kerno tests each tool your MCP server exposes as its own endpoint. It calls the tool with arguments and checks what comes back, the returned content, the structured output against its schema, and whether the call succeeded or returned an error. For each tool, Kerno generates: * **A happy-path call** with valid arguments, checked against the tool's output contract. * **A case per required argument**, omitted or given the wrong type, expecting the tool to return an error. * **Type and boundary cases** for constrained fields. * **Side-effect checks** for tools that change state, which assert the change happened and then clean up. The baseline records the result the tool returns and masks non-deterministic fields, so a fresh id or timestamp never counts as a diff. The `validation` and `security` tags apply as they do for HTTP, so a tool can carry both functional and OWASP security coverage. To point Kerno at a tool, ask your agent to test it by name. `kerno_list_mcp_tools` lists what is testable, and tools also appear in your normal endpoint list. {% hint style="info" %} Kerno connects to your MCP server over Streamable HTTP. Point your `sut_url` at the server's MCP path, for example `http://localhost:9300/mcp`. Discovery and planning work from your source with the server stopped. Implementing and running scenarios need it up, the same rule that applies to HTTP endpoints. {% endhint %} ### Testing background consumers Kerno also tests the background work your application runs on its own. A consumer is started by a queued job or a broker message instead of an HTTP request, so it has no caller identity and returns no response. Its observable outcome is a store write or a message it publishes in turn, and that outcome is what the baseline records. Address a consumer by its trigger verb and its registered name. | Trigger | `endpoint_method` | `endpoint_path` | | ---------------------------------------------------------------- | ----------------- | --------------------------------------------------------------------------------- | | A Celery task, in a Python application | `TASK` | the dotted name a worker registers it under, such as `orders.tasks.process_order` | | A Graphile Worker job, in a TypeScript or JavaScript application | `TASK` | the identifier the worker registers, such as `send_welcome_email` | | A handler a broker message triggers | `MESSAGE` | the handler's registered name, such as `payment-service.paymentRequestConsumer` | Celery tasks and Graphile jobs share the `TASK` verb. Kerno tells them apart by the language of the file it found the task in, and dispatches each the way its own worker expects, through the broker for Celery and through `add_job` for Graphile. The path is the registered name, so it carries no leading slash and looks nothing like a URL. Kerno discovers consumers from your source. For Celery it reads one endpoint per decorated function, recording the queue each is dispatched on and the parameter list a dispatched message has to satisfy. For Graphile Worker it reads all three places a real application registers tasks, which are the crontab file, a task directory where the file name is the identifier, and a task-list object where the key is the identifier. Consumers appear in your normal endpoint list alongside HTTP routes and MCP tools. {% hint style="info" %} Graphile Worker support lands in the next agent release. Celery and broker-message consumers work today. {% endhint %} A consumer scenario enqueues work that satisfies the handler's declared input, then asserts the side effect the handler produced. Because that outcome is a side effect rather than a response, verifying it usually needs the dependency access described in [Test environment](/docs/core-concepts/environment-setup.md). ### Effort levels When Kerno creates baseline tests, the effort level sets how hard it tries to prove each one is solid, trading speed for rigor. It defaults to high, so you only lower it when you want a faster, lighter run. * **Low.** Each test is written and run once. The fastest, cheapest signal. * **Medium.** Each test runs twice in a row and both runs must pass. This proves the test is genuinely repeatable and exercises a second sample of randomized data. * **High.** Everything medium does, plus a critique-and-repair pass that reviews each finished test and sends it back until it holds up. The second run is meaningful in itself. If a test passes once and fails the next, that is a real finding, either the test is not isolated or fresh random data hit an edge case worth knowing about. ### How Kerno ensures test quality Tests run in Kerno's own sandbox against your live application, and every verdict comes from what the checks actually did there. Because the run is independent of your coding agent, the result reflects your application's real behavior. Two mechanisms guard the tests themselves: * **Tests are proven repeatable.** At `medium` and `high` effort each one runs twice in a row, so a test that only passed because of leftover state gets caught. * **Tests are critiqued and repaired.** At `high` effort, each finished test goes through a review that checks it against what it was meant to test and sends it back for repair until it holds up. The review rejects checks weakened or rigged to pass no matter what, errors swallowed silently, tests that verify less than they were meant to, matching loose enough to pass on the wrong value, setup that claims state it never created, hardcoded identifiers where each run needs a fresh one, and teardown that claims to delete something without confirming it did. ### Reading test results Each scenario reports one of four verdicts. * **Passed.** The scenario ran and the endpoint behaved as expected. * **Failed.** The endpoint did something the scenario did not expect. * **Blocked.** The scenario could not run, usually because a dependency it needs is not configured. Blocked is neither a pass nor a fail; nothing was tested. * **Not implemented.** Kerno could not produce a working scenario for this case. This is reported honestly rather than counted as a pass. A passing scenario may also carry a **potential bug**, where Kerno found the endpoint doing something unexpected, confirmed it against your source, and documented the real behaviour. See [Security testing](/docs/references/security-testing.md) for how to review and dismiss these. #### Potential bugs A baseline test can carry a **potential bug tag**. Kerno noticed the endpoint doing something unexpected, confirmed it against your source code, and wrote the test to document that real behavior. The flag names what deviated, its root cause with a specific code location, and what a future change would signify. A potential bug flags behavior worth your attention on a test that otherwise passes. If you review it and decide the behavior is known or intended, tell your agent to ignore it: > "The `role` field Kerno flagged on `GET /users/:id` is intentional, we keep returning it for backwards compatibility. Have Kerno ignore that potential bug." An ignore decision is recorded in Kerno's memory for that endpoint, alongside the code revision you made it against. There is currently no way to reverse it through Kerno, so read the flag carefully before you dismiss it. See [Memory and learning](/docs/core-concepts/memory-and-learning.md).