> For the complete documentation index, see [llms.txt](https://kerno.gitbook.io/docs/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://kerno.gitbook.io/docs/guides/run-tests-in-ci.md). # How to Run your Baseline Tests in CI Learn how to replay your committed baseline tests against your application on every pull request, and read the result as a check. ### Introduction The baseline tests Kerno writes are files in your repository. Once they are committed, anything that can run them can run them, including your CI. This guide wires up the Kerno GitHub Action. On every pull request it replays the baseline tests you have committed against your running application and reports the result as a check, so behaviour that moved is caught where your team already reviews changes rather than on one developer's machine. It needs **no Kerno account, no API key and no agent**. The action pulls one public image and runs the tests already in your repository. Nothing calls a language model, which is also why it is fast and free to run. Add your Kerno API key to also follow each run in the Kerno portal ([Step 6](#step-6-seeing-runs-in-the-kerno-portal)) and track your default branch ([Step 7](#step-7-tracking-your-default-branch)). ### Prerequisites Before you begin, you will need: * Baseline tests committed under `/.kerno/scenarios`. See [How to Create Baseline Tests for your Entry Points](/docs/guides/capture-a-baseline.md). * A way to start your application in CI, and a URL it answers on. * A Linux runner with Docker. `ubuntu-latest` works as-is. {% hint style="info" %} Tests that read a database, mint tokens from a shared secret, or call a downstream service need values you would never commit. [Step 4](#step-4-tests-that-need-configuration) covers passing those in from secrets. {% endhint %} ### Step 1. Adding the workflow Create `.github/workflows/kerno.yaml`: ```yaml name: kerno on: pull_request: push: branches: [main] # your default branch paths: ['**/.kerno/**'] workflow_dispatch: # run once by hand, to send the first snapshot permissions: contents: read checks: write # the reporter step creates a check run pull-requests: write # the action comments its results on the pull request jobs: kerno: runs-on: ubuntu-latest steps: - uses: actions/checkout@v5 # Your application. Start it however you already do. Kerno only connects to it. - if: github.event_name == 'pull_request' run: | docker compose up -d --wait for _ in $(seq 1 30); do curl -fsS -o /dev/null http://localhost:8080/health && exit 0 sleep 2 done echo "app did not become ready" >&2; exit 1 - uses: kernoio/kerno-check@v1 with: mode: ${{ github.event_name == 'pull_request' && 'replay' || 'sync' }} sut-url: http://localhost:8080 api-key: ${{ secrets.KERNO_API_KEY }} - uses: mikepenz/action-junit-report@v6 if: always() && github.event_name == 'pull_request' with: report_paths: kerno-junit.xml ``` Run it once by hand after you add it (**Actions → kerno → Run workflow**) to send your default branch's tests to Kerno for the first time. A few things about that file are worth knowing: * **One file does both jobs.** On a pull request it starts your application and replays your tests. After a merge to `main` that changes `.kerno`, it runs `mode: sync`, which sends your default branch's tests to Kerno and needs no running application ([Step 7](#step-7-tracking-your-default-branch)). * **The API key is optional.** Create it under **Settings → API Key** in the portal and store it as the `KERNO_API_KEY` repository secret. Without it, the pull request check still runs and nothing is sent to Kerno. * **You start your application; Kerno connects to it.** The action never starts, builds or tears down your app. Whatever you already use, whether Compose, a service container, or `./gradlew bootRun &`, keep using it. * **Poll for readiness, not just for health.** `compose up --wait` waits for container health, which is not the same as your application serving requests. The loop above is worth copying. * **`localhost` is fine.** The tests execute inside a container, where `localhost` would mean the container itself, so the action rewrites a `localhost` or `127.0.0.1` URL to reach your runner. `@v1` follows every 1.x release. If you need a reference that never moves, pin a commit SHA. Versions before `v1.1.0` lack the `forward-env` and `apps` inputs this guide uses. ### Step 2. Reading the check The action writes a summary to the job page: ``` ⚠️ Kerno scenarios — my-service 30/47 passed — 17 skipped (never executed) against http://host.docker.internal:8080 ``` A run with nothing failed and nothing skipped is marked ✅; one with skips is marked ⚠️ so they are visible at a glance, and a failure is marked ❌. A failure is listed with the assertion that did not match and a diff of expected against actual, so you can see what moved without opening the logs. The JUnit report lands at `kerno-junit.xml`, which the reporter step turns into a check run with each test's result. On a pull request, the action also posts its totals as a comment, and updates the same comment on every run. The comment and the check run need the `pull-requests: write` and `checks: write` permissions in the workflow above. Without them, they don't appear, and the action only logs a warning. Each test comes back as one of the four verdicts described in [Baseline tests](/docs/core-concepts/scenarios-and-baselines.md#reading-test-results). JUnit has three states, so they map like this: | Kerno verdict | In the report | Fails the check | | --------------- | ------------- | --------------- | | Passed | passed | no | | Failed | failed | **yes** | | Blocked | skipped | no | | Not implemented | skipped | no | **Skipped tests never fail the check, and they are always counted separately.** A blocked test is a known state, meaning a dependency it needs is not configured, and a not-implemented one is reported honestly rather than counted as a pass. That is why the summary reads `30/47 passed — 17 skipped` rather than `47 passed`: a suite that asserts nothing should not look like coverage. ### Step 3. Choosing what runs The snippets in Steps 3 to 5 show the inputs they change. Keep `mode` and `api-key` from Step 1 alongside them. By default the action discovers every `/.kerno/scenarios` tree in your repository and replays all of them against `sut-url`, which is usually what a monorepo wants. When it finds several applications, `report-path` becomes a directory with one report per application, so set the reporter's `report_paths` to `kerno-junit.xml/*.xml`. To replay one application: ```yaml - uses: kernoio/kerno-check@v1 with: sut-url: http://localhost:8080 app-dir: services/orders ``` To replay a subset, filter on the path relative to the scenarios directory: ```yaml - uses: kernoio/kerno-check@v1 with: sut-url: http://localhost:8080 scenarios: endpoints/GET/** ``` `*` stays within a path segment and `**` crosses them. ### Step 4. Tests that need configuration Tests that read or seed a database, mint tokens from a shared secret, or call a downstream service need values that must not live in your repository. Name them in `forward-env` and supply them from secrets: ```yaml - uses: kernoio/kerno-check@v1 with: sut-url: http://localhost:8080 forward-env: | DATABASE_URL JWT_SECRET env: DATABASE_URL: ${{ secrets.DATABASE_URL }} JWT_SECRET: ${{ secrets.JWT_SECRET }} ``` Two properties worth relying on: * **Only the names you list are forwarded.** The rest of the runner's environment does not reach the container. * **A name with no value stops the run**, before the container starts, and says which variable is missing. So does a name that isn't a valid variable name, or one of the runner's own variables (`PATH`, `HOME`, `NODE_OPTIONS`, `LD_PRELOAD` and `LD_LIBRARY_PATH`). That matters more than it sounds: a test that needed `DATABASE_URL` and did not get it fails at its database step while every HTTP assertion around it still passes, which reads as a partial success rather than a configuration mistake. ### Step 5. A monorepo with services on different ports `sut-url` gives every application the same address. When your services listen on different ports, map each one instead: ```yaml - uses: kernoio/kerno-check@v1 with: apps: | services/orders=http://localhost:8080 services/billing=http://localhost:8081 forward-env: | DATABASE_URL env: DATABASE_URL: ${{ secrets.DATABASE_URL }} ``` Each directory is the one containing `.kerno`, relative to the repository root, and each application is replayed against its own URL. Use `.` for an application at the repository root. It can sit in the same list as the others. `apps` replaces both `sut-url` and `app-dir`, and setting `apps` with either one stops the run with a configuration error. One report is written per application, so `report-path` is a **directory** in this mode, and the reporter step takes a glob: ```yaml - uses: kernoio/kerno-check@v1 with: apps: | services/orders=http://localhost:8080 services/billing=http://localhost:8081 report-path: kerno-reports - uses: mikepenz/action-junit-report@v6 if: always() with: report_paths: kerno-reports/*.xml ``` The counts are summed across every application, so one failing test in one service fails the check. ### Step 6. Seeing runs in the Kerno portal With `api-key` set, the action opens a run in the [Kerno portal](/docs/portal/runs.md) for each entry point and user flow it replays. The pull request comment then lists each one with its passed and failed counts and a link that opens its run. `api-key` is the only credential the action needs. Each key belongs to one organization, and Kerno finds it from the key. A key Kerno refuses, Kerno being unreachable, or a run that can't be opened logs a warning and never fails the check. The key is yours. If you leave the organization or delete the key, Kerno stops receiving data from the repository until a teammate adds theirs. The summary also says how many entry points your team marked critical in the portal, and where that count came from. With an API key, the action reads it from the portal. Without one, it reads the `criticality.json` committed under each application's `.kerno/` directory. ### Step 7. Tracking your default branch The portal follows how your default branch's tests grow over time. Kerno reads them in one of two ways. **With the Kerno GitHub App connected**, Kerno records your default branch on every push and backfills its history itself (see [Integrations and API keys](/docs/portal/integrations.md#integrations)). Your workflow then only replays. Keep `pull_request` as its only trigger and set `mode: replay`. The portal's CI card hands out this file once GitHub is connected. **Without the GitHub App**, the `mode: sync` run from Step 1 does it. After every merge to `main` that changes `.kerno`, it runs no tests and needs no running application. It sends the tests committed for each entry point and user flow to Kerno, with the commit's time, as that commit's snapshot. The job summary lists the entry points, user flows and tests the merge added, removed or changed. * It records only your default branch. A run on any other branch logs a notice and sends nothing. * Without `api-key`, it sends nothing and the job stays green. A refused key or a failed request logs a warning and never fails the job. * It warns about the test files it leaves out. These are files outside `.kerno/scenarios/endpoints/` and every user flow folder, files whose entry point it can't read, and repeated test ids. * A commit that is already recorded, for example by the GitHub App, is kept once, and the summary says so. ### What this does not do * **It does not create or update baseline tests.** That happens on a developer's machine, where the proposed tests can be reviewed before they are committed. CI replays what is already in the repository and nothing else. * **It does not start your application.** You start it and pass `sut-url`. Kerno connects to a system under test; it never manages one. * **It only forwards the environment variables you name.** Nothing else from the runner reaches your tests, and a name with no value stops the run rather than being quietly dropped. ### Reference **Inputs** | Input | Required | Default | | | ----------------- | ------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | `mode` | no | `replay` | `replay` runs your tests. `sync` sends your default branch's tests to Kerno ([Step 7](#step-7-tracking-your-default-branch)). | | `sut-url` | unless `apps` | | Base URL of your running application. A `localhost` URL is rewritten so the container can reach it. | | `apps` | no | | One `=` per line, replaying each application against its own URL. `.` is the repository root. Replaces `sut-url`. | | `forward-env` | no | | Environment variable names to pass to your tests, one per line, valued from the step's own `env:`. | | `app-dir` | no | *(repository root)* | Replay one application's tests. Unset discovers every `/.kerno/scenarios` tree. | | `scenarios` | no | *(all)* | Glob filter on the path relative to the scenarios directory. | | `image` | no | *(pinned digest)* | The runner image. Pinned by digest so a given version of the action always runs the same code. | | `report-path` | no | `kerno-junit.xml` | Where the JUnit report lands. A directory of one report per application when `apps` is used or several applications are found. | | `fail-on-failure` | no | `true` | Set `false` to report without gating. | | `api-key` | no | | Your Kerno API key, from **Settings → API Key**. Opens portal runs, and sends the snapshot in `mode: sync`. | | `organization-id` | no | *(the key's organization)* | Kerno reads the organization from the key, so you can leave it unset. Needs `api-key`. | **Outputs**: `junit-path`, `total`, `passed`, `failed`, `skipped`, and `portal-run-urls` (one portal run link per entry point and user flow, one per line, when `api-key` is set). **Exit codes** | Code | | | ---- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `0` | Nothing failed. Tests may have been skipped, so check the counts. | | `1` | At least one test failed. | | `2` | Configuration error: no tests found, an unparseable or empty report, or an input problem such as a missing URL, a malformed `apps` line, a missing scenarios directory, conflicting inputs, a bad `forward-env` name, a directory listed twice in `apps`, an unknown `mode`, or `organization-id` without `api-key`. Never reported as a test failure. | | `3` | The runner could not start. A broken container must not look like a failing test. | {% hint style="info" %} The action gives the runner the `NET_ADMIN` capability so Kerno can intercept outbound HTTPS from your application. That is how a test can exercise a path that calls a third-party API without that API being reachable from CI. {% endhint %} ### Conclusion Your committed baseline tests now run on every pull request, and their exit code is your gate. Tests are still authored and reviewed locally, covered in [How to Review your Code Changes](/docs/guides/validate-code-changes.md), and CI is where the whole team finds out when behaviour moves.