How to Run your Baseline Tests in CI

Learn how to replay your committed baseline tests against your application on every pull request, and read the result as a check.

Introduction

The baseline tests Kerno writes are files in your repository. Once they are committed, anything that can run them can run them, including your CI.

This guide wires up the Kerno GitHub Action. On every pull request it replays the baseline tests you have committed against your running application and reports the result as a check, so behaviour that moved is caught where your team already reviews changes rather than on one developer's machine.

It needs no Kerno account, no API key and no agent. The action pulls one public image and runs the tests already in your repository. Nothing calls a language model, which is also why it is fast and free to run. To also follow each run in the Kerno portal, add an API key (Step 6).

Prerequisites

Before you begin, you will need:

• Baseline tests committed under <app>/.kerno/scenarios. See How to Create Baseline Tests for your Entry Points.

• A way to start your application in CI, and a URL it answers on.

• A Linux runner with Docker. ubuntu-latest works as-is.

Tests that read a database, mint tokens from a shared secret, or call a downstream service need values you would never commit. Step 4 covers passing those in from secrets.

Step 1. Adding the workflow

Create .github/workflows/kerno.yaml:

name: kerno

on: pull_request

permissions:
  contents: read
  checks: write          # the reporter step creates a check run
  pull-requests: write   # the action comments its results on the pull request

jobs:
  kerno:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5

      # Your application. Start it however you already do — Kerno only connects to it.
      - run: docker compose up -d --wait
      - run: |
          for _ in $(seq 1 30); do
            curl -fsS -o /dev/null http://localhost:8080/health && exit 0
            sleep 2
          done
          echo "app did not become ready" >&2; exit 1

      - uses: kernoio/kerno-check@v1
        with:
          sut-url: http://localhost:8080

      - uses: mikepenz/action-junit-report@v6
        if: always()
        with:
          report_paths: kerno-junit.xml

Three things about that file are worth knowing:

• You start your application; Kerno connects to it. The action never starts, builds or tears down your app. Whatever you already use, whether Compose, a service container, or ./gradlew bootRun &, keep using it.

• Poll for readiness, not just for health. compose up --wait waits for container health, which is not the same as your application serving requests. The loop above is worth copying.

• localhost is fine. The tests execute inside a container, where localhost would mean the container itself, so the action rewrites a localhost or 127.0.0.1 URL to reach your runner.

@v1 follows every 1.x release. If you need a reference that never moves, pin a full version such as @v1.2.2, or a commit SHA. Versions before v1.1.0 lack the forward-env and apps inputs this guide uses.

Step 2. Reading the check

The action writes a summary to the job page:

⚠️ Kerno scenarios — my-service
30/47 passed — 17 skipped (never executed) against http://host.docker.internal:8080

A run with nothing failed and nothing skipped is marked ✅; one with skips is marked ⚠️ so they are visible at a glance, and a failure is marked ❌.

A failure is listed with the assertion that did not match and a diff of expected against actual, so you can see what moved without opening the logs. The JUnit report lands at kerno-junit.xml, which the reporter step turns into a check run with each test's result.

On a pull request, the action also posts its totals as a comment, and updates the same comment on every run. The comment and the check run need the pull-requests: write and checks: write permissions in the workflow above. Without them, they don't appear, and the action only logs a warning.

Each test comes back as one of the four verdicts described in Baseline tests. JUnit has three states, so they map like this:

Kerno verdict	In the report	Fails the check
Passed	passed	no
Failed	failed	yes
Blocked	skipped	no
Not implemented	skipped	no

Skipped tests never fail the check, and they are always counted separately. A blocked test is a known state, meaning a dependency it needs is not configured, and a not-implemented one is reported honestly rather than counted as a pass. That is why the summary reads 30/47 passed — 17 skipped rather than 47 passed: a suite that asserts nothing should not look like coverage.

Step 3. Choosing what runs

By default the action discovers every <app>/.kerno/scenarios tree in your repository and replays all of them, which is usually what a monorepo wants.

To replay one application:

      - uses: kernoio/kerno-check@v1
        with:
          sut-url: http://localhost:8080
          app-dir: services/orders

To replay a subset, filter on the path relative to the scenarios directory:

      - uses: kernoio/kerno-check@v1
        with:
          sut-url: http://localhost:8080
          scenarios: endpoints/GET/**

* stays within a path segment and ** crosses them.

Step 4. Tests that need configuration

Tests that read or seed a database, mint tokens from a shared secret, or call a downstream service need values that must not live in your repository. Name them in forward-env and supply them from secrets:

      - uses: kernoio/kerno-check@v1
        with:
          sut-url: http://localhost:8080
          forward-env: |
            DATABASE_URL
            JWT_SECRET
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          JWT_SECRET: ${{ secrets.JWT_SECRET }}

Two properties worth relying on:

• Only the names you list are forwarded. The rest of the runner's environment does not reach the container.

• A name with no value stops the run, before the container starts, and says which variable is missing. So does a name that isn't a valid variable name, or one of the runner's own variables (PATH, HOME, NODE_OPTIONS, LD_PRELOAD and LD_LIBRARY_PATH). That matters more than it sounds: a test that needed DATABASE_URL and did not get it fails at its database step while every HTTP assertion around it still passes, which reads as a partial success rather than a configuration mistake.

Step 5. A monorepo with services on different ports

sut-url gives every application the same address. When your services listen on different ports, map each one instead:

      - uses: kernoio/kerno-check@v1
        with:
          apps: |
            services/orders=http://localhost:8080
            services/billing=http://localhost:8081
          forward-env: |
            DATABASE_URL
        env:
          DATABASE_URL: ${{ secrets.DATABASE_URL }}

Each directory is the one containing .kerno, relative to the repository root, and each application is replayed against its own URL. apps replaces both sut-url and app-dir, and setting apps with either one stops the run with a configuration error.

One report is written per application, so report-path is a directory in this mode, and the reporter step takes a glob:

      - uses: kernoio/kerno-check@v1
        with:
          apps: |
            services/orders=http://localhost:8080
            services/billing=http://localhost:8081
          report-path: kerno-reports

      - uses: mikepenz/action-junit-report@v6
        if: always()
        with:
          report_paths: kerno-reports/*.xml

The counts are summed across every application, so one failing test in one service fails the check.

Step 6. Seeing runs in the Kerno portal

Add your Kerno API key and organization, and the action opens a run in the Kerno portal for each entry point it replays. The pull request comment then lists each run with a link to its report:

      - uses: kernoio/kerno-check@v1
        with:
          sut-url: http://localhost:8080
          api-key: ${{ secrets.KERNO_API_KEY }}
          organization-id: ${{ vars.KERNO_ORGANIZATION_ID }}

Create the key under Settings → API Key in the portal (see Integrations and API keys), store it as a repository secret, and set both inputs together. Setting only one of them stops the run with a configuration error. A run link that can't be opened never fails the check.

The summary also says how many entry points your team marked critical in the portal, and where that count came from. With credentials, the action reads it from the portal. Without them, it reads the criticality.json committed under each application's .kerno/ directory.

Step 7. Tracking your default branch

To let the portal follow how your default branch's tests grow over time, add a second workflow that runs mode: sync after every merge:

name: kerno-sync

on:
  push:
    branches: [main]
    paths: ['**/.kerno/**']
  workflow_dispatch:

permissions:
  contents: read

jobs:
  sync:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: kernoio/kerno-check@v1
        with:
          mode: sync
          api-key: ${{ secrets.KERNO_API_KEY }}
          organization-id: ${{ vars.KERNO_ORGANIZATION_ID }}

sync runs no tests and needs no running application. It sends the tests committed on your default branch to Kerno as that commit's snapshot, and lists in the job summary which entry points and tests the merge added or removed. It records only your default branch. Run it once by hand after you add it, to send the first snapshot.

What this does not do

• It does not create or update baseline tests. That happens on a developer's machine, where the proposed tests can be reviewed before they are committed. CI replays what is already in the repository and nothing else.

• It does not start your application. You start it and pass sut-url. Kerno connects to a system under test; it never manages one.

• It only forwards the environment variables you name. Nothing else from the runner reaches your tests, and a name with no value stops the run rather than being quietly dropped.

Reference

Inputs

Input	Required	Default
mode	no	replay	replay runs your tests. sync sends your default branch's tests to the portal instead (Step 7).
sut-url	unless apps		Base URL of your running application. A localhost URL is rewritten so the container can reach it.
apps	no		One <dir>=<url> per line, replaying each application against its own URL. Replaces sut-url.
forward-env	no		Environment variable names to pass to your tests, one per line, valued from the step's own env:.
app-dir	no	(repository root)	Replay one application's tests. Unset discovers every <app>/.kerno/scenarios tree.
scenarios	no	(all)	Glob filter on the path relative to the scenarios directory.
image	no	(pinned digest)	The runner image. Pinned by digest so a given version of the action always runs the same code.
report-path	no	kerno-junit.xml	Where the JUnit report lands. A directory when apps is used.
fail-on-failure	no	true	Set false to report without gating.
api-key	no		Your Kerno API key. Set it with organization-id to open portal runs, and for mode: sync.
organization-id	no		Your Kerno organization. Set it with api-key.

Outputs: junit-path, total, passed, failed, skipped, and portal-run-urls (one portal run link per line when api-key is set).

Exit codes

Code
0	Nothing failed. Tests may have been skipped, so check the counts.
1	At least one test failed.
2	Configuration error: no tests found, an unparseable or empty report, or an input problem such as a missing URL, a malformed apps line, a missing scenarios directory, conflicting inputs, a bad forward-env name, or api-key without organization-id. Never reported as a test failure.
3	The runner could not start. A broken container must not look like a failing test.

The action gives the runner the NET_ADMIN capability so Kerno can intercept outbound HTTPS from your application. That is how a test can exercise a path that calls a third-party API without that API being reachable from CI.

Conclusion

Your committed baseline tests now run on every pull request, and their exit code is your gate. Tests are still authored and reviewed locally, covered in How to Review your Code Changes, and CI is where the whole team finds out when behaviour moves.