Automated Testing

The test command runs automated snapshot-based tests for your templates. Unlike try (which runs templates interactively for manual development), test executes predefined test cases from test.cyan.yaml and compares the output against expected snapshots.

Quick Start with `test init`

Generate Test Cases

Run test init to automatically generate test cases by walking your template's Q&A tree:

cyanprint test init .

This explores all answer combinations (Select options, Confirm true/false, Checkbox subsets) and generates a test case for each, capped at 30 combinations by default.

Review Generated Files

test init creates two things:

test.cyan.yaml

test.cyan.yaml — test case definitions with answer states
fixtures/expected/ — snapshot directories with expected output for each test case

Run Tests

cyanprint test template .

All tests should pass since the snapshots were just generated from your template.

How `test init` Works

test init walks your template's question tree using depth-first exploration:

Question Type	Exploration Strategy
Select	One branch per option
Confirm	Two branches: `true` and `false`
Checkbox	Subsets: empty, each individual option, and all selected
Text	Single value using `--text-seed` (default: `"dummy"`)
Password	Single value using `--password-seed` (default: `"secret"`)
Date	Single value using `--date-seed` (default: today's date)

The --max-combinations flag (default: 30) caps the total number of test cases. Once the limit is reached, in-progress branches complete but no new branches are started.

test init is only available for templates. Processors, plugins, and resolvers require manually creating test.cyan.yaml.

Interactive Mode

For large templates, exhaustive exploration may produce too many combinations. Use interactive mode to selectively choose branches:

cyanprint test init . -i

This prompts you at each question branch, letting you pick which options to explore.

Custom Seeds

Override default seed values for deterministic answers:

cyanprint test init . --text-seed "my-project" --password-seed "s3cret" --date-seed "2025-06-15"

Understanding `test.cyan.yaml`

Each test case defines the answers to provide and the expected output:

test.cyan.yaml

tests:
  - name: typescript-project
    expected:                          // !mark
      type: snapshot                   // !callout "Snapshot comparison mode"
      value:
        path: ./fixtures/expected/typescript-project
    answer_state:                      // !callout "Maps question IDs to answers"
      project_name:
        type: String
        value: "my-project"
      use_typescript:
        type: Bool
        value: true
      features:
        type: StringArray
        value:
          - linting
          - testing
    deterministic_state:               // !mark
      seed: "12345"
    validate:                          // !callout "Commands to verify output"
      - "bun install --dry-run"

Answer State Types

The answer_state maps question IDs to their answers:

Type	Used For	Value Format
`String`	Text, Password, Date inputs	Single string
`Bool`	Confirm prompts	`true` or `false`
`StringArray`	Checkbox / multi-select	List of strings

Deterministic State

The deterministic_state field pins non-deterministic values (like random seeds) to ensure reproducible output across test runs.

Validation Commands

The validate field lists shell commands that run in the generated output directory after the template executes. If any command exits with a non-zero code, the test fails.

validate:
  - "bun install --dry-run"    # Verify package.json is valid
  - "bun run build"            # Verify project builds successfully

validate:
  - "pip install --dry-run -r requirements.txt"  # Verify requirements.txt is valid
  - "python -m py_compile src/main.py"           # Verify Python syntax

validate:
  - "dotnet restore"           # Verify csproj is valid
  - "dotnet build --no-restore" # Verify project builds successfully

validate:
  - "nix flake check"          # Verify Nix flake structure
  - "nix eval .#packages"      # Verify package definitions

Project Structure

A template project with automated tests looks like this:

cyan.yaml

test.cyan.yaml

index.ts

Dockerfile

package.json

README.md

Running Tests

Run All Tests

cyanprint test template .

Run a Specific Test

cyanprint test template . --test typescript-project

Parallel Execution

Speed up test runs by executing multiple test cases in parallel:

cyanprint test template . --parallel 4

Updating Snapshots

When you intentionally change your template's output, update the expected snapshots:

cyanprint test template . --update-snapshots

This replaces the contents of each fixtures/expected/ directory with the actual output. Always review the changes before committing.

Always review snapshot changes with git diff before committing. Incorrect snapshots can mask bugs.

Snapshot Comparison Rules

File Type	Comparison Method
`.json` files	Deep comparison (field order ignored)
Other text files	Exact string match (trailing whitespace trimmed)
Binary files	Skipped (reported but not compared)

Extra files in actual output or missing files from expected output cause test failure.

CI/CD Integration

Generate JUnit XML Reports

Generate JUnit XML reports for CI systems:

cyanprint test template . --junit test-results.xml --disable-daemon-autostart

GitHub Actions Example

.github/workflows/test.yml

name: Template Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup CyanPrint
        run: |                            // !mark
          curl -sSL https://get.cyanprint.dev | bash

      - name: Run Tests
        run: |                            // !callout "Generate JUnit report for CI"
          cyanprint test template . \
            --junit test-results.xml \
            --disable-daemon-autostart

      - name: Publish Test Results        // !mark
        uses: dorny/test-reporter@v1
        if: always()
        with:
          name: Template Tests
          path: test-results.xml
          reporter: java-junit

GitLab CI Example

.gitlab-ci.yml

template-tests:
  stage: test
  image: atomicloud/cyanprint:latest
  script:
    - cyanprint test template . --junit test-results.xml --disable-daemon-autostart  // !mark
  artifacts:
    when: always
    reports:
      junit: test-results.xml

Use --disable-daemon-autostart in CI environments where you manage the coordinator separately, or ensure Docker is available for automatic daemon startup.

The JUnit report follows the standard format supported by most CI platforms (GitHub Actions, GitLab CI, Jenkins).

Writing Custom Test Cases

Manual Test Case Creation

For processors, plugins, and resolvers, manually create test.cyan.yaml:

test.cyan.yaml

tests:
  - name: default-config
    expected:
      type: snapshot
      value:
        path: ./fixtures/expected/default-config
    answer_state:
      project_name:
        type: String
        value: "test-project"
      use_typescript:
        type: Bool
        value: true
    deterministic_state:
      seed: "test-seed-123"
    validate:
      - "bun install --dry-run"

test.cyan.yaml

tests:
  - name: default-config
    expected:
      type: snapshot
      value:
        path: ./fixtures/expected/default-config
    answer_state:
      project_name:
        type: String
        value: "test-project"
      use_python:
        type: Bool
        value: true
    deterministic_state:
      seed: "test-seed-123"
    validate:
      - "pip install --dry-run -r requirements.txt"

test.cyan.yaml

tests:
  - name: default-config
    expected:
      type: snapshot
      value:
        path: ./fixtures/expected/default-config
    answer_state:
      project_name:
        type: String
        value: "TestProject"
      use_csharp:
        type: Bool
        value: true
    deterministic_state:
      seed: "test-seed-123"
    validate:
      - "dotnet restore"

Generate Initial Snapshots

After creating your test cases, generate the initial snapshots:

cyanprint test template . --update-snapshots

Verify and Commit

Review the generated snapshots and commit them to version control:

git add fixtures/expected/ test.cyan.yaml
git commit -m "test: add automated test cases"

Next Steps

CLI Commands Reference — Full command options and flags
Test Locally with Try — Interactive development testing
Push to Registry — Publish your template

Automated Testing

On this page