CI/CD Integration

Integrate curl-runner with CI/CD pipelines using CI-friendly exit codes and configurable failure thresholds.

Overview

curl-runner provides CI-friendly exit codes that integrate seamlessly with popular CI/CD platforms. By default, the tool follows standard conventions where exit code 0 indicates success and exit code 1 indicates failure.

--strict-exit

Strict Mode

Exit 1 if any validation fails

--fail-on

Count Threshold

Allow N failures before failing

--fail-on-percentage

Percentage Threshold

Allow N% failures before failing

Exit Code Options

Control how curl-runner determines its exit code based on test results.

terminal

# CI/CD exit code options

# Strict mode - exit 1 if ANY validation fails
curl-runner tests/ --strict-exit

# Run all tests, then fail if any failed
curl-runner tests/ --continue-on-error --strict-exit

# Allow up to 2 failures before failing
curl-runner tests/ --fail-on 2

# Allow up to 10% of tests to fail
curl-runner tests/ --fail-on-percentage 10

# Combine options for comprehensive CI setup
curl-runner tests/ \
  --continue-on-error \
  --strict-exit \
  --output-format json \
  --output results.json

# CI/CD exit code options

# Strict mode - exit 1 if ANY validation fails
curl-runner tests/ --strict-exit

# Run all tests, then fail if any failed
curl-runner tests/ --continue-on-error --strict-exit

# Allow up to 2 failures before failing
curl-runner tests/ --fail-on 2

# Allow up to 10% of tests to fail
curl-runner tests/ --fail-on-percentage 10

# Combine options for comprehensive CI setup
curl-runner tests/ \
  --continue-on-error \
  --strict-exit \
  --output-format json \
  --output results.json

Option Reference

Option	Description
`--strict-exit`	Exit with code 1 if any validation fails, regardless of --continue-on-error
`--fail-on <count>`	Exit with code 1 only if failures exceed the specified count
`--fail-on-percentage <pct>`	Exit with code 1 only if failure percentage exceeds the threshold (0-100)

Exit Code Behavior

Understanding how different configurations affect the exit code.

Configuration	No Failures	Failures Exist
Default (no options)	Exit 0	Exit 1
`--continue-on-error`	Exit 0	Exit 0
`--strict-exit`	Exit 0	Exit 1
`--continue-on-error --strict-exit`	Exit 0	Exit 1
`--fail-on N`	Exit 0	Exit 0 if ≤N, Exit 1 if >N
`--fail-on-percentage P`	Exit 0	Exit 0 if ≤P%, Exit 1 if >P%

Tip: Use --continue-on-error --strict-exit together to run all tests before determining the final exit code. This ensures you get complete test results while still failing the pipeline when needed.

Environment Variables

All CI options can be configured via environment variables for easy integration with CI/CD platforms.

terminal

# Environment variables for CI/CD

# Enable strict exit mode
export CURL_RUNNER_STRICT_EXIT=true

# Set failure threshold (count)
export CURL_RUNNER_FAIL_ON=5

# Set failure threshold (percentage)
export CURL_RUNNER_FAIL_ON_PERCENTAGE=10

# Combine with other settings
export CURL_RUNNER_STRICT_EXIT=true
export CURL_RUNNER_CONTINUE_ON_ERROR=true
export CURL_RUNNER_OUTPUT_FORMAT=json
curl-runner tests/

# Environment variables for CI/CD

# Enable strict exit mode
export CURL_RUNNER_STRICT_EXIT=true

# Set failure threshold (count)
export CURL_RUNNER_FAIL_ON=5

# Set failure threshold (percentage)
export CURL_RUNNER_FAIL_ON_PERCENTAGE=10

# Combine with other settings
export CURL_RUNNER_STRICT_EXIT=true
export CURL_RUNNER_CONTINUE_ON_ERROR=true
export CURL_RUNNER_OUTPUT_FORMAT=json
curl-runner tests/

Environment Variable Reference

Variable	Type	Description
`CURL_RUNNER_STRICT_EXIT`	boolean	Enable strict exit mode (true/false)
`CURL_RUNNER_FAIL_ON`	number	Maximum allowed failures
`CURL_RUNNER_FAIL_ON_PERCENTAGE`	number	Maximum allowed failure percentage (0-100)

Platform Examples

GitHub Actions

Complete workflow for running API tests with artifact upload.

.github/workflows/api-tests.yml

name: API Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Bun
        uses: oven-sh/setup-bun@v2

      - name: Install curl-runner
        run: bun install -g curl-runner

      - name: Run API Tests
        env:
          CURL_RUNNER_STRICT_EXIT: true
          CURL_RUNNER_CONTINUE_ON_ERROR: true
        run: curl-runner tests/ --output-format json --output results.json

      - name: Upload Results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: results.json

name: API Tests
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Setup Bun
        uses: oven-sh/setup-bun@v2

      - name: Install curl-runner
        run: bun install -g curl-runner

      - name: Run API Tests
        env:
          CURL_RUNNER_STRICT_EXIT: true
          CURL_RUNNER_CONTINUE_ON_ERROR: true
        run: curl-runner tests/ --output-format json --output results.json

      - name: Upload Results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: test-results
          path: results.json

GitLab CI

GitLab CI configuration with test artifacts.

.gitlab-ci.yml

stages:
  - test

api-tests:
  stage: test
  image: oven/bun:latest
  variables:
    CURL_RUNNER_STRICT_EXIT: "true"
    CURL_RUNNER_CONTINUE_ON_ERROR: "true"
    CURL_RUNNER_OUTPUT_FORMAT: json
  before_script:
    - bun install -g curl-runner
  script:
    - curl-runner tests/ --output test-results.json
  artifacts:
    when: always
    paths:
      - test-results.json

stages:
  - test

api-tests:
  stage: test
  image: oven/bun:latest
  variables:
    CURL_RUNNER_STRICT_EXIT: "true"
    CURL_RUNNER_CONTINUE_ON_ERROR: "true"
    CURL_RUNNER_OUTPUT_FORMAT: json
  before_script:
    - bun install -g curl-runner
  script:
    - curl-runner tests/ --output test-results.json
  artifacts:
    when: always
    paths:
      - test-results.json

Jenkins Pipeline

Declarative Jenkins pipeline with artifact archiving.

Jenkinsfile

pipeline {
    agent any

    environment {
        CURL_RUNNER_STRICT_EXIT = 'true'
        CURL_RUNNER_CONTINUE_ON_ERROR = 'true'
        CURL_RUNNER_OUTPUT_FORMAT = 'json'
    }

    stages {
        stage('Install') {
            steps {
                sh 'npm install -g curl-runner'
            }
        }

        stage('Test') {
            steps {
                sh 'curl-runner tests/ --output test-results.json'
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: 'test-results.json', allowEmptyArchive: true
        }
    }
}

pipeline {
    agent any

    environment {
        CURL_RUNNER_STRICT_EXIT = 'true'
        CURL_RUNNER_CONTINUE_ON_ERROR = 'true'
        CURL_RUNNER_OUTPUT_FORMAT = 'json'
    }

    stages {
        stage('Install') {
            steps {
                sh 'npm install -g curl-runner'
            }
        }

        stage('Test') {
            steps {
                sh 'curl-runner tests/ --output test-results.json'
            }
        }
    }

    post {
        always {
            archiveArtifacts artifacts: 'test-results.json', allowEmptyArchive: true
        }
    }
}

CircleCI

CircleCI configuration with test artifact storage.

.circleci/config.yml

version: 2.1

executors:
  node:
    docker:
      - image: cimg/node:20.0

jobs:
  test:
    executor: node
    environment:
      CURL_RUNNER_STRICT_EXIT: "true"
      CURL_RUNNER_CONTINUE_ON_ERROR: "true"
    steps:
      - checkout
      - run:
          name: Install curl-runner
          command: npm install -g curl-runner
      - run:
          name: Run API Tests
          command: curl-runner tests/ --output-format json --output test-results.json
      - store_artifacts:
          path: test-results.json
          destination: test-results

workflows:
  test:
    jobs:
      - test

version: 2.1

executors:
  node:
    docker:
      - image: cimg/node:20.0

jobs:
  test:
    executor: node
    environment:
      CURL_RUNNER_STRICT_EXIT: "true"
      CURL_RUNNER_CONTINUE_ON_ERROR: "true"
    steps:
      - checkout
      - run:
          name: Install curl-runner
          command: npm install -g curl-runner
      - run:
          name: Run API Tests
          command: curl-runner tests/ --output-format json --output test-results.json
      - store_artifacts:
          path: test-results.json
          destination: test-results

workflows:
  test:
    jobs:
      - test

Best Practices

Run All Tests First

Use --continue-on-error with --strict-exit to get complete test results before failing.

Use JSON Output

JSON format is easier to parse and integrate with CI reporting tools.

Archive Test Results

Always save results as artifacts, even when tests fail, for debugging.

Use Environment Variables

Configure via env vars for flexibility across environments.

Troubleshooting

Exit Code Always 0

If the exit code is always 0 even with failures, check that you're using CI options correctly:

# Wrong: exits 0 with failures when using -c alone
curl-runner tests/ -c

# Right: exits 1 if any failures occur
curl-runner tests/ -c --strict-exit

Tests Pass Locally but Fail in CI

Common causes include network issues, missing environment variables, or timing differences. Try increasing timeout and retry settings:

curl-runner tests/ --timeout 30000 --retries 3 --retry-delay 2000