e2e

E2E Tests - Godog Framework

This directory contains end-to-end (e2e) tests, written using the Godog framework.

Overview

What is Godog/BDD/Cucumber?

Godog is a Behavior-Driven Development (BDD) framework that allows you to write tests in a human-readable format called Gherkin. Tests are written as scenarios using Given-When-Then syntax, making them accessible to both technical and non-technical stakeholders.

Benefits:

Readable: Tests serve as living documentation
Maintainable: Reusable step definitions reduce code duplication
Collaborative: Product owners and developers share the same test specifications
Structured: Clear separation between test scenarios and implementation

Project Structure

test/e2e/
├── README.md                   # This file
├── features_test.go            # Test runner and suite initialization
├── features/                   # Gherkin feature files
│   ├── install.feature         # ClusterExtension installation scenarios
│   ├── update.feature          # ClusterExtension update scenarios
│   ├── recover.feature         # Recovery scenarios
│   ├── status.feature          # ClusterExtension status scenarios
│   └── metrics.feature         # Metrics endpoint scenarios
└── steps/                      # Step definitions and test utilities
    ├── steps.go                # Step definition implementations
    ├── hooks.go                # Test hooks and scenario context
    └── testdata/               # Test data (RBAC templates)
        ├── serviceaccount-template.yaml
        ├── olm-sa-helm-rbac-template.yaml
        ├── olm-sa-boxcutter-rbac-template.yaml
        ├── pvc-probe-sa-boxcutter-rbac-template.yaml
        ├── cluster-admin-rbac-template.yaml
        └── metrics-reader-rbac-template.yaml

Architecture

1. Test Runner (`features_test.go`)

The main test entry point that configures and runs the Godog test suite.

2. Feature Files (`features/*.feature`)

Gherkin files that describe test scenarios in natural language.

Structure:

Feature: [Feature Name]
  [Feature description]

  Background:
  [Common setup steps for all scenarios]

  Scenario: [Scenario Name]
    Given [precondition]
    When [action]
    Then [expected result]
    And [additional assertions]

Example:

Feature: Install ClusterExtension

  Background:
    Given OLM is available
    And "test" catalog serves bundles
    And Service account "olm-sa" with needed permissions is available in test namespace

  Scenario: Install latest available version from the default channel
    When ClusterExtension is applied
      """
      apiVersion: olm.operatorframework.io/v1
      kind: ClusterExtension
      metadata:
        name: ${NAME}
      spec:
        namespace: ${TEST_NAMESPACE}
        serviceAccount:
          name: olm-sa
        source:
          sourceType: Catalog
          catalog:
            packageName: test
            selector:
              matchLabels:
                "olm.operatorframework.io/metadata.name": test-catalog
        ...
      """
    Then ClusterExtension is rolled out
    And ClusterExtension is available

3. Step Definitions (`steps/steps.go`)

Go functions that implement the steps defined in feature files. Each step is registered with a regex pattern that matches the Gherkin text.

Registration:

func RegisterSteps(sc *godog.ScenarioContext) {
sc.Step(`^OLM is available$`, OLMisAvailable)
sc.Step(`^bundle "([^"]+)" is installed in version "([^"]+)"$`, BundleInstalled)
sc.Step(`^ClusterExtension is applied$`, ResourceIsApplied)
// ... more steps
}

Step Implementation Pattern:

func BundleInstalled(ctx context.Context, name, version string) error {
    sc := scenarioCtx(ctx)
    waitFor(ctx, func() bool {
        v, err := kubectl("get", "clusterextension", sc.clusterExtensionName, "-o", "jsonpath={.status.install.bundle}")
        if err != nil {
          return false
        }
        var bundle map[string]interface{}
        json.Unmarshal([]byte(v), &bundle)
        return bundle["name"] == name && bundle["version"] == version
    })
    return nil
}

4. Hooks and Context (`steps/hooks.go`)

Manages test lifecycle and scenario-specific context.

Hooks:

CheckFeatureTags: Skips scenarios based on feature gate tags (e.g., @WebhookProviderCertManager)
CreateScenarioContext: Creates unique namespace and names for each scenario
ScenarioCleanup: Cleans up resources after each scenario

Variable Substitution:

Replaces ${TEST_NAMESPACE}, ${NAME}, ${SCENARIO_ID}, ${PACKAGE:<name>}, and ${CATALOG:<name>} with scenario-specific values.

Writing Tests

1. Create a Feature File

Create a new .feature file in test/e2e/features/:

Feature: Your Feature Name
  Description of what this feature tests

  Background:
    Given OLM is available
    And "test" catalog serves bundles

  Scenario: Your scenario description
    When [some action]
    Then [expected outcome]

2. Implement Step Definitions

Add step implementations in steps/steps.go:

func RegisterSteps(sc *godog.ScenarioContext) {
    // ... existing steps
    sc.Step(`^your step pattern "([^"]+)"$`, YourStepFunction)
}

func YourStepFunction(ctx context.Context, param string) error {
    sc := scenarioCtx(ctx)
    // Implementation
    return nil
}

3. Use Existing Steps

Leverage existing steps for common operations:

Setup: Given OLM is available, And "test" catalog serves bundles
Resource Management: When ClusterExtension is applied, And resource is applied
Assertions: Then ClusterExtension is available, And bundle "..." is installed
Conditions: Then ClusterExtension reports Progressing as True with Reason Retrying:

4. Variable Substitution

Use these variables in YAML templates:

${NAME}: Scenario-specific ClusterExtension name (e.g., ce-123)
${COS_NAME}: Scenario-specific ClusterObjectSet name (e.g., cos-123; for applying ClusterObjectSets directly)
${TEST_NAMESPACE}: Scenario-specific namespace (e.g., ns-123)
${SCENARIO_ID}: Unique scenario identifier used for resource name isolation
${PACKAGE:<name>}: Parameterized package name (e.g., ${PACKAGE:test} expands to test-<scenario-id>)
${CATALOG:<name>}: Catalog resource name (e.g., ${CATALOG:test} expands to test-catalog-<scenario-id>)

5. Feature Tags

Tags can be used for different purposes in the test suite:

Feature Gate Tags

Use tags to conditionally run scenarios based on feature gates:

@WebhookProviderCertManager
Scenario: Install operator having webhooks

Scenarios are skipped if the feature gate is not enabled on the deployed controller.

Serial Execution Tag

By default, scenarios run concurrently (up to 100 parallel scenarios). However, some tests must run serially, typically because they:

Modify shared cluster resources (e.g., cluster-wide TLS configuration)
Have resource constraints that prevent parallel execution
Require exclusive access to a resource

To mark a test for serial execution, add the @Serial tag:

@Serial
Feature: TLS profile enforcement on metrics endpoints

  Scenario: Test TLS configuration
    Given the "catalogd" deployment is configured with custom TLS settings
    ...

The Makefile automatically separates scenarios when run without additional GODOG_ARGS:

Scenarios without @Serial run concurrently in the first test phase
Scenarios with @Serial run sequentially in a separate serial test phase

Running Tests

Run All Tests

make test-e2e                      # Standard features
make test-experimental-e2e         # Experimental features

Run Standard and Experimental in Parallel

Both suites can run simultaneously on separate KIND clusters (see the Makefile comment for Linux prerequisites):

make -j2 test-e2e test-experimental-e2e

Custom godog arguments can be modified by setting the following:

GODOG_ARGS=--godog.tags=@WebhookProviderCertManager make test-experimental-e2e

Note that when this is done the make target will no longer automatically split the test run into parallel and serial runs, and test execution time may increase. If you wish to add concurrency back into the arguments, it is recommended to also disable the @Serial tests:

GODOG_ARGS="--godog.tags=~@Serial --godog.concurrency=100" make test-experimental-e2e

Iterative Development

For iterating on individual scenarios without full suite setup/teardown each time, use the persistent cluster and single-scenario targets:

1. Set up a persistent cluster (once):

make e2e-setup                     # Standard features
make experimental-e2e-setup        # Experimental features

This builds images, creates a KIND cluster, deploys OLM, and waits for readiness. The cluster persists until explicitly torn down.

2. Run individual scenarios:

# Run all scenarios in a feature file
make e2e/install

# Run scenarios matching a name prefix (case-insensitive)
make e2e/install/Install                      # all "Install ..." scenarios
make "e2e/install/Install latest"             # prefix with spaces (use quotes)
make e2e/install/Boxcutter                    # single matching scenario

# Override timeout or kubeconfig
make e2e/install/Install E2E_TIMEOUT=30m
make e2e/install KUBECONFIG=~/.kube/config

The prefix matches scenario names from the start. If multiple scenarios match, all of them run. If no scenario matches, the command fails with a list of available scenario names.

When using experimental-e2e-setup, override KUBECONFIG to point at the experimental cluster:

make e2e/install/Install KUBECONFIG=.kubeconfig/operator-controller-experimental-e2e.kubeconfig

3. Tear down when done:

make e2e-teardown                  # Standard cluster
make experimental-e2e-teardown     # Experimental cluster

Run Specific Feature

go test test/e2e/features_test.go -- features/install.feature

Run Specific Scenario by Tag

go test test/e2e/features_test.go --godog.tags="@WebhookProviderCertManager"

Run with Debug Logging

go test -v test/e2e/features_test.go --log.debug

CLI Options

Godog options can be passed after --:

go test test/e2e/features_test.go \
  --godog.format=pretty \
  --godog.tags="@WebhookProviderCertManager"

Available formats: pretty, cucumber, progress, junit

Custom Flags:

--e2e.scenario=<prefix>: Run scenarios whose name starts with the given prefix (case-insensitive). Used by make e2e/<feature>/<prefix> internally.
--log.debug: Enable debug logging (development mode)
--k8s.cli=<path>: Specify path to Kubernetes CLI (default: kubectl)
- Useful for using oc or a specific kubectl binary

Example:

go test test/e2e/features_test.go --log.debug --k8s.cli=oc

Environment Variables

KUBECONFIG: Path to kubeconfig file (defaults to ~/.kube/config)
E2E_SUMMARY_OUTPUT: Path to write test summary (optional)
PROMETHEUS_URL: Prometheus endpoint for summary generation (set automatically by make; skips summary if unset)
CLUSTER_REGISTRY_HOST: In-cluster registry host for pulling catalog images

Design Patterns

1. Scenario Isolation

Each scenario runs in its own namespace with unique resource names, ensuring complete isolation:

Namespace: ns-{scenario-id}
ClusterExtension: ce-{scenario-id}

2. Test-Identifying Annotations

Every resource applied during a scenario is automatically annotated with the feature file name and scenario name:

e2e.olm.operatorframework.io/feature: derived from the feature file path (e.g., install, update, recover)
e2e.olm.operatorframework.io/scenario: the scenario name (e.g., Install latest available version)

These annotations are added to all resources created within a scenario.

These annotations make it possible to identify which test scenario produced a given resource when debugging failures on a cluster:

kubectl get clusterextension -o json | jq '.items[] | {name: .metadata.name, feature: .metadata.annotations["e2e.olm.operatorframework.io/feature"], scenario: .metadata.annotations["e2e.olm.operatorframework.io/scenario"]}'

3. Automatic Cleanup

The ScenarioCleanup hook ensures all resources are deleted after each scenario:

Deletes ClusterExtensions and (when BoxcutterRuntime gate is enabled) ClusterObjectSets
Deletes ClusterCatalogs
Deletes namespaces
Deletes added resources

4. Declarative Resource Management

Resources are managed declaratively using YAML templates embedded in feature files as docstrings:

When ClusterExtension is applied
"""
  apiVersion: olm.operatorframework.io/v1
  kind: ClusterExtension
  metadata:
    name: ${NAME}
  spec:
    ...
  """

5. Polling with Timeouts

All asynchronous operations use waitFor with consistent timeout (300s) and tick (1s):

waitFor(ctx, func() bool {
    // Check condition
    return conditionMet
})

6. Feature Gate Detection

Tests automatically detect enabled feature gates from the running controller and skip scenarios that require disabled features.

Common Step Patterns

A list of available, implemented steps can be obtained by running:

go test test/e2e/features_test.go -d

When working with Claude Code, run the /list-e2e-steps command to get a categorized reference of all step definitions including parameters, DocString expectations, polling behavior, and handler locations. This is useful when writing new feature files or debugging existing scenarios.

Best Practices

Keep scenarios focused: Each scenario should test one specific behavior
Use Background wisely: Common setup steps belong in Background
Reuse steps: Leverage existing step definitions before creating new ones
Meaningful names: Scenario names should clearly describe what is being tested
Avoid implementation details: Focus on behavior, not implementation

Navigation Menu

Uh oh!

README.md

E2E Tests - Godog Framework

Overview

What is Godog/BDD/Cucumber?

Project Structure

Architecture

1. Test Runner (`features_test.go`)

2. Feature Files (`features/*.feature`)

3. Step Definitions (`steps/steps.go`)

4. Hooks and Context (`steps/hooks.go`)

Writing Tests

1. Create a Feature File

2. Implement Step Definitions

3. Use Existing Steps

4. Variable Substitution

5. Feature Tags

Feature Gate Tags

Serial Execution Tag

Running Tests

Run All Tests

Run Standard and Experimental in Parallel

Iterative Development

Run Specific Feature

Run Specific Scenario by Tag

Run with Debug Logging

CLI Options

Environment Variables

Design Patterns

1. Scenario Isolation

2. Test-Identifying Annotations

3. Automatic Cleanup

4. Declarative Resource Management

5. Polling with Timeouts

6. Feature Gate Detection

Common Step Patterns

Best Practices

References

Name		Name	Last commit message	Last commit date
parent directory ..
features		features
steps		steps
README.md		README.md
features_test.go		features_test.go
support-bundle.yaml		support-bundle.yaml

Uh oh!

FilesExpand file tree

e2e

Directory actions

More options

Directory actions

More options

Latest commit

History

e2e

Folders and files

parent directory

README.md

E2E Tests - Godog Framework

Overview

What is Godog/BDD/Cucumber?

Project Structure

Architecture

1. Test Runner (features_test.go)

2. Feature Files (features/*.feature)

3. Step Definitions (steps/steps.go)

4. Hooks and Context (steps/hooks.go)

Writing Tests

1. Create a Feature File

2. Implement Step Definitions

3. Use Existing Steps

4. Variable Substitution

5. Feature Tags

Feature Gate Tags

Serial Execution Tag

Running Tests

Run All Tests

Run Standard and Experimental in Parallel

Iterative Development

Run Specific Feature

Run Specific Scenario by Tag

Run with Debug Logging

CLI Options

Environment Variables

Design Patterns

1. Scenario Isolation

2. Test-Identifying Annotations

3. Automatic Cleanup

4. Declarative Resource Management

5. Polling with Timeouts

6. Feature Gate Detection

Common Step Patterns

Best Practices

References

1. Test Runner (`features_test.go`)

2. Feature Files (`features/*.feature`)

3. Step Definitions (`steps/steps.go`)

4. Hooks and Context (`steps/hooks.go`)