Skip to main content

Bitbucket Pipelines

Levo's security/contract tests can be embedded in quality gates via Bitbucket Pipelines.

This integration uses a PowerShell script (levo-cli-runner.ps1) that handles Python environment setup, Levo CLI installation, and test execution on Windows self-hosted runners.

Table of Contents​

Prerequisites​

Before setting up Levo in your Bitbucket Pipelines, ensure you have:

  • A runnable Levo Test Plan - Create a test plan in the Levo console. Refer to Test Plans for more information
  • Windows Self-Hosted Runner - Bitbucket Pipelines must be configured to use a Windows self-hosted runner
  • Python 3.12+ - The PowerShell script will detect and use Python 3.12 or higher if available
  • Levo CLI Authorization Key - Refer to Generating CLI Authorization Keys for instructions
  • Organization ID - Refer to Accessing Organization IDs for instructions
  • Google Cloud Access Token - Required for accessing Levo CLI from Google Artifact Registry (GAR)

Setup​

Step 1: Download the PowerShell Script​

The levo-cli-runner.ps1 script is available on GitHub.

You can either:

  • Download it manually and place it on your Windows self-hosted runner (e.g., C:\Scripts\levo-cli-runner.ps1)
  • Download it automatically in your pipeline using the raw GitHub URL (recommended)

The recommended approach is to download it in your pipeline to ensure you always use the latest version.

Step 2: Configure Repository Variables​

Navigate to your Bitbucket repository:

  1. Go to Repository settings > Repository variables
  2. Add the required variables (see Repository Variables section below)
  3. Mark sensitive variables (API keys, tokens) as Secured

Step 3: Create Pipeline Configuration​

Create or update your

Loading...
file in the root of your repository. See Pipeline Configuration for examples and a complete reference configuration.

Repository Variables​

Configure these variables in Repository settings > Repository variables.

Required Variables​

VariableDescriptionSecured
LEVOAI_AUTH_KEYLevo Auth key for authentication. Refer to Generating CLI Authorization KeysYes
LEVOAI_ORG_IDLevo organization ID. Refer to Accessing Organization IDsNo
PYPI_USERNAMESet to _json_key_base64 for Google Artifact RegistryNo
PYPI_PASSWORDGoogle Cloud access token for accessing GAR. Please contact support@levo.aiYes

Optional Variables​

VariableDescriptionDefault
LEVOAI_CLI_VERSIONSpecific CLI version to install (e.g., 1.2.3). If not set, latest version is installedlatest
LEVOAI_BASE_URLCustom Levo API URL. Default: https://api.levo.ai.https://api.levo.ai

Configuration via Parameters or Environment Variables​

The script accepts configuration either as command-line parameters or environment variables. Parameters take precedence over environment variables.

ParameterEnvironment VariableDescriptionDefault
-AppNameAPP_NAMEApplication name for test runs. If not set, auto-detected from repositoryAuto-detected
-EnvironmentENVIRONMENTTarget environment (e.g., staging, production, development)staging
-TestMethodsTEST_METHODSHTTP methods to test (comma-separated: GET,POST,PUT,DELETE). Mutually exclusive with -ExcludeMethodsGET,POST
-ExcludeMethodsEXCLUDE_METHODSHTTP methods to exclude from testing (comma-separated: GET,POST,PUT,DELETE). Mutually exclusive with -TestMethodsNone
-EndpointPatternENDPOINT_PATTERNRegex pattern to match endpoints to be tested (e.g., ^/api/v1/.*)None (all endpoints)
-ExcludeEndpointPatternEXCLUDE_ENDPOINT_PATTERNRegex pattern to exclude endpoints from testing (e.g., ^/health)None
-TestUsersTEST_USERSComma-separated list of test user names to use for testing. Users must be configured in the app's IAM configurationAuto-detected from app config
-TargetUrlTARGET_URLTarget URL for the test run. If not provided, will attempt to use target URL from tracesAuto-detected from app config
-CategoriesCATEGORIESComma-separated list of test categories to run (e.g., CORS, Broken User Authentication, Fuzzing, SQL Injection). See the Levo console for the complete list.All available categories
-FailScopeFAIL_SCOPEWhen to fail the build: new (new vulnerabilities), any (any vulnerabilities), none (never fail)new
-FailSeverityFAIL_SEVERITYMinimum severity to fail: critical, high, medium, low, nonehigh
-FailThresholdFAIL_THRESHOLDOptional: fail if vulnerability count exceeds this threshold (integer)None (no threshold)

Note: You can pass these as parameters directly to the script (e.g., levo-cli-runner.ps1 test -AppName myapp -Environment production) or set them as environment variables in your pipeline. -TestMethods and -ExcludeMethods are mutually exclusive—use only one.

Pipeline Configuration​

Basic Pipeline Structure​

Configure your

Loading...
file with caching for virtual environments and pip. The pipeline should run on Windows self-hosted runners and execute the PowerShell script with the test command. Include artifacts for levo-test-output.log and pip-install.log for debugging.

Complete Pipeline Example​

For a complete reference implementation, see the example

Loading...
file. This example includes:

  • Pull request validation pipelines
  • Branch-based testing (main/master)
  • Custom pipelines for audits and environment-specific testing
  • Proper caching and artifact configuration

Using the Script​

Download the levo-cli-runner.ps1 script from GitHub in your pipeline using PowerShell. This ensures you always use the latest version:

script:
  - powershell -ExecutionPolicy Bypass -Command "Invoke-WebRequest -Uri 'https://raw.githubusercontent.com/levoai/downloads/main/bitbucket/levo-cli-runner.ps1' -OutFile 'levo-cli-runner.ps1'"
  - powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test

Command Reference​

The levo-cli-runner.ps1 PowerShell script supports the following commands:

install​

Installs Levo CLI into the virtual environment. Run with powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' install.

Use Cases:

  • Initial setup
  • Debugging installation issues
  • Updating to a specific version (set LEVOAI_CLI_VERSION)

test​

Runs security tests. This is the primary command for CI/CD pipelines. Execute with powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test.

Usage:

Basic Examples:

# Using environment variables (default)
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test

# Using parameters
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -AppName myapp -Environment production -FailSeverity critical

# Mix of environment variables and parameters
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -Environment production -TestMethods "GET,POST,PUT"

Endpoint Filtering:

# Filter endpoints by pattern
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -EndpointPattern "^/api/v1/.*"

Test Configuration:

# Specify test users
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -TestUsers "admin,user1"

# Specify target URL
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -TargetUrl "https://api.example.com"

# Run specific test categories
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -Categories "CORS,Broken User Authentication"

Failure Configuration:

# Set failure threshold
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -FailThreshold 10

Complex Example:

# Complex example with multiple options
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -AppName myapp -Environment staging -TestMethods "GET,POST" -EndpointPattern "^/api/.*" -Categories "CORS,Rate Limit" -FailScope new -FailSeverity high -FailThreshold 5

Behavior:

  • Auto-installs Levo CLI if not present
  • Runs tests based on parameters or environment variables (parameters take precedence)
  • Fails the build if vulnerabilities are found (based on FAIL_SCOPE, FAIL_SEVERITY, and FAIL_THRESHOLD)
  • If TARGET_URL is not provided, attempts to use target URL from app configuration or traces
  • If TEST_USERS is not provided, uses the default user from app's IAM configuration
  • If CATEGORIES is not provided, runs all available test categories

audit​

Runs comprehensive security audit without failing the build. Execute with powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' audit.

Behavior:

  • Runs all HTTP methods (GET, POST, PUT, DELETE, PATCH)
  • Sets FAIL_SCOPE=none and FAIL_SEVERITY=none
  • Always returns exit code 0 (never fails build)
  • Useful for comprehensive security assessments

version​

Displays the installed Levo CLI version. Run with powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' version.

help​

Displays help information and usage examples. Run with powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' help.

Pipeline Examples​

Example 1: Pull Request Validation​

Run security tests on pull requests with strict criteria (only fail on new critical vulnerabilities). Configure the pipeline to set FAIL_SCOPE=new and FAIL_SEVERITY=critical for pull request triggers. Include all required environment variables in the script execution.

See the example

Loading...
for a complete pull request validation configuration.

Example 2: Branch-Based Testing​

Run different test configurations for different branches. For the main branch, configure staging tests with ENVIRONMENT=staging, FAIL_SCOPE=new, and FAIL_SEVERITY=high. Add a separate production step with manual trigger, using ENVIRONMENT=production, FAIL_SCOPE=any, and FAIL_SEVERITY=high.

See the example

Loading...
for branch-based testing configurations.

Example 3: Custom Pipeline - Security Audit​

Run a comprehensive audit that never fails the build. Use the audit command which automatically sets FAIL_SCOPE=none and FAIL_SEVERITY=none, ensuring the build always succeeds while providing comprehensive security assessment results.

See the example

Loading...
for the full-security-audit custom pipeline.

Example 4: Custom Pipeline - Install Only​

Useful for debugging installation issues. Create a custom pipeline that runs the install command followed by version to verify the installation without running tests.

See the example

Loading...
for the install-levo-cli custom pipeline.

Example 5: Environment-Specific Testing​

Test against different environments with a custom pipeline variable. Define a TARGET_ENV variable with allowed values (staging, production, development) and use it to set the ENVIRONMENT variable dynamically in your pipeline.

See the example

Loading...
for the test-environment custom pipeline.

Example 6: Complete Pipeline with Multiple Stages​

Create a complete pipeline with build, test, and deploy stages. Add a security testing step between build and deployment that runs the Levo security tests, ensuring vulnerabilities are caught before deployment.

For a complete reference implementation with all examples, see the example

Loading...
file.

Example 7: Advanced Filtering and Configuration​

Use advanced options to fine-tune your test runs:

Endpoint Filtering:

# Test only API v1 endpoints
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -EndpointPattern "^/api/v1/.*"

# Exclude health check and metrics endpoints
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -ExcludeEndpointPattern "^/health|^/metrics|^/status"

HTTP Method Filtering:

# Test only GET and POST methods
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -TestMethods "GET,POST"

# Exclude DELETE and PATCH methods
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -ExcludeMethods "DELETE,PATCH"

Test Categories:

# Run specific categories (e.g., CORS and Broken User Authentication)
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -Categories "CORS,Broken User Authentication"

# Run multiple categories
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -Categories "CORS,Fuzzing,Rate Limit,SQL Injection"

Test Users:

# Use specific test users (must be configured in app's IAM config)
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -TestUsers "admin,user1"

Target URL Override:

# Override target URL from app configuration
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -TargetUrl "https://staging-api.example.com"

Failure Threshold:

# Fail if more than 10 vulnerabilities are found
powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -FailThreshold 10

Combined Example:

powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test `
  -AppName myapp `
  -Environment staging `
  -TestMethods "GET,POST" `
  -EndpointPattern "^/api/v1/.*" `
  -ExcludeEndpointPattern "^/health" `
  -Categories "CORS,Broken User Authentication,Rate Limit" `
  -TestUsers "admin" `
  -TargetUrl "https://staging-api.example.com" `
  -FailScope new `
  -FailSeverity high `
  -FailThreshold 5

Troubleshooting​

Python Not Found​

Error: Python 3.12 or higher is required but not found

Solutions:

  1. Install Python 3.12+ from python.org
  2. Ensure Python is in your PATH
  3. Use the Python launcher: py -3.12 should work
  4. Verify installation: python --version or py --version

Virtual Environment Creation Fails​

Error: Failed to create virtual environment

Solutions:

  1. Check Python installation: python -m venv --help
  2. Ensure sufficient disk space
  3. Check write permissions in the repository directory
  4. Try deleting .levo-venv and recreating

Levo CLI Installation Fails​

Error: pip install failed

Solutions:

  1. Check pip-install.log for detailed error messages
  2. Verify PYPI_USERNAME is set to _json_key_base64
  3. Verify PYPI_PASSWORD contains a valid Google Cloud access token
  4. Please contact support@levo.ai for assistance with access tokens
  5. Check network connectivity to us-python.pkg.dev

Authentication Errors​

Error: LEVOAI_AUTH_KEY is required but not set or LEVOAI_ORG_ID is required but not set

Solutions:

  1. Verify repository variables are set in Bitbucket with the correct names: LEVOAI_AUTH_KEY and LEVOAI_ORG_ID
  2. Ensure variables are not marked as "Secured" if they shouldn't be (except for sensitive values)
  3. Check variable names match exactly (case-sensitive)
  4. Verify variables are available in the pipeline context

Test Execution Fails​

Error: Test runs but exits with non-zero code

Solutions:

  1. Check levo-test-output.log for detailed error messages
  2. Verify your Test Plan LRN is correct
  3. Ensure the target API is accessible from the runner
  4. Check FAIL_SCOPE and FAIL_SEVERITY settings
  5. Review test results in the Levo console

C Extension Compilation Issues​

Error: Warnings about C extension compilation

Solutions:

  1. The script automatically reinstalls packages with C extensions
  2. Ensure Visual C++ Build Tools are installed (for Windows)
  3. Check pip-install.log for specific compilation errors
  4. The script should handle most cases automatically

Script Not Found​

Error: The system cannot find the file specified

Solutions:

  1. Verify the script path in your
    Loading...
    file
  2. Ensure the download command uses the correct URL for levo-cli-runner.ps1
  3. If using a local script, use absolute paths: C:\Scripts\levo-cli-runner.ps1
  4. Or use forward slashes: C:/Scripts/levo-cli-runner.ps1
  5. Ensure the script exists on the runner
  6. Check file permissions

Execution Policy Errors​

Error: Execution of scripts is disabled on this system

Solutions:

  1. The pipeline uses -ExecutionPolicy Bypass flag
  2. If issues persist, check PowerShell execution policy on the runner
  3. Run: Get-ExecutionPolicy to check current policy
  4. Contact your system administrator if needed

Best Practices​

1. Use Caching​

Always enable caching for virtual environments and pip. Configure levo-venv: .levo-venv and pip: ~/.cache/pip in your pipeline definitions. This significantly speeds up pipeline execution.

2. Store Artifacts​

Save test output logs for debugging. Include levo-test-output.log and pip-install.log in your artifacts section to review test results and troubleshoot installation issues.

3. Environment-Specific Configuration​

Use different FAIL_SCOPE and FAIL_SEVERITY values for different environments. You can set these as environment variables or pass them as parameters:

  • Pull Requests: FAIL_SCOPE=new, FAIL_SEVERITY=critical (strict)
  • Staging: FAIL_SCOPE=new, FAIL_SEVERITY=high (moderate)
  • Production: FAIL_SCOPE=any, FAIL_SEVERITY=high (strict)

Example using parameters:

powershell -ExecutionPolicy Bypass -File 'levo-cli-runner.ps1' test -FailScope new -FailSeverity critical

4. Secure Sensitive Variables​

Always mark sensitive variables as "Secured" in Bitbucket:

  • LEVOAI_AUTH_KEY
  • PYPI_PASSWORD
  • Any tokens or secrets

5. Use Custom Pipelines for Audits​

For comprehensive security audits that shouldn't fail builds, use the audit command in a custom pipeline. This allows you to run comprehensive security assessments without blocking your deployment process.

6. Pin CLI Version (Optional)​

For production pipelines, consider pinning the CLI version by setting the LEVOAI_CLI_VERSION repository variable to a specific version (e.g., 1.2.3). This ensures consistent behavior across pipeline runs.

7. Monitor Test Results​

  • Review levo-test-output.log artifacts after each run
  • Check the Levo console for detailed test results
  • Set up notifications for failed tests

8. Separate Install and Test Steps​

For debugging, you can separate installation and testing into different pipeline steps. This allows you to verify the installation independently before running tests.

9. Use Deployment Environments​

Leverage Bitbucket's deployment environments for better organization. Use deployment: staging or deployment: production to categorize your pipeline steps and track deployments.

10. Handle Script Updates​

The levo-cli-runner.ps1 script is hosted on GitHub. By downloading it in your pipeline, you automatically get the latest version. If you need to pin to a specific version, you can reference a specific commit or tag in the raw GitHub URL.

Additional Resources​

Need Help?​

For further assistance, please reach out to support@levo.ai.