Catcher e2e tests tool for beginners

What is Catcher?

It is an end-to-end test tool, specially designed to test systems containing many components. Initially developed as end-to-end microservices test tool it perfectly fits needs of data pipeline testing. Here we describe basic Catcher concepts.

Catcher is a modular system. It has core modules which are shipped with Catcher itself and additional modules which can be installed separately on demand.

Steps

In Catcher terminology module and step is the same.
Every Catcher test can have next types of steps:

built-in core Catcher steps. They include basic steps like checks, sh, loops, http, wait and others;
extended Catcher steps. They have more complex implementation and sometimes require dependencies or drivers to be installed in the system. For example: kafka, elastic, docker, mongo, s3 and others.
your custom steps, written in Python, Java, sh or any other language you prefer.

Simple example of how test can look like:

steps:
  - http:  # load answers via post body
      post:
        url: 'http://127.0.0.1:8080/load'
        body_from_file: "data/answers.json"
  - elastic:  # check in the logs that answers were loaded
      search:
        url: 'http://127.0.0.1:9200'
        index: test
        query:
          match: {payload : "answers loaded successfully"}

Variables

Steps with hard-coded values are not that useful. Variables is one of Catcher’s key features. Jinja2 templates are fully supported. Try to use as much variables as you can to avoid code duplication and make things flexible.

Test-local variables

Every step can use existing variables by defining them in the variables section. They can be either static (as token) or computed (as user_email).

variables:
  user_email: '{{ random("email") }}'
  token: 'my_secret_token'
steps:
  - http:
      post:
        headers: {Content-Type: 'application/json', Authorization: '{{ token }}'}
        url: 'http://127.0.0.1:8080?user={{ user_email }}'
        body: {'foo': bar}

Every step can also register it’s output (or part of it) as a new variable:

variables: 
  user_email: '{{ random("email") }}'
  token: 'my_secret_token'
steps:
  - mongo:  # search MongoDb for user 
      request:
        conf:
            database: test
            username: test
            password: test
            host: localhost
            port: 27017
        collection: 'users'
        find_one: {'user_id': '{{ user_email }}'}
      register: {found_user: '{{ OUTPUT }}'}
  - check:  # check token was saved
      equals: {the: '{{ found_user["token"] }}', is: "{{ token }}"}

Let’s take closer look at this line: register: {found_user: '{{ OUTPUT }}'}. Here OUTPUT system variable stores mongo step’s output. We register it as a new variable found_user.

OUTPUT is the system variable which is being used to store every step’s output.

System variables

Catcher can also access your system environment variables. F.e. run export MY_ENV_HOST=localhost && catcher my_test.yml and your environment variable will be picked up. You can disable this behavior by running Catcher with -s flag: catcher -s false my_test.yml.

steps:
  - http:  # load answers via post body
      post:
        url: 'http://{{ MY_ENV_HOST }}/load'

To override variables use -e param while running tests:

catcher -e var=value -e other_var=other_value tests/Code language: JavaScript (javascript)

Override priority is:

command line variables override everything below
newly registered variables override everything below
test-local variables override everything below
inventory variables override everything below
environment variables override nothing

Inventories

As you may notice hard-coding services configuration either in the test or in test’s variables is not that flexible, as they are different for every environment. Catcher uses inventory files for environment-specific configuration. Inventories are stored in the inventory folder. They also support templates.

You can create inventory file: local.yml and set variables there:

airflow_web: 'http://127.0.0.1:8080'
s3_config:
    url: http://127.0.0.1:9001
    key_id: minio
    secret_key: minio123

And create develop_aws.yml with variables:

airflow_web: 'http://my_airflow:8080'
s3_config:
    key_id: my_real_aws_key_id
    secret_key: my_real_aws_secrey

When you run Catcher you can specify inventory via -i param:

catcher -i inventory/local.yml test

Reports

Sometimes, when you write your test, you need to see what is going on in your system after each step. At the moment Catcher supports only json output. Run it with -p json option:

catcher -p json my_test.ymlCode language: CSS (css)

And check reports directory for a json step-by-step report. It will contain all in and out variables for every step.

Installation

Docker

For a quick start you can use Catcher Docker image. It includes catcher-core, all extended modules, drivers and dependencies. It comes ready to use – all you need to do is to mount your local directories with your tests, inventories, etc.
The full command to run it is:

docker run -v $(pwd)/inventory:/opt/catcher/inventory
           -v $(pwd)/resources:/opt/catcher/resources
           -v $(pwd)/test:/opt/catcher/tests
           -v $(pwd)/steps:/opt/catcher/steps
           -v $(pwd)/reports:/opt/catcher/reports
            catcher -i inventory/my_inventory.yml tests

Catcher in Docker is usually used in CI while developers prefer Catcher local installation for writing tests.

Local

For a local installation ensure you have Python 3.5+ version. You can use miniconda if you don’t have it.

Run pip install catcher to install catcher-core. It will install the Catcher and basic steps. If you need any extended steps you need to install them separately. F.e. for kafka and postgres you have to run pip install catcher_modules[kafka,postgres]

Some extended steps also require drivers and system requirements to be installed first. F.e. libclntsh.dylib library for the Oracle database.