Catcher e2e tests tool for beginners
What is Catcher? It is an end-to-end test tool, specially designed to test systems containing many components. Initially developed as end-to-end microservices test tool it perfectly fits needs of data pipeline testing. Here we describe basic Catcher concepts. Catcher is a modular system. It has core modules which are shipped with Catcher itself and additional …
Rollback for microservices with Ansible and Jenkins
Imagine your project consists of 4 microservices (3 backends, 1 frontend). Yesterday you introduced several new features and made a release. Unfortunately, your users have just reported a bug. Some of the old important features are not working. You need to do a rollback for all services. Ah, if it could be done with one …
Microservices and continuous delivery
Imagine a typical situation – yesterday your devops engineer was eaten by a tiger. You a very sad because he didn’t finish the release system for your project. It contains 4 repositories: 2 back-end, 1 front-end, 1 data pipeline. And now it is you who should set up a deploy pipeline for your project tomorrow. …
End-to-end microservices testing with Catcher
I would like to introduce a new tool for end-to-end testing – Catcher. What is an e2e test? End-to-end test usually answers the questions like: “Was this user really created, or service just returned 200 without any action?”. In comparison with project level tests (unit/functional/integration) e2e runs against the whole system. They can call your …
Ansible and Jenkins – automate your scritps
The topic I’d like to reveal in this article may seem obvious, but I was surprised how many companies don’t follow this best practice. For impatient: Automate every action you’ve done more than once. Don’t use Jenkins static groovy library. Use Jenkins + Ansible + Python for automation. The problem Any developer in his work …
HowTo: Run Cloudera Quickstart in Docker
The best way to familiarize yourself with Hadoop ecosystem or to do POC is to play with it in a sandbox. For that Cloudera provides 2 Quick Start options: one is the image for VirtualMashine and another is the Docker image. Both of them contain a small CDH cluster with one DataNode. But this is …
Scala as backend language. Tips, tricks and pain
I’ve got a legacy service, written in Scala. Stack was: Play2, Scala, Slick, Postgres. Here is described why such technology stack is not the best option, what should be done to make it work better with less efforts and how to avoid underwater rocks. For impatient: If you have choice – don’t use Slick. If …
Python & Graphql. Tips, tricks and performance improvements.
Recently I’ve finished another back-end with GraphQL, but now on Python. In this article I would like to tell you about all difficulties I’ve faced and narrow places which can affect the performance. Technology stack: graphene + flask and sqlalchemy integration. Here is a piece of requirements.txt: graphene graphene_sqlalchemy flask flask-graphql flask-sqlalchemy flask-cors injector flask-injector …
How to create avro based table in Impala
Consider the following situation: A bundle of .avro files is stored on HDFS. They need to be converted to Impala tables. Schemas are not provided with files, at least not externally (it’s contained as first line of any avro file). But Impala has a known-issue with avro tables, and its usage is pretty limited: we …