Development

Development

To get info about new technologies, perspective products and useful services

BigData

BigData

To know more about big data, data analysis techniques, tools and projects

Refactoring

Refactoring

To improve your code quality, speed up development process

JFrog Artifactory on Travis CI for Python tests

JFrog Artifactory on Travis CI for Python tests

Disclaimer:

  • It is not about storing python packages in Artifactory.
  • It is not about creating python packages in TravisCI and publishing them to Artifactory.

In this article I’ll show you how to set up Artifactory in Travis CI for using it in your python project’s tests.

Why do I (and may be you) need this?

Imagine you develop an application, which talks to Artifactory. If you can’t – take a look at Enot – advanced Erlang package manager. In a few words – it is a hybrid of maven and pip but for Erlang.

One of the things it can do: is to compile Erlang project into a package and load it to a remote repository. Once loaded, it can be used everywhere without compiling. Enot will just download package and link it to your project. Compile once – use many.

Great! How is it related to Artifactory? It’s simple – Enot use several backend types for remote repositories. Artifactory is one of them. And I have tests for a class which works with Artifactory, to ensure that everything is ok.

All my tests are run through make tests and I set up Travis easily by placing .travis.yml in my repository root.

sudo: required
dist: trusty
language: python
python:
  - "3.3"
  - "3.4"
  - "3.5"
  - "3.6"
before_install:
  - sudo apt-get -y install erlang
install: "python setup.py install"
script:
  - make tests

Now Travis will run tests against 4 python versions on push to every branch and pull request opened on github.

Problem – no local Artifactory service

If your tests use local Artifactory there can be a problem. Travis doesn’t have such service locally. You need to add it somehow. Actualy Travis is poor for existent connection services. There are ready some, but only to a few popular databases. So you need to install, run and configure artifactory before running tests.

Solution – installing Artifactory before tests

Option 1 – installing Artifactory service from deb

dist: trusty tells Travis to create Ubuntu Trusty build environment. It is quite easy to install Artifactory on Ubuntu Trusty. From JFrogs installation guide:

echo "deb https://jfrog.bintray.com/artifactory-pro-debs xenial main" | sudo tee -a /etc/apt/sources.list

curl https://bintray.com/user/downloadSubjectPublicKey?username=jfrog | sudo apt-key add -

sudo apt-get update
sudo apt-get install jfrog-artifactory-oss

But this means every time you run tests – Artifactory will be downloaded and installed. This is not cool, as it takes some time to download artifactory and about 2 minutes for thre first run.

Option 2 – manual Artifactory installation from zip

As I didn’t want to download and set-up Artifactory every time I run tests, I decided to download Artifactory zip package and cache it between test runs.

Travis allows caching some data in directories, specified in .travis.yml file. The content of these directories is packed in tar and uploaded to S3. Before building Travis check if a cached archive exists. If it does, it will be pulled and unpacked to the specified location. Travis has one cache per branch and language version/compiler version/JDK version/Gemfile location/etc.

Modify .travis.yml to set up cache directory $HOME/artifactory and include install_deps.sh running before installation. Also you will need to add Artifactory starting in a background via $HOME/artifactory/artifactory/bin/artifactory.sh & and 2 min delay to wait for the first start. Later delay can be decreased to 10 sec.

sudo: required
dist: trusty
language: python
python:
  - "3.3"
  - "3.4"
  - "3.5"
  - "3.6"
before_install:
  - sudo apt-get -y install erlang
  - bash install_deps.sh
install: "python setup.py install"
addons:
cache:
  directories:
  - $HOME/artifactory
script:
  - $HOME/artifactory/artifactory/bin/artifactory.sh &
  - sleep 120
  - make tests

Add install_deps.sh which should download and untar Artifactory zip package if cache directory doesn’t exsist.

#!/usr/bin/env bash
if [ ! -d  "$HOME/artifactory/artifactory" ]; then
  mkdir -p $HOME/artifactory
  artifactorywget https://bintray.com/jfrog/artifactory/download_file?file_path=jfrog-artifactory-oss-5.3.0.zip \
       -O $HOME/artifactory/artifactory.zip
  unzip $HOME/artifactory/artifactory.zip -d $HOME/artifactory
  mv $HOME/artifactory/artifactory-oss-* $HOME/artifactory/artifactory
else
  echo "Using cached artifactory."
fi

If $HOME/artifactory exists in system – it will be used:

Zip package config makes Artifactory treat dir it was downloaded to (cache dir in our case) as installation dir. So – first run configuration results are also cached!

Any other time it takes about 9 seconds for Artifactory to start:

As for configuration, provided in etc/artifactory.config.xml shipped with artifactory.zip – generic example-repo-local will be created with default credentials mentioned in README.md:

username: admin
password: password

That is what I exactly need! Now Artifactory test repo is available for Travis at http://localhost:8081/artifactory/example-repo-local and my Artifactory tests run without problem.

Leave a Reply

Your email address will not be published. Required fields are marked *