Development

Development

To get info about new technologies, perspective products and useful services

BigData

BigData

To know more about big data, data analysis techniques, tools and projects

Refactoring

Refactoring

To improve your code quality, speed up development process

Dynamic environment configuration (Erlang)

Dynamic environment configuration (Erlang)

Most of Erlang applications requires configuration variables. It can be: database credentials, metrics report hosts, some system limits and so on. Such variables can be set in application resource configuration file. They can be accessed with application:get_env. You should prefer dynamic configuration instead of hardcoded one. But what if you need different properties in configuration depend on environment?

Assume you have your compound service, which uses database, some microservices and also sends push notifications to mobile phones. All these configuration properties rely on environment. You have different microservice hosts/ports, auth tokens, certificates, database passwords in production, staging and testing. You don’t use same credentials everywhere, do you? How to deal with it and handle configuration changes gracefully? (Quick hint, the most cool stuff is at the end of the article, so, if you don’t like reading the comparison of different ways – just skip it). Lets explore all options.

1. Static configuration with git

The most simple (and not recommended) approach is to save configuration in app.src under the VCS. In master branch you will always have production configuration, while in develop you have staging and so on. Everything is simple, but:

  • You should always remember to change configuration when merging branches. And this can be the most common reason of failures in starting your service after the merge. One of your developers forget to change config and prod was started with staging or developing configurations.
  • You test one branch (develop) and deploy another branch on production. It can also be a reason for instability of your service.

This approach is not good, but it is suitable, if you have tiny configuration without lots of thirdparty services. But when your environment grows – you should use something else.

2. Dynamic configuration

Dynamic configuration is a completely another way of configuring your application. You remove all environment dependent config from app.src file and move them somewhere else. It can be third-party service, your own configuration module or configuration file.

2.1 Third-party service

Third party service is used for storing configuration. It can be Consul, Etcd, Zookeeper or even a database. You add url for accessing config service, where all environment dependent variables can be taken:

{env, [{config_agent_url, “127.0.0.1:8500”, …some_env_independent_config…}]}

where config_agent_url is static and remain the same for all environments. Agents use storages with different configurations.

This approach allow you to do a lot of cool things, like service auto discovering, when you obtain hosts of all your services at runtime. They start, go to config_agent_url and register there. Your main service can query them just like this:

GET config_agent_url/services/database

[ {db.host.1.your.domain, 27018}, {db.host.2.your.domain, 27018} …]

But there are also cons:

  • You have to maintain additional service in your environment.
  • You need to be sure service exists and is accessible on every environment.
  • Your configuration doesn’t come along with source code – so you need to remember its format and values.
  • It can be insecure to store credentials on third-party services.
  • You should modify your application to load configuration remotely (although you can use existing libraries like Seagull).

Summing up – if you don’t need service discovery, it’s probably not what you want. External service will bring additional difficulties, which can overweight profit from per-environment configuration.

2.2 Service integrated configuration

It is nearly the same, as third party service, except, there is no additional service. You have your static file/embedded databse with configurations saved in filesystem. In your app.src file you also have something like this:

{env, [{config_path, “/opt/your_cervice/conf”, …some_env_independent_config…}]}

where config_path is a static and remain the same just with different configuration under it. You just load this file on application start .

So, with this approach you neutralize one of the biggest concern: a need to maintanin an external service. But as well there is no longer an advantage of service discovery, as a trade off.

Also, there appeared another complexity – chanding configuration. With external service you just login and change configuration once. After what it will be synchronized by all agents. Here configuratoin is not shared. Each server has its own configuration file. You need to access every server to apply configuration changes. Of course it is not a problem, if you use Asible (which adds additional complexity).

Summing up cons:

  • You have to be sure that configuration file exists on every environment.
  • Your configuration doesn’t come along with source code, so you need to remember its format and value.
  • You should modify your application to load files.
  • You have additional difficulties when changing configuration.

Don’t like this approach? Read next.

2.3 Environment variables configuration

So, you are here if you don’t like to create additional dependencies for your service. No dynamic remote configuration services, no files and embedded databases. Enough.

This approach is really simple – rely on system defined environmental variables. Modify your app.src to contain variables for the most important environment (production, I guess). Let system environment variables override it.

Just don’t forget to set up these variables on all systems except production. It will work with its default configuration and if someone forget to override production configuration in his own environment – this is not critical. Cost of mistake won’t be high.

This approach has less disadvantiges than above one:

  • You need to be sure configuration variables exists on every non-prod environment.
  • You should modify source code of your application to load files.
  • You have additional difficulties while changing non-prod configuration.

Cool, it is a real evolution of approaches! Can it be even better? Yes.

3. Environment based native configuration

I tried all these options in my different projects and found them a little bit inconvenient. It is better when your configuration go along with your code, in one place. When some parts of configuration variables are splitted to system environmental variables or external services – it leads to a mess. Sometimes you can forget to add or modify some variable, or you find yourself connecting to remote configuration service when you forgot variable format.

It also goes without saying, that external dependency adds additional point of failure to your system. What if dynamic config agent fails to connect to service, or its cluster broke? What if config file is unaccessible by user running your service? And what is the first place every Erlang developer will search for configuration?

Sure, it will be much better, if environment based configuration will present in .app.src file. But how it will fit there?

Java programmers found solutions for their projects. You can build for different environments with Maven, use environment based configuration as a part of frameworks such as Spring and PlayFramework. So, how they are doing it?
They have all their configurations in the same repo with code, in different files, or even in one, but with environment prefix usage.

Part of PlayFramework’s application.conf:

# Configuration for scr

%scr.http.port=9500

# Production configuration

%production.http.port=80

It’s more comfortable: you have one place for all your configurations. If you need to add/modify/check some variable – you can do it fast, nearly in a moment.

Can we do the same neat thing in Erlang? Yes, and Enot build tool will help us. One of the cool things Enot can do it Jinja2 templating support in generation .app file from .app.src. That is exactly what we need!

Assume we have this simple app.src file

{application, super_service, [
  {description, "Your description"},
  {vsn, {{ app.vsn}} },
  {registered, []},
  {applications, {{ app.std_apps + app.apps }} },
  {modules, {{ modules }}},
  {mod, {'super_service_app', []}},
  {env, [
    {static_var, “environment_independent_value”}
  ]}
]}.

Here we’ve used the basics of Jinja2 templating. We access Enot’s Package class and get app.std_apps which are ['kernel', 'stdlib'] and app.apps which are dependent applications used by your super_service. Modules template is a list of all modules, belongs to application. About 5 years ago programmers have to add modules to .app files manually, but in modern world all build systems do this for you.

Also we access application version from config. It is much more comfortable to store it as JSON in enot_config.json file and let Enot insert it everywhere (relx.conf, .app file).

But these all are basics, not connected to the topic of the article. So, dig deeper?

In .app.src you can also use {{ hostname }} template. It is exactly what we need! Lets add dynamic variables configuration to env.

{application, super_service, [
    {description, "Your description"},
    {vsn, {{ app.vsn}} },
    {registered, []},
    {applications, {{ app.std_apps + app.apps }} },
    {modules, {{ modules }}},
    {mod, {'super_service_app', []}},
    {env, [
        {static_var, “environment_independent_value”},
        {% set prod_hosts = ['super_prod1', 'super_prod2', 'super_prod3'] -%}
        {% set stage_hosts = ['super_stage1', 'super_stage2', 'super_stage3'] -%}
        {% if hostname in prod_hosts %}
                {database, [
                        {host, "prod_db"},
                        {password, "super_secude_password"},
                        {pool_size, 400}
                ]}
        {% elif hostname in stage_hosts %}
                {database, [
                        {host, "staging_db"},
                        {password, "other_secude_password"},
                        {pool_size, 200}
                ]}
        {% elif hostname == ‘bob_home_computer’ %}
                {database, [
                        {host, "127.0.0.1"},
                        {password, "super_bob"},
                        {pool_size, 1}
                ]}
        {% else %}
                {database, [
                        {host, "dev_host"},
                        {password, "dev_password"},
                        {pool_size, 10}
                ]}
        {% endif %}
    ]}
]}.

What have we done here? Well, just in .app.src file we have mentioned all our prod and staging hosts. Then we add a condition, that, if current hostname belongs to prod group – prod database configuration will be written to .app file. If hostname is in staging group – staging configuration will be written. If it is Bob’s home machine, local database will be used, as Bob has individual development environment at home. Default development configuration will be used for any other hosts.

Do you feel the power of templating? But what about the logs?

You had tested your business logic already, so log level should be different on prod and staging.

Add this to your environment:

{% if app.git_branch == 'master' %}
    {log, info}
{% else %}
    {log, debug}
{% endif %}

Now when you finish testing application tests and merge to master – logs will automatically switch to info level.

Summing up

I have described 5 different approaches of dynamic environment based configuration. It is up to you which approach to use, but remember, it will be better for you to:

  • automate as much as possible. Let tools do this work for you so you can spend your time on coding;
  • have all configuration in one place. Just to be sure, that, if all your developers suddently die, newcomers won’t be confused searching and gathering configuration pieces all over the service;
  • don’t use self-made configuration files and modules which work with it. Your OPS won’t be happy with the unknown file’s format;
  • don’t bring third-party services you don’t really need. More services – more points of failure.

Have a good luck and keep your configuration clean! 🙂

Leave a Reply

Your email address will not be published. Required fields are marked *