BLOG

Random thoughts on AWS and Information Technology
  1. Home
  2. Blog
  3. Article
  4. Using DynamoDB as a Django settings store

CATEGORIES

TAGS

ARCHIVE

Using DynamoDB as a Django settings store

In this article, I will show you how you can use DynamoDB as a settings store for your Django application. Some people prefer to store settings in the environment variables of the instance itself, but DynamoDB can be a quite good alternative.

Table of contents

Django settings overview

When you don't specify a settings-module for your Django project, the settings.py which is located in your project folder will be used. You can override the settings module in two ways:

  • Via the command line using the --settings= parameter.
  • Via the DJANGO_SETTINGS_MODULE environment variable.

In the past, I used a separate settings module for each environment, which resulted in multiple settings modules in my codebase:

/project/settings.py (for local development)
/project/settings_test.py (for test environment)
/project/settings_prod.py (for production environment)

I think most people who started developing with Django did it this way initially, however, there are a few drawbacks to this technique:

  • Disclosure of sensitive information: Keeping settings in your codebase directly, means you also have sensitive information like database usernames and passwords in your codebase.

  • Subtle changes in test vs production: You add a parameter in your test environment settings module, but you forget to add the parameter in your production settings module.

  • Changing settings requires deployment: This one speaks for itself. Changing settings should not require a new deployment of your application.

Test vs. production state

While settings on your local development machine can differ from your production environment (after all, during development we experiment with new things), for actual deployment, we want our test environment to match our production environment as close as possible. In order to make this possible, we need the following three conditions:

  • We need to dynamically determine our environment during application startup.
  • Based on the information we got in the previous step, load the configuration associated with the environment we are currently running in.
  • Our settings loading mechanism should be identical in both test and production.

Let's bring in AWS

Let's see how we can establish those three steps, with a little help from AWS :-)

Determine our environment

Since we run our application on EC2 instances, we can use tags to identify our environment. For example, for every instance we launch in our test environment, we can add the following tags:

Environment: test-myapp01
Environment-role: test-myapp01-website

The Environment-role tag is added to quickly identify EC2 instances when your application consists of multiple components. For example, you might also have a role called test-myapp01-mailgateway if your application sends out email and the mail server is on a different instance.

If you use tools like CloudFormation or TerraForm (and I really recommend you do), you can have those tags added automatically every time you make a change to your infrastructure.

During startup of our application, we can determine our environment by querying the meta-data of the instance we are running on.

Loading the configuration associated with our environment

Since our environment is now identified, we can easily load our configuration. I choose DynamoDB as the repository for the application settings, since it's highly-available in your AWS region, it's cheap, and you can manage it via the AWS console.

Unifying our settings loader

In this setup I only have two settings modules in my codebase:

/project/settings.py (for local development)
/project/settings_deploy.py (for test and production environment)

settings_deploy.py will retrieve the EC2 tags associated with the instance it is running on, and retrieve the settings from the DynamoDB table.

Implementation

DISCLAIMER: This is just a proof-of-concept, and not production-quality code.

Creating the DynamoDB table

The name of the table should be related to the environment-role we run in. For example, if our environment-role is test-myapp-website, we need to create a DynamoDB table that is called test-myapp-website-config.

We will use the AWS command-line tools to do this:

aws dynamodb create-table --table-name=test-myapp-website-config \
--attribute-definitions AttributeName=Parameter,AttributeType=S  \
--key-schema AttributeName=Parameter,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

Next, we will fill this table with some default settings. Create the file test-myapp-website-config.json with the following content:

{
    "test-myapp-website-config": [
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "debug"},
                    "Value": {"BOOL": true}
                }
            }
        },
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "db_host"},
                    "Value": {"S": "my.test.server"}
                }
            }
        },
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "db_name"},
                    "Value": {"S": "my_db"}
                }
            }
        },
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "db_user"},
                    "Value": {"S": "my_username"}
                }
            }
        },
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "db_pass"},
                    "Value": {"S": "my_password"}
                }
            }
        },
        {
            "PutRequest": {
                "Item": {
                    "Parameter": {"S": "db_port"},
                    "Value": {"S": "5432"}
                }
            }
        }
    ]
}

Next step, load this file into your DynamoDB table:

aws dynamodb batch-write-item --request-items file://test-myapp-website-config.json

Loading our settings from Django

Make sure you have Boto and Requests installed:

pip install requests
pip install boto

At the top of our settings_deploy.py file, we can add the following code to retrieve the value of our Environment-role tag:

# get environment
r = requests.get('http://169.254.169.254/latest/meta-data/instance-id')
if r.status_code == requests.codes.ok:
    instance_id = r.text
    conn = ec2.connect_to_region(AWS_REGION_NAME)
    reservations = conn.get_all_instances()
    for res in reservations:
        for inst in res.instances:
            if inst.__dict__['id'] == instance_id:
                AWS_ENV = inst.__dict__['tags']['Environment-role']

Now we can construct the name of our table, and connect to it:

dynamo_conn = dynamodb.connect_to_region(AWS_REGION_NAME)
config_table = dynamo_conn.get_table('{}-config'.format(AWS_ENV))

After we connected to the table, we can retrieve our settings:

DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.postgresql_psycopg2',
            'NAME': config_table.get_item(hash_key='db_name')['Value'],
            'USER': config_table.get_item(hash_key='db_user')['Value'],
            'PASSWORD': config_table.get_item(hash_key='db_pass')['Value'],
            'HOST': config_table.get_item(hash_key='db_host')['Value'],
            'PORT': config_table.get_item(hash_key='db_port')['Value'],
    }
}

IAM access role

Our instances need some additional permissions, to read the EC2 tags, and read the DynamoDB table. Add the following to your instance's IAM role:

{
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeInstances",
                "ec2:DescribeTags"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:BatchGetItem"
            ],
            "Resource": [
                "<arn of your dynamoDB table>"
            ]
        }

Repeat for your production environment

You can now create a similar table for your production environment, and tag your production instances in the same way.

Final words

This is just a quick example, and you might want to do some extra work before you start implementing this:

  • Provide error checking to make sure the table and values exist in DynamoDB.
  • Use BatchGetItem to retrieve all settings in one go.

Also take a look at Dynamodb-config-store: https://github.com/sebdah/dynamodb-config-store.

References

The Twelve-Factor App: http://12factor.net
How to manage production/staging/dev Django settings: https://discussion.heroku.com/t/how-to-manage-production-staging-dev-django-settings/21
An Introduction to boto’s DynamoDB interface: http://boto.readthedocs.org/en/2.3.0/dynamodb_tut.html