When packaging any app in a Docker image, you run into the issue of how to protect the secrets, like database credentials or API keys.

We all know about not checking in secrets to Github (don’t we?), but we also don’t want to package secrets in the image. Anyone who can get at the image can also get at the secrets, and since the image proliferates through CI/CD pipeline and image repo, it’s near impossible to limit access effectively.

What we found to work best is to store secrets in a well protected Cloud Storage bucket, and have the app download the secrets on startup. This decouples the secrets from Docker images, and rotating secrets becomes a cinch.

The example here from an app written to run on Google Cloud Platform.

Layering Secrets in Flask

Flask gives you multiple ways to load configuration. Every time it loads a configuration, it overlays the existing configuration.

We can use the default settings (i.e., dev settings or reasonable defaults) in the code, then download and merge secrets.

from my_lib import secrets

app = Flask(__name__)

# Load default config
app.config.from_pyfile('config/default_settings.py')

# Overlay secure secrets
secrets.load(app)

Loading Secrets from Cloud Storage

Here we download secrets from a Cloud Storage bucket. The bucket should be restricted to allow read access from a specific service account assigned to the app.

import os
import uuid
from google.cloud import storage, kms_v1


def load(app):
    config_file_name = "secrets-%s.cfg" % str(uuid.uuid4())

    storage_client = storage.Client()
    bucket = storage_client.get_bucket('vault')
    blob = bucket.blob('secrets.cfg')
    blob.download_to_filename(config_file_name)

    try:
        app.config.from_pyfile(config_file_name)

    finally:
        os.remove(config_file_name)

Extra Paranoid Secrets Loading

As a paranoid implementation, you can encrypt the secrets in the bucket. The app decrypts the downloaded secrets then loads it in. Both encrypted and decrypted files get deleted. The app needs permission for the bucket as well as the crypto keyring in KMS.

def load_from_encrypted(app):
    tempname = str(uuid.uuid4())
    encrypted_file_name = "secrets-%s.cfg.encrypted" % tempname
    decrypted_file_name = "secrets-%s.cfg.decrypted" % tempname

    storage_client = storage.Client()
    bucket = storage_client.get_bucket('vault')
    blob = bucket.blob('secrets.cfg.encrypted')
    blob.download_to_filename(encrypted_file_name)

    client = kms_v1.KeyManagementServiceClient()
    key_name = client.crypto_key_path('app', 'global', 'keyring', 'secrets')

    try:
        with open(encrypted_file_name, 'rb') as content_file:
            cipher_text = content_file.read()
            decrypted = client.decrypt(key_name, cipher_text)
            with open(decrypted_file_name, 'w') as decrypted_file:
            	decrypted_file.write(\
 							decrypted.plaintext.decode("utf-8"))
        		app.config.from_pyfile(decrypted_file_name)

    finally:
        os.remove(encrypted_file_name)
        os.remove(decrypted_file_name)

Encrypting Secrets

You need to encrypt the secret file before uploading to the bucket. The shell script below encrypts and uploads a secrets.cfg file. To run this script, you need access to KMS keyring as well as write permission for the vault bucket

#!/usr/bin/env bash
gcloud services enable cloudkms.googleapis.com

KEYRING=keyring
KEY=secrets

gcloud kms keys list --location global --keyring $KEYRING

if [[ $? -ne 0 ]]; then
    gcloud kms keyrings create ${KEYRING} --location global
    gcloud kms keys create ${KEY} --location global \
      --keyring ${KEYRING} --purpose encryption
fi

rm -f secrets.cfg.encrypted

gcloud kms encrypt --location global \
  --keyring ${KEYRING} \
  --key ${KEY} \
  --plaintext-file secrets.cfg \
  --ciphertext-file secrets.cfg.encrypted

gsutil cp sectets.cfg.encrypted gs://vault

Flask + Gunicorn Caveat

When running a Flask app under gunicorn, loading configuration file using relative path (i.e. app.config.from_pyfile('config/default_settings.py')) can fail. This is because gunicorn doesn’t change to the app directory. You need to supply--chdir flag to make gunicorn change to the directory that app expects.

Dockerfile example:

ENV APP_HOME /app
ENV PORT 8080

CMD exec gunicorn --bind 0.0.0.0:$PORT --workers 1 --chdir $APP_HOME --threads 1 app:app