Rubynetes: Kubernetes config the easy way

Kubernetes is a useful system for managing modern deployments, but it is a big beast whose flexibility brings with it a new twist on some old, old problems - how do we tell the machine what to do and keep that up to date over time? Configuration and Release Management don’t go away just because we’ve wrapped the application in a container. In many ways we’ve just added another layer to the onion.

This short series of posts suggests a way to manage your sites may be hiding in plain sight right on your machine.

The Problem

Getting your applications to run on Kubernetes means creating YAML files. Big, hairy, heavily nested YAML files that you constantly get wrong. So you look for a tool to help you, which you have to find a way of installing and that turns out to be configured by a different set of big, hairy, nested YAML files that you constantly get wrong. So then you look for a tool to help you…

Turtles all the way down.

Now what if this incidental difficulty could be addressed in a nice straightforward way using well tested tooling, approaches that have been battled tested in production for a decade or more, and which is probably already installed on your machine?

Turns out there is. Just code up your configs in Ruby.

Config Parameters

def config
  {
    prefix: 'cloud-controller-build',
    image: 'gcr.io/kaniko-project/executor:latest',
    requestsCpu: '1700m',
    requestsMemory: '1Gi',
    limitsCpu: '2',
    limitsMemory: '1890Mi',
    gitRepo: 'git://github.com/brightbox/brightbox-cloud-controller-manager.git',
    dockerTarget: 'brightbox/brightbox-cloud-controller-manager',
    secretName: 'regcred',
    holdTime: 600
  }
end

This is a set of parameters for a Kubernetes Job resource - used to build the Brightbox Cloud Controller on a Kubernetes cluster. We need to generate several jobs relating to the latest active versions of Kubernetes and push the output to Docker Hub. This allows customers to build Kubernetes clusters on Brightbox Cloud for the version of Kubernetes they require.

With Ruby we can easily pull those details directly from the git repository

Config as Ruby Code

require 'active_support/core_ext/hash/keys'
require 'yaml'

def create_job(release, config)
  version = `git describe --always release-#{release}`
    .strip.sub(/^v([0-9]+\.[0-9]+\.[0-9]+).*$/, '\1')
  name = "#{config[:prefix]}-#{version.tr('.', '-')}"
  create_job_manifest(release, config, version, name)
    .to_hash.deep_stringify_keys.to_yaml
end

Here we grab the details from the git repo surrounding this script and grab the data we require using Ruby’s extensive string manipulation facilities. Then we leverage the Ruby ActiveSupport hash extensions plus the standard YAML libary to generate the output in the format we want.

Generating the output for several releases just requires a pretty ordinary Ruby enumeration

first_release = 14
last_release = 17
jobs = (first_release..last_release).inject('') do |memo, release|
  memo << create_job("1.#{release}", config)
end

puts jobs

Deploying the Config

So how do we get the YAML onto the Kubernetes cluster? Well we could parse the kubeconfig, locate the necessary credentials and then mess around with an HTTP library before POSTing the YAML to the correct part of the API.

But frankly life is too short.

require 'open3'
stdout_str, status = Open3.capture2('kubectl apply -f -', stdin_data: jobs)
puts stdout_str
exit status.exitstatus

The Config

Now what about the Job manifest itself? That ends up as a parameterised Ruby Hash literal

def create_job_manifest(release, config, version, name)
  {
    apiVersion: 'batch/v1',
    kind: 'Job',
    metadata: {
      name: name,
      labels: {
        build: config[:prefix]
      }
    },
    spec: {
      ttlSecondsAfterFinished: config[:holdTime],
      template: {
        spec: {
          containers: [
            {
              name: name,
              image: config[:image],
              resources: {
                requests: {
                  memory: config[:requestsMemory],
                  cpu: config[:requestsCpu]
                },
                limits: {
                  memory: config[:limitsMemory],
                  cpu: config[:limitsCpu]
                }
              },
              args: [
                '--dockerfile=Dockerfile',
                "--context=#{config[:gitRepo]}#refs/heads/release-#{release}",
                "--destination=#{config[:dockerTarget]}:#{version}"
              ],
              volumeMounts: [
                {
                  name: config[:secretName],
                  mountPath: '/root'
                }
              ]
            }
          ],
          volumes: [
            {
              name: config[:secretName],
              secret: {
                secretName: config[:secretName],
                items: [
                  {
                    key: '.dockerconfigjson',
                    path: '.docker/config.json'
                  }
                ]
              }
            }
          ]
        }
      }
    }
  }
end

Testing the Config

How do we know this is correct? Well we could write a set of test specs for it. Here’s an example.

describe '#create_job_manifest' do
  let(:job) {
    load 'create_docker_jobs'
    create_job_manifest('1.16', config, '1.16.1', 'job-1.16')
  }

  it 'has a restart Policy of never' do
    expect(job[:spec][:template][:spec][:restartPolicy]).to eq 'Never'
  end
end

Run it with rspec

$ rspec
F

Failures:

  1) #create_job_manifest has a restart Policy of never
     Failure/Error: expect(job[:spec][:template][:spec][:restartPolicy]).to eq 'Never'

       expected: "Never"
            got: nil

       (compared using ==)
     # ./spec/create_docker_jobs_spec.rb:6:in `block (2 levels) in <top (required)>'

Finished in 0.05696 seconds (files took 0.14845 seconds to load)
1 example, 1 failure

Failed examples:

rspec ./spec/create_docker_jobs_spec.rb:5 # #create_job_manifest has a restart Policy of never

Dang. Always forget to set that one with Job manifests. Better get it fixed.

And for my Next Trick…

But perhaps you don’t fancy writing all those specs by hand. Is there another way that will check you have the format correct? Yes there is, and one that checks all the resources you have loaded in your Kubernetes Cluster - including the custom ones.

See part 2, where we look at using OpenAPI to validate Kubernetes configs.

Postscript

Ruby offers all the tools necessary to handle that awkward moment when declarative purity hits the real world of delivering on time and budget. You can test your configs with Specs, you can modularise your configs with Gems - storing them in their own repository if you wish, and you can DRY up your deployment code very easily using Ruby’s inherent scriptiness.

Give it a go. Ruby is beautiful. And Fun. And Practical. And, most importantly for this job, Mature.

Interested in Managed Kubernetes?

Brightbox have been managing web deployments large and small for over a decade. If you’re interested in the benefits of Kubernetes but want us to handle managing and monitoring it for you, drop us a line.