How to test k8s upgrades

juju
1

Overview

The mechanics of how upgrades work for k8s deployments is similar to what happens for non-k8s:

  • Juju checks to see what agent binaries are available, using agent-stream to select between develop, proposed, and released binaries
  • the requested image is downloaded and used

A key difference is where the agent binaries are sourced. For non-k8s, it’s simplestreams. For k8s, it’s dockerhub.

The default dockerhub account used to store the Juju j8s operator images is jujusolutions. Tags are used to represent the Juju version, 2.7.5, 2.8-beta1, 2.8-beta1.1 etc.

Production upgrades

When a user types juju upgrade-controller, Juju will:

  • list all of the jujud-operator images on dockerhub
  • filter the images based on the agent-stream value
  • instruct k8s to pull the most recent image, unless --agent-version is used to pick a specific image
  • k8s will update the controller pod with the selected image

The agent-stream filtering is:

  • devel selects all images tagged beta and higher
  • proposed selects all images tagged rc and higher
  • 'released` (the default) only selects images without tags, eg 2.7.5

Upgrades in development

makefile targets

We’ll use 2 makefile targets:

  • operator-image
  • push-operator-image

Procedure

For developing Juju, you want to use you own dockerhub namespace to host the binaries, not jujusolutions. So start by setting DOCKER_USERNAME=, eg

export DOCKER_USERNAME=wallyworld

Since Juju pulls all docker images it needs from this docker username, you’ll need to make sure the Mongo DB image is copied there first up.

docker pull jujusolutions/juju-db:4.0 
docker tag jujusolutions/juju-db:4.0 wallyworld/juju-db:4.0
docker push wallyworld/juju-db:4.0

For the controller to use your dockerhub username, you’ll need to set the caas-image-repo controller config. This can be done at bootstrap time:
juju bootstrap microk8s --config caas-image-repo=wallyworld

Assuming you have a 2.7.5 controller deployed, and you want to test upgrading to a 2.8-beta1 version you are working on…

  1. push a locally complied 2.8-beta1 image to dockerhub
    make push-operator-image

  2. upgrade the controller
    juju upgrade-controller --agent-stream=develop

If it were an rc, you could use --agent-stream=proposed.
Or you could pick a specific version if --dry-run showed something other than what was wanted
juju upgrade-controller --agent-stream=develop --agent-version=2.8-beta3

–build-agent replacement

Say you’ve deployed the latest version of Juju from develop branch and you want to make a change and try out a new build number. For k8s , --build-agent cannot work, so the steps are:

  1. build an operator image from source
    (it will always have the juju version from the branch, eg 2.8-beta1)
    make operator-image

  2. tag the operator image with a build number eg 1
    docker tag wallyworld/jujud-operator:2.8-beta1 wallyworld/jujud-operator:2.8-beta1.1

  3. push this tagged image to dockerhub so Juju upgrades can see it
    docker push wallyworld/jujud-operator:2.8-beta1.1

  4. upgrade as normal
    juju upgrade-controller --agent-stream=develop

If you’re curious what images you’ve got built locally taking up space, you can always list them
docker image list

Speeding up microk8s bootstrapping

To cut out the step to push the operator image to dockerhub and require microk8s to download it again, you can seed the microk8s image cache directly.

make microk8s-image-update
juju bootstrap microk8s

You can also use a different namespace to jujusolutions as already covered above.

export DOCKER_USERNAME=wallyworld
make microk8s-operator-update
juju bootstrap microk8s --config caas-image-repo=wallyworld
2 Likes
2

If you are using your own repo for pulling images (via the caas-image-repo config option), do you also need to push the juju-db image to your own repo or is there a fallback in place to pull it from jujusolutions/juju-db?

3

Good question. Everything comes from the same repo.

I had pushed the juju-db image to my dockerhub repo a while ago and forgot that I had done it.
So yeah, you’ll need to:

docker pull jujusolutions/juju-db:4.0 
docker tag jujusolutions/juju-db:4.0 wallyworld/juju-db:4.0
docker push wallyworld/juju-db:4.0
4

There is arg for setting the build number like this

DOCKER_USERNAME=your-repo JUJU_BUILD_NUMBER=1 make microk8s-operator-update|push-operator-image
5

I’ve noticed an issue with recent testing, where I have been:

  • Building Juju from HEAD.
  • Bootstrapping to AWS or Azure.
  • Installing Charmed Kubernetes with the appropriate overlay such as for AWS.
  • Running juju scp kubernetes-master/0:/home/ubuntu/config ~/.kube/config.
  • Running juju add-k8s to add the cluster (cloud) to the controller.
  • Adding a model for this cloud.

When deploying K8s units, the OperatorProvisioningInfo returned from the API looks like this:

{
    ImagePath: manadart/jujud-operator:2.9-rc13 
    Version: 2.9-rc13.1 
    ...
}

This is due to the multi-cloud setup where the controller is not in a pod with an operator image built using JUJU_BUILD_NUMBER, but rather my local dev agent build running in a VM.

What it means is that I always need 2 operator images in my repository. One for 2.9-rc13 and one for 2.9-rc13.1, which the agent immediately tries to upgrade to, because that is what is in model config.