I’m working on filling out the series upgrade support in Charmed Kubernetes and I was wondering how people deal with automated testing of the series upgrade support. Specifically, the do-release-upgrade has a fair amount of manual intervention required that seems like it would be very difficult and error-prone to drive in an automated fashion.
Has anyone worked around this and done automated testing of series upgrades? If so, what approaches did you use, and are there any other issues that you ran into?
Trusty to Xenial is more problematic because some older packages still have bugs in startup, etc., unhelpfully making some of them not-non-interactive.
In our operational procedure for live clouds, we do the following for Xenial->Bionic
First, drop a file in /etc/apt/apt-conf.d/local with the following:
DPkg::options { “–force-confdef”; “–force-confnew”; }
Then run the following on the units:
purge old kernels and stop cron/landscape-client from interfering with upgrade
juju run --machine {} ‘sudo service cron stop; sudo service landscape-client stop; sudo cp /home/ubuntu/local /etc/apt/apt.conf.d/local; sudo apt-get dist-upgrade -y; sudo apt-get autoremove -y’
juju run --timeout=180m0s --machine {} ‘sudo do-release-upgrade -f DistUpgradeViewNonInteractive’
Remove the /etc/apt/apt-conf.d/local file after the do-release-upgrade finishes
juju run --machine {} ‘sudo init 6’
The local file plus the -f DistUpgradeViewNonIInteractive ends up bypassing all of the interactive bits of d-r-u, but you need to make sure those settings are appropriate for the applications you’re installing and running. Obviously if you force-confnew, you have to be certain that the configuration files that you’re concerned with will be regenerated on the new series upon juju unit restart. This is typically handled by the series-upgrade “complete” step. that calls post-series-upgrade hooks, IIRC.
For posterity, there was a lot of extra logic in the Zaza cases that didn’t end up being necessary for the simpler K8s case, so the K8s test ended up being mostly an implementation of @afreiberger’s distillation above.
