I have been experimenting with release upgrades. I know this is unsupported, but bionic->focal, which often does not work so well Several times I have come across the issue where the command e.g.
juju upgrade-series 0 complete
never actually completes. In this case, machine 0 had a neutron-gateway on it, but it seems that the ntp charm was responsible for this failure;
:~$ juju ssh 0 tail /var/log/juju/unit-ntp-1.log
2020-07-07 16:35:53 DEBUG start Could not find platform independent libraries <prefix>
2020-07-07 16:35:53 DEBUG start Could not find platform dependent libraries <exec_prefix>
2020-07-07 16:35:53 DEBUG start Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
2020-07-07 16:35:53 DEBUG start Fatal Python error: Py_Initialize: Unable to get the locale encoding
2020-07-07 16:35:53 DEBUG start ModuleNotFoundError: No module named 'encodings'
2020-07-07 16:35:53 DEBUG start
2020-07-07 16:35:53 DEBUG start Current thread 0x00007fb8cb8fb740 (most recent call first):
2020-07-07 16:35:54 ERROR juju.worker.uniter.operation runhook.go:136 hook "start" (via explicit, bespoke hook script) failed: signal: aborted (core dumped)
So I get it doesnāt work yet, I donāt really care about that. The question is, can I recover from this somehow? As it stands the machine is still locked for the release upgrade and I canāt seem to do anything with it.
I came across this yesterday with another unit and am working on fixing it.
To unlock the machine, you can manually connect to the MongoDB (see here) and remove the machineās series-upgrade lock from the machineUpgradeSeriesLocks collection. As to the charm issue itself, I am not yet sure as to a resolution.
I noticed the same problem with the placement charm, but I just rebuild that. This unit being on ātinā would have been a bit harder, but you have saved me the hassle!
To the charm, I guessed that the unit has a virtualenv that does not survive the release upgrade?
One thing Iāve found useful for determining why something is not completing in the complete phase is to use the juju_machine_lock command to check what charm/hook is holding up the line, and then determining the issues itās causing. It seems you were able to narrow it down to ntp charm failing to run itās post-upgrade hook successfully.
Can you comment on which revision of the ntp charm you are running? Iām wondering if the ntp charm revision youāre running supports focal. One must be running cs:ntp-38 or cs:ntp-39 (and all subordinate charms youāre running should be checked for series support) before performing the focal upgrade on your principal unit machines.
Also, the workaround in this bug may help you to get through your woes.
You essentially need to delete the wheelhouse/.bootstrapped and .venv files/dirs in the charms affected and then force a bootstrap manually by calling the python methods used during the install hook of layer-basic.
Several times I have come across the issue where the command e.g.