Deploy openstack "default Failed deployment: Loading ephemeral"

[riccardo-magrini]

Hi Dear
I’m trying to deploy Openstack base via Juju (2.7.6) on my virtual lab (Maas 2.8.2 and Ubuntu 20.04) but after I run the command

$: juju deploy cs:bundle/openstack-base-70

the machine started the deploy of OS (20.04)

but after a couple of minutes obtained this issue

on their log I saw these line:

curtin: Installation started. (20.1-2-g42a9667f-0ubuntu1~18.04.1)

is it normal seeing the 18.04??? while at the end of file:

Generating grub debconf_selections for devices=[’/dev/vda’] uefi=False
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/install-grub: FAIL: installing grub to target devices
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks/configuring-bootloader: FAIL: configuring target system bootloader
finish: cmd-install/stage-curthooks/builtin/cmd-curthooks: FAIL: curtin command curthooks
Traceback (most recent call last):
File “/curtin/curtin/commands/main.py”, line 202, in main
ret = args.func(args)
File “/curtin/curtin/commands/curthooks.py”, line 1770, in curthooks
builtin_curthooks(cfg, target, state)
File “/curtin/curtin/commands/curthooks.py”, line 1736, in builtin_curthooks
setup_grub(cfg, target, osfamily=osfamily)
File “/curtin/curtin/commands/curthooks.py”, line 689, in setup_grub
configure_grub_debconf(instdevs, target, uefi_bootable)
File “/curtin/curtin/commands/curthooks.py”, line 509, in configure_grub_debconf
link = block.disk_to_byid_path(dev)
File “/curtin/curtin/block/init.py”, line 869, in disk_to_byid_path
mapping = get_dev_disk_byid()
File “/curtin/curtin/block/init.py”, line 861, in get_dev_disk_byid
return _get_dev_disk_by_prefix(’/dev/disk/by-id’)
File “/curtin/curtin/block/init.py”, line 846, in _get_dev_disk_by_prefix
for path in os.listdir(prefix)]
FileNotFoundError: [Errno 2] No such file or directory: ‘/dev/disk/by-id’
[Errno 2] No such file or directory: ‘/dev/disk/by-id’

and on Juju status the situation is that

I’ve also tried to deploy another charm and the result is ok

someone can help me? thanks a lot

[wallyworld (Ian Booth)]

@sparkiegeek can we get a MAAS person to weigh in here to help diagnose where to look next?

[routergod]

Are you using vmware or suchlike? If you don’t have a proper disk id the default storage layout does not work now. Try with LVM.

[erik-lonroth (Erik Lönroth)]

Can you deploy the servers with maas only to verify that your nodes can be deployed at all?

If not, try to commission them again and then try again until you know your maas works before you get juju in to the picture.

[riccardo-magrini]

I’ll try to use LVM instead of default storage. I’ll inform you soon

[riccardo-magrini]

@erik-lonroth

an you deploy the servers with maas only to verify that your nodes can be deployed at al

Making the deploy via MAAS I haven’t any issue, the VM called ubuntu.maas has been made in that way.

[pmatulis (Peter Matulis)]

Probably due to LP #1876258. This is a curtin issue that occurs when a KVM-based disk does not have a serial number. A fix has been committed so I guess it’s a matter of when a backport can be made to Ubuntu 20.04 LTS. A workaround is to manually add serial numbers.

[riccardo-magrini]

@pmatulis

A fix has been committed so I guess it’s a matter of when a backport can be made to Ubuntu 20.04 LTS. A workaround is to manually add serial numbers.

@pmatulis @routergod
maybe I’ve resolved that, but I’ve to try the same procedure made for the bootstrap also for the VM dedicated to Openstack, soon my feedback

[riccardo-magrini]

@pmatulis @routergod

after the changes on all the VMs and run the deploy of its bundle base, the situation is completed different

I’m waiting that the procedure finish. At this point I don’t know if that issue is a bug or the VMs have to be set via LVM storage.

[riccardo-magrini]

sorry, the result after the deploy

but on its status there are that error:

keystone/0*                  blocked   idle   0/lxd/2  10.0.0.20       5000/tcp           Incomplete relations: database, Allowed_units list provided but this unit not present
  keystone-mysql-router/0*   waiting   idle            10.0.0.20                          'db-router' incomplete, MySQL Router not yet bootstrapped
mysql-innodb-cluster/0       error     idle   0/lxd/3  10.0.0.19                          hook failed: "install"

I have to wait???see too many waiting

[routergod]

@riccardo-magrini it appears you have resolved or worked-around the issue you were having with your virtual disk ids and now you have a separate problem with your Mysql8 cluster. The nature of the error is not clear without some logs.

Please try to identify the error from the debug log

juju debug-log --include mysql-innodb-cluster

It might take a while for the charm to wake up and do something if stuck in “error, idle”, you might want to give it a kick like this;

juju resolve mysql-innodb-cluster/0

I hope you find some clues!

[riccardo-magrini]

after 3hrs the situation of the deploy is that:

$:juju status
Model      Controller             Cloud/Region        Version  SLA          Timestamp
openstack  maas-cloud-controller  maas-cloud/default  2.8.1    unsupported  17:36:34Z
App                     Version  Status   Scale  Charm                   Store       Rev  OS      Notes
ceph-mon                15.2.3   blocked      3  ceph-mon                jujucharms   49  ubuntu  
ceph-osd                15.2.3   active       3  ceph-osd                jujucharms  304  ubuntu  
ceph-radosgw            15.2.3   waiting      1  ceph-radosgw            jujucharms  290  ubuntu  
cinder                  16.1.0   active       1  cinder                  jujucharms  304  ubuntu  
cinder-ceph             16.1.0   active       1  cinder-ceph             jujucharms  257  ubuntu  
cinder-mysql-router     8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
dashboard-mysql-router  8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
glance                  20.0.0   active       1  glance                  jujucharms  298  ubuntu  
glance-mysql-router     8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
keystone                17.0.0   active       1  keystone                jujucharms  317  ubuntu  
keystone-mysql-router   8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
mysql-innodb-cluster    8.0.21   active       3  mysql-innodb-cluster    jujucharms    1  ubuntu  
neutron-api             16.0.0   active       1  neutron-api             jujucharms  287  ubuntu  
neutron-api-plugin-ovn  16.0.0   waiting      1  neutron-api-plugin-ovn  jujucharms    1  ubuntu  
neutron-mysql-router    8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
nova-cloud-controller   21.0.0   active       1  nova-cloud-controller   jujucharms  346  ubuntu  
nova-compute            21.0.0   active       3  nova-compute            jujucharms  319  ubuntu  
nova-mysql-router       8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
ntp                     3.5      active       3  ntp                     jujucharms   41  ubuntu  
openstack-dashboard     18.3.2   active       1  openstack-dashboard     jujucharms  305  ubuntu  
ovn-central             20.03.0  waiting      3  ovn-central             jujucharms    1  ubuntu  
ovn-chassis             20.03.0  waiting      3  ovn-chassis             jujucharms    3  ubuntu  
placement               3.0.0    active       1  placement               jujucharms   12  ubuntu  
placement-mysql-router  8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  
rabbitmq-server         3.8.2    active       1  rabbitmq-server         jujucharms  104  ubuntu  
vault                   1.1.1    blocked      1  vault                   jujucharms   40  ubuntu  
vault-mysql-router      8.0.21   active       1  mysql-router            jujucharms    3  ubuntu  

Unit                         Workload  Agent  Machine  Public address  Ports              Message
ceph-mon/0*                  blocked   idle   0/lxd/0  10.0.0.86                          Unit not clustered (no quorum)
ceph-mon/1                   active    idle   1/lxd/0  10.0.0.96                          Unit is ready and clustered
ceph-mon/2                   active    idle   2/lxd/0  10.0.0.89                          Unit is ready and clustered
ceph-osd/0                   active    idle   0        10.0.0.22                          Unit is ready (1 OSD)
ceph-osd/1                   active    idle   1        10.0.0.23                          Unit is ready (1 OSD)
ceph-osd/2*                  active    idle   2        10.0.0.21                          Unit is ready (1 OSD)
ceph-radosgw/0*              waiting   idle   0/lxd/1  10.0.0.88       80/tcp             Incomplete relations: mon
cinder/0*                    active    idle   1/lxd/1  10.0.0.97       8776/tcp           Unit is ready
  cinder-ceph/0*             active    idle            10.0.0.97                          Unit is ready
  cinder-mysql-router/0*     active    idle            10.0.0.97                          Unit is ready
glance/0*                    active    idle   2/lxd/1  10.0.0.93       9292/tcp           Unit is ready
  glance-mysql-router/0*     active    idle            10.0.0.93                          Unit is ready
keystone/0*                  active    idle   0/lxd/2  10.0.0.84       5000/tcp           Unit is ready
  keystone-mysql-router/0*   active    idle            10.0.0.84                          Unit is ready
mysql-innodb-cluster/0       active    idle   0/lxd/3  10.0.0.98                          Unit is ready: Mode: R/O
mysql-innodb-cluster/1*      active    idle   1/lxd/2  10.0.0.100                         Unit is ready: Mode: R/O
mysql-innodb-cluster/2       active    idle   2/lxd/2  10.0.0.99                          Unit is ready: Mode: R/W
neutron-api/0*               active    idle   1/lxd/3  10.0.0.92       9696/tcp           Unit is ready
  neutron-api-plugin-ovn/0*  waiting   idle            10.0.0.92                          'certificates' awaiting server certificate data, 'ovsdb-cms' incomplete
  neutron-mysql-router/0*    active    idle            10.0.0.92                          Unit is ready
nova-cloud-controller/0*     active    idle   0/lxd/4  10.0.0.102      8774/tcp,8775/tcp  Unit is ready
  nova-mysql-router/0*       active    idle            10.0.0.102                         Unit is ready
nova-compute/0               active    idle   0        10.0.0.22                          Unit is ready
  ntp/0*                     active    idle            10.0.0.22       123/udp            chrony: Ready
  ovn-chassis/0*             waiting   idle            10.0.0.22                          'certificates' awaiting server certificate data, 'ovsdb' incomplete
nova-compute/1               active    idle   1        10.0.0.23                          Unit is ready
  ntp/2                      active    idle            10.0.0.23       123/udp            chrony: Ready
  ovn-chassis/2              waiting   idle            10.0.0.23                          'certificates' awaiting server certificate data, 'ovsdb' incomplete
nova-compute/2*              active    idle   2        10.0.0.21                          Unit is ready
  ntp/1                      active    idle            10.0.0.21       123/udp            chrony: Ready
  ovn-chassis/1              waiting   idle            10.0.0.21                          'certificates' awaiting server certificate data, 'ovsdb' incomplete
openstack-dashboard/0*       active    idle   1/lxd/4  10.0.0.91       80/tcp,443/tcp     Unit is ready
  dashboard-mysql-router/0*  active    idle            10.0.0.91                          Unit is ready
ovn-central/0                waiting   idle   0/lxd/5  10.0.0.94       6641/tcp,6642/tcp  'ovsdb-peer' incomplete, 'certificates' awaiting server certificate data
ovn-central/1*               waiting   idle   1/lxd/5  10.0.0.90       6641/tcp,6642/tcp  'ovsdb-peer' incomplete, 'certificates' awaiting server certificate data
ovn-central/2                waiting   idle   2/lxd/3  10.0.0.87       6641/tcp,6642/tcp  'ovsdb-peer' incomplete, 'certificates' awaiting server certificate data
placement/0*                 active    idle   2/lxd/4  10.0.0.85       8778/tcp           Unit is ready
  placement-mysql-router/0*  active    idle            10.0.0.85                          Unit is ready
rabbitmq-server/0*           active    idle   2/lxd/5  10.0.0.101      5672/tcp           Unit is ready
vault/0*                     blocked   idle   0/lxd/6  10.0.0.95       8200/tcp           Vault needs to be initialized
  vault-mysql-router/0*      active    idle            10.0.0.95                          Unit is ready

Machine  State    DNS         Inst id              Series  AZ       Message
0        started  10.0.0.22   openstack02          focal   default  Deployed
0/lxd/0  started  10.0.0.86   juju-ab0a09-0-lxd-0  focal   default  Container started
0/lxd/1  started  10.0.0.88   juju-ab0a09-0-lxd-1  focal   default  Container started
0/lxd/2  started  10.0.0.84   juju-ab0a09-0-lxd-2  focal   default  Container started
0/lxd/3  started  10.0.0.98   juju-ab0a09-0-lxd-3  focal   default  Container started
0/lxd/4  started  10.0.0.102  juju-ab0a09-0-lxd-4  focal   default  Container started
0/lxd/5  started  10.0.0.94   juju-ab0a09-0-lxd-5  focal   default  Container started
0/lxd/6  started  10.0.0.95   juju-ab0a09-0-lxd-6  focal   default  Container started
1        started  10.0.0.23   openstack03          focal   default  Deployed
1/lxd/0  started  10.0.0.96   juju-ab0a09-1-lxd-0  focal   default  Container started
1/lxd/1  started  10.0.0.97   juju-ab0a09-1-lxd-1  focal   default  Container started
1/lxd/2  started  10.0.0.100  juju-ab0a09-1-lxd-2  focal   default  Container started
1/lxd/3  started  10.0.0.92   juju-ab0a09-1-lxd-3  focal   default  Container started
1/lxd/4  started  10.0.0.91   juju-ab0a09-1-lxd-4  focal   default  Container started
1/lxd/5  started  10.0.0.90   juju-ab0a09-1-lxd-5  focal   default  Container started
2        started  10.0.0.21   openstack01          focal   default  Deployed
2/lxd/0  started  10.0.0.89   juju-ab0a09-2-lxd-0  focal   default  Container started
2/lxd/1  started  10.0.0.93   juju-ab0a09-2-lxd-1  focal   default  Container started
2/lxd/2  started  10.0.0.99   juju-ab0a09-2-lxd-2  focal   default  Container started
2/lxd/3  started  10.0.0.87   juju-ab0a09-2-lxd-3  focal   default  Container started
2/lxd/4  started  10.0.0.85   juju-ab0a09-2-lxd-4  focal   default  Container started
2/lxd/5  started  10.0.0.101  juju-ab0a09-2-lxd-5  focal   default  Container started

there are also some “waiting” and a stopped…

anyway. opening Horizon the gui is ok

[routergod]

Looks ok, you just need to initialize vault to complete the certificates relations, see https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/latest/app-vault.html