[SOLVED] Help with vault hacluster deployment

Hi,

I have a problem with vault - hacluster deployment, in the docs there is a relation for ha and yes I can deploy it without any errors, but VIP is not setup, neither there are any resources defined when doing crm status. Could someone clear it up for me a bit, how it should work, whether it’s active + standby, should I expect VIP and some resources been defined in corosync/pacemaker, there is very little documentation in charm store.

My test config, please note it’s a cut-down test version of my openstack deployment just so I can narrow down issues quicker, hence the keystone and placement only as other components.

  keystone-vip:        &keystone-vip        "10.10.20.203 10.10.30.203 10.10.40.203"
  vault-vip:   &vault-vip   "10.10.20.207"
  placement-vip:   &placement-vip   "10.10.20.208"

machines:
  '0':
    constraints: 'zones=az-01 tags=infra'
    series: focal
  '1':
    constraints: 'zones=az-02 tags=infra'
    series: focal
  '2':
    constraints: 'zones=az-03 tags=infra'
    series: focal
applications:
  mysql-innodb-cluster:
    charm: cs:mysql-innodb-cluster
    num_units: 3
    bindings:
      "": *oam-space
      db-router: *internal-space
      cluster: *internal-space
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    
  keystone-mysql-router:
    charm: cs:mysql-router
    bindings:
      "": *oam-space
      db-router: *internal-space
      shared-db: *internal-space
  keystone:
    charm: cs:keystone
    num_units: 3
    bindings:
      "": *oam-space
      public: *public-space
      admin: *admin-space
      internal: *internal-space
      shared-db: *internal-space
      ha: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      vip: *keystone-vip
      preferred-api-version: 3
    to:
    - lxd:0
    - lxd:1
    - lxd:2
  hacluster-keystone:
    charm: cs:hacluster
    bindings:
      "": *oam-space
      ha: *internal-space
    options:
      cluster_count: 3
  placement-mysql-router:
    charm: cs:mysql-router
    bindings:
      "": *oam-space
      db-router: *internal-space
      shared-db: *internal-space
  placement:
    charm: cs:placement
    num_units: 3
    bindings:
      "": *oam-space
      shared-db: *internal-space
      ha: *internal-space
    options:
      worker-multiplier: *worker-multiplier
      openstack-origin: *openstack-origin
      vip: *placement-vip
    to:
    - lxd:0
    - lxd:1
    - lxd:2
  hacluster-placement:
    charm: cs:hacluster
    bindings:
      "": *oam-space
      ha: *internal-space
    options:
      cluster_count: 3
      
  vault-mysql-router:
    charm: cs:mysql-router
    bindings:
      "": *oam-space
      db-router: *internal-space
      shared-db: *internal-space
  vault:
    charm: cs:vault
    num_units: 3
    bindings:
      "": *oam-space
      shared-db: *internal-space
      ha: *internal-space
    options:
      auto-generate-root-ca-cert: true
      vip: &vault-vip
    to:
    - lxd:0
    - lxd:1
    - lxd:2
  hacluster-vault:
    charm: cs:hacluster
    bindings:
      "": *oam-space
      ha: *internal-space
    options:
      cluster_count: 3
      
relations:
  - [ keystone:ha, hacluster-keystone:ha ]
  - [ keystone:shared-db, keystone-mysql-router:shared-db ]
  - [ keystone-mysql-router:db-router, mysql-innodb-cluster:db-router ]
  - [ vault:certificates, keystone:certificates ]
  - [ vault:ha, hacluster-vault:ha ]
  - [ vault:shared-db, vault-mysql-router:shared-db ]
  - [ vault-mysql-router:db-router, mysql-innodb-cluster:db-router ]
  - [ placement-mysql-router:db-router, mysql-innodb-cluster:db-router ]
  - [ placement:shared-db, placement-mysql-router:shared-db ]
  - [ placement, keystone ]
  - [ placement:ha, hacluster-placement:ha ]
  - [ vault:certificates, placement:certificates ]



juju status
Model      Controller       Cloud/Region      Version  SLA          Timestamp
ceph-demo  maas-controller  lab-maas/default  2.8.0    unsupported  18:11:35Z

App                     Version  Status   Scale  Charm                 Store       Rev  OS      Notes
hacluster-keystone               active       3  hacluster             jujucharms   68  ubuntu  
hacluster-placement              waiting      3  hacluster             jujucharms   68  ubuntu  
hacluster-vault                  active       3  hacluster             jujucharms   68  ubuntu  
keystone                17.0.0   active       3  keystone              jujucharms  316  ubuntu  
keystone-mysql-router   8.0.20   active       3  mysql-router          jujucharms    2  ubuntu  
mysql-innodb-cluster    8.0.20   active       3  mysql-innodb-cluster  jujucharms    0  ubuntu  
placement               3.0.0    error        3  placement             jujucharms   11  ubuntu  
placement-mysql-router  8.0.20   waiting      3  mysql-router          jujucharms    2  ubuntu  
vault                   1.1.1    blocked      3  vault                 jujucharms   39  ubuntu  
vault-mysql-router      8.0.20   active       3  mysql-router          jujucharms    2  ubuntu  

Unit                         Workload  Agent  Machine   Public address  Ports     Message
keystone/15                  active    idle   39/lxd/0  10.10.20.15     5000/tcp  Unit is ready
  hacluster-keystone/13      active    idle             10.10.20.15               Unit is ready and clustered
  keystone-mysql-router/7    active    idle             10.10.20.15               Unit is ready
keystone/16                  active    idle   40/lxd/0  10.10.20.14     5000/tcp  Unit is ready
  hacluster-keystone/14      active    idle             10.10.20.14               Unit is ready and clustered
  keystone-mysql-router/8    active    idle             10.10.20.14               Unit is ready
keystone/17*                 active    idle   41/lxd/0  10.10.20.7      5000/tcp  Unit is ready
  hacluster-keystone/12*     active    idle             10.10.20.7                Unit is ready and clustered
  keystone-mysql-router/6*   active    idle             10.10.20.7                Unit is ready
mysql-innodb-cluster/9       active    idle   39/lxd/1  10.10.20.11               Unit is ready: Mode: R/O
mysql-innodb-cluster/10      active    idle   40/lxd/1  10.10.20.12               Unit is ready: Mode: R/O
mysql-innodb-cluster/11*     active    idle   41/lxd/1  10.10.20.6                Unit is ready: Mode: R/W
placement/9                  error     idle   39/lxd/2  10.10.20.13     8778/tcp  hook failed: "identity-service-relation-joined"
  hacluster-placement/10     active    idle             10.10.20.13               Unit is ready and clustered
  placement-mysql-router/4   waiting   idle             10.10.20.13               'shared-db' incomplete, Waiting for proxied DB creation from cluster
placement/10                 error     idle   40/lxd/2  10.10.20.8      8778/tcp  hook failed: "identity-service-relation-joined"
  hacluster-placement/11     waiting   idle             10.10.20.8                Resource: res_placement_9dbe3ee_vip not yet configured
  placement-mysql-router/5   active    idle             10.10.20.8                Unit is ready
placement/11*                error     idle   41/lxd/2  10.10.20.5      8778/tcp  hook failed: "identity-service-relation-joined"
  hacluster-placement/9*     active    idle             10.10.20.5                Unit is ready and clustered
  placement-mysql-router/3*  waiting   idle             10.10.20.5                'shared-db' incomplete, Waiting for proxied DB creation from cluster
vault/9                      blocked   idle   39/lxd/3  10.10.20.9      8200/tcp  Vault needs to be initialized
  hacluster-vault/10         active    idle             10.10.20.9                Unit is ready and clustered
  vault-mysql-router/4       active    idle             10.10.20.9                Unit is ready
vault/10                     blocked   idle   40/lxd/3  10.10.20.10     8200/tcp  Vault needs to be initialized
  hacluster-vault/11         active    idle             10.10.20.10               Unit is ready and clustered
  vault-mysql-router/5       active    idle             10.10.20.10               Unit is ready
vault/11*                    blocked   idle   41/lxd/3  10.10.20.4      8200/tcp  Vault needs to be initialized
  hacluster-vault/9*         active    idle             10.10.20.4                Unit is ready and clustered
  vault-mysql-router/3*      active    idle             10.10.20.4                Unit is ready

Machine   State    DNS          Inst id               Series  AZ     Message
39        started  10.10.20.1   lab-03                focal   az-01  Deployed
39/lxd/0  started  10.10.20.15  juju-0ff478-39-lxd-0  focal   az-01  Container started
39/lxd/1  started  10.10.20.11  juju-0ff478-39-lxd-1  focal   az-01  Container started
39/lxd/2  started  10.10.20.13  juju-0ff478-39-lxd-2  focal   az-01  Container started
39/lxd/3  started  10.10.20.9   juju-0ff478-39-lxd-3  focal   az-01  Container started
40        started  10.10.20.2   lab-04                focal   az-02  Deployed
40/lxd/0  started  10.10.20.14  juju-0ff478-40-lxd-0  focal   az-02  Container started
40/lxd/1  started  10.10.20.12  juju-0ff478-40-lxd-1  focal   az-02  Container started
40/lxd/2  started  10.10.20.8   juju-0ff478-40-lxd-2  focal   az-02  Container started
40/lxd/3  started  10.10.20.10  juju-0ff478-40-lxd-3  focal   az-02  Container started
41        started  10.10.20.3   lab-07                focal   az-03  Deployed
41/lxd/0  started  10.10.20.7   juju-0ff478-41-lxd-0  focal   az-03  Container started
41/lxd/1  started  10.10.20.6   juju-0ff478-41-lxd-1  focal   az-03  Container started
41/lxd/2  started  10.10.20.5   juju-0ff478-41-lxd-2  focal   az-03  Container started
41/lxd/3  started  10.10.20.4   juju-0ff478-41-lxd-3  focal   az-03  Container started

sudo crm status
sudo: setrlimit(RLIMIT_CORE): Operation not permitted
Cluster Summary:

  • Stack: corosync
  • Current DC: juju-0ff478-41-lxd-3 (version 2.0.3-4b1f869f0f) - partition with quorum
  • Last updated: Sat Jul 18 17:53:35 2020
  • Last change: Sat Jul 18 17:29:16 2020 by hacluster via crmd on juju-0ff478-40-lxd-3
  • 3 nodes configured
  • 0 resource instances configured

Node List:

  • Online: [ juju-0ff478-39-lxd-3 juju-0ff478-40-lxd-3 juju-0ff478-41-lxd-3 ]

Full List of Resources:

  • No resources

The correct HA implementation is documented on openstack website. What worked for me is:

  etcd:
    charm: cs:etcd
    num_units: 3
    bindings:
      "": *oam-space
      db: *internal-space
    to:
    - lxd:0
    - lxd:1
    - lxd:2
    
  easyrsa:
    charm: cs:~containers/easyrsa-318
    num_units: 1
    bindings:
      "": *oam-space
    to:
    - lxd:0
      
  vault-mysql-router:
    charm: cs:mysql-router
    bindings:
      "": *oam-space
      db-router: *internal-space
      shared-db: *internal-space
  vault:
    charm: cs:vault
    num_units: 3
    bindings:
      "": *oam-space
      shared-db: *internal-space
      ha: *internal-space
    options:
      auto-generate-root-ca-cert: true
      vip: &vault-vip
    to:
    - lxd:0
    - lxd:1
    - lxd:2
  hacluster-vault:
    charm: cs:hacluster
    bindings:
      "": *oam-space
      ha: *internal-space
    options:
      cluster_count: 3