Telco CLoud/EPA : Issues with Cloud init and Self-service

Hello i have issues using Openstack deployed with MaaS/Juju .
My set-up is based on DVR (3 Compute nodes).
Each node are using one Ge interface for management/API, one DPDK bond interface (2*10Ge) and 2 (10Ge) SRIOV interfaces Data-plane.

I’m facing 2 issues :

  1. Cloudinit
  2. Overlay with selfservice network

For the 1st issue, only config-drive works. No responses from Metadata services either from the VM or from the Compute namespace when using cloud-init images (both for self-services or provider networks).

For 2nd issue, only self-service instances spawned on the same compute where the DHCP service resides will have connectivity. Traffic over VXLAN don’t seems to pass between instances on other compute nodes and the compute node where the dhcp service is located.
No network connectivity or MTU issue between Compute Nodes on the Overlay VLAN.
By the way instances on provider networks work fine whatever the compute node is.

Any help or guidance will be appreciated for both issues.
Thanks.

I’m using the following bundle:

xxx:~$ more Juju/openstack-base-65/bundle-dvr-ha-mem.yaml
variables:
  openstack-origin: &openstack-origin cloud:bionic-train
  worker-multiplier: &worker-multiplier 0.25
  osd-devices: &osd-devices /dev/vdb /dev/vdc /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh
  vlan-ranges: &vlan-ranges prov_dpdk:200:300 prov_sriov_1:200:300 prov_sriov_2:200:300
  expected-osd-count: &expected-osd-count 3
  expected-mon-count: &expected-mon-count 3
machines:
  "0":
    series: bionic
    constraints: tags=virtual,ctrl zones=DC1-AZ2
  '1':
    series: bionic
    constraints: tags=compute,dual-core-ht-off,hdd zones=DC1-AZ1
  '2':
    series: bionic
    constraints: tags=virtual,ctrl zones=DC1-AZ3
  '3':
    series: bionic
    constraints: tags=compute,single-core,hdd zones=DC1-AZ2
  '4':
    series: bionic
    constraints: tags=network,single-core,hdd zones=DC1-AZ3
  '5':
    series: bionic
    constraints: tags=virtual,ctrl zones=DC1-AZ3
relations:
- - nova-compute:amqp
  - rabbitmq-server:amqp
- - keystone:shared-db
  - mysql:shared-db
- - nova-cloud-controller:identity-service
  - keystone:identity-service
- - glance:identity-service
  - keystone:identity-service
- - neutron-api:identity-service
  - keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
  - neutron-api:neutron-plugin-api
- - neutron-api:shared-db
  - mysql:shared-db
- - neutron-api:amqp
  - rabbitmq-server:amqp
- - glance:shared-db
  - mysql:shared-db
- - glance:amqp
  - rabbitmq-server:amqp
- - nova-cloud-controller:image-service
  - glance:image-service
- - nova-compute:image-service
  - glance:image-service
- - nova-cloud-controller:cloud-compute
  - nova-compute:cloud-compute
- - nova-cloud-controller:amqp
  - rabbitmq-server:amqp
- - nova-compute:neutron-plugin
  - neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
  - rabbitmq-server:amqp
- - openstack-dashboard:identity-service
  - keystone:identity-service
- - nova-cloud-controller:shared-db
  - mysql:shared-db
- - nova-cloud-controller:neutron-api
  - neutron-api:neutron-api
- - cinder:image-service
  - glance:image-service
- - cinder:amqp
  - rabbitmq-server:amqp
- - cinder:identity-service
  - keystone:identity-service
- - cinder:cinder-volume-service
  - nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
  - cinder:storage-backend
- - ceph-mon:client
  - nova-compute:ceph
- - nova-compute:ceph-access
  - cinder-ceph:ceph-access
- - cinder:shared-db
  - mysql:shared-db
- - ceph-mon:client
  - cinder-ceph:ceph
- - ceph-mon:client
  - glance:ceph
- - ceph-osd:mon
  - ceph-mon:osd
- - ntp:juju-info
  - nova-compute:juju-info
- - ceph-radosgw:mon
  - ceph-mon:radosgw
- - ceph-radosgw:identity-service
  - keystone:identity-service
- - placement
  - mysql
- - placement
  - keystone
- - placement
  - nova-cloud-controller
- - nova-cloud-controller:memcache
  - memcached:cache
series: bionic
services:
  ceph-mon:
    bindings:
      public: storage-data
      cluster: storage-cluster
    annotations:
      gui-x: '750'
      gui-y: '500'
    charm: cs:ceph-mon-44
    num_units: 3
    options:
      expected-osd-count: *expected-osd-count
      monitor-count: *expected-mon-count
      source: *openstack-origin
    to:
    - lxd:1
    - lxd:3
    - lxd:4
  ceph-osd:
    bindings:
      public: storage-data
      cluster: storage-cluster
    annotations:
      gui-x: '1000'
      gui-y: '500'
    charm: cs:ceph-osd-294
    num_units: 3
    #num_units: 4
    options:
      osd-devices: *osd-devices
      source: *openstack-origin
    to:
    - '1'
    - '3'
    - '4'
  ceph-radosgw:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
    annotations:
      gui-x: '1000'
      gui-y: '250'
    charm: cs:ceph-radosgw-283
    num_units: 1
    options:
      source: *openstack-origin
    to:
    - lxd:0
  cinder:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '750'
      gui-y: '0'
    charm: cs:cinder-297
    num_units: 1
    options:
      block-device: None
      glance-api-version: 2
      openstack-origin: *openstack-origin
      worker-multiplier: *worker-multiplier
    to:
    - 0
  cinder-ceph:
    annotations:
      gui-x: '750'
      gui-y: '250'
    charm: cs:cinder-ceph-251
    num_units: 0
    options:
      ceph-osd-replication-count: 3
  glance:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '250'
      gui-y: '0'
    charm: cs:glance-291
    num_units: 1
    options:
      openstack-origin: *openstack-origin
      worker-multiplier: *worker-multiplier
    to:
    - 5
  keystone:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '500'
      gui-y: '0'
    charm: cs:keystone-309
    num_units: 1
    options:
      openstack-origin: *openstack-origin
      worker-multiplier: *worker-multiplier
    to:
    - lxd:2
  mysql:
    bindings:
      shared-db: internal-api
    annotations:
      gui-x: '0'
      gui-y: '250'
    charm: cs:percona-cluster-281
    num_units: 1
    options:
      innodb-buffer-pool-size: 256M
      max-connections: 1000
      performance-schema: true
    to:
    - lxd:5
  neutron-api:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '500'
      gui-y: '500'
    charm: cs:neutron-api-282
    num_units: 1
    options:
      flat-network-providers: prov_dpdk prov_sriov_1 prov_sriov_2
      global-physnet-mtu: 9000
      neutron-security-groups: true
      overlay-network-type: vxlan
      enable-dvr: true
      enable-l3ha: true
      max-l3-agents-per-router : 2
      enable-sriov: true
      openstack-origin: *openstack-origin
      vlan-ranges: *vlan-ranges
      worker-multiplier: *worker-multiplier
    to:
    - lxd:4
  neutron-openvswitch:
    bindings:
      data: compute-data
    annotations:
      gui-x: '250'
      gui-y: '500'
    charm: cs:neutron-openvswitch-269
    num_units: 0
    options:
      bridge-mappings: prov_dpdk:br-data
      data-port: br-data:bond-dpdk
      use-dvr-snat: true
      enable-local-dhcp-and-metadata: true
      enable-dpdk: true
      dpdk-socket-cores: 1
      dpdk-socket-memory: 1024
      dpdk-driver: vfio-pci
      dpdk-bond-mappings: "bond-dpdk:e4:43:4b:b6:f5:dc bond-dpdk:f8:f2:1e:82:2a:80 bond-dpdk:e4:43:4b:ab:65:76 bond-dpdk:f8:f2:1e:82:31:80 bond-dpdk:e4:43:4b:b6:7e:ba bond-dpdk:f8:f2:1e:82:
31:70 bond-dpdk:e4:43:4b:ab:27:74 bond-dpdk:f8:f2:1e:82:32:30"
      dpdk-bond-config: ":balance-tcp:active:fast"
      enable-sriov: true
      sriov-device-mappings: "prov_sriov_1:eno2 prov_sriov_2:enp179s0f1 prov_sriov_2:enp94s0f1"
      sriov-numvfs: "eno2:16 enp179s0f1:16 enp94s0f1:16"
  nova-cloud-controller:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '0'
      gui-y: '500'
    charm: cs:nova-cloud-controller-339
    num_units: 1
    options:
      network-manager: Neutron
      console-access-protocol: vnc
      openstack-origin: *openstack-origin
      worker-multiplier: *worker-multiplier
    to:
    - 2
  nova-compute:
    annotations:
      gui-x: '250'
      gui-y: '250'
    charm: cs:nova-compute-309
    num_units: 3
    options:
      config-flags: default_ephemeral_format=ext4
      enable-live-migration: true
      enable-resize: true
      migration-auth-type: ssh
      cpu-mode: host-passthrough
      reserved-host-memory: 2048
      pci-passthrough-whitelist: '[{"devname":"eno2", "physical_network":"prov_sriov_1"},{"devname":"enp94s0f1","physical_network":"prov_sriov_2"},{"devname":"enp179s0f1", "physical_network
":"prov_sriov_2"},{"address":"*:17:00.01", "physical_network":"prov_sriov_1"},{"address":"*:18:00.01", "physical_network":"prov_sriov_1"},{"address":"*:b3:00.01", "physical_network":"prov_s
riov_2"},{"address":"*:5e:00.01", "physical_network":"prov_sriov_2"}]'
      openstack-origin: *openstack-origin
    to:
    - '1'
    - '3'
    - '4'
  ntp:
    annotations:
      gui-x: '1000'
      gui-y: '0'
    charm: cs:ntp-36
    num_units: 0
  openstack-dashboard:
    bindings:
      public: public-api
    annotations:
      gui-x: '500'
      gui-y: '-250'
    charm: cs:openstack-dashboard-297
    num_units: 1
    options:
      openstack-origin: *openstack-origin
    to:
    - lxd:0
  placement:
    bindings:
      public: public-api
      internal: internal-api
      admin: admin-api
      shared-db: internal-api
    annotations:
      gui-x: '0'
      gui-y: '500'
    charm: cs:~openstack-charmers/bionic/placement
    num_units: 1
    options:
      openstack-origin: *openstack-origin
      worker-multiplier: *worker-multiplier
    to:
    - lxd:5
  rabbitmq-server:
    bindings:
      amqp: internal-api
    annotations:
      gui-x: '500'
      gui-y: '250'
    charm: cs:rabbitmq-server-97
    num_units: 1
    to:
    - lxd:3
  memcached:
    bindings:
      "": internal-api
    charm: cs:memcached
    num_units: 1
    to:
    - lxd:2

Juju status:

   xxx:~$ juju status
    Model    Controller     Cloud/Region      Version  SLA          Timestamp
    default  maas-ctrl-dc1  maas-tdf/default  2.7.6    unsupported  10:35:43+02:00

    App                    Version  Status  Scale  Charm                  Store       Rev  OS      Notes
    ceph-mon               14.2.11  active      3  ceph-mon               jujucharms   44  ubuntu
    ceph-osd               14.2.11  active      3  ceph-osd               jujucharms  294  ubuntu
    ceph-radosgw           14.2.11  active      1  ceph-radosgw           jujucharms  283  ubuntu
    cinder                 15.3.0   active      1  cinder                 jujucharms  297  ubuntu
    cinder-ceph            15.3.0   active      1  cinder-ceph            jujucharms  251  ubuntu
    glance                 19.0.4   active      1  glance                 jujucharms  291  ubuntu
    keystone               16.0.1   active      1  keystone               jujucharms  309  ubuntu
    memcached                       active      1  memcached              jujucharms   30  ubuntu
    mysql                  5.7.20   active      1  percona-cluster        jujucharms  281  ubuntu
    neutron-api            15.1.0   active      1  neutron-api            jujucharms  282  ubuntu
    neutron-openvswitch    15.1.0   active      3  neutron-openvswitch    jujucharms  269  ubuntu
    nova-cloud-controller  20.3.0   active      1  nova-cloud-controller  jujucharms  339  ubuntu
    nova-compute           20.3.0   active      3  nova-compute           jujucharms  309  ubuntu
    ntp                    3.2      active      3  ntp                    jujucharms   36  ubuntu
    openstack-dashboard    16.2.0   active      1  openstack-dashboard    jujucharms  297  ubuntu
    placement              2.0.0    active      1  placement              jujucharms   15  ubuntu
    rabbitmq-server        3.6.10   active      1  rabbitmq-server        jujucharms   97  ubuntu

    Unit                      Workload  Agent  Machine  Public address  Ports              Message
    ceph-mon/0*               active    idle   1/lxd/0  10.40.0.210                        Unit is ready and clustered
    ceph-mon/1                active    idle   3/lxd/0  10.40.0.207                        Unit is ready and clustered
    ceph-mon/2                active    idle   4/lxd/0  10.40.0.203                        Unit is ready and clustered
    ceph-osd/0                active    idle   1        10.10.0.204                        Unit is ready (8 OSD)
    ceph-osd/1                active    idle   3        10.10.0.205                        Unit is ready (4 OSD)
    ceph-osd/2*               active    idle   4        10.10.0.206                        Unit is ready (4 OSD)
    ceph-radosgw/0*           active    idle   0/lxd/0  10.10.0.209     80/tcp             Unit is ready
    cinder/0*                 active    idle   0        10.10.0.212     8776/tcp           Unit is ready
      cinder-ceph/0*          active    idle            10.10.0.212                        Unit is ready
    glance/0*                 active    idle   5        10.10.0.214     9292/tcp           Unit is ready
    keystone/0*               active    idle   2/lxd/0  10.10.0.243     5000/tcp           Unit is ready
    memcached/0*              active    idle   2/lxd/1  10.20.0.219     11211/tcp          Unit is ready
    mysql/0*                  active    idle   5/lxd/0  10.20.0.218     3306/tcp           Unit is ready
    neutron-api/0*            active    idle   4/lxd/1  10.10.0.242     9696/tcp           Unit is ready
    nova-cloud-controller/0*  active    idle   2        10.10.0.213     8774/tcp,8775/tcp  Unit is ready
    nova-compute/0            active    idle   1        10.10.0.204                        Unit is ready
      neutron-openvswitch/0*  active    idle            10.10.0.204                        Unit is ready
      ntp/0*                  active    idle            10.10.0.204     123/udp            chrony: Ready
    nova-compute/1            active    idle   3        10.10.0.205                        Unit is ready
      neutron-openvswitch/2   active    idle            10.10.0.205                        Unit is ready
      ntp/1                   active    idle            10.10.0.205     123/udp            chrony: Ready
    nova-compute/2*           active    idle   4        10.10.0.206                        Unit is ready
      neutron-openvswitch/1   active    idle            10.10.0.206                        Unit is ready
      ntp/2                   active    idle            10.10.0.206     123/udp            chrony: Ready
    openstack-dashboard/0*    active    idle   0/lxd/1  10.30.0.209     80/tcp,443/tcp     Unit is ready
    placement/0*              active    idle   5/lxd/1  10.10.0.252     8778/tcp           Unit is ready
    rabbitmq-server/0*        active    idle   3/lxd/1  10.20.0.221     5672/tcp           Unit is ready

    Machine  State    DNS          Inst id              Series  AZ       Message
    0        started  10.10.0.212  cp1-2                bionic  DC1-AZ2  Deployed
    0/lxd/0  started  10.10.0.209  juju-2a779e-0-lxd-0  bionic  DC1-AZ2  Container started
    0/lxd/1  started  10.30.0.209  juju-2a779e-0-lxd-1  bionic  DC1-AZ2  Container started
    1        started  10.10.0.204  copernic             bionic  DC1-AZ1  Deployed
    1/lxd/0  started  10.40.0.210  juju-2a779e-1-lxd-0  bionic  DC1-AZ1  Container started
    2        started  10.10.0.213  cp1-3                bionic  DC1-AZ3  Deployed
    2/lxd/0  started  10.10.0.243  juju-2a779e-2-lxd-0  bionic  DC1-AZ3  Container started
    2/lxd/1  started  10.20.0.219  juju-2a779e-2-lxd-1  bionic  DC1-AZ3  Container started
    3        started  10.10.0.205  sirius               bionic  DC1-AZ2  Deployed
    3/lxd/0  started  10.40.0.207  juju-2a779e-3-lxd-0  bionic  DC1-AZ2  Container started
    3/lxd/1  started  10.20.0.221  juju-2a779e-3-lxd-1  bionic  DC1-AZ2  Container started
    4        started  10.10.0.206  polaris              bionic  DC1-AZ3  Deployed
    4/lxd/0  started  10.40.0.203  juju-2a779e-4-lxd-0  bionic  DC1-AZ3  Container started
    4/lxd/1  started  10.10.0.242  juju-2a779e-4-lxd-1  bionic  DC1-AZ3  Container started
    5        started  10.10.0.214  cp1-4                bionic  DC1-AZ3  Deployed
    5/lxd/0  started  10.20.0.218  juju-2a779e-5-lxd-0  bionic  DC1-AZ3  Container started
    5/lxd/1  started  10.10.0.252  juju-2a779e-5-lxd-1  bionic  DC1-AZ3  Container started

–>Nova services

xxx:~$ openstack compute service list
+----+----------------+-----------------------+----------+---------+-------+----------------------------+
| ID | Binary         | Host                  | Zone     | Status  | State | Updated At                 |
+----+----------------+-----------------------+----------+---------+-------+----------------------------+
|  1 | nova-conductor | cp1-3                 | internal | enabled | up    | 2020-10-05T10:30:55.000000 |
|  2 | nova-scheduler | cp1-3                 | internal | enabled | up    | 2020-10-05T10:30:47.000000 |
|  4 | nova-compute   | eno4.10.polaris.maas  | nfv2     | enabled | up    | 2020-10-05T10:30:55.000000 |
|  6 | nova-compute   | eno4.10.copernic.maas | nfv1     | enabled | up    | 2020-10-05T10:30:55.000000 |
|  7 | nova-compute   | eno4.10.sirius.maas   | nfv2     | enabled | up    | 2020-10-05T10:30:55.000000 |
+----+----------------+-----------------------+----------+---------+-------+----------------------------+
xxx:~$ openstack host list
+-----------------------+-----------+----------+
| Host Name             | Service   | Zone     |
+-----------------------+-----------+----------+
| cp1-3                 | conductor | internal |
| cp1-3                 | scheduler | internal |
| eno4.10.polaris.maas  | compute   | nfv2     |
| eno4.10.copernic.maas | compute   | nfv1     |
| eno4.10.sirius.maas   | compute   | nfv2     |
+-----------------------+-----------+----------+
xxx:~$ openstack hypervisor list
+----+---------------------+-----------------+-------------+-------+
| ID | Hypervisor Hostname | Hypervisor Type | Host IP     | State |
+----+---------------------+-----------------+-------------+-------+
|  1 | polaris.maas        | QEMU            | 10.10.0.206 | up    |
|  2 | copernic.maas       | QEMU            | 10.10.0.204 | up    |
|  3 | sirius.maas         | QEMU            | 10.10.0.205 | up    |
+----+---------------------+-----------------+-------------+-------+

   xxx:~$ openstack network list --long
+--------------------------------------+----------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+
| ID                                   | Name           | Status | Project                          | State | Shared | Subnets                              | Network Type | Router Type | Availability Zones | Tags |
+--------------------------------------+----------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+
| 0141d8e6-4afe-4654-b310-5122b09425da | ext_net_10.200 | ACTIVE | 0abf6175646545ec97732059317d3d31 | UP    | True   | 5c8b9870-72a3-4b6b-b4d7-0b2406f21520 | None         | External    | nova               |      |
| 18faf4d6-ed38-4bca-8be7-b2ac162da789 | int_net_172.16 | ACTIVE | 7ab86a607037437488805b599bec396c | UP    | False  | f5c65fc6-7365-452a-b1d4-9eefb3aad4aa | None         | Internal    | nova               |      |
| 69700bed-fa39-4c55-bf8c-e8d003db7314 | int_net_10.220 | ACTIVE | 0abf6175646545ec97732059317d3d31 | UP    | True   | 3bb33507-dbc5-480c-b400-fd87103a576a | None         | Internal    | nova               |      |
| b47b7d33-4349-4e6d-b452-2cfe51e2087c | int_net_10.210 | ACTIVE | 0abf6175646545ec97732059317d3d31 | UP    | True   | f378fe16-fd98-4975-ae03-5c395be9601e | None         | Internal    | nova               |      |
| bdeef42f-30d1-40db-8977-7141d5dc7e3c | int_net_10.220 | ACTIVE | 0abf6175646545ec97732059317d3d31 | UP    | True   | 93f7dd98-0796-4aef-ac4d-9abb85504d05 | None         | Internal    | nova               |      |
+--------------------------------------+----------------+--------+----------------------------------+-------+--------+--------------------------------------+--------------+-------------+--------------------+------+
xxx:~$ openstack network list
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| ID                                   | Name                                               | Subnets                              |
+--------------------------------------+----------------------------------------------------+--------------------------------------+
| 0141d8e6-4afe-4654-b310-5122b09425da | ext_net_10.200                                     | 5c8b9870-72a3-4b6b-b4d7-0b2406f21520 |
| 18faf4d6-ed38-4bca-8be7-b2ac162da789 | int_net_172.16                                     | f5c65fc6-7365-452a-b1d4-9eefb3aad4aa |
| 69700bed-fa39-4c55-bf8c-e8d003db7314 | int_net_10.220                                     | 3bb33507-dbc5-480c-b400-fd87103a576a |
| 764c6d0b-0f52-4622-89f9-4712890db30e | HA network tenant 7ab86a607037437488805b599bec396c | 87d190da-ee90-4ffa-a667-4b0291032880 |
| b47b7d33-4349-4e6d-b452-2cfe51e2087c | int_net_10.210                                     | f378fe16-fd98-4975-ae03-5c395be9601e |
| bdeef42f-30d1-40db-8977-7141d5dc7e3c | int_net_10.220                                     | 93f7dd98-0796-4aef-ac4d-9abb85504d05 |
+--------------------------------------+----------------------------------------------------+--------------------------------------+

xxx:~$ openstack network segment list
+--------------------------------------+------+--------------------------------------+--------------+---------+
| ID                                   | Name | Network                              | Network Type | Segment |
+--------------------------------------+------+--------------------------------------+--------------+---------+
| 18bd7b1a-c5fd-4aac-87a7-40953aab3628 | None | 0141d8e6-4afe-4654-b310-5122b09425da | vlan         |     200 |
| 4076e1bd-ea85-4643-a529-0850d80a4322 | None | b47b7d33-4349-4e6d-b452-2cfe51e2087c | vlan         |     210 |
| 63a4cd92-ebbc-4869-a2f4-053bd85983cd | None | bdeef42f-30d1-40db-8977-7141d5dc7e3c | vlan         |     220 |
| 8cf09acb-6670-4df9-bb4a-e3ba2c6d079d | None | 18faf4d6-ed38-4bca-8be7-b2ac162da789 | vxlan        |    1001 |
| 92534297-1df8-4322-853d-f3d15a57c57d | None | 69700bed-fa39-4c55-bf8c-e8d003db7314 | vlan         |     220 |
| ad5a7259-4e72-4305-a378-c73a0f62d8ad | None | 764c6d0b-0f52-4622-89f9-4712890db30e | vxlan        |    1002 |
+--------------------------------------+------+--------------------------------------+--------------+---------+

–>Compute/0

   xxx:~$ juju ssh nova-compute/0
    ubuntu@copernic:~$ ip ro
    default via 10.14.0.254 dev eno3 proto static
    10.10.0.0/24 dev eno4.10 proto kernel scope link src 10.10.0.204
    10.14.0.0/24 dev eno3 proto kernel scope link src 10.14.0.204
    10.20.0.0/24 dev eno4.20 proto kernel scope link src 10.20.0.204
    10.30.0.0/24 dev eno4.30 proto kernel scope link src 10.30.0.204
    10.40.0.0/24 dev br-eno4-40 proto kernel scope link src 10.40.0.204
    10.50.0.0/24 dev br-eno4-50 proto kernel scope link src 10.50.0.204
    10.100.0.0/24 dev vif100 proto kernel scope link src 10.100.0.204
    10.200.0.0/24 dev vif200 proto kernel scope link src 10.200.0.204
    10.206.182.0/24 dev lxdbr0 proto kernel scope link src 10.206.182.1
    10.210.0.0/24 dev vif210 proto kernel scope link src 10.210.0.204
    10.220.0.0/24 dev vif220 proto kernel scope link src 10.220.0.204

    ubuntu@copernic:~$ sudo ovs-vsctl show
    00eebfb8-68c0-4c03-8573-732e3f9f2561
    Manager "ptcp:6640:127.0.0.1"
        is_connected: true
    Bridge br-data
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: netdev
        Port br-data
            Interface br-data
                type: internal
        Port "vif200"
            tag: 200
            Interface "vif200"
                type: internal
        Port "vif220"
            tag: 220
            Interface "vif220"
                type: internal
        Port "vif100"
            tag: 100
            Interface "vif100"
                type: internal
        Port phy-br-data
            Interface phy-br-data
                type: patch
                options: {peer=int-br-data}
        Port bond-dpdk
            Interface "dpdk-5fe3176"
                type: dpdk
                options: {dpdk-devargs="0000:18:00.0"}
            Interface "dpdk-0ce9eee"
                type: dpdk
                options: {dpdk-devargs="0000:5e:00.0"}
        Port "vif210"
            tag: 210
            Interface "vif210"
                type: internal
    Bridge br-ex
        datapath_type: netdev
        Port br-ex
            Interface br-ex
                type: internal
    Bridge br-int
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: netdev
        Port "fg-c15ebf73-12"
            tag: 3
            Interface "fg-c15ebf73-12"
                type: internal
        Port "qr-ba93feeb-ed"
            tag: 1
            Interface "qr-ba93feeb-ed"
                type: internal
        Port "vhuafd9edf6-7f"
            tag: 4
            Interface "vhuafd9edf6-7f"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhuafd9edf6-7f"}
        Port "qg-663a571a-e0"
            tag: 3
            Interface "qg-663a571a-e0"
                type: internal
        Port "ha-fd964971-93"
            tag: 2
            Interface "ha-fd964971-93"
                type: internal
        Port "vhu6694bae2-57"
            tag: 4
            Interface "vhu6694bae2-57"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhu6694bae2-57"}
        Port br-int
            Interface br-int
                type: internal
        Port "vhu1ce6fe1d-df"
            tag: 1
            Interface "vhu1ce6fe1d-df"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhu1ce6fe1d-df"}
        Port "tap190d775e-0a"
            tag: 1
            Interface "tap190d775e-0a"
        Port int-br-data
            Interface int-br-data
                type: patch
                options: {peer=phy-br-data}
        Port "vhu650e5335-0a"
            tag: 4
            Interface "vhu650e5335-0a"
                type: dpdkvhostuserclient
                options: {vhost-server-path="/run/libvirt-vhost-user/vhu650e5335-0a"}
        Port "sg-2f7665c6-2f"
            tag: 1
            Interface "sg-2f7665c6-2f"
                type: internal
        Port patch-tun
            Interface patch-tun
                type: patch
                options: {peer=patch-int}
    Bridge br-tun
        Controller "tcp:127.0.0.1:6633"
            is_connected: true
        fail_mode: secure
        datapath_type: netdev
        Port patch-int
            Interface patch-int
                type: patch
                options: {peer=patch-tun}
        Port "vxlan-0a6400ce"
            Interface "vxlan-0a6400ce"
                type: vxlan
                options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.100.0.204", out_key=flow, remote_ip="10.100.0.206"}
        Port "vxlan-0a6400cd"
            Interface "vxlan-0a6400cd"
                type: vxlan
                options: {df_default="true", egress_pkt_mark="0", in_key=flow, local_ip="10.100.0.204", out_key=flow, remote_ip="10.100.0.205"}
        Port br-tun
            Interface br-tun
                type: internal
    ovs_version: "2.12.0"

    ubuntu@copernic:~$ ip netns
    fip-0141d8e6-4afe-4654-b310-5122b09425da (id: 4)
    qrouter-122a9c91-6b53-4453-869e-be3faaf312f5 (id: 3)
    snat-122a9c91-6b53-4453-869e-be3faaf312f5 (id: 2)
    qdhcp-18faf4d6-ed38-4bca-8be7-b2ac162da789 (id: 1) *-->self-service dhcp namespace* 

    ubuntu@copernic:~$ sudo ip netns exec qdhcp-18faf4d6-ed38-4bca-8be7-b2ac162da789 ip a
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
    2: ns-190d775e-0a@if65: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 8950 qdisc noqueue state UP group default qlen 1000
    link/ether fa:16:3e:f0:0a:3d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 169.254.169.254/16 brd 169.254.255.255 scope global ns-190d775e-0a
       valid_lft forever preferred_lft forever
    inet 172.16.0.10/24 brd 172.16.0.255 scope global ns-190d775e-0a
       valid_lft forever preferred_lft forever
    inet6 fe80::f816:3eff:fef0:a3d/64 scope link
       valid_lft forever preferred_lft forever

    Ping compute/1 on vlan 100 (for vxlan overlay)

    ubuntu@copernic:~$ ping -M do -s 8972 10.100.0.205 -I vif100
    PING 10.100.0.205 (10.100.0.205) from 10.100.0.204 vif100: 8972(9000) bytes of data.
    8980 bytes from 10.100.0.205: icmp_seq=1 ttl=63 time=0.668 ms
    8980 bytes from 10.100.0.205: icmp_seq=2 ttl=63 time=0.440 ms

–>Instances (vm-ubuntu-ci-3 is on compute/0(dhcp service), vm-ubuntu-ci-4 on compute/1.

xxx:~$ openstack server list
+--------------------------------------+----------------+--------+------------------------------------------+--------------------------------+--------+
| ID                                   | Name           | Status | Networks                                 | Image                          | Flavor |
+--------------------------------------+----------------+--------+------------------------------------------+--------------------------------+--------+
| c42ecf9b-56bc-4489-a443-0964aefd6c46 | vm-ubuntu-ci-4 | ACTIVE | int_net_172.16=172.16.0.19, 10.200.0.191 | ubuntu_bionic_18_04            | s.nvf2 |
| c80f714c-3ca3-4785-94f7-12171f80293c | vm-ubuntu-ci-3 | ACTIVE | int_net_172.16=172.16.0.15, 10.200.0.107 | ubuntu_bionic_18_04            | s.nvf1 |
+--------------------------------------+----------------+--------+------------------------------------------+--------------------------------+--------+

xxx:~$ ping 10.200.0.107
PING 10.200.0.107 (10.200.0.107) 56(84) bytes of data.
64 bytes from 10.200.0.107: icmp_seq=1 ttl=62 time=476 ms
64 bytes from 10.200.0.107: icmp_seq=2 ttl=62 time=0.374 ms
64 bytes from 10.200.0.107: icmp_seq=3 ttl=62 time=0.375 ms
^C
--- 10.200.0.107 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2023ms
rtt min/avg/max/mdev = 0.374/158.922/476.017/224.220 ms
admintdf@omega:~$ ping 10.200.0.191
PING 10.200.0.191 (10.200.0.191) 56(84) bytes of data.
^C
--- 10.200.0.191 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5124ms