I am actually struggling around this issue, context below:
- Openstack deployed with MaaS/Juju [1]
- 1 provider (with dhcp enabled) and 1 self service network [2]
- 1 instance A adressed in the self service network, with a flaoting IP [3]
- 1 instance B adressed directly in the provider network
A is fully reachable, B is not (both network and metadata, IP configuration is done with config-drive method)
I have tried to ping inside namespace on the neutron-gateway:
#### qrouter-cd77fe8a-46bc-4c91-9efe-71a173a82b6c
lo UNKNOWN 127.0.0.1/8 ::1/128
gre0@NONE DOWN
gretap0@NONE DOWN
erspan0@NONE DOWN
qg-6b3c82c9-4e@if70 UP 10.142.1.197/24 10.142.1.196/32 fe80::f816:3eff:fea7:caf0/64
qr-56376ad0-de@if71 UP 192.168.199.1/24 fe80::f816:3eff:fe0c:f019/64
#### qdhcp-d6420097-fe20-43f5-b39b-0236ee364f72
lo UNKNOWN 127.0.0.1/8 ::1/128
gre0@NONE DOWN
gretap0@NONE DOWN
erspan0@NONE DOWN
tap2a99d602-87 UNKNOWN 192.168.199.100/24 fe80::f816:3eff:fe03:9409/64```
# ip netns list | awk '{print $1}' | while read ns ; do echo "#### $ns" ; ip netns exec $ns ping -c 1 10.142.1.199 ; done
#### qrouter-cd77fe8a-46bc-4c91-9efe-71a173a82b6c
PING 10.142.1.199 (10.142.1.199) 56(84) bytes of data.
From 10.142.1.197 icmp_seq=1 Destination Host Unreachable
--- 10.142.1.199 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
#### qdhcp-d6420097-fe20-43f5-b39b-0236ee364f72
PING 10.142.1.199 (10.142.1.199) 56(84) bytes of data.
From 10.142.1.197 icmp_seq=1 Destination Host Unreachable
--- 10.142.1.199 ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
I also tried to disable “juju config neutron-openvswitch disable-security-groups=true”, but same issue. (in doubt, all Openstack ressources are destroy/re-create with Terraform after each modifications).
Tcpdump on nova and neutron-gateway (on any interfaces): ARP request are seen on both (ping comming from outside, via neutron-gateway, to nova), but no replies.
It seems the instance is completly isolated on the Nova host.
Any help or guidance will be appreciated.
[1] Juju bundle
variables:
openstack-origin: &openstack-origin cloud:bionic-train
data-port: &data-port br-data:enp3s0f1
worker-multiplier: &worker-multiplier 1
bridge-mappings: &bridge-mappings physnet1:br-data
# bridge-mappings: &bridge-mappings physnet1:br-data
osd-devices: &osd-devices /dev/sdb /dev/vdb
expected-osd-count: &expected-osd-count 3
expected-mon-count: &expected-mon-count 3
mysql-connections: &mysql-connections 1000
machines:
'0':
series: bionic
constraints: zones=default tags=test
'1':
series: bionic
constraints: zones=default tags=test
# '2':
# series: bionic
# constraints: zones=default tags=test
# '3':
# series: bionic
# constraints: zones=default tags=test
"4":
series: bionic
constraints: zones=default tags=virtual
"5":
series: bionic
constraints: zones=default tags=virtual
"6":
series: bionic
constraints: zones=default tags=virtual
"7":
series: bionic
constraints: zones=default tags=virtual
relations:
- - nova-compute:amqp
- rabbitmq-server:amqp
- - neutron-gateway:amqp
- rabbitmq-server:amqp
- - keystone:shared-db
- mysql:shared-db
- - nova-cloud-controller:identity-service
- keystone:identity-service
- - glance:identity-service
- keystone:identity-service
- - neutron-api:identity-service
- keystone:identity-service
- - neutron-openvswitch:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - neutron-api:shared-db
- mysql:shared-db
- - neutron-api:amqp
- rabbitmq-server:amqp
- - neutron-gateway:neutron-plugin-api
- neutron-api:neutron-plugin-api
- - glance:shared-db
- mysql:shared-db
- - glance:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:image-service
- glance:image-service
- - nova-compute:image-service
- glance:image-service
- - nova-cloud-controller:cloud-compute
- nova-compute:cloud-compute
- - nova-cloud-controller:amqp
- rabbitmq-server:amqp
- - nova-cloud-controller:quantum-network-service
- neutron-gateway:quantum-network-service
- - nova-compute:neutron-plugin
- neutron-openvswitch:neutron-plugin
- - neutron-openvswitch:amqp
- rabbitmq-server:amqp
- - openstack-dashboard:identity-service
- keystone:identity-service
- - openstack-dashboard:shared-db
- mysql:shared-db
- - nova-cloud-controller:shared-db
- mysql:shared-db
- - nova-cloud-controller:neutron-api
- neutron-api:neutron-api
- - cinder:image-service
- glance:image-service
- - cinder:amqp
- rabbitmq-server:amqp
- - cinder:identity-service
- keystone:identity-service
- - cinder:cinder-volume-service
- nova-cloud-controller:cinder-volume-service
- - cinder-ceph:storage-backend
- cinder:storage-backend
- - ceph-mon:client
- nova-compute:ceph
- - nova-compute:ceph-access
- cinder-ceph:ceph-access
- - cinder:shared-db
- mysql:shared-db
- - ceph-mon:client
- cinder-ceph:ceph
- - ceph-mon:client
- glance:ceph
- - ceph-osd:mon
- ceph-mon:osd
- - ntp:juju-info
- nova-compute:juju-info
- - ntp:juju-info
- neutron-gateway:juju-info
- - ceph-radosgw:mon
- ceph-mon:radosgw
- - ceph-radosgw:identity-service
- keystone:identity-service
- - placement
- mysql
- - placement
- keystone
- - placement
- nova-cloud-controller
- - nova-cloud-controller:memcache
- memcached:cache
series: bionic
services:
ceph-mon:
annotations:
charm: cs:~openstack-charmers-next/bionic/ceph-mon
num_units: 3
options:
expected-osd-count: *expected-osd-count
monitor-count: *expected-mon-count
source: *openstack-origin
to:
- 'lxd:4'
- 'lxd:5'
- 'lxd:6'
bindings:
"" : maas-infralab
ceph-osd:
annotations:
charm: cs:~openstack-charmers-next/bionic/ceph-osd
num_units: 3
options:
osd-devices: *osd-devices
source: *openstack-origin
to:
- '4'
- '5'
- '6'
bindings:
"" : maas-infralab
ceph-radosgw:
annotations:
charm: cs:~openstack-charmers-next/bionic/ceph-radosgw
num_units: 1
options:
source: *openstack-origin
to:
- 'lxd:6'
bindings:
"" : maas-infralab
cinder:
annotations:
charm: cs:~openstack-charmers-next/bionic/cinder
num_units: 1
options:
block-device: None
glance-api-version: 2
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
to:
- 'lxd:2'
bindings:
"" : maas-infralab
cinder-ceph:
annotations:
charm: cs:~openstack-charmers-next/bionic/cinder-ceph
num_units: 0
glance:
annotations:
charm: cs:~openstack-charmers-next/bionic/glance
num_units: 1
options:
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
to:
- 'lxd:7'
bindings:
"" : maas-infralab
keystone:
annotations:
charm: cs:~openstack-charmers-next/bionic/keystone
num_units: 1
options:
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
admin-password: Superpass01
to:
- 'lxd:7'
bindings:
"" : maas-infralab
mysql:
annotations:
charm: cs:~openstack-charmers-next/bionic/percona-cluster
num_units: 1
options:
max-connections: *mysql-connections
innodb-buffer-pool-size: 256M
performance-schema: True
to:
- 'lxd:4'
bindings:
"" : maas-infralab
neutron-api:
annotations:
charm: cs:~openstack-charmers-next/bionic/neutron-api
num_units: 1
options:
manage-neutron-plugin-legacy-mode: true
neutron-security-groups: true
flat-network-providers: physnet1
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
to:
- '0'
bindings:
"" : maas-infralab
placement:
annotations:
charm: cs:~openstack-charmers-next/bionic/placement
num_units: 1
options:
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
to:
- 'lxd:5'
bindings:
"" : maas-infralab
neutron-gateway:
annotations:
charm: cs:~openstack-charmers-next/bionic/neutron-gateway
num_units: 1
options:
bridge-mappings: *bridge-mappings
data-port: *data-port
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
to:
- '0'
bindings:
"" : maas-infralab
neutron-openvswitch:
bridge-mappings: *bridge-mappings
data-port: *data-port
enable-local-dhcp-and-metadata: true
annotations:
charm: cs:~openstack-charmers-next/bionic/neutron-openvswitch
num_units: 0
bindings:
"" : maas-infralab
nova-cloud-controller:
annotations:
charm: cs:~openstack-charmers-next/bionic/nova-cloud-controller
num_units: 1
options:
network-manager: Neutron
worker-multiplier: *worker-multiplier
openstack-origin: *openstack-origin
console-access-protocol: vnc
to:
- 'lxd:7'
bindings:
"" : maas-infralab
nova-compute:
annotations:
charm: cs:~openstack-charmers-next/bionic/nova-compute
num_units: 1
options:
config-flags: default_ephemeral_format=ext4
enable-live-migration: true
enable-resize: true
migration-auth-type: ssh
openstack-origin: *openstack-origin
# cpu-mode: host-passthrough
bindings:
"" : maas-infralab
to:
- '1'
# - '2'
# - '3'
ntp:
annotations:
charm: cs:ntp
num_units: 0
openstack-dashboard:
annotations:
charm: cs:~openstack-charmers-next/bionic/openstack-dashboard
num_units: 1
options:
openstack-origin: *openstack-origin
to:
- 'lxd:6'
bindings:
"" : maas-infralab
rabbitmq-server:
annotations:
charm: cs:~openstack-charmers-next/bionic/rabbitmq-server
num_units: 1
to:
- 'lxd:5'
bindings:
"" : maas-infralab
memcached:
charm: cs:memcached
num_units: 1
to:
- lxd:7
bindings:
"" : maas-infralab
[2] Openstack networks and subnet
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2020-09-23T15:09:33Z |
| description | |
| dns_domain | None |
| id | d6420097-fe20-43f5-b39b-0236ee364f72 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | None |
| is_vlan_transparent | None |
| mtu | 1458 |
| name | net-42-1 |
| port_security_enabled | False |
| project_id | 38ba8a1a65564951a01f0f0cb6d963e7 |
| provider:network_type | gre |
| provider:physical_network | None |
| provider:segmentation_id | 1 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | Internal |
| segments | None |
| shared | False |
| status | ACTIVE |
| subnets | 218ecbe5-8ea7-4965-9df2-702a8d8a0ca6 |
| tags | |
| updated_at | 2020-09-23T15:09:39Z |
+---------------------------+--------------------------------------+
+---------------------------+--------------------------------------+
| Field | Value |
+---------------------------+--------------------------------------+
| admin_state_up | UP |
| availability_zone_hints | |
| availability_zones | nova |
| created_at | 2020-09-23T15:09:33Z |
| description | |
| dns_domain | None |
| id | d87f8d99-e0e3-41b6-bad8-deec52746408 |
| ipv4_address_scope | None |
| ipv6_address_scope | None |
| is_default | False |
| is_vlan_transparent | None |
| mtu | 1500 |
| name | net-dmz-ext |
| port_security_enabled | False |
| project_id | 38ba8a1a65564951a01f0f0cb6d963e7 |
| provider:network_type | vlan |
| provider:physical_network | physnet1 |
| provider:segmentation_id | 1606 |
| qos_policy_id | None |
| revision_number | 2 |
| router:external | External |
| segments | None |
| shared | True |
| status | ACTIVE |
| subnets | 86f37edd-4a39-41cc-beb9-f3529ebf8874 |
| tags | |
| updated_at | 2020-09-23T15:09:39Z |
+---------------------------+--------------------------------------+
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 192.168.199.100-192.168.199.199 |
| cidr | 192.168.199.0/24 |
| created_at | 2020-09-23T15:09:39Z |
| description | |
| dns_nameservers | 1.1.1.1, 8.8.8.8 |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 192.168.199.1 |
| host_routes | |
| id | 218ecbe5-8ea7-4965-9df2-702a8d8a0ca6 |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | sub-42-1 |
| network_id | d6420097-fe20-43f5-b39b-0236ee364f72 |
| prefix_length | None |
| project_id | 38ba8a1a65564951a01f0f0cb6d963e7 |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2020-09-23T15:09:39Z |
+----------------------+--------------------------------------+
+----------------------+--------------------------------------+
| Field | Value |
+----------------------+--------------------------------------+
| allocation_pools | 10.142.1.190-10.142.1.200 |
| cidr | 10.142.1.0/24 |
| created_at | 2020-09-23T15:09:39Z |
| description | |
| dns_nameservers | 8.8.4.4, 8.8.8.8 |
| dns_publish_fixed_ip | None |
| enable_dhcp | True |
| gateway_ip | 10.142.1.1 |
| host_routes | |
| id | 86f37edd-4a39-41cc-beb9-f3529ebf8874 |
| ip_version | 4 |
| ipv6_address_mode | None |
| ipv6_ra_mode | None |
| name | sub-dmz-ext |
| network_id | d87f8d99-e0e3-41b6-bad8-deec52746408 |
| prefix_length | None |
| project_id | 38ba8a1a65564951a01f0f0cb6d963e7 |
| revision_number | 0 |
| segment_id | None |
| service_types | |
| subnetpool_id | None |
| tags | |
| updated_at | 2020-09-23T15:09:39Z |
+----------------------+--------------------------------------+
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
| ID | Agent Type | Host | Availability Zone | Alive | State | Binary |
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
| 174fce95-3fc1-4d4f-b0ee-390ecb8e932c | Open vSwitch agent | PM-865 | None | :-) | UP | neutron-openvswitch-agent |
| 178ff5ff-c216-4972-b500-82fea7085ab0 | DHCP agent | PM-866.maas | nova | :-) | UP | neutron-dhcp-agent |
| 72b706b9-8b96-4c2b-bcfa-c9ac531fd414 | Metadata agent | PM-866.maas | None | :-) | UP | neutron-metadata-agent |
| 75d1d34c-16e0-43fb-b2b6-5a4c3f00d3b7 | Open vSwitch agent | PM-866.maas | None | :-) | UP | neutron-openvswitch-agent |
| a5aa6eb1-ae50-4181-922e-3f889cf3cd58 | Metering agent | PM-865 | None | :-) | UP | neutron-metering-agent |
| c8586881-83e7-4d2a-89a3-d3628d8cf3bf | L3 agent | PM-865 | nova | :-) | UP | neutron-l3-agent |
| d3faa78c-1c1a-4ae9-ad6d-a80c1ce8e485 | DHCP agent | PM-865 | nova | :-) | UP | neutron-dhcp-agent |
| dcd81b88-bb7e-459b-934e-8160367d2b9d | Metadata agent | PM-865 | None | :-) | UP | neutron-metadata-agent |
+--------------------------------------+--------------------+-------------+-------------------+-------+-------+---------------------------+
[3] Instances
+--------------------------------------+------------+--------+----------------------------------------+--------------+-----------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+------------+--------+----------------------------------------+--------------+-----------+
| b0cad3c3-05da-4bd2-a9f0-7f75633e1b73 | bulk-vm3-0 | ACTIVE | net-dmz-ext=10.142.1.199 | cirros-amd64 | m1.medium |
| 525dee3f-d0cc-4f32-bb83-c3f06d4b7387 | bulk-vm1-0 | ACTIVE | net-42-1=192.168.199.143, 10.142.1.196 | bionic-amd64 | m1.medium |
+--------------------------------------+------------+--------+----------------------------------------+--------------+-----------+