Primary-network and external-network in vSphere deployments

Hi all,

what is the purpose for primary-network and external-network in a vSphere environment? I’m deploying Kubernetes clusters (with CDK and CORE) and thought the primary-network was used to communicate nodes between them and external for accesing from outside, but I can see, when I deploy the bundle, that is not in that way.

Thanks in advanced.

1 Like

I had the same thoughts around this. We only use one network so we had the luxury to not understand this yet =D

I’m not sure what properties the primary network needs to have, like internet connectivity or what else… Etc.

1 Like

The intent when modeling primary vs external was very much that. (Internal communication vs access from outside of the cloud.) I’m guessing anything that diverges from that tends to be “well I can route to external from everywhere, so I’ll just always use that” (shortcuts that weren’t intended).

It would be good to know where you’re seeing divergence from internal communication vs external communication.

2 Likes

Logical way is to use primary network to communicate between nodes inside, i.e., Kubernetes deployment and external network to gain access to Internet/other and to publish services.

When I declare the two kind of networks and deploy a CDK via juju the behavior is different at any time the bundle is deployed: sometimes the public address is the external one and others the public address is the primary in juju status.

By now, I’ll keep only the primary-network to deploy Kubernetes deployments, but it should be revised…

1 Like

I know this is a bit of thread-necromancy, but I see the same behavior here - vSphere 7.0U2 + Juju 2.9.18

I have two networks defined:

primary-network: nv-22-jujuinternal
external-network: nv-23-jujuexternal

nv-22 is 10.50.22.0/24

nv-23 is 10.50.23.0/24

Both networks are routable and my client browser can hit IP’s in either subnet. There are no firewalls between either of these subnets, the subnet that vCenter is running on, or the subnet that I’m attempting to access the service from.

Both primary-network: nv-22-jujuinternal and external-network: nv-23-jujuexternal are configured as such in the controller bootstrap.yaml and in my model-config yaml which I’m passing prior to running juju deploy in a model.

When deploying juju-gui, I can see from status that “Public address” is showing as an address which is on the primary-network (nv-22)

Unit         Workload  Agent  Machine  Public address  Ports           Message
juju-gui/2*  unknown   idle   2        10.50.22.30     80/tcp,443/tcp 

If I go to that address in a browser on http:// or https:// then I eventually get an ERR_CONNECTION_TIMED_OUT response, however if I go to vCenter and grab the IP that was provided via DHCP on the external-network then the Juju GUI loads up.

I see the same behavior with landscape-scalable too

Unit                 Workload  Agent  Machine  Public address  Ports           Message
haproxy/0*           active    idle   0        10.50.22.31     80/tcp,443/tcp  Unit is ready
landscape-server/0*  active    idle   1        10.50.22.32
postgresql/0*        active    idle   2        10.50.22.33     5432/tcp        Live master (10.19)
rabbitmq-server/0*   active    idle   3        10.50.22.34     5672/tcp        Unit is ready

Machine  State    DNS          Inst id        Series  AZ  Message
0        started  10.50.22.31  juju-dee5d2-0  bionic      poweredOn
1        started  10.50.22.32  juju-dee5d2-1  bionic      poweredOn
2        started  10.50.22.33  juju-dee5d2-2  bionic      poweredOn
3        started  10.50.22.34  juju-dee5d2-3  bionic      poweredOn

haproxy is listening on 80/443 on the ‘external-network’ address and responds when attempting access

https://10.50.23.21/new-standalone-user

but juju status haproxy shows the public address as being the DHCP assigned address from the primary-network

App      Version  Status  Scale  Charm    Store       Channel  Rev  OS      Message
haproxy           active      1  haproxy  charmstore  stable    61  ubuntu  Unit is ready

Unit        Workload  Agent  Machine  Public address  Ports           Message
haproxy/0*  active    idle   0        10.50.22.31     80/tcp,443/tcp  Unit is ready

Machine  State    DNS          Inst id        Series  AZ  Message
0        started  10.50.22.31  juju-dee5d2-0  bionic      poweredOn

According to this here document I should be able to run juju status haproxy and use the IP address that it provides, but this has so far been false and I can only access 80 and 443 by going to vCenter and getting the external-network assigned address.

My Juju experience can be counted in minutes on one hand at this point, and it’s very possible that I’m just doing something wrong or missing a step, but from what I understand so far this isn’t expected or desired behavior?

Any advice would be most appreciated :slight_smile:

Thx

stu

1 Like

Welcome @dotalchemy I hope you can get some help on this topic.

1 Like

Not sure how much info is too much info, but for what it’s worth, SSH seems to know which address to use:

Machine  State    DNS          Inst id        Series  AZ  Message
0        started  10.50.22.14  juju-799d5a-0  bionic      poweredOn
1        started  10.50.22.15  juju-799d5a-1  bionic      poweredOn
2        started  10.50.22.16  juju-799d5a-2  bionic      poweredOn
3        started  10.50.22.17  juju-799d5a-3  bionic      poweredOn

 ># juju ssh 0                                                                                     

Welcome to Ubuntu 18.04.6 LTS (GNU/Linux 4.15.0-162-generic x86_64)
~omitted for brevity~
Memory usage: 20%               IP address for ens192: 10.50.22.14
Swap usage:   0%                IP address for ens224: 10.50.23.13

and netstat to confirm

># netstat | grep 10.50.22.14
># netstat | grep 10.50.23.13
 tcp4       0      0  10.169.16.249.50710    10.50.23.13.ssh        ESTABLISHED

Juju tries all addresses to connect to a unit when wrapping ssh, which explains why it works.

1 Like

@afreiberger actually gave me an idea on this post about DNS IP’s and so I did a bit of testing.

I swapped around the vSphere switches that were assigned to Juju, so

primary-network: (old) nv-22-jujuinternal --> (now)nv-23-jujuexternal
external-network: (old) nv-23-jujuexternal --> (now)nv-22-jujuinternal

It does appear that on vSphere the Public address of a unit is numerically the lowest available.

As a reminder to save people scrolling up

nv-22-jujuinternal == 10.50.22.0/24
nv-23-jujuexternal == 10.50.23.0/24
># juju model-config | grep network | grep nv
external-network              model    nv-22-jujuinternal
primary-network               model    nv-23-jujuexternal

># juju status
Unit                 Workload  Agent  Machine  Public address  Ports           Message
haproxy/0*           active    idle   0        10.50.22.13     80/tcp,443/tcp  Unit is ready
landscape-server/0*  active    idle   1        10.50.22.14
postgresql/0*        active    idle   2        10.50.22.15     5432/tcp        Live master (10.19)
rabbitmq-server/0*   active    idle   3        10.50.22.16     5672/tcp        Unit is ready

But then I thought there may be opportunity for a little more testing, so I swapped around the subnet range associated with each VLAN

Now, back to the original mappings of

primary-network: nv-22-jujuinternal
external-network: nv-23-jujuexternal
nv-22-jujuinternal  == 10.50.23.0/24
nv-23-jujuexternal == 10.50.22.0/24

In theory, we should see that even though nv-22-jujuinternal is the primary network and the initial issue was that the primary interface was presented as the “Public address”, if my suspicion of “lowest numerical” is true, then the address associated with the external-network configurable should be presented by units as the public address, which now would be 10.50.22.0/24

Let’s test!

Waiting for address
Attempting to connect to 10.50.23.10:22
Connected to 10.50.23.10
~snip~

api-endpoints: ['10.50.23.10:17070']

># juju deploy landscape-scalable
~snip~

># juju status
Unit                 Workload  Agent  Machine  Public address  Ports     Message
haproxy/0*           active    idle   0        10.50.22.13               Unit is ready
landscape-server/0*  active    idle   1        10.50.22.14
postgresql/0*        active    idle   2        10.50.22.15     5432/tcp  Live master (10.19)
rabbitmq-server/0*   active    idle   3        10.50.22.16     5672/tcp  Unit is ready

And there we go, lowest numerical wins for the “Public address” on vSphere.

Now, on the downside, haproxy decided it didn’t want to expose ports there, but I suspect that’s likely due to the charm not handling multiple interfaces? haproxy is accessible now on the private-network IP…

Anyway, I think at this point I should probably file an issue report vs. filling up this thread. That’ll be the next thing I go learn how to do.

Thx

stu