Using VMware vSphere with Juju

Overview

To register a VMware ESXi cluster with vSphere as a “vSphere cloud”, there are three main steps:

  1. juju add-cloud registers the cluster with Juju, providing Juju with details such as the host’s IP address
  2. juju add-credential provides credentials to Juju to access the vSphere API
  3. juju bootstrap creates a Juju controller within the cluster

Requirements

vSphere cloud you will need to have an existing vSphere installation which supports, or has access to, the following:

  • VMware Hardware Version 8 (or greater)
  • ESXi 5.0 (or greater)
  • Internet access
  • DNS and DHCP

Juju supports both high-availability vSAN deployments as well and standard deployments.

Adding a vSphere cloud

There are two mechanisms for registering a cluster with Juju. You can either use a guided interactive prompt or create a configuration file with equivalent information and provide that.

During the process, you will be asked to provide the following information:

  • A name that Juju will refer to the cluster as
  • IP address of your cluster’s vCenter
  • The name(s) of any Datacenter(s) that you want to enable Juju to be able to deploy to

Tip: Finding Datacenter names in vSphere

Your Datacenters are available through the vSphere web client by selecting vCenter Inventory Lists > Resources > Datacenters from the hierarchical menu at the top left.

The values you need are listed in the ‘Name’ column, such as the ‘dc0’ and ‘dc1’ Datacenters shown here:

vSphere web client showing Datacenters

Using an interactive prompt

Use the add-cloud command to interactively add your vSphere cloud to Juju’s list of clouds:

juju add-cloud --local

Why --local?: Juju controllers have the ability to manage workloads that span multiple clouds. Adding the --local option indicates that the information should be stored on the machine where you are executing the commands and not passed to a controller.

An session with multiple datacenters:

Cloud Types
  lxd
  maas
  manual
  openstack
  vsphere

Select cloud type: vsphere

Enter a name for your vsphere cloud: vsp-cloud

Enter the vCenter address or URL: 178.18.42.10

Enter datacenter name: dc0

Enter another datacenter? (y/N): y

Enter datacenter name: dc1

Enter another datacenter? (y/N): n

Cloud "vsp-cloud" successfully added

You will need to add credentials for this cloud (`juju add-credential vsp-cloud`)
before creating a controller (`juju bootstrap vsp-cloud`).

Manually adding a vSphere cloud with a configuration file

Write out a YAML-formatted configuration file with the sections in angle brackets (<…>) replaced with the relevant information for your system.

clouds:
 <cloud-name>:
  type: vsphere
  auth-types: [userpass]
  endpoint:  <vcenter-ip-address-or-hostname>
  regions:
    <datacenter-1>: {}
    [<datacenter-n>: {}]

Here is an example:

clouds:
 vsp-cloud:
  type: vsphere
  auth-types: [userpass]
  endpoint: 178.18.42.10
  regions:
   dc0: {}  # these empty maps
   dc1: {}  # are necessary

Adding a cloud manually can either be done locally or the cloud can be stored on a current controller). Here, we’ll show how to do it locally (client cache).

To add cloud ‘vsp-cloud’, assuming the configuration file is vsp-cloud.yaml in the current directory, we would run:

juju add-cloud --local vsp-cloud vsp-cloud.yaml

Why --local?: Juju controllers have the ability to manage workloads that span multiple clouds. Adding the --local option indicates that the information should be stored on the machine where you are executing the commands and not passed to a controller.

Adding credentials

Use the juju add-credential command to register security credentials to the new cloud.

juju add-credential vsp-cloud

Example user session:

Enter credential name: vsp-cloud-creds

Using auth-type "userpass".

Enter user: jlaurin@juju.example.com

Enter password: ********

Enter vmfolder (optional): juju-root

Credential "vsp-cloud-creds" added locally for cloud "vsp-cloud".

We’ve called the new security credential ‘vsp-cloud-creds’. You will need to provide your VMware account username–this looks like an email address–and the associated password.

Locked out?: Credentials for the vSphere cloud have been reported to occasionally stop working over time.

If this occurs, then you can “remind” vSphere of your credentials. See Dealing with inert credentials for guidance.

Further information

For background information on adding a cloud to Juju, see these pages:

Creating a controller

You are now ready to create a Juju controller for cloud ‘vsp-cloud’:

juju bootstrap vsp-cloud vsp-controller

Above, the name given to the new controller is ‘vsp-controller’. vSphere will provision an instance to run the controller on.

For a detailed explanation and examples of the bootstrap command see the Creating a controller page.

There are three VMware-specific options available for specifying the network and datastore to use:

  • primary-network
    The primary network that VMs will be connected to. If this is not specified, Juju will look for a network named VM Network.
  • external-network
    An external network that VMs will be connected to. The resulting IP address for a VM will be used as its public address. An external network provides the interface to the internet for virtual machines connected to external organization vDC networks.
  • datastore
    The datastore in which to create VMs. If this is not specified, the process will abort unless there is only one datastore available.

For example:

juju bootstrap vsp-cloud vsp-controller \
    --config primary-network=$PRIMARY_NET \
    --config external-network=$EXTERNAL_NET \
    --config datastore=$DATA_STORE

The above --config options will only apply to the ‘controller’ and ‘default’ models. Use option --model-default instead if you want any newly-created models to be affected. You can also use the model-defaults command once the controller is created to do the same thing.

Initial bootstrap duration: When creating a controller with vSphere, a cloud image is downloaded to the client and then uploaded to the ESX host. This depends on your network connection and can take a while.

vSphere specific features

When creating a controller, there are three levels of placement that Juju understands: cloud, region, and availability zone. In vSphere, these are mapped in two different ways depending on your topology:

  • cloud (vSphere endpoint), region (Datacenter), availability zone (host)
  • cloud (vSphere endpoint), region (Datacenter), availability zone (cluster)

If your topology has a cluster without a host, Juju will see this as an Availability Zone and may fail silently. To solve this, either make sure the host is within the cluster, or use a placement directive:

juju bootstrap vsphere/<datacenter> <controllername> --to zone=<cluster|host>

To create a controller using Datacenter ‘dc1’ you would enter the following:

juju bootstrap vsp-cloud/dc1 vsp-controller

Specifying a datastore when deploying an application

There is a constraint called ‘root-disk-source’ that can stipulate the name of a vSphere datastore to house the root disk:

juju deploy myapp --constraints root-disk-source=mydatastore

Deploying applications to a specific host or cluster

Resource groups within a host or cluster can be specified with the ‘zones’ constraint:

juju deploy myapp --constraints zones=mycluster/mygroup
juju deploy myapp --constraints zones=mycluster/myparent/mygroup

Removing the cluster

To remove the ability for Juju to manage workloads on your cluster, use the following commands. You will

  • juju destroy-controller --destroy-all-models <controller-name>
  • juju remove-cloud --client <cloud-name>
  • juju remove-credential --client <credential-name>

Next steps

A controller is created with two models - the ‘controller’ model, which should be reserved for Juju’s internal operations, and a model named ‘default’, which can be used for deploying user workloads.

Example Workload: Charmed Kubernetes

Your cluster is now set up to deploy Charmed Kubernetes, a “pure upstream” Kubernetes distribution that is standards-compliant and easy to upgrade. Follow these instructions to deploy Kubernetes to vSphere (start at the “Add model” step).

Further reading

1 Like

Question: "How do I add a “centos7” image to vsphere?

Question: “How do I make use of additional datastores apart from the one configured in the controller or model?”

Question: “Can juju handle NFS type datastores which exists in vsphere?” Very useful for providing NAS to juju units.

Question: “How do I make use of networks in vsphere?” - Is spaces available yet for vsphere?

What is a correct example for

juju deploy myapp --constraints zones=mycluster/mygroup
juju deploy myapp --constraints zones=mycluster/myparent/mygroup

Because I cannot get it to work

We have a mapping as:

(vsphere) datacenter = (juju) region

According to docs for vsphere, this mapping can be different depending on your vsphere configuration.

Hope it helps.

It would be nice to explain the minimum required privileges need in vSphere to bootstrap/deploy a juju environment to a vSphere cloud. Based on this commit [1] we know at least System.Read is required on the Datacenter entity. Clearly you need more than just read.

Thanks,
Nick

[1] https://github.com/juju/juju/commit/c9eeea11702d1172e5f68deff2c298b9199e8299#diff-4f35754aeeeeb70bd4f4e59b409336ab

Hey Nick

We are running juju with vsphere and have some experience.

The latest release of juju came with alot of changes to improve the situation with vsphere but I agree that it’s unclear just exactly how to setup any cloud for a proper juju situation. Lxd might be the best documented cloud in my opinion.

We have experimented our way forward and are not super happy about that.

Perhaps we missed something, but, yeah.

@rick_h

Erik,
When giving juju an admin account everything is fine, but in the spirit of security and least privileged access it would be good to know exactly what that least privilege is. We also have customers that have this requirement and would like to see this documentation.

Nick

1 Like

@nniehoff I feel with you.

I am not an expert in VMware, so I rely on some people at work to deal with these things which in our case also relates to Active directory I think.

But, the main issue for me at the moment is to be able to properly separate users views on provisioned vms etc. This is what seems to be partially solved/addressed/mitigated with later versions of juju.

The previous situation was that every user had a view of all vms for everyone since user access was enabled for the whole datacenter in vsphere.

There certainly is missing information for best practice. I particularly hate that we ask people to give Juju full admin privileges on AWS: Using Amazon AWS with Juju

To address some of these concerns, I hope to expand the post below into a full guide. It sounds like guide should be supplemented for guides for each cloud.

1 Like