Use VMware Container Service Extension with a corporate proxy

Ludovic Rivallain Nov 26, 2020 · 6 min read ·

Share on:

vmware kubernetes cse vcloud director extension

Overview

The current post is a quickstart to VMware open-source project Container Service Extension (CSE), a Kubernetes as a Service for VMware vCloud Director.

The project is already well documented (CSE documentation) and you should not have any trouble to set it up by following the installation steps.

Except... if you plan to use it behind a corporate proxy to access to Internet. The goal of this post is to be a reminder to me for this kind of setup.

CSE server appliance

The first step to setup CSE is to prepare an appliance that will host the CSE server component. In my case, I used a freshly deployed Ubuntu 20.04 LTS server, deployed from the ubuntu cloud images repository: https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.ova

Once the appliance is up and running, I setup the proxy information:

W.X.Y.Z is the IP address of my HTTP based proxy.

1echo "HTTP_PROXY=W.X.Y.Z:3128
2HTTPS_PROXY=W.X.Y.Z:3128
3NO_PROXY=.vlab.lcl,192.168.0.0/16,127.0.0.1,localhost" | sudo tee -a /etc/environment >/dev/null
4
5export HTTP_PROXY="W.X.Y.Z:3128"
6export HTTPS_PROXY="W.X.Y.Z:3128"
7export NO_PROXY=".vlab.lcl,192.168.0.0/16,127.0.0.1,localhost"
8
9echo "Acquire::http::proxy \"http://W.X.Y.Z:3128\";" | sudo tee -a /etc/apt/apt.conf >/dev/null

A quick test:

1curl https://google.com
2<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
3<TITLE>301 Moved</TITLE></HEAD><BODY>
4<H1>301 Moved</H1>
5The document has moved
6<A HREF="https://www.google.com/">here</A>.
7</BODY></HTML>

It works.

CSE server components

Let's install software components required for CSE:

1# vcd-cli + CSE
2sudo apt-get install python3-pip gcc -y
3sudo pip3 install vcd-cli
4sudo pip3 install container-service-extension==2.6.1

Yep, still using CSE 2.6 for backwards compatibility.

Next, we install kubectl:

1# kubectl
2curl -LO "https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl"
3chmod +x ./kubectl
4sudo mv ./kubectl /usr/local/bin/kubectl
5# Test it:
6kubectl version --client

`vcd` CLI integration

Quick method to integrate the cse CLI to the vcd CLI:

1mkdir -p ~/.vcd-cli/
2echo "extensions:
3- container_service_extension.client.cse" >> ~/.vcd-cli/profiles.yaml
4# Test it:
5vcd cse version

Authentication key pairs

If needed, create a SSH key pair to be used from this appliance, to the K8S nodes for maintenance:

1ssh-keygen

CSE configuration

Once our appliance is ready to host CSE server, we can configure it:

1mkdir ~/.cse
2cse sample -o ~/.cse/decrypted-config.yaml

This command will create a ~/.cse/decrypted-config.yaml file to configure according to the reference documentation.

To ease the testing, I made a fork of the official templates repository to my GitHub workspace with only one ubuntu based template:

 1templates:
 2  - compute_policy: ""
 3    cpu: 2
 4    deprecated: false
 5    description: "Ubuntu 16.04, Docker-ce 19.03.12, Kubernetes 1.18.6, weave 2.6.5"
 6    mem: 2048
 7    name: ubuntu-16.04_k8-1.18_weave-2.6.5
 8    revision: 1
 9    kind: native
10    sha256_ova: 3c1bec8e2770af5b9b0462e20b7b24633666feedff43c099a6fb1330fcc869a9
11    source_ova: "https://cloud-images.ubuntu.com/releases/xenial/release-20180418/ubuntu-16.04-server-cloudimg-amd64.ova"
12    source_ova_name: ubuntu-16.04-server-cloudimg-amd64.ova
13    os: "ubuntu-16.04"
14    docker_version: "19.03.12"
15    kubernetes: "upstream"
16    kubernetes_version: "1.18.6"
17    cni: "weave"
18    cni_version: "2.6.5"
19    upgrade_from:
20    - "ubuntu-16.04_k8-1.17_weave-2.6.0"
21    - "ubuntu-16.04_k8-1.18_weave-2.6.5"

Now we encrypt the file:

1cse encrypt ~/.cse/decrypted-config.yaml --output ~/.cse/config.yaml
2chmod 600 config.yaml
3rm ~/.cse/decrypted-config.yaml # otherwise it will be useless to encrypt it

If you need to decrypt it (for example to edit the content):

1cse decrypt  ~/.cse/config.yaml --output ~/.cse/decrypted-config.yaml

Then you can run the install process...

!!!BUT!!! We will kill it when the process will download the OVA file(s) in order to hack the content of customization scripts:

1cse install -c ~/.cse/config.yaml --ssh-key ~/.ssh/id_rsa.pub
2# !! cut the execution when the OVA is downloading !! CTRL+C

When the script is stopped, it is possible to edit the content of customization scripts to insert the proxy settings. In my case:

1vi ~/.cse_scripts/ubuntu-16.04_k8-1.18_weave-2.6.5_rev1/cust.sh

I added the following lines at the beginning of the file, just after the line set -e:

 1# proxy setup
 2echo "HTTP_PROXY=W.X.Y.Z:3128
 3HTTPS_PROXY=W.X.Y.Z:3128
 4NO_PROXY=.vlab.lcl,192.168.0.0/16,127.0.0.1,localhost" >> /etc/environment
 5echo "Acquire::http::proxy \"http://W.X.Y.Z:3128\";" >> /etc/apt/apt.conf
 6export HTTP_PROXY="W.X.Y.Z:3128"
 7export HTTPS_PROXY="W.X.Y.Z:3128"
 8export NO_PROXY=".vlab.lcl,192.168.0.0/16,127.0.0.1,localhost"
 9echo "[Service]
10Environment=\"HTTP_PROXY=http://W.X.Y.Z:3128\"
11Environment=\"HTTPS_PROXY=http://W.X.Y.Z:3128\"
12Environment=\"NO_PROXY=.vlab.lcl,192.168.0.0/16,127.0.0.1,localhost\"
13" >> /etc/systemd/system/docker.service.d/override.conf

Save+quit etc. And we re-run the CSE initialisation command:

1cse install -c ~/.cse/config.yaml --ssh-key ~/.ssh/id_rsa.pub

And your template will now being built using the HTTP proxy you specified. After the template preparation, the template is added to the available ones:

1cse template list -c ~/.cse/config.yaml
2Password for config file decryption:
3Decrypting 'config.yaml'
4name                                revision  compute_policy    local    remote
5--------------------------------  ----------  ----------------  -------  --------
6ubuntu-16.04_k8-1.18_weave-2.6.5           1                    Yes      Yes

(I removed some columns to ease the post reading)

Patching Pika for Python 3.8

In the next steps, if you use Python version 3.8 (you can check it by running python3 -V command), you may have an issue with an error message like:

1vcd cse template list
2Usage: vcd cse template list [OPTIONS]
3Try "vcd cse template list -h" for help.
4
5Error: maximum recursion depth exceeded

You can patch the Pika library by applying a patch made from this Pull request from @lukebakken.

Two choices:

Download manually this patch file and run patch command:

1sudo patch /usr/local/lib/python3.8/dist-packages/pika/compat.py < pika-1254.patch

All in one command (you should check the content of a downloaded file before applying it to you environment):

1curl -s 'https://gist.githubusercontent.com/lrivallain/be77cd8ffd731649705ae7b1e139d8d3/raw/d35069fab35f179dd1a76f29607367424d87314a/pika-1254.patch' | sudo patch /usr/local/lib/python3.8/dist-packages/pika/compat.py

Tests

The following command will run the CSE server services in foreground mode

1cse run -c ~/.cse/config.yaml

After a series of checks, the process should display a message like:

waiting for requests (ctrl+c to close)

You can test it by running the following commands from another terminal instance:

1vcd login vcd.vlab.lcl <org> <username>
2vcd cse template list

And you should get a list of templates ready for deployment:

1name                                revision  is_default    catalog
2--------------------------------  ----------  ------------  ---------
3ubuntu-16.04_k8-1.18_weave-2.6.5           1  Yes           cse

(I removed some columns to ease the post reading)

Finalize installation

Using the CSE server in foreground mode is not the easiest way for day-to-day operations, so we will enable it as a system service:

1# get the service files from the CSE repository
2curl -L https://github.com/vmware/container-service-extension/raw/2.6.1_ga/cse.service > ~/.cse/cse.service
3curl -L https://github.com/vmware/container-service-extension/raw/2.6.1_ga/cse.sh > ~/.cse/cse.sh
4chmod +x ~/.cse/cse.sh

Edit the CSE_CONFIG_PATH value of ~/.cse/cse.sh according to the path where is stored your configuration file. For my setup:

1CSE_CONFIG_PATH=/home/ubuntu/.cse/config.yaml

Edit the ExecStart, User, WorkingDirectory value of ~/.cse/cse.service according to your environment. I also add an Environment statement to store the password of the configuration file and my proxy settings.

For my setup:

 1[Unit]
 2Description=Container Service Extension for VMware vCloud Director
 3Wants=network-online.target
 4After=network-online.target
 5
 6[Service]
 7ExecStart=/home/ubuntu/.cse/cse.sh
 8User=ubuntu
 9WorkingDirectory=/home/ubuntu/.cse/
10Type=simple
11Restart=always
12Environment=CSE_CONFIG_PASSWORD="VMware1!"
13Environment=HTTP_PROXY="W.X.Y.Z:3128"
14Environment=HTTPS_PROXY="W.X.Y.Z:3128"
15Environment=NO_PROXY=".vlab.lcl,192.168.0.0/16,127.0.0.1,localhost"
16
17[Install]
18WantedBy=default.target

Then you can enable, start and check this new service:

1sudo cp .cse/cse.service /etc/systemd/system/
2sudo systemctl enable cse
3sudo systemctl start cse
4sudo systemctl status cse

Conclusion

You now have a fully working CSE appliance, running behind a corporate proxy and you can use it from any location with an access to the vCD instance.

You can find more details on the way to create/manage CSE K8S clusters in the CSE documentation.

Of course, if you can afford to store a fork of the official templates repository with your proxy informations and your customisations, it could ease the deployment of new templates when needed: for example by setting up a private or internal git repository.