When you install a VMware vCloud Director instance, it comes with an embedded JRE (Java Runtime Environment). For example, vCloud Director includes JRE 1.8.0_121 :

1$VCLOUD_HOME/jre/bin/java -version
2java version "1.8.0_121"
3Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
4Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

JRE includes it's own keystore for trusted SSL certificates, the cacerts file. This file is depending on both oracle/Java and VMware/vCloud releasing and if your certification authority root certificate isn't included, you may experienced issue when trying to communicate with products securized with SSL certificates (example : Catalog synchronization between two vCloud Director entities).

It's possible to find SSL related issues by looking at the $VCLOUD_HOME/logs/vcloud-container-debug.log file where you may find errors like:

1...
2javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
3...

The following post shows how to add a new trusted (root and/or intermediate) certificates to the JRE cacerts keystore.

Is you certification authority already trusted ?

According to the authority that issued your certificates, get the corresponding crt files and their sha1 fingerprint.

For example, if you use Let's Encrypt :

In Let's Encrypt case, you need 2 certificates:

• DST Root CA X3
• Let's Encrypt Authority X3

1# get crt files
2curl -s "https://pastebin.com/raw/63dQV36A" > DSTRootCAX3.crt
3curl -s "https://letsencrypt.org/certs/lets-encrypt-x3-cross-signed.pem.txt" > LetsEncryptAuthorityX3.crt
4# calculate fingerprint
5intermediate=$(openssl x509 -in /tmp/LetsEncryptAuthorityX3.crt -fingerprint -sha1 -noout | cut -d'=' -f2)
6root=$(openssl x509 -in /tmp/DSTRootCAX3.crt -fingerprint -sha1 -noout | cut -d'=' -f2)

Now you have two variables $root and $intermediate with cert authority sha1:

1echo "$root
2$intermediate"
3DA:C9:02:4F:54:D8:F6:DF:94:93:5F:B1:73:26:38:CA:6A:D7:7C:13
4E6:A3:B4:5B:06:2D:50:9B:33:82:28:2D:19:6E:FE:97:D5:95:6C:CB

Let's look in the keystore to check if the certificates are already trusted. For this, we use the Java keytool script that allow to manage the Java's keystore :

1$VCLOUD_HOME/jre/bin/keytool -list -keystore $VCLOUD_HOME/jre/lib/security/cacerts -storepass changeit | grep -i -B1 $root
2$VCLOUD_HOME/jre/bin/keytool -list -keystore $VCLOUD_HOME/jre/lib/security/cacerts -storepass changeit | grep -i -B1 $intermediate

If there is no result, for both commands, you should add your certification authority to the trusted ones.

By default, the JRE keystore is protected with a password: changeit.

Note : The letsencrypt sample is purely fictional as the Root certificate DSTRootCAX3 is already included in the included JRE keystore. But for the post, we will act as it is not.

Update JRE keystore

So your certification authority, possible internal and self-signed, is not known from the JRE keystore and you need to communicate between your vCloud Director and product using certificates issued from this authority. Therefore an update of the trusted certificates is required.

We still use the (fabulous) Java keytool script to insert new certificates to the keystore.

These operations needs to be done on each cells of your vCloud Director instance as the JRE keystore is not shared between members.

Take a backup of cacerts file

Take a backup of the embedded keystore, to be able to rollback if necessary (md5sum is used to quiclky check that we copy the file without error):

1cp -f $VCLOUD_HOME/jre/lib/security/cacerts ~/cacerts_jre.backup
2md5sum ~/cacerts_jre.backup $VCLOUD_HOME/jre/lib/security/cacerts

1#2ecfd7e5a8789c3f0e68ae85a26dea23  ~/cacerts_jre.backup
2#2ecfd7e5a8789c3f0e68ae85a26dea23  $VCLOUD_HOME/jre/lib/security/cacerts

Add the CA certs to the keystore

Time to update keystore content :

 1$VCLOUD_HOME/jre/bin/keytool \
 2    -import -trustcacerts \
 3    -alias identrustdstx3 \
 4    -file /tmp/DSTRootCAX3.crt \
 5    -keystore $VCLOUD_HOME/jre/lib/security/cacerts \
 6    -storepass changeit
 7
 8$VCLOUD_HOME/jre/bin/keytool \
 9    -import -trustcacerts \
10    -alias letsencryptx3 \
11    -file /tmp/LetsEncryptAuthorityX3.crt \
12    -keystore $VCLOUD_HOME/jre/lib/security/cacerts \
13    -storepass changeit

And we check:

1$VCLOUD_HOME/jre/bin/keytool -list -keystore $VCLOUD_HOME/jre/lib/security/cacerts -storepass changeit | grep -i $root -c
21
3$VCLOUD_HOME/jre/bin/keytool -list -keystore $VCLOUD_HOME/jre/lib/security/cacerts -storepass changeit | grep -i $intermediate -c
41

One result is found for each certificate : good.

It's also possible to look in keystore for a specific alias entry without grep, but depending on used alias, I consider that it's possible to miss a match (that's why I prefer to calculate the fingerprint and made a search on it):

1$VCLOUD_HOME/jre/bin/keytool -list -keystore $VCLOUD_HOME/jre/lib/security/cacerts -storepass changeit -alias "identrustdstx3"
2identrustdstx3, Aug 25, 2016, trustedCertEntry,
3Certificate fingerprint (SHA1): DA:C9:02:4F:54:D8:F6:DF:94:93:5F:B1:73:26:38:CA:6A:D7:7C:13

Shutdown cell

Once done, each cell needs to be 'service-restarted' in order to use the new keystore:

Note :Do not forget to move the vCenter proxy role before restarting cells.

Quiesce the cell, to not accept new incoming job on this specific cell:

1$VCLOUD_HOME/bin/cell-management-tool -u administrator -p '********' cell -quiesce true

Check that there is no more running job:

1$VCLOUD_HOME/bin/cell-management-tool -u administrator -p '********' cell -status

Note: Since a couple of versions, when you use the quiesce true command, the command is not returning prompt until there is no more job left on the cell. The status command could be considered as useless.

Shutdown services:

1$VCLOUD_HOME/bin/cell-management-tool -u administrator -p '********' cell -shutdown

Then restart services:

1service vmware-vcd start

Have a look on the startup of the service through the logs:

1tail -f $VCLOUD_HOME/logs/cell.log | grep -i "Application Initialization"

Note : A better documentation about the cell shutdown is available in official VMware documentation about vCD

When all cells are restarted, your issue with certificate chain should be solved.