No healthy upstream ошибка vcenter

After upgrading to vCenter 7 Update 1 , when I tried to browse vCenter HTML5 UI, I faced “no healthy upstream” error. I could access to vCenter Management Interface (VAMI) https://vCenter-IPaddress:5480 without any issues. I could also connect to vCenter Server through  SSH but I realized couple of vCenter Server services could not start.

You can also check the details status of services by connecting to vCenter through SSH and run the following command:

#service-control --list 

Then I tried to force to start services by below commands:

#service-control --start --all
#service-control –-start {service-name}

After waiting for a while, I got the underneath error.

After spending couple of hours reading logs and a bit of googling, I have been pointed towards different answers. First of all I went through all DNS, NTP and IP checks and in my case everything was working as it should.

In my scenario, vCenter’s SSL certificate were
replaced with a valid signed certificate and it was one of the reason that
points me to check certification validity. Beside this SSL certificate, there
are couple of other certificates that vCenter server uses. To get familiar with
vSphere certificates you can read the following vSphere documentation:

https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.authentication.doc/GUID-3AF7757E-A30E-4EEC-8A41-28DA72102520.html

In my case  “Trusted root certificate, Machine SSL Certificate and SMS” were still valid . But ” Machine, vpxd, vpxd-extension and vsphere-webclient” were expired.    

You can check the validity of each certificate by running below commands in vCenter server:

# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SSL_CERT --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store SMS --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store machine --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vpxd-extension --text | less
# /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store vsphere-webclient --text | less

Below you can find the expired certificate screen shot:

In this case you need to update the expired certificates with use of vCenter certificate manager through running following command on vCenter CLI.

#/usr/lib/vmware-vmca/bin/certificate-manager

choose number 6 to replace Solution User certificates.

Then you need to answer the required information

• Do you wish to generate all certificates using configuration file : Option[Y/N] ? : Y
• Please provide valid SSO and VC privileged user credential to perform certificate operations.Enter username [Administrator@vsphere.local]:

Note: this is an example how to address each question you need to fill it out based on your environment.

• Enter proper value for ‘Country’ [Default value : US] :US
• Enter proper value for ‘Name’ [Default value : CA] : CA
• Enter proper value for ‘Organization’ [Default value : VMware] : “ vElements lab”
• Enter proper value for ‘OrgUnit’ [Default value : VMware Engineering] : VELEMENTSIT
• Enter proper value for ‘State’ [Default value : California]: California
• Enter proper value for ‘Locality’ [Default value : Palo Alto] : Palo Alto
• Enter proper value for ‘IPAddress’ (Provide comma separated values for multiple IP addresses) [optional] : you can press Enter or provide the required information
• Enter proper value for ‘Email’ [Default value : email@acme.com] : Press Enter
• Enter proper value for ‘Hostname’ (Provide comma separated values for multiple Hostname entries) [Enter valid Fully Qualified DomainName(FQDN), For Example : example.domain.com] : vc.velements.net
• Enter proper value for VMCA ‘Name’ : vc.velements.net You are going to regenerate Solution User Certificates using VMCA
• Continue operation : Option[Y/N] ? : Y

After I successfully updated the certificates , vCenter
services got started and I could reach the vCenter UI.

Below you can also find other solutions I found
when I was googling  

Suggested answers to check

• Upgrade VMware Hardware version and choose the
  correct OS for vCenter

Note: Take a snapshot from
vCenter Server VM before hardware version upgrade, as it’s none reversible  task to previous versions.

• Shutdown the vCenter > right click on the VM
  > Compatibility > Upgrade VM Compatibility.
• Right click on the vCenter and choose Edit
  settings > VM Options > General Options > Select VMware Photon OS
• Check DNS (you should be able to resolve FQDN
  names from vCenter)
• Check NTP (Time should be synced and correct
  between ESXi hosts and vCenter Server)
• vCenter Server IP address should be set Static

All of the services which are set to Automatic start are running without any errors or warnings. Hopefully this will help you to solve your issue.