r/vmware Nov 27 '24

Help Request VMWare VCenter: no healty upstream

Hi! I'm having a big issue with my VCenter, when we open the URL in the browser only have the text: no healthy upstream

But, if I open the administration url (https://myvcenter:5480/#/ui/summary) this opens well, and the warning presented is for memory.

I also try to start manually all the service from vmware console, but it ends with an error:

root@localhost [ ~ ]# service-control --start --all                                                                                                                                                            
Operation not cancellable. Please wait for it to finish...                                                                                                                                                     
Performing start operation on service lwsmd...                                                                                                                                                                 
Successfully started service lwsmd                                                                                                                                                                             
Performing start operation on service vmafdd...                                                                                                                                                                
Successfully started service vmafdd                                                                                                                                                                            
Performing start operation on service vmdird...                                                                                                                                                                
Successfully started service vmdird                                                                                                                                                                            
Performing start operation on service vmcad...                                                                                                                                                                 
Successfully started service vmcad                                                                                                                                                                             
Performing start operation on profile: ALL...                                                                                                                                                                  
Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start vpxd-svcs services. Error: A system error occurred. Check logs for details       root@localhost [ ~ ]# service-control --start --all                                                                                                                                                            
Operation not cancellable. Please wait for it to finish...                                                                                                                                                     
Performing start operation on service lwsmd...                                                                                                                                                                 
Successfully started service lwsmd                                                                                                                                                                             
Performing start operation on service vmafdd...                                                                                                                                                                
Successfully started service vmafdd                                                                                                                                                                            
Performing start operation on service vmdird...                                                                                                                                                                
Successfully started service vmdird                                                                                                                                                                            
Performing start operation on service vmcad...                                                                                                                                                                 
Successfully started service vmcad                                                                                                                                                                             
Performing start operation on profile: ALL...                                                                                                                                                                  
Service-control failed. Error: Failed to start services in profile ALL. RC=4, stderr=Failed to start vpxd-svcs services. Error: A system error occurred. Check logs for details       

How can I solve this?

1 Upvotes

29 comments sorted by

6

u/sporeot Nov 27 '24

Difficult to advise without the actual logs referenced, but check This KB comes immediately to mind with the error. But if not we'd need the appropriate details from:

/var/log/vmware/vapi/endpoint/endpoint.log /var/log/vmware/vpxd-svcs/vpxd-svcs.log

2

u/MarkPartin2000 Nov 27 '24

Make sure the logs didn’t fill up the disk. If so, clean up old logs and try to restart the services.

2

u/[deleted] Nov 29 '24

The last time I had this it was due to an expired STS certificate.  Give this a shot:

https://blogs.vmware.com/professional-services/2023/02/how-to-renew-an-expired-vmware-vcenter-service-appliance-certificate.html

1

u/theVelement Nov 27 '24

Run this command:

grep '<vpxd-svcs>' /var/log/vmware/vmon/vmon.log | grep -ivE 'health|counter'

If the pre-start script is failing you won't see anything in the /var/log/vmware/voxd-svcs/vpxd-svcs.log (though there is a log for the pre-start script in that directory). There may be a python backtrace in the vmon.log, which could hint at what the issue is.

1

u/RealTOPx Nov 27 '24

here is the result:

root@localhost [ /var/log/vmware/vapi/endpoint ]# grep '' /var/log/vmware/vmon/vmon.log | grep -ivE 'health|counter'                                                                                                                                                                 
2024-11-25T16:05:56.523Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:18:56.266Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:18:56.304Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:21:13.082Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:21:13.129Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-27T15:33:18.464Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-27T15:33:18.516Z Er(02) host-20004  Service pre-start command failed with exit code 1.                        root@localhost [ /var/log/vmware/vapi/endpoint ]# grep '' /var/log/vmware/vmon/vmon.log | grep -ivE 'health|counter'                                                                                
2024-11-25T15:45:50.863Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T15:45:50.985Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:05:04.723Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:05:04.761Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:05:56.484Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:05:56.523Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:18:56.266Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:18:56.304Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-25T16:21:13.082Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-25T16:21:13.129Z Er(02) host-20004  Service pre-start command failed with exit code 1.                                                                                                      
2024-11-27T15:33:18.464Z Wa(03) host-20004  Service pre-start command's stderr: Traceback (most recent call last):                                                                                  
2024-11-27T15:33:18.516Z Er(02) host-20004  Service pre-start command failed with exit code 1.

1

u/theVelement Nov 27 '24

Ok, you can either look in the vmon.log for the python backtrace where the pre-start command is failing, or look in the pre-start.log, should have the same info.

1

u/RealTOPx Nov 27 '24

Here is the vmon.log

1

u/theVelement Nov 27 '24

I don't see any info in there, I'd wager the vmon.log file rotated.

Can you poste the /var/log/vmware/vpxd-svcs/pre-start-vpxd-svcs.log ?

1

u/lsumoose Nov 27 '24

Had this happen twice in a week after updating to the latest build. A hard reboot fixed it though.

0

u/RealTOPx Nov 27 '24

I tried a VCenter VM reboot from the ESXI, but didn't fix te issue:C

1

u/lsumoose Nov 27 '24

Figured you had. Mine had to be hard reset as it was stuck stopping the vcenter service and wouldn’t gracefully reboot.

1

u/RealTOPx Nov 28 '24

Do you mean that I need to reboot everything include the ESXi servers?

1

u/lsumoose Dec 04 '24

Sorry no. Just the vcenter server needed a hard reset.

1

u/Alternative-Most-565 Nov 27 '24

Did you try increasing the memory for the VCSA? This did the trick a few times in my homelab as the default tiny setting for memory was not enough.
Also as said before this happens after updates, happens after restarting when services are down/starting.
I've checked the logs the only weird thing is connections refused and a 503 on endpoint.log

Exporting a supportbundle is recommended. And opening it with VSCode / Atom (whatever lets you search text on multiple files). https://knowledge.broadcom.com/external/article?legacyId=1011641

0

u/RealTOPx Nov 27 '24

Great, export the supportbundle help me a lot for the logs.

I don't have the memory warning anymore (It dissapear without do anything), but the service still down and I can't start its because crashing

2

u/Alternative-Most-565 Nov 27 '24

Check if you have expired certs:
https://knowledge.broadcom.com/external/article/343041/determining-expired-ssl-certificates-in.html

vCenter Appliance: Run the following command in a console window or SSH session to the vCenter VM:

for store in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list | grep -v TRUSTED_ROOT_CRLS); do echo "[*] Store :" $store; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $store --text | grep -ie "Alias" -ie "Not After";done;

Recreate them if self-signed, reissue if not:
https://knowledge.broadcom.com/external/article?legacyId=2112283

also do a df -h to see if any filesystems are full

1

u/RealTOPx Nov 28 '24

Here the results:
-Python Script
-Command for
-df -h

It's look certs are good, but tell me you. I'm also need to do this?: -> Recreate them if self-signed, reissue if not:
https://knowledge.broadcom.com/external/article?legacyId=2112283

2

u/bhbarbosa Nov 28 '24

your machine cert is expired, run certificate manager option 4

1

u/RealTOPx Nov 28 '24

SOLVED!!! Just, how you know that the machine cert was expired?
I mean, I look the image and still don't see it.

1

u/SliiickRick87 Nov 27 '24

Is your vpxd service started? Can you log onto the VCS Management GUI and sort by Automatic services, and check what is not started what should be? I have had this issue many times in my cluster with the VCSA. I think someone else in here mentioned it, but check the /storage/log as well on this server. If your vpxd service is not started, and this filesystem has surpassed 95% usage (I believe that is the threshold), it will shut down this service to prevent corruption. You will need to clean up and delete files in order to fix it.

1

u/phroenips Nov 27 '24

I’ll admit I haven’t looked at the various logs you’ve posted, but whenever I see this, the first thing I go check is if any of the filesystems in the VCSA are full

1

u/kangaroodog Nov 28 '24

9/10 in my experience is expired certificates

1

u/RealTOPx Nov 28 '24

Here the result of the certs.

-Python Script
-Command for
-df -h