In a previous blog post, I discussed how I found a VSC plugin submitting an old password for my account and locking me out.
Today, I have a similar story but with a few different tools and culprits.
I changed my password on 11/1 and didn’t immediately see any issues. The next day, 11/2, I noticed that my account was locked out. I proceeded to use a in-house Account Lockout Tool to query the DCs to determine what sites the bad logins were coming from.
At that point I asked my AD team to comb the security logs using Qradar to find the source of the bad logins.
They told me that it was coming from one my vCenter servers.
Based on the last time that this happened, I first wanted to check if vCenter logged the bad login. I have Log Insight integrated with all of my vCenters, so it should be collecting any failed logins. Log Insight has a few pre-extracted fields that are exactly what you need. I added a filter for vmw_vc_auth_failed_user and grouped by vmw_vc_auth_failed_source.
You can see in this image that the failures stopped at the point (when I found the culprit). Since I chose to group by the source, the IP is in the Key in the top right. You can see that the IP ends in 189
When I looked up the IP address, it pointed to an old vCOPs 5.8.1 instance that we were keeping around for historical data (we have a 6.0 instance as well). That set off some alarms in my head, I remember having to re-register this instance for whatever reason, so maybe my account was still cached there.
I tried to login to the admin page at into https://<ip>/admin but the page was stuck at “Verifying”. I found this KB that detailed how to reset the admin password.
Once In, I could see that my account was definitely used at one time to register a vCenter. I have no idea why vCOPS would keep that password though past the initial registration? There was a collection user defined, so the registration user was not needed. Regardless, I changed the registration account to another vCenter admin account and saw that the bad passwords from that vCenter away.
The next morning I looked in the Account Lockout tool to see if there were any more bad passwords. Three more submitted at 6:33am! I went back into Log Insight and found a different IP as the source. That server ended up had Veeam Backup and Recovery Free edition installed. I probably used my account to link it to a vCenter at some point. Once that was uninstalled, all of my bad logins ceased