License issues after vCenter 6.0u2 Upgrade

By | July 21, 2016

I had a weird issue at work with two vCenters in two different sso domains. The vCenters were upgraded to 6.0u1b and then upgraded to 6.0u2. After they were upgraded to 6.0u2, we started seeing the following licensing issues:

  1. After upgrading from vCenter 6.0u1b to version 6.0u2, vCenter shows all the ESXi hosts as unlicensed.
  2. Logging directly into the ESXi hosts shows they are licensed.
  3. The upgrade activity completed without errors
  4. Unable to add hosts.
  5. Unable to create clusters.
  6. License view displays nothing.
  7. Refreshing license view generates an error: GetLicenses Authorization result: User does not have admin rights to perform the operation.
  8. For symptom #4 & 5 the account used is administrator@vsphere.local and any other accounts that are members of the administrator role.
  9. In web client it shows the ESXi hosts as licensed but we are unable to add hosts or create cluster.
  10. In web client the license view displays the licenses.

It took quite a bit of time working with support to get this resolved.

The issue is that there is a local sso service account for vpxd (vpxd-guid) that should be a member of the LicenseService.Administrators group in LDAP, if that entry is missing then you will start to see these issues. Note that you will not see the membership issue in the groups editor in vCenter, you will only see it using a ldap editor.

 
To resolve

  1. Determine vpxd-guid acct that is missing
    1. Log on to the PSC and browse to %programdata%/vmware/vcenter/log/cis-license/license.log. You will see messages like this: 

      2016-07-17T09:35:08.399+08:00 pool-3-thread-1  WARN  common.vmomi.authz.impl.PrivilegeAuthorizerImpl authorize: Authorization result: User does not have admin rights to perform the operation (4885b2a7-62f6-4408-a247-457b9b89388e): sessionNonce: ‘4885b2a7-62f6-4408-a247-457b9b89388e’ sessionUser: ‘vpxd-f3adb4e1-ece7-62f6-b91e-215167b7429e@vsphere.local’ requestUri: ‘/ls/sdk’ requestContext: ‘{operationID=F78B23E1-000003FC-84, realUser=<username>}’

      2016-07-17T09:35:08.399+08:00 pool-3-thread-1  INFO  vim.vmomi.server.impl.ValidatorFutureImpl Validation failed for 48: Authorization result: User does not have admin rights to perform the operation (4885b2a7-62f6-4408-a247-457b9b89388e)

      2016-07-17T09:35:08.408+08:00 pool-3-thread-1  WARN  common.vmomi.authz.impl.PrivilegeAuthorizerImpl authorize: Authorization result: User does not have admin rights to perform the operation (4885b2a7-62f6-4408-a247-457b9b89388e): sessionNonce: ‘4885b2a7-62f6-4408-a247-457b9b89388e’ sessionUser: ‘vpxd-f3adb4e1-ece7-62f6-b91e-215167b7429e@vsphere.local’ requestUri: ‘/ls/sdk’ requestContext: ‘{operationID=F78B23E1-000003FC-84, realUser=<username>}’

      2016-07-17T09:35:08.409+08:00 pool-3-thread-1  INFO  vim.vmomi.server.impl.ValidatorFutureImpl Validation failed for 49: Authorization result: User does not have admin rights to perform the operation (4885b2a7-62f6-4408-a247-457b9b89388e)

      2016-07-17T09:35:08.553+08:00 pool-3-thread-1  WARN  common.vmomi.authz.impl.PrivilegeAuthorizerImpl authorize: Authorization result: User does not have admin rights to perform the operation (4885b2a7-62f6-4408-a247-457b9b89388e): sessionNonce: ‘4885b2a7-62f6-4408-a247-457b9b89388e’ sessionUser: ‘vpxd-f3adb4e1-ece7-62f6-b91e-215167b7429e@vsphere.local’ requestUri: ‘/ls/sdk’ requestContext: ‘{operationID=F78B23E1-000003FC-84, realUser=<username>}’

    2. In this case the user is vpxd-f3adb4e1-ece7-62f6-b91e-215167b7429e
  2. Install jxplorer
  3. Connect to the PSC
  4. Open LicenseService.Administrators
  5. Click on Table Editor
  6. Right click on one of the ‘member’ entries in the table and choose “Add Another Value”
  7. licenseserviceadministrators1
  8. Another ‘member” line show appear with a blank value
  9. Click in the value area and enter in CN=vpxd-<guid>,CN=SerivcePrincipals,DC=vsphere,DC=local
  10. Scroll to the bottom of jXplorer and click submit
  11. Restart service on PSC (if external)
  12. Restart services on vCenter

2 thoughts on “License issues after vCenter 6.0u2 Upgrade

  1. Mayank Kumar Singh

    + In our scenario we had valid member ids for vcenter in SSO > LicenseService.Administrators when verified in jxplorer.
    + It was a working setup for 6 months post upgrade to 6.0 U2.
    + We observed that restart of cm service was hung.
    + Graceful restart of SSO node was hung. We had to use power reset.
    + Manual start of cm service failed with error ‘WARNING: VMware Component Manager may have failed to start.
    + we found that one of the file system on SSO node was full at 100 %. We removed the log files under that partition and then we were able to start cm service successfully.
    + Without having to restart vc service , we just refreshed Licensing Page in VI client and it loaded license details of all assets.

    Reply
  2. John

    We experienced this issue, although ours was an appliance as opposed to a Windows server.
    It turns out that the logs do not purge correctly, which is fixed in 6.0u3, and when the drive fills up, you cannot even add licences.
    Support sent this fix:
    https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2143565
    The remedy worked for us, although it required a restart of the PSC in order for it to work out that it now had some disk space

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *