VVOLs vs. the Expired Certificate

Hi all, I’m writing this to document a fix to an interesting challenge that has pretty much been my life for the last 24 hours or so. Through a comedy of errors and other things happening, we had a situation where the upstream CA from our VMware Certificate Authority (and other things) became very unavailable and the certificate authorizing it to manage certificates expired. Over the course of the last couple of days I’ve had to reissue certificates for just about everything including my Nimble Storage array and as far as vSphere goes we’ve had to revert all the certificate infrastructure to essentially the same as the out of the box self-signed guys and then reconfigure the VMCA as a subordinate again under the Root CA.

Even after all that I continued to have an issue where my Production VVOLs storage was inaccessible to the hosts. That’s not to say they weren’t working, amazingly and as a testament to the design of how VVOLs works my VMs on it ran throughout the process, but I was very limited in terms of the management of those VMs. Snapshots didn’t work, backups didn’t work, for a time even host migrations didn’t work until we reverted to the self-signed certs.

Thanks for a great deal of support and help from both VMware support and Nimble Storage Support we were finally able to come up with a runbook in dealing with a VVOL situation where major certificate changes occurred on the vSphere side. There is an assumption to this process that by the time you’ve got here all of your certificates both throughout vSphere as well as with the Nimble arrays are good and valid.

  1. Unregister the VASA provider and Web Client integration from the Nimble array. This can be done either through the GUI in Administration>VMware Integration by editing your vCenter, unchecking the boxes for the Web Client and VASA Provider and hitting save. This can also be done via the CLI using the command
  2. Register the integrations back in. Again, from the GUI simply just check the boxes back and hit save. If successful you should see a couple of little green bars briefly appear at the top of the screen saying the process was successful. From the CLI the commands are pretty similar
  3. Verify that your VASA provider is available in vCenter and online. This is just to make sure that the integration was successful. In either the Web Client or the HTML5 client go to vCenter> Configure> Storage Provider and look for the entry that matches the name of your array group and in the URL has the IP address of your array’s management interface. This should show as online. As you have been messing with certificates its probably worth looking at the Certificate Info tab as well while you are here to verify that the certificate is what you expect.
  4. Refresh the CA Certificates on each your hosts. Next, we need to ensure that all of the CA certificates are available on the hosts to ensure they can verify the certificates presented to them by the storage array. To do this you can either right-click each host > Certificates > Refresh CA Certificates or if you navigate to the configuration tab of each host, go to Certificate there is a button there as well. While in the window it is worth looking at the Status of each host’s certificate and ensure that it is Good.
  5. Restart the vvold service on each host. This final step was evidently the hardest one to nail down and find in the documentation. The simplest method may be to simply reboot each of your hosts as long as you can put them into maintenance mode and evacuate them first. The quicker way and the way that will let you keep things running is to enter a shell session on each of your hosts and simply run the following command:

    Once done you should see a response like the feature image on this post and a short while later your VVOLs array will again become available for each host as you work on them.

That’s about it. I really cannot thank the engineers at VMware (Sujish) and Nimble (Peter) enough for their assistance in getting me back to good. Also I’d like to thank Pete Flecha for jumping in at the end, helping me and reminding me to blog this.

If nothing else I hope this serves as a reminder to you (as well as myself) that certificates should be well tended to, please watch them carefully. ūüėČ

Support Adobe Digital ID Signing with Automated Microsoft CA User Certificate Generation

Just a quick how to, wanting to document a task I have recently had need of. This process has a perquisite of you having a Microsoft Certificate Authority already available in your environment.

  1. Start > Run >mmc
    1. Add Remove Snap-ins and choose the following
      – Certificate Authority (when prompted add the name of your CA)
      – Certificate Templates
      – Group Policy Management
  2. In Certificate Templatesright click on “User” and choose “Duplicate Template”
    1. Set compatibility settings as needed. If you have a 2008 R2 pure Active Directory environment make it match. In terms of Certificate Recipient make it match the oldest OS you have in use.
    2. Under General Change the Name to something meaningful as you’ll be referencing it later.
    3. Under the Security Tab set Domain Users to have both Enroll and Autoenroll permissions
  3. In Certificate Authorityright click on the “Certificate Templates”subfolder and choose New> “Certificate Template to Issue”
    1. Choose your newly created Certificate Template
  4. In Group Policy Management we are going to do a couple of things; setup your domain for certificate auto enrollmentand also define registry settings for Adobe Acrobat and Acrobat Reader.
    1. In any GPO that will hit the users you wish to have certificates (Default Domain Policy for example) choose to edit.
    2. Navigate to User Configuration> Windows Settings> Security Settings> Public Key Policies
    3. Double click on Certificate Services Client- Auto-Enrollment and set
      – Configuration Model: Enabled
      – Check Renew expired certificates…
      – Check Update certificates that use certificate templates
      – Hit OK
  5. Digital Signature Verification PreferencesBy default Adobe Acrobat and Reader only recognize certificates that are signed by the usual public authorities as trusted, so you have to tell it to look at what is available in the local Windows Certificate Store. In Adobe Acrobat or Acrobat Reader you can do this in Preferences, under Signatures>Verification and¬†enable “Validating Signatures” under Windows Integration. This can be cumbersome across the enterprise but luckily this data is saved in a registry key, which means that through Group Policy Preferences we can manage this setting. ¬†The fix below will work for all Acrobat or Acrobat Reader versions 7 or later
    1. Select the GPO of your choice to edit (again, I recommend the Default Domain Policy) and navigate to User Configuration> Preferences> Registry
    2. Right click in the window New> Registry Item
    3. You will need to create an entry with the following attributes:
      – Hive: HKEY_CURRENT_USER
      РKey Path: Software\Adobe\product\versionnumber\Security\cASPKI\cMSCAPI_DirectoryProvider
      * (Example for Acrobat Pro 11: Software\Adobe\Adobe Acrobat\11.0\Security\cASPKI\cMSCAPI_DirectoryProvider)
      – Value name: iMSStoreTrusted
      – Value type: REG_DWORD
      – Value data: 60 (hexidecimal)
      – Hit OK
    4. gp-prefRepeat steps B & C for each product/version combination you have in your environment. For example, in our environment we only have one version of Reader, but 3 different major versions of Acrobat Pro, so I needed 4 variants of this key to cover each of them.

And that’s it! It will probably take a little while for these policy changes to naturally propagate, but once it does so it works very slickly. Once done you and your users will be able to use their generated certificate as a Digital ID to sign any documents with a digital signature field in a fillable form. Do keep in mind that while this will work and absolutely can and should be trusted within your organization, if you or your users are in need of this type of service between organizations you will probably want to call the fine folks at Verisign or Thawte.

To for more information check out