Fixing the SSL Certificate with Project Honolulu

So if you haven’t heard of it yet Microsoft is doing some pretty cool stuff in terms of Local Server management in what they are calling Project Honolulu. The latest version, 1802, was released March 1, 2018, so it is as good a time as any to get off the ground with it if you haven’t yet. If you’ve worked with Server Manager in versions newer than Windows Server 2008 R2 then the web interface should be comfortable enough that you can feel your way around so this post won’t be yet another “cool look at Project Honolulu!” but rather it will help you with a hiccup in getting it up and running well.

I was frankly a bit amazed that this is evidently a web service from Microsoft not built upon IIS. As such your only GUI based opportunity to get the certificate right is during installation, and that is based on the thumbprint at that, so still not exactly user-friendly. In this post, I’m going to talk about how to find that thumbprint in a manner that copies well (as opposed to opening the certificate) and then replacing the certificate on an already up and running Honolulu installation. Giving props where they do this post was heavily inspired by How to Change the Thumbprint of a Certificate in Microsoft Project Honolulu by Charbel Nemnom.

Step 0: Obtain a certificate: A good place to start would be to obtain or import a certificate to the server where you’ve installed Project Honolulu. If you want to do a public one, fine, but more likely you’ll have a certificate authority available to you internally. I’m not going to walk you through this again, my friend Luca Dell’Oca has a good write up on it here. Just do steps 1-3.

Make note of the Application ID here, you’ll use it later

Step 1: Shut it down and gather info: Next we need to shut down the Honolulu service. As most of what we’ll be doing here today is going to be in Powershell let’s just do this by CLI as well.

Now let’s take a look at what’s currently in place. You can do this with the following command, the output should look like the figure to the right. The relevant info we want to take note of here is 1) The port that we’ve got Honolulu listening on and 2) The Application ID attached to the certificate. I’m just going to reuse the one there but as Charbel points out this is generic and you can just generate a new one to use by using a generator.

Pick a cert, not any cert

Finally, in our quest to gather info let’s find the thumbprint of our newly loaded certificate. You can do this by using the Get-ChildItem command like this

As you can see in the second screenshot that will give you a list of the certificates with thumbprints installed on your server. You’ll need the thumbprint of the certificate you imported earlier.

Step 2: Make it happen: Ok now that we’ve got all our information let’s get this thing swapped. All of this seems to need to be done from the legacy command prompt. First, we want to delete the certificate binding in place now and the ACL. For the example shown above where I’m using port 443 it would look like this:

Now we need to put it back into place and start things back up. Using the port number, certificate thumbprint, and appid from our example the command to re-add the SSL certificate would look like this. You, of course, would need to sub in your own information.  Next, we need to put the URL ACL back in place. Finally, we just need to start the service back up from PowerShell.

Conclusion

At this point, you should be getting a shiny green padlock when you go the site and no more nags about a bad certificate. I hope as this thing progresses out of Tech Preview and into production quality this component gets easier but at least there’s a way.

VVOLs vs. the Expired Certificate

Hi all, I’m writing this to document a fix to an interesting challenge that has pretty much been my life for the last 24 hours or so. Through a comedy of errors and other things happening, we had a situation where the upstream CA from our VMware Certificate Authority (and other things) became very unavailable and the certificate authorizing it to manage certificates expired. Over the course of the last couple of days I’ve had to reissue certificates for just about everything including my Nimble Storage array and as far as vSphere goes we’ve had to revert all the certificate infrastructure to essentially the same as the out of the box self-signed guys and then reconfigure the VMCA as a subordinate again under the Root CA.

Even after all that I continued to have an issue where my Production VVOLs storage was inaccessible to the hosts. That’s not to say they weren’t working, amazingly and as a testament to the design of how VVOLs works my VMs on it ran throughout the process, but I was very limited in terms of the management of those VMs. Snapshots didn’t work, backups didn’t work, for a time even host migrations didn’t work until we reverted to the self-signed certs.

Thanks for a great deal of support and help from both VMware support and Nimble Storage Support we were finally able to come up with a runbook in dealing with a VVOL situation where major certificate changes occurred on the vSphere side. There is an assumption to this process that by the time you’ve got here all of your certificates both throughout vSphere as well as with the Nimble arrays are good and valid.

  1. Unregister the VASA provider and Web Client integration from the Nimble array. This can be done either through the GUI in Administration>VMware Integration by editing your vCenter, unchecking the boxes for the Web Client and VASA Provider and hitting save. This can also be done via the CLI using the command
  2. Register the integrations back in. Again, from the GUI simply just check the boxes back and hit save. If successful you should see a couple of little green bars briefly appear at the top of the screen saying the process was successful. From the CLI the commands are pretty similar
  3. Verify that your VASA provider is available in vCenter and online. This is just to make sure that the integration was successful. In either the Web Client or the HTML5 client go to vCenter> Configure> Storage Provider and look for the entry that matches the name of your array group and in the URL has the IP address of your array’s management interface. This should show as online. As you have been messing with certificates its probably worth looking at the Certificate Info tab as well while you are here to verify that the certificate is what you expect.
  4. Refresh the CA Certificates on each your hosts. Next, we need to ensure that all of the CA certificates are available on the hosts to ensure they can verify the certificates presented to them by the storage array. To do this you can either right-click each host > Certificates > Refresh CA Certificates or if you navigate to the configuration tab of each host, go to Certificate there is a button there as well. While in the window it is worth looking at the Status of each host’s certificate and ensure that it is Good.
  5. Restart the vvold service on each host. This final step was evidently the hardest one to nail down and find in the documentation. The simplest method may be to simply reboot each of your hosts as long as you can put them into maintenance mode and evacuate them first. The quicker way and the way that will let you keep things running is to enter a shell session on each of your hosts and simply run the following command:

    Once done you should see a response like the feature image on this post and a short while later your VVOLs array will again become available for each host as you work on them.

That’s about it. I really cannot thank the engineers at VMware (Sujish) and Nimble (Peter) enough for their assistance in getting me back to good. Also I’d like to thank Pete Flecha for jumping in at the end, helping me and reminding me to blog this.

If nothing else I hope this serves as a reminder to you (as well as myself) that certificates should be well tended to, please watch them carefully. ūüėČ

Support Adobe Digital ID Signing with Automated Microsoft CA User Certificate Generation

Just a quick how to, wanting to document a task I have recently had need of. This process has a perquisite of you having a Microsoft Certificate Authority already available in your environment.

  1. Start > Run >mmc
    1. Add Remove Snap-ins and choose the following
      – Certificate Authority (when prompted add the name of your CA)
      – Certificate Templates
      – Group Policy Management
  2. In Certificate Templatesright click on “User” and choose “Duplicate Template”
    1. Set compatibility settings as needed. If you have a 2008 R2 pure Active Directory environment make it match. In terms of Certificate Recipient make it match the oldest OS you have in use.
    2. Under General Change the Name to something meaningful as you’ll be referencing it later.
    3. Under the Security Tab set Domain Users to have both Enroll and Autoenroll permissions
  3. In Certificate Authorityright click on the “Certificate Templates”subfolder and choose New> “Certificate Template to Issue”
    1. Choose your newly created Certificate Template
  4. In Group Policy Management we are going to do a couple of things; setup your domain for certificate auto enrollmentand also define registry settings for Adobe Acrobat and Acrobat Reader.
    1. In any GPO that will hit the users you wish to have certificates (Default Domain Policy for example) choose to edit.
    2. Navigate to User Configuration> Windows Settings> Security Settings> Public Key Policies
    3. Double click on Certificate Services Client- Auto-Enrollment and set
      – Configuration Model: Enabled
      – Check Renew expired certificates…
      – Check Update certificates that use certificate templates
      – Hit OK
  5. Digital Signature Verification PreferencesBy default Adobe Acrobat and Reader only recognize certificates that are signed by the usual public authorities as trusted, so you have to tell it to look at what is available in the local Windows Certificate Store. In Adobe Acrobat or Acrobat Reader you can do this in Preferences, under Signatures>Verification and¬†enable “Validating Signatures” under Windows Integration. This can be cumbersome across the enterprise but luckily this data is saved in a registry key, which means that through Group Policy Preferences we can manage this setting. ¬†The fix below will work for all Acrobat or Acrobat Reader versions 7 or later
    1. Select the GPO of your choice to edit (again, I recommend the Default Domain Policy) and navigate to User Configuration> Preferences> Registry
    2. Right click in the window New> Registry Item
    3. You will need to create an entry with the following attributes:
      – Hive: HKEY_CURRENT_USER
      РKey Path: Software\Adobe\product\versionnumber\Security\cASPKI\cMSCAPI_DirectoryProvider
      * (Example for Acrobat Pro 11: Software\Adobe\Adobe Acrobat\11.0\Security\cASPKI\cMSCAPI_DirectoryProvider)
      – Value name: iMSStoreTrusted
      – Value type: REG_DWORD
      – Value data: 60 (hexidecimal)
      – Hit OK
    4. gp-prefRepeat steps B & C for each product/version combination you have in your environment. For example, in our environment we only have one version of Reader, but 3 different major versions of Acrobat Pro, so I needed 4 variants of this key to cover each of them.

And that’s it! It will probably take a little while for these policy changes to naturally propagate, but once it does so it works very slickly. Once done you and your users will be able to use their generated certificate as a Digital ID to sign any documents with a digital signature field in a fillable form. Do keep in mind that while this will work and absolutely can and should be trusted within your organization, if you or your users are in need of this type of service between organizations you will probably want to call the fine folks at Verisign or Thawte.

To for more information check out