VMworld 2017 US: T -2

By | | ,
1 Comment

I write this while traveling to sunny and amazingly hot Las Vegas for the 2017 edition of VMworld US. I hope to provide feedback and news throughout the conference, highlighting not only the excellent content and programs but also the best the virtualization community has to offer.

Today will be a travel day as well as a day to meet up with friends, new and old. Tomorrow, the Sunday before the conference, is when the real fun begins with things like Opening Acts for me, TAM and partner content for others as well as a number of social events.

What We Know So Far

Yesterday was the day that Vmware went on a killing spree, announcing the depreciation of Windows based vCenter, the flash based vSphere web client and the vmkLinux APIs and its associated driver ecosystem. All of these enter the depreciated state with the next major version of vSphere and then will be gone for ever and ever in the revision after that. Each of these are significant steps towards the evolution of vSphere as we know it, and when coupled with the advances in PowerCLI in version 6.5 the management of our in house infrastructure has been changed for the better.

These announcements came rapid fire on the Friday before Vmworld with the death of the Windows based vCenter coming first. As we have had versions of varying success of the vCenter Server Appliances (VCSA) for over 5 years now it’s been a long time coming. I myself migrated two years ago and while it was good then with the latest 6.5 version, with its PhotonOS base, excellent migration wizard and in appliance vCenter Update Manager support it has show it is definitely the way forward.

The flash … Go Read More →

Notes on Migrating from an “All in One” Veeam Backup & Replication Server to a Distributed System

By | | , ,
Leave a comment

One of the biggest headaches I not only have and have heard about from other Veeam Backup & Replication administrators have is backup server migrations. In the past I have always gone the “All-in-One” approach, have one beefy physical server with Veeam directly installed and housing all the roles. This is great! It runs fast and it’s a fairly simple system to manage, but the problem is every time you need more space or your upgrading an old server you have to migrate all the parts and all the data. With my latest backup repository upgrade I’ve decided to go to a bit more of a distributed architecture, moving the command and control part out to a VM with an integrated SQL server and then letting the physical box handle the repository and proxy functions producing a best of both worlds setup, the speed and simplicity of all the data mover and VM access happening from the single physical server while the setup and brains of the operation reside in a movable, upgradable VM.

This post is mostly composed of my notes from the migration of all parts of VBR. The best way to think of this is to split the migration into 3 major parts; repository migration, VBR migration, proxy migration, and VBR migration. These notes are fairly high level, not going too deep into the individual steps. As migrations are complex if any of these parts don’t make sense to you or do not provide enough … Go Read More →

A VMworld US 2017 To Do List

By | | ,
Leave a comment

If you work in the virtualization or datacenter field (are they really different anymore?) you probably know that VMworld US 2017 is next week, August 27-31. While VMware may not be the only option out there when it comes to virtualization anymore VMworld is still the defacto event for people in the field. This conference’s definition of community is unrivaled in scope with just as much if not more going on outside of the conference agenda as  in it.

As with all things worth doing conference attendance probably needs a checklist. Have you done yours? If not here are the high points of mine. I’m not going to bore you with “Jim will be attending session so and so”; well except for VMTN6699U and VMTN6700U you should totally join me at those sessions, but these are pretty general things I try to do each time.

  • Take Your Vitamins– I hate to say it but the Vegas Flu is a real thing. Between being in the recirculated air of a jumbo jet for any number of hours to bookend event and being in the recirculated air of a Vegas hotel/casino/conference center I always seem to get at least a mild head cold at some point during the week. Start about now taking whatever version of Vitamin C supplement you like and do so throughout the event to help head this issue off.
  • Bring Sharable Power- The average conference attendee has 3 devices on them at all times, phone, tablet and laptop. These things will start to get low on battery about midday and that just won’t do. In theory lots of places will have power outlets but with 25,000+ attendees they are still in short supply. I typically bring a big battery pack, a Go Read More →

Tech Conferences in Las Vegas for Newbies

By | | , ,
Leave a comment

As June is here we are deep into tech conference season already so I find myself behind the curve somewhat with this post, but here we are. I am extremely fortunate to have an employer who understands the value of attending Tech Conferences for IT Professionals and I’ve been able to attend at least one each year since 2014; going back and forth between CiscoLive and VMworld with a sprinkling of VeeamON and more local events such as vBrisket and VMUGs for good measure. As a “Hyper-Converged Admin” my choice of which “biggie” conference is done each year by looking at where my projects land; last year was CiscoLive due to a lot of Voice and Security Projects, this year VMworld due to lots of updates coming down the pike there and a potential VDI project.

The problem when you have a conference with north of 25,000 attendees is that you are limited in where you put these on. While Cisco does tend to move around some, VMworld has typically either been in San Francisco or Las Vegas. With the Moscone Center closed again this year for renovation we find pretty much all of the big guys are back in Las Vegas, with both CiscoLive and VMworld at Mandalay Bay once again as well as AWS re:Invent and Dell/EMC World in town this year as well. If you haven’t been to one of these Tech Conferences before or to Las Vegas both can be both exciting and overwhelming, but with a little help from others and some decent tips neither are that big of a deal.

Las Vegas Basics

Go Read More →

Cisco Voice Servers Version 11.5 Could Not Load modules.dep

By | |
Leave a comment

About 6 months ago we updated 3/4 of our Cisco Telephony environment from 8.5 to 11.5. The only reason we didn’t do it all is because UCCX 11.5 wasn’t out yet so it went to 11. While there were a few bumps in the road; resizing VMs, some COP files, etc. the update went well. Unfortunately once it was done we starting having a glorious issue where after a reboot the servers sometimes failed to boot, presenting “FATAL: Could not load /lib/modules/2.6.32-573.18.1.el6.x86_64/modules.dep: No such file or directory”. Any way you put it, this sucked.

The first time this happened I call TAC and while they had seen it, they had no good answer except for rebuild the VM, restore from backup. Finally after the 3rd time (approximately 3 months after install) the bug had been officially documented and (yay) it included a work around. The good news is that the underlying issue at this point has been fixed in 11.5(1.11900.5) and forward so if you are already there, no problems.

The issue lies with the fact that the locked down build of RHEL 6 that any of the Cisco Voice server platforms are built on don’t handle VMware Tools updates well. It’s all good when you perform a manual update from their CLI and use their “utils vmtools refresh” utility, but many organizations, mine included, choose to make life easier and enable vCenter Update Manager to automatically upgrade the VMware tools each time a new version is available and the VM is rebooted.

So how do you fix it? While the bug ID has the fix in it, if you aren’t a VMware regular they’ve left out a few steps and it may not be the easiest thing to follow. So here I’m going to run down the entire process and … Go Read More →

Learning To Pick The Right Tech Conference at vBrisket- TOMORROW!

By | | ,
Leave a comment

Hey all, just a quick post to mention that the fine folks at vBrisket will be having a get together February 24th at 2 PM at Grist House Craft Brewery in Pittsburgh. If you work in the virtualization industry and haven’t heard of vBrisket yet you should get to know them because they have a great thing going.  vBrisket takes the typical User Group back to its vendor independence roots, allowing you to focus more on your general virtualization career and less on the path of any particular vendor. At the same time it gives Clint, Gabe, Jaison, and John a great reason to bring out the smokers and prepare enough meat to feed a brewery full of techies.

I’m honored to have been invited to join the panel discussion this time. The topic is “Tech Conferences – What are the right ones for you?” This will be moderated by the vBrisket team and includes myself, John White, Mike Muto, and Justin Paul. As I see my attendance at various conferences as a big driver in the success of my career and my growth as a technology worker I’m excited to be included.

Of course this meeting wouldn’t be possible without the sponsorship from Zerto. At the meeting they’ll be talking I’m sure about their new conference, ZertoCON in Boston May 22-24th.

So if you are in the Pittsburgh area tomorrow and would like to attend just be there at 2, I look forward to meeting up!

Windows Server Deduplication, Veeam Repositories, and You!

By | |
1 Comment

Backup, among other things, is very good at creating multiple copies of giant buckets of data that don’t change much and tend to sit for long periods of time. Since we are in modern times, we have a number of technologies to deal with this problem, one of which is called deduplication with quite a few implementations of it. Microsoft has had server-based storage versions since Windows 2008 R2 that has gotten better with each release, but as any technology still has its pitfalls to be mindful of. In this post I’m going to look a very specific use case of Windows server deduplication, using it as the storage beneath your Veeam Backup and Replication repositories, covering some basic tips to keep your data healthy and performance optimized.

What is Deduplication Anyway?

For those that don’t work with it much imagine you had a copy of War and Peace stored as a Word document with an approximate file size 1 MB. Each day for 30 days you go into the document and change 100 KB worth of the text in the document and save it as a new file on the same volume. With a basic file system like NTFS this would result in you having 31 MB tied up in the storage of these files, the original and then the full file size of each additional copy.

Now let’s look at the same scenario on a volume with deduplication enabled. The basic idea of deduplication replaces identical blocks of data with very small pointers back to a common copy of the data. In this case after 30 days instead of having 31 MB of data sitting on disk you would approximately 4 MB; the original 1 MB plus just the 100 KB of incremental updates. As far as the … Go Read More →

Veeam Vanguard Again in 2017

By | | , ,
1 Comment

It has been a great day here because today I learned that I have once again been awarded acceptance into the excellent Veeam Vanguard program, my third time. This program, above any others that I am or have been involved with takes a more personal approach to creating a group of awardees who not only deserve anything good they get out of it but give back just as much to the community itself. In only its 3rd year the group has grown; from 31 the first year, 50(ish) the second, to a total of 62 this year. There are 21 new awardees in that 62 number so there really isn’t a rubber stamp to stay included, it is legitimately awarded each year. The group has grown each year but as you can see not by the leaps and bounds others have, and for good reason. There is no way this experience could be had with a giant community.

At this point in the post I would typically tell you a bit about what the Vanguard program is and isn’t but honestly, Veeam’s own Dmitry Kniazev really put it best in a couple recent posts, “Veeam Vanguard Part 1: WTH Is This?” and “Veeam Vanguard Part 2: What It’s Not.”  What I will add is that as nice as some of the perks are, as DK says in the Part 1 post the true perk is the intangibles; a vibrant community full of some of the smartest, most passionate people in the industry and in many cases access right to the people approving and disapproving changes to their software. These are the thing that made me sweat approval time.

Once again I would give a giant thank you to Veeam Software and especially the whole Vanguard crew. This includes Rick … Go Read More →

Why Is My Nimble Storage Firmware Update Not Available

By | |
Leave a comment

Today, like everyday as a technology professional, I got the opportunity to learn something new. After seeing posts on social media and articles that Nimble Storage with their NimbleOS version 3.6 supports the shiny new features of VMware’s vSphere 6.5 release including VVOLs 2.0 and VASA 3.0. After reading through the release notes and not seeing anything to really stress me out in the known issues I went to begin the download for an update in the off hours. To my early adopter horror I saw there was no download available! Had I misread the releases, did I imagine that the release notes really were for 3.6? No, those were real and it should be there. After asking around I learned that Nimble in a notable effort to save us from ourselves will from time to time blacklist you from receiving updates due to things they observe through their excellent InfoSight analytics system.

The problem with this is they don’t really make easily apparent that you are blacklisted from anywhere close to the download screen. In order to see if you are blacklisted  you have to switch over from the array management screen to InfoSight, go to Manage > Assets > Click on the Array, and then at the top where it says “Version: ….” click on the version link. There finally you will either see the new version in black if you are good to upgrade or as shown in my image, in red if blacklisted. Even with this it still doesn’t tell you why you are blacklisted, you have to call support to learn that.

Conclusion

The idea of blacklisting arrays that show signs of things known not to play well with future versions of software is a noble idea and has the potential to keep … Go Read More →

Fixing Domain Controller Boot in Veeam SureBackup Labs

By | | ,
1 Comment

We’ve been dealing with an issue for past few runs of our monthly SureBackup jobs where the Domain Controller boots into Safe Mode and stays there. This is no good because without the DC booting normally you have no DNS, no Global Catalog or any of the other Domain Controller goodness for the rest of your servers launching behind it in the lab. All of this seems to have come from a change in how domain controller recover is done in Veeam Backup and Replication 9.0, Update 2 as discussed in a post on the Veeam Forums. Further I can verify that if you call Veeam Support you get the same answer as outlined here but there is no public KB about the issue. There are a couple of ways to deal with this, either each time or permanently, and I’ll outline both in this post.

The booting into Safe Mode is totally expected, as a recovered Domain Controller object should boot into Directory Services Restore mode the first time. What is missing though is that as long as you have the Domain Controller box checked for the VM in your application group setup then once booted Veeam should modify the boot setup and reboot the system before presenting it to you as a successful launch. This in part explains why when you check the Domain Controller box it lengthens the boot time allowed from 600 seconds to 1800 seconds by default.

On the Fly Fix

If you are like me and already have the lab up and need to get it fixed without tearing it back down you simply need to clear the Safe Boot bit and reboot from Remote Console. I prefer to

  • Make a Remote Console connection to the  lab booted VM and login
  • Go to … Go Read More →