VMworld 2017 US: T -2

I write this while traveling to sunny and amazingly hot Las Vegas for the 2017 edition of VMworld US. I hope to provide feedback and news throughout the conference, highlighting not only the excellent content and programs but also the best the virtualization community has to offer.

Today will be a travel day as well as a day to meet up with friends, new and old. Tomorrow, the Sunday before the conference, is when the real fun begins with things like Opening Acts for me, TAM and partner content for others as well as a number of social events.

What We Know So Far

Yesterday was the day that Vmware went on a killing spree, announcing the depreciation of Windows based vCenter, the flash based vSphere web client and the vmkLinux APIs and its associated driver ecosystem. All of these enter the depreciated state with the next major version of vSphere and then will be gone for ever and ever in the revision after that. Each of these are significant steps towards the evolution of vSphere as we know it, and when coupled with the advances in PowerCLI in version 6.5 the management of our in house infrastructure has been changed for the better.

These announcements came rapid fire on the Friday before Vmworld with the death of the Windows based vCenter coming first. As we have had versions of varying success of the vCenter Server Appliances (VCSA) for over 5 years now it’s been a long time coming. I myself migrated two years ago and while it was good then with the latest 6.5 version, with its PhotonOS base, excellent migration wizard and in appliance vCenter Update Manager support it has show it is definitely the way forward.

The flash client was the next announcement to come and again, we are looking at an depreciation that needs to happen and is most definitely going to be a good thing but does come with some apprehension. With most things that have been depreciated by Vmware we’ve had at least 1 feature rich version of the replacement out and stable before they announced the predecessor’s demise. This isn’t the case with the flash based web client. While the latest builds are getting very, very good there are still major things that either are quirky or simply aren’t there yet. The good news to this is we have been given almost immediately assurances by everyone involved with the product management that we the vSphere admins will never be left without a GUI management ability for any given task we have today and I for one believe them. The last components of what is known as the HTML5 client in my opinion simply can’t come enough, I’m tired of having to hop through multiple GUIs and browsers to be able to perform basic tasks in my daily work life.

Finally the day was finished with the announced depreciation of the non-native Linux drivers. To be honest I didn’t know that these were even still a thing as every Linux VM I’ve rolled for the past many years have been able to work with the native drivers. I’m sure there are those that at this point may still need additional time but the as the removal is still a couple of versions off this should be something can be mitigated now that the end is known.


With all of these preconference announcements related to Vmware’s flagship product is this going to be the year where Vmworld is chocked full of improvements to vSphere. This will be my 3rd one in 4 years and each year I’ve felt their focus was elsewhere. While vSAN, NSX, and the like are definitely where the company’s seeing growth all of these things rely on vSphere as an underlay. I for one would be happy to see a little love shown here.

With that happy thought I’m going to shut it down and land. For those coming to Vmworld this weekend safe travels and for those at home look for more info as its known here on koolaid.info.

Notes on Migrating from an “All in One” Veeam Backup & Replication Server to a Distributed System

One of the biggest headaches I not only have and have heard about from other Veeam Backup & Replication administrators have is backup server migrations. In the past I have always gone the “All-in-One” approach, have one beefy physical server with Veeam directly installed and housing all the roles. This is great! It runs fast and it’s a fairly simple system to manage, but the problem is every time you need more space or your upgrading an old server you have to migrate all the parts and all the data. With my latest backup repository upgrade I’ve decided to go to a bit more of a distributed architecture, moving the command and control part out to a VM with an integrated SQL server and then letting the physical box handle the repository and proxy functions producing a best of both worlds setup, the speed and simplicity of all the data mover and VM access happening from the single physical server while the setup and brains of the operation reside in a movable, upgradable VM.

This post is mostly composed of my notes from the migration of all parts of VBR. The best way to think of this is to split the migration into 3 major parts; repository migration, VBR migration, proxy migration, and VBR migration. These notes are fairly high level, not going too deep into the individual steps. As migrations are complex if any of these parts don’t make sense to you or do not provide enough detail I would recommend that you give the fine folks at Veeam support a call to ride along as you perform your migration.

I. Migrating the Repository

  1. Setup 1 or more new repository servers
  2. Add new repository pointing to a separate folder (i.e. D:\ConfigBackups) on the new repository server to your existing VBR server exclusively for Configuration Backups. These cannot be included in a SOBR. Change the Config Backup Settings (File > Config Backup) to point to the new repository. This is also probably a good time to go ahead and run a manual Config Backup while you are there to snapshot your existing setup.
  3. Create one or more new backup repositories on your new repository server(s) to your existing VBR server configuration.
  4. Create Scale Out Backup Repository (SOBR), adding your existing repository and new repository or repositories as extents.
  5. All of your backup jobs should automatically be changed to point to the SOBR during the setup but check each of your jobs to ensure they are pointing at the SOBR.
  6. If possible go ahead and do a regular run of all jobs or wait until your regularly scheduled run.
  7. After successful run of jobs put the existing extent repository into Maintenance Mode and evacuate backups.
  8. Remove existing repository from the SOBR configuration and then from the Backup Repositories section. At this point no storage of any jobs should actually be flowing through your old server. It is perfectly fine for a SOBR to only contain a single extent from a data locality standpoint.

II. Migrate the Backup and Guest Interaction Proxies

  1. Go to each of your remaining repositories and set proxy affinity to the new repository server you have created. If you have previously scaled out your backup proxies then you can ignore this step.
  2. Under Backup Proxy in Backup Infrastructure remove the Backup Proxy installation on your existing VBR server.  Again, if possible you may want to run a job at this point to ensure you haven’t broken anything in the process.
  3. Go to each of your backup jobs that are utilizing the Guest Processing features. Ensure the guest interaction proxy at the bottom of the screen is set to either your new repository server, auto or if scaled out another server in your infrastructure.

III. Migrate the Veeam Backup & Replication Server

  1. Disable all backup, Backup Copy and Agent jobs on your old server that have a schedule.
  2. Run a Config Backup on the old server. If you have chosen to Encrypt your configuration backup the process below is going to be a great test to see if you remember or documented it. If you don’t know what this is go ahead and change it under File>Manage Passwords before running this final configuration backup.
  3. Shutdown all the Veeam services on your existing backup server or go ahead and power it down. This ensures you won’t have 2 servers accessing the same components.
  4. If not already done, create your new Veeam Backup and Replication server/VM. Be sure to follow the guidelines on sizing available in the Best Practices Guide.
  5. Install Veeam Backup, ensuring that you use the same version and update as production server. Safest bet is to just have both patched to the latest level of the latest version.
  6. Add a backup repository on your new server pointing to the Config Backup repository folder you created in step 2 of the Migrating the Repository step.
  7. Go to Config Backup and hit the “Restore” button.
  8. As the wizard begins choose the Migrate option.
  9. Change the backup repository to the repository created in step 5, choose your latest backup file which should be the same as the one created in step 2 above.
  10. If encrypted, specify your backup password and then choose to overwrite the existing VeeamBackup database you created when you installed Veeam in step 4. The defaults should do this.
  11. Choose any Restore Options you may want. I personally chose to check all 4 of the boxes but each job will have its own requirements.
  12. Click the Finish button to begin the migration. From this point if any screens or messages pop up about errors or issues in processing it is a good idea go to ahead and contact support. All this process does is move the database from the old server to the new, changing any references to the old server to the new along the way. If something goes wrong it is most likely going to have a cascade effect and you are going to want them involved sooner than later.

IV. Verification and Cleanup

  1. Now that your server has been migrated it’s a good idea to go through all the tabs in your Backup Infrastructure section, ensuring that all your information looks correct.
  2. Go ahead and run a Config Backup at this point. That’s a nice low-key way to ensure that all of the basic Veeam components are working correctly.
  3. Re-enable your disabled backup, backup copy and Agent jobs. If possible go ahead and run one and ensure that everything is hunky dory there.


This process when working correctly is extremely smooth. I’ll be honest and admit that I ran into a what I believe is a new bug in the VBR Migration wizard. We had a few SureBackup jobs that had been setup and while they had been run they have never been modified again since install. When this happens VBR notes the job_modified field of the job config database record as NUL. During the migration the wizard left those fields blank in the restored database, which is evidently something that is checked when you start the Veeam Backup Service. While the Service in the basic services.msc screen appears to be running under the hood you are only getting partial functionality. In my case support was able to go in and modify the database and re-include the NUL data to the field, but if you think you might have this issue it might be worth changing something minor on all of your jobs before the final configuration backup.


If you’ve made it this far, congrats! You should be good to go. While the process seems daunting it really wasn’t all that bad. If I hadn’t run into an issue it wouldn’t have been bad at all. The good news is that at this point you should be able to scale your backup system much easier without the grip and rip that used to be required.

A VMworld US 2017 To Do List

If you work in the virtualization or datacenter field (are they really different anymore?) you probably know that VMworld US 2017 is next week, August 27-31. While VMware may not be the only option out there when it comes to virtualization anymore VMworld is still the defacto event for people in the field. This conference’s definition of community is unrivaled in scope with just as much if not more going on outside of the conference agenda as  in it.

As with all things worth doing conference attendance probably needs a checklist. Have you done yours? If not here are the high points of mine. I’m not going to bore you with “Jim will be attending session so and so”; well except for VMTN6699U and VMTN6700U you should totally join me at those sessions, but these are pretty general things I try to do each time.

  • Take Your Vitamins– I hate to say it but the Vegas Flu is a real thing. Between being in the recirculated air of a jumbo jet for any number of hours to bookend event and being in the recirculated air of a Vegas hotel/casino/conference center I always seem to get at least a mild head cold at some point during the week. Start about now taking whatever version of Vitamin C supplement you like and do so throughout the event to help head this issue off.
  • Bring Sharable Power- The average conference attendee has 3 devices on them at all times, phone, tablet and laptop. These things will start to get low on battery about midday and that just won’t do. In theory lots of places will have power outlets but with 25,000+ attendees they are still in short supply. I typically bring a big battery pack, a travel surge protector and USB power cables for everything under the sun so that I can plug in and share at sessions and keynotes.
  • Get There Early and Be Ready To Learn– While the conference doesn’t start in earnest until Monday the 27th I always try to arrive midday Saturday because there is so much going on before the conference starts. One of the highlights of the entire conference to me each year is Opening Acts, a series of panel sessions put on by VMunderground and vBrownBag on Sunday afternoon. These sessions always prove to be insightful and are traditionally more career-centric or more wide-ranging than your typical VMworld session. The fact that this is followed by the always awesome VMunderground party that night is not lost on me either. Also, if you are a VMware TAM customer there is exclusive content for you on Sunday afternoon.
  • Be Comfortable Being Yourself– So what do you wear? My friend Matt Crape covered this well in his recent post but I would like to add that go with what makes you most comfortable networking with your peers. If you are good with shorts and a t-shirt, go for it. Me personally I’m a golf shirt and jeans kind of guy so that’s most of what you’ll see from me. Your days at VMworld are most likely going to be between 15-20 hours so go with what feels good unless that’s naked. Nobody needs to see that. 😉
  • Get Out and Be Social– This is not a “Woo Hoo, It’s Vegas So Let’s Party” topic. Yes, you can do that if that’s your prerogative, but keep in mind some of the smartest minds in your chosen career are going to be here and out at both events in the evening as well as in the hang space during the day. Go meet people as they are typically pretty nice and cool. While the VMworld sessions are what’s being sold as the content of the conference I will book very few of those, choosing instead to spend my time learning from others how they are dealing with many of the same issues that I have and make connections that can prove helpful down the road.
    Where to go be social? During the day the HangSpace/ VM Village is the place to go. In the evenings there is a never-ending list of gatherings to find your way to. I personally will be making sure I attend the Veeam party and VMunderground as they are my 2 evening must do’s each year and are typically among the biggest. Past that I’ll just go with the flow.
  • Be Social Online Too– If you are a tweeter be sure to use not only the #VMworld hashtag but also that of whatever session or event you are currently in. If you look around it will typically be on a wall somewhere. This will help you extend the conversation during the session. If you aren’t on twitter yet you may want to consider that, often this is a great way to see what your colleagues are saying about announcements and such in real time. It also serves a great way to meet up with others at the conference.
  • Get Some Sleep When Possible– I know this sounds counter-intuitive to the previous topic but if you are a 40-year-old like me this week will catch up to you. It is definitely possible to do events and conference from 7:30 AM to after midnight each day and while that’s a lot of fun, by Wednesday there are so many zombies walking around Mandalay Bay it looks like an episode of the Walking Dead. If you’ve been working on the session builder already take a look at your schedule and make room for you to sleep in a morning sometime midweek. You can catch up on the sessions once you get back.

While there’s more than that for me those are the basics. If you are going please hit me up @k00laidIT on twitter, I’d love to have a coffee, a beer or just a conversation with you. Have a great time!

P.S. Wear comfortable shoes!

Learning To Pick The Right Tech Conference at vBrisket- TOMORROW!

Hey all, just a quick post to mention that the fine folks at vBrisket will be having a get together February 24th at 2 PM at Grist House Craft Brewery in Pittsburgh. If you work in the virtualization industry and haven’t heard of vBrisket yet you should get to know them because they have a great thing going.  vBrisket takes the typical User Group back to its vendor independence roots, allowing you to focus more on your general virtualization career and less on the path of any particular vendor. At the same time it gives Clint, Gabe, Jaison, and John a great reason to bring out the smokers and prepare enough meat to feed a brewery full of techies.

I’m honored to have been invited to join the panel discussion this time. The topic is “Tech Conferences – What are the right ones for you?” This will be moderated by the vBrisket team and includes myself, John White, Mike Muto, and Justin Paul. As I see my attendance at various conferences as a big driver in the success of my career and my growth as a technology worker I’m excited to be included.

Of course this meeting wouldn’t be possible without the sponsorship from Zerto. At the meeting they’ll be talking I’m sure about their new conference, ZertoCON in Boston May 22-24th.

So if you are in the Pittsburgh area tomorrow and would like to attend just be there at 2, I look forward to meeting up!

Windows Server Deduplication, Veeam Repositories, and You!

Backup, among other things, is very good at creating multiple copies of giant buckets of data that don’t change much and tend to sit for long periods of time. Since we are in modern times, we have a number of technologies to deal with this problem, one of which is called deduplication with quite a few implementations of it. Microsoft has had server-based storage versions since Windows 2008 R2 that has gotten better with each release, but as any technology still has its pitfalls to be mindful of. In this post I’m going to look a very specific use case of Windows server deduplication, using it as the storage beneath your Veeam Backup and Replication repositories, covering some basic tips to keep your data healthy and performance optimized.

What is Deduplication Anyway?

For those that don’t work with it much imagine you had a copy of War and Peace stored as a Word document with an approximate file size 1 MB. Each day for 30 days you go into the document and change 100 KB worth of the text in the document and save it as a new file on the same volume. With a basic file system like NTFS this would result in you having 31 MB tied up in the storage of these files, the original and then the full file size of each additional copy.

Now let’s look at the same scenario on a volume with deduplication enabled. The basic idea of deduplication replaces identical blocks of data with very small pointers back to a common copy of the data. In this case after 30 days instead of having 31 MB of data sitting on disk you would approximately 4 MB; the original 1 MB plus just the 100 KB of incremental updates. As far as the user experience goes, the user just sees the 31 files they expect to see and they open like they normally would.

So that’s great when you are talking about a 1 MB file but what if we are talking about file storage in the virtualization world, one where we talking about terabytes of data multi gigabyte changes daily? If you think about the basic layout of a computer’s disk it is very similar to our working copy of War and Peace, a base system that rarely changes, things we add that then sit forever, and then a comparatively few things we change throughout the course of our day. This is why for virtual machine disk files and backup files deduplication works great as long as you set it up correctly and maintain it.

Jim’s Basic Rules of Windows Server Deduplication for Backup Repositories

I have repeated these a few times as I’ve honed them over the years. If you feel like you’ve read or heard this before its been part of my VeeamON presentations in both 2014 and 2015 as well as part of blog posts both here and on 4sysops.com. In any case here are the basics on care and feeding your deduplicated repositories.

  1. Format the Volume Correctly. Doing large-scale deduplication is not something that should be done without getting it right from the start. Because when we talk about backup files, or virtual disks in general for that matter, we are talking about large files we always want to format the volume through the command line so we can put some modifiers in there. The two attributes we really want to look at is /L and /A:64k. The /L  is an NTFS only attribute which overrides the default (small) size of the file record. The /A controls the allocation unit size, setting the block size. So for a given partition R: your format string may look like this:
  2. Control File Size As Best You Can. Windows Server 2012 R2 Deduplication came with some pretty stringent recommendations when it came to maximum file size and using deduplication, 1 TB. With traditional backup files blowing past that is extremely easy to do when you have all of your VMDKs rolled into a single backup file even after compression. While I have violated that recommendation in the past without issue I’ve also heard many horror stories of people who found themselves with corrupted data due to this. Your best bet is to be sure to enable Per-VM  backup chains on your Backup Repository (Backup Infrastructure> Backup Repositories> [REPONAME] > Repository> Advanced).
  3. Schedule and Verify Weekly Defragmentation. While by default Windows schedules weekly defragmentation jobs on all volumes these days the one and only time I came close to getting burnt but using dedupe was when said job was silently failing every week and the fragmentation became too much. I found out because my backup job began failing due to corrupted backup chain, but after a few passes of defragmenting the drive it was able to continue without error and test restores all worked correctly. For this reason I do recommend having the weekly job but make sure that it is actually happening.
  4. Enable Storage-Level Corruption Guard. Now that all of these things are done we should be good, but a system left untested can never be relied upon. With Veeam Backup & Replication v9 we now have the added tool on our backup jobs of being able to do periodic backup corruption checks. When you are doing anything even remotely risky like this it doesn’t hurt to make sure this is turned on and working. To enable this go to the Maintenance tab of the Advanced Storage settings of your job and check the top box. If you have a shorter retention time frame you may want to consider setting this to weekly.
  5. Modify Deduplication Schedule To Allow for Synthetic Operations. Finally the last recommendation has to do more with performance than with integrity of data. If you are going to be doing weekly synthetic fulls I’ve found performance is greatly decreased if you leave the default file age before deduplication setting (3 or 5 days depending on version of Windows) enabled. This is because in order to do the operation it has to reinflate each of the files before doing the operation. Instead set the deduplication age to 8 days to allow for the files to already be done processing before they were deduplicated.  For more information on how to enable deduplication as well as how to modify this setting see my blog over on 4sysops.com.

Well with that you now know all I know about deduplicating VBR repositories with Windows Server. Although there is currently a bug in the wild with Server 2016 deduplication, with a fix available, the latest version of Windows Server shows a lot of promise in its storage deduplication abilities. Among other things it pushes the file size limit up and does quite a bit to increase performance and stability.

Veeam Vanguard Again in 2017

It has been a great day here because today I learned that I have once again been awarded acceptance into the excellent Veeam Vanguard program, my third time. This program, above any others that I am or have been involved with takes a more personal approach to creating a group of awardees who not only deserve anything good they get out of it but give back just as much to the community itself. In only its 3rd year the group has grown; from 31 the first year, 50(ish) the second, to a total of 62 this year. There are 21 new awardees in that 62 number so there really isn’t a rubber stamp to stay included, it is legitimately awarded each year. The group has grown each year but as you can see not by the leaps and bounds others have, and for good reason. There is no way this experience could be had with a giant community.

At this point in the post I would typically tell you a bit about what the Vanguard program is and isn’t but honestly, Veeam’s own Dmitry Kniazev really put it best in a couple recent posts, “Veeam Vanguard Part 1: WTH Is This?” and “Veeam Vanguard Part 2: What It’s Not.”  What I will add is that as nice as some of the perks are, as DK says in the Part 1 post the true perk is the intangibles; a vibrant community full of some of the smartest, most passionate people in the industry and in many cases access right to the people approving and disapproving changes to their software. These are the thing that made me sweat approval time.

Once again I would give a giant thank you to Veeam Software and especially the whole Vanguard crew. This includes Rick Vanover, Clint Wyckoff, Michael White, Michael Cade, Anthony Spiteri, Kirsten Stoner, Dmitry Kniazev, Andrew Zhelezko and finally Doug Hazelman. Without these people it wouldn’t be nearly as nice.

Fixing Domain Controller Boot in Veeam SureBackup Labs

We’ve been dealing with an issue for past few runs of our monthly SureBackup jobs where the Domain Controller boots into Safe Mode and stays there. This is no good because without the DC booting normally you have no DNS, no Global Catalog or any of the other Domain Controller goodness for the rest of your servers launching behind it in the lab. All of this seems to have come from a change in how domain controller recover is done in Veeam Backup and Replication 9.0, Update 2 as discussed in a post on the Veeam Forums. Further I can verify that if you call Veeam Support you get the same answer as outlined here but there is no public KB about the issue. There are a couple of ways to deal with this, either each time or permanently, and I’ll outline both in this post.

The booting into Safe Mode is totally expected, as a recovered Domain Controller object should boot into Directory Services Restore mode the first time. What is missing though is that as long as you have the Domain Controller box checked for the VM in your application group setup then once booted Veeam should modify the boot setup and reboot the system before presenting it to you as a successful launch. This in part explains why when you check the Domain Controller box it lengthens the boot time allowed from 600 seconds to 1800 seconds by default.

On the Fly Fix

If you are like me and already have the lab up and need to get it fixed without tearing it back down you simply need to clear the Safe Boot bit and reboot from Remote Console. I prefer to

  1. Make a Remote Console connection to the  lab booted VM and login
  2. Go to Start, Run and type “msconfig”
  3. Click on the Boot tab and uncheck the “Safe boot” box. You may notice that Active Directory repair option is selected
  4. Hit Ok and select to Restart

Alternatively if you are command inclined a method is available via Veeam KB article 1277  where you just run these commands

it will reboot itself into normal operation. Just to be clear, either of these fixes are temporary. If you tear down the lab and start it back to the same point in time you will experience the same issue.

The Permanent Fix

The problem with either of the above methods is that while they will get you going on a lab that is already running about 50% of the time I find that once I have my DC up and running well I have to reboot all the other VMs in the lab to fix dependency issues. By the time I’m done with that I could have just relaunched the whole thing. To permanently fix the root issue is you can revert the way DCs are handled by creating a single registry entry as shown below on the production copy of each Domain Controller you run in the lab.

Once you have this key in place on your production VM you won’t have any issues with it going forward as long as the labs you launch are from backups made after that change is put in use. My understanding is this is a known issue and will eventually be fixed but at least as of 9.5 RTM it is not.

The Most Magical Time of Year: Influencer Program Selection Season!

Each year many of the major companies in the tech industry allow people to be nominated, by themselves or by others, to be recognized for the contributions to the community that surrounds that company’s products. These people are typically active on social media, in both online and in person forums and user groups and often will write blogs about their experiences with the products. In return for what is essentially free, grass-roots type marketing the companies will provide awardees any number of benefits; access to licenses for products for homelabbing as well as sometimes access to engineers, preferred experiences at conferences, NDA level information, etc but in some cases the biggest benefit is the recognition itself.

As of today (November 10, 2016) two of the bigger and in my opinion one of the best programs are all open for nominations.

Program Name Program Leader Nomination Link
Cisco Champions Lauren Friedman Nomination Link
VMware vExpert Corey Romero Nominations Accepted until 12/16
Veeam Vanguards Rick Vanover Nominations Accepted until 12/9

I’m honored to be both a vExpert and a Veeam Vanguard and like to think of myself as an honorary Cisco Champion (they can’t accept government employees) so I have some experience with each of these programs. Let’s take a look at all three.

vexpert-624x111VMware vExpert may not necessarily be the oldest influencers program but it is probably the one socially active technical people know except possibly the Microsoft MVP program. In many ways vExpert is not only an honorary of its own right but a launch pad towards acceptance into other programs. vExperts are as far as I know the largest such group with around 1500 members world-wide, it also boasts some really good benefits not only from VMware but from other companies in the virtualization ecosphere. There are many webinars and meet and greets throughout the calendar year which are either vExpert only or vExpert preferred and the vExpert party at VMworld is well-known as one of the best. The distinction I make most about vExpert is that while it is for and by VMware, some years much of the educational focus is on the ecosphere and community that surrounds it.

The vExpert program offers 4 paths to membership. The one most are in is the Evangelist path. These may be customers, partners or VMware employees themselves, but they are people speaking the good word of VMware. There are also specific paths for Partners and Customers but I don’t know that I’ve ever met anyone who was awarded in those tracks. Finally if you have achieved the highest level of VMware certification, VCDX, you automatically are awarded vExpert status.

ciscochampion2016-512-nodateCisco Champions contrasts from vExpert most because it is a self-contained program with all the educational opportunities and benefits being from Cisco Systems itself. With the Champions there aren’t so many of the freebies with the notable exception of some nice perks if you attend CiscoLive, but what they do offer is exposure of your personal brand. Between the weekly Cisco Champions Radio podcast and the regularly featured blogs on Cisco’s website if you are working to make a name for yourself in the industry for whatever reason it is a very good program for that. Further Cisco gives you access to developers and program  managers within the company so that you can not only gain greater understanding of the products but in many cases have the opportunity to weigh in on technology decisions during the development process.

Cisco breaks their program down into business segments in regards to your qualification for the program with tracks in Collaboration, Data Center, Enterprise Networks, IoT, and Security. If you have expertise in any of these by all means apply.
veeam_vanguard-700x224In my mind I’m saving the best for last. The Veeam Vanguard program opened its nominations up today for its 3rd year and I’ve been honored to have awarded each year (so far). It is by far the most exclusive; there are currently only 50 members worldwide and I believe the philosophy is to keep it on the small side with only people who truly understand what the company is about. There are a lot of swag type benefits to the Vanguard to be sure, most noticeably something really special that revolves around  their VeeamON conference (NOLA this year baby!), but to be honest what I most get out of the program is the distributed brain of not only the Veeam employees affiliated with the group but the group itself. On a daily basis it seems sometimes somebody’s technology issues, Veeam related or not, are being sorted out through Vanguard communication methods. Long story short, in the Vanguard program they simply take care of you and I’m happy to call all of them not just my peers but friends.

Because Veeam is a much tighter set of products than the other two there aren’t any official tracks within the program. That said they are very good about selecting members who affiliate themselves with each of the hypervisor companies they support, VMware’s vSphere and Microsoft’s Hyper-V. This diversity is part of what makes the discussions between us so good.


Over the course of the past week I’ve heard various people talking about strategies regarding getting awarded to any number of these. I’m not going to do this one so I can focus on that one and so forth, and honestly all I can recommend to you if you are interested in applying to any of them is look at where your focus is or where you focus should be and apply. There is no thing that says “you belong to too many programs” or anything like that; if you feel you are qualified for any of these or any other by all means go apply. The name of the game is to grow your involvement with the technology community, regardless of what type of technology it is.

Setting Up External Access To A Veeam SureBackup Virtual Lab

Hey y’all, happy Friday! One of the things that seems to still really fly under the radar in regards to Veeam Backup & Replication is its SureBackup feature. This feature is designed to allow for automated testing via scripts of groups of your backups. An example would be if you have a critical web application. You can create an application group that includes both the database server and the web server and when the SureBackup job is run Veeam will connect a section of its backup repository to a specified ESXi host as a datastore and, start the VMs within a NAT protected segment of your vSphere infrastructure, run either the role based scripts included or custom ones you specify to ensure that the VMs are connecting to the applications correctly, and then when done shut the lab down and fire off an e-mail.

That workflow is great an all but it only touches on the edge of the power of what SureBackup can do for you. In our environment not only do we have a mandate to provide backup tests that allow for end-user interaction, but we also use SureBackup for test bed type applications such as patch tests. An example of the latter here is when I was looking to upgrade our internal Windows-based CA to Server 2012 R2. I was able to launch the server in the lab, perform the upgrade and ensure that it behaved as expected WITHOUT ANY IMPACT ON PRODUCTION first and then tear down the lab and it was like it never happened. Allowing the VMs to stay up and running after the job starts requires nothing more than checking a box in your job setup.

By default access to a running lab is fairly limited. When you launch a lab from your Veeam server a route to the NAT’d network is injected to the Veeam server itself to allow access, but that doesn’t help you all that much if you are wanting others to be able to interact; we need to expand that access outwards. This post is going to walk you through the networking setup for a Virtual Lab that can be accessed from whatever level of access you are looking for, in my case from anywhere within my production network.

Setting Up the Virtual Lab


The first step if you haven’t setup SureBackup in your environment at all is to set up your Virtual Lab.  The first of two parts here that are critical to this task is setting up the Proxy IP, which is the equivalent to your outside NAT address if you’ve ever worked on a firewall. This IP is going to essentially be the production network side of the Lab VM that is created when you setup a Veeam Virtual Lab.


Next we need to set up an isolated network for each production port group you need to support. While I use many VLANs in my datacenter I try to keep the application groups I need to test on the same VLAN to make this setup simple, but it doesn’t need to be, you can support as many as you need. Simply hit add, browse out and find the production network port group you need to support, give the isolated network a name and specify a VLAN.


The last step of setting up the Virtual Lab in this regard is creating a virtual NIC to map to each of your isolated networks. So where I see a lot of people get tripped up with this is always make the proxy appliance IP address here map to the default gateway of the production network it is reflecting. If you don’t do that the launched lab VMs will never be able to talk outside of the lab. Second, in regard to the Masquerade IP try to aim for some consistency. Notice that in my production network I am using a Class B private address space but with a class C mask. By default this will throw off the automatic generation of the Masquerade IP and I’ve found it isn’t always consistent across multiple Virtual NIC setups.  If you setup multiple isolated networks above you need to repeat this process for each network. Once you are done with this you can complete your Lab Setup and hit Finish to have it build or rebuild the appliance.


Tweaking the SureBackup Job

For the sake of brevity I’m assuming at this point that you’ve got your Application Groups setup without issue and are ready to proceed to fixing your SureBackup job to stay up and running. To do so on the Application Group screen All you have to do is check the “Keep the application group running after the job completes” box. That’s it. Really. Once you do that this lab will stay up and running until you right click on the job in the Veeam Backup & Replication Console and choose stop. I’ve been lobbying for year for a “stop after X hours” option but still haven’t got very far with that one, but really the concern there is more performance impact from doubling a part of your load since you are essentially running 2 copies of a segment of your datacenter. If you have plenty to burn it isn’t an issue.


Fixing the Routing

Now the final step is to either talk to your network guy or go yourself to where your VLAN routing is taking place and add a static route to the IP range of your inside the lab into the routing table through the Proxy Appliance’s IP. For the example we’ve been working through in this post our Proxy appliance has an IP of and all of our Lab networks are within the network. If you are using a IOS based Cisco switch to handle your VLAN routing the command would be

After that is done, from anywhere that route is accessible from you should now be able to pass whatever traffic inbound to the lab network addresses. So sticking with our example, for a production VM with the IP address, you would interact with the IP in whatever way needed. Keep in mind this is for lack of a better word one way traffic. You can connect in to any of the hosts within the lab network but you can’t really have them reach directly out and have them interact on the production network.


One More Thing…

One final tip that I can give you on this if you are going to let others in to play in your labs is to have at least one workstation grade VM that you include in each of your Applications Groups with the software needed to test with loaded. This way you can enable RDP on that VM and they user can just double-click an icon and connect into the lab, running their tests from there. Otherwise if you have locally installed applications that need to connect to hosts that are now inside the lab you are either going to need to reconfigure the application with the corrected address or modify the user’s hosts file temporarily so that they connect to the right place, neither of which is particularly easy to manage. The other nice thing about a modern RDP session is you can cut and paste files in and out of it, which is handy if the user wants to run reports and the like.


As an aside I’m contemplating doing a video run through of the setting up a SureBackup environment to be added to the blog next week. Would you find such a thing helpful? If so please let me know on twitter @k00laidIT.

Fun with the vNIC Shuffle with Cisco UCS

Here at This Old Datacenter we’ve recently made the migration to using Cisco UCS for our production compute resources. UCS offers a great number of opportunity for system administrators, both in deployment as well as on going maintenance, making updating the physical as manageable as we virtualization admins are getting used to with the virtualized layer of the DC. Of course like any other deployment there is always going to be that one “oh yeah, that” moment. In my case after I had my servers up I realized I needed another virtual NIC, or vNIC in UCS world. This shouldn’t be a big deal because a big part of what UCS does for you is it abstracts the hardware configuration away from the actual hardware.

For those more familiar with standard server infrastructure, instead of having any number of physical NIC in the back of the host for specific uses (iSCSI, VM traffic, specialized networking, etc) you have a smaller number of connections as part of the Fabric Interconnect to the blade chassis that are logically split to provide networking to the individual blades. These Fabric Interconnects (FI) not only have multiple very high-speed connections (10 or 40 GbE) but each chassis typically will have multiple FI to provide redundancy throughout the design. All this being said, here’s a very basic design utilizing a UCS Mini setup with Nexus 3000 switches and a copper connected storage array:


So are you starting to thing this is a UCS geeksplainer? No, no my good person, this is actually the story of a fairly annoying hiccup when it comes to the relationship between UCS and VMware’s ESXi. You see while adding a vNIC should be as simple as create your vNICs in the Server Profile, reboot the effected blades and new NIC(s) are shown as available within ESXi, it of course is not that simple. What happens in reality when you add new NICs to an existing Physical NIC to vSwitch layout is that the relationships are shuffled. So for example if you started with a vNIC (shown as vmnicX in ESXi), vSwitch layout that looks like this to start with


After you add NICs and reboot it looks like this


Notice the vmnic to MAC address relationship in the 2. So while all the moving pieces are still there different physical devices map to different vSwitches than as designed. This really matters when you think about all the differences that usually exist in the VLAN design that underlay networking in an ESXi  setup. In this example vSwitch0 handles management traffic, HQProd-vDS handles all the VM traffic (so just trunked VLANS) and vSwitch1 handles iSCSI traffic. Especially when things like iSCSI that require specialized networking setup are involved does this become a nightmare; frankly I couldn’t imagine having to do this will a more complex design.

The Fix

So I’m sure you are sitting here like I was thinking “I’ll call support and they will have some magic that with either a)fix this, b) prevent it from happening in the future, or preferably c) both. Well, not so much. The answer from both VMware and Cisco support is to figure out which NICs should be assigned to which vSwitch by reviewing the MAC to vNIC assignment in UCS Manager as shown and then manually manage the vSwitch Uplink assignment for each host.



As you may be thinking, yes this is a pain in the you know what. I only had to do this with 4 hosts, I don’t want to think about what this looks like in a bigger environment. Further, as best I can get answers from either TAC or VMware support there is no way to make this go better in the future; this was not an issue with my UCS setup, this is just the way it is. I would love it if some of my “Automate All The Things!!!” crew could share a counterpoint to this on how to automate your way out of this but I haven’t found it yet. Do you have a better idea? Feel free to share it in the comments or tweet me @k00laidIT.