Fixing Domain Controller Boot in Veeam SureBackup Labs

  By | | ,
1 Comment

We’ve been dealing with an issue for past few runs of our monthly SureBackup jobs where the Domain Controller boots into Safe Mode and stays there. This is no good because without the DC booting normally you have no DNS, no Global Catalog or any of the other Domain Controller goodness for the rest of your servers launching behind it in the lab. All of this seems to have come from a change in how domain controller recover is done in Veeam Backup and Replication 9.0, Update 2 as discussed in a post on the Veeam Forums. Further I can verify that if you call Veeam Support you get the same answer as outlined here but there is no public KB about the issue. There are a couple of ways to deal with this, either each time or permanently, and I’ll outline both in this post.

The booting into Safe Mode is totally expected, as a recovered Domain Controller object should boot into Directory Services Restore mode the first time. What is missing though is that as long as you have the Domain Controller box checked for the VM in your application group setup then once booted Veeam should modify the boot setup and reboot the system before presenting it to you as a successful launch. This in part explains why when you check the Domain Controller box it lengthens the boot time allowed from 600 seconds to 1800 seconds by default.

On the Fly Fix

If you are like me and already have the lab up and need to get it fixed without tearing it back down you simply need to clear the Safe Boot bit and reboot from Remote Console. I prefer to

  • Make a Remote Console connection to the  lab booted VM and login
  • Go to … Go Read More →

    • The Most Magical Time of Year: Influencer Program Selection Season!

      By | | , , , ,
    4 Comments

    Each year many of the major companies in the tech industry allow people to be nominated, by themselves or by others, to be recognized for the contributions to the community that surrounds that company’s products. These people are typically active on social media, in both online and in person forums and user groups and often will write blogs about their experiences with the products. In return for what is essentially free, grass-roots type marketing the companies will provide awardees any number of benefits; access to licenses for products for homelabbing as well as sometimes access to engineers, preferred experiences at conferences, NDA level information, etc but in some cases the biggest benefit is the recognition itself.

    As of today (November 10, 2016) two of the bigger and in my opinion one of the best programs are all open for nominations.

    Program Name Program Leader Nomination Link Cisco Champions Lauren Friedman Nomination Link VMware vExpert Corey Romero Nominations Accepted until 12/16 Veeam Vanguards Rick Vanover Nominations Accepted until 12/9

    I’m honored to be both a vExpert and a Veeam Vanguard and like to think of myself as an honorary Cisco Champion (they can’t accept government employees) so I have some experience with each of these programs. Let’s take a look at all three.

    vexpert-624x111VMware vExpert may not necessarily be the oldest influencers program but it is probably the one socially active technical people know except possibly the Microsoft MVP program. In many ways vExpert is not only an honorary of its … Go Read More →

    Installing .Net 3.5 on Server 2012/ Windows 8 and above

      By | |
    Leave a comment

    Hi all, just a quick post to serve as both a reminder to me and hopefully something helpful for you. For some reason Microsoft has decided to make installing .Net 3.5 on anything after Windows Server 2012 (or Windows 8 on the client side) harder than it has to be. While it is included in the regular Windows Features GUI it is not included in the on-disk sources for features to be installed automatically. In a perfect world you just choose to source from Windows Update and go about your day, but in my experience this is a hit or miss solution as many times for whatever reason it errors out when attempting to access.

    The fix is to install via the Deployment Image Servicing and Management tool better known as DISM and provide a local source for the file. .Net 3.5 is included in every modern Windows CD/ISO under the sources\sxs directory. When I do this installation I typically use the following command set from an elevated privilege command line or PowerShell window:

    installedWhen done the window should look like the window to the left. Pretty simple, right? While this is all you really need to know to get it installed let’s go over what all these parameters are that you just fed into your computer.

    Setting Up External Access To A Veeam SureBackup Virtual Lab

      By | | ,
    1 Comment

    Hey y’all, happy Friday! One of the things that seems to still really fly under the radar in regards to Veeam Backup & Replication is its SureBackup feature. This feature is designed to allow for automated testing via scripts of groups of your backups. An example would be if you have a critical web application. You can create an application group that includes both the database server and the web server and when the SureBackup job is run Veeam will connect a section of its backup repository to a specified ESXi host as a datastore and, start the VMs within a NAT protected segment of your vSphere infrastructure, run either the role based scripts included or custom ones you specify to ensure that the VMs are connecting to the applications correctly, and then when done shut the lab down and fire off an e-mail.

    That workflow is great an all but it only touches on the edge of the power of what SureBackup can do for you. In our environment not only do we have a mandate to provide backup tests that allow for end-user interaction, but we also use SureBackup for test bed type applications such as patch tests. An example of the latter here is when I was looking to upgrade our internal Windows-based CA to Server 2012 R2. I was able to launch the server in the lab, perform the upgrade and ensure that it behaved as expected WITHOUT ANY IMPACT ON PRODUCTION first and then tear down the lab and it was like it never happened. Allowing the VMs to stay up and running after the job starts requires nothing more than checking a box in your job setup.

    By default access to a running lab is fairly limited. When you launch a lab from your Veeam … Go Read More →

    Upgrading Cisco Agent Desktop on Windows 10

      By | | ,
    Leave a comment

    So we recently had the joys of upgrading our Cisco Voice setup to version 11, including our Unified Contact Center Express (UCCX) system. In the process of our upgrade we had to do a quick upgrade of UCCX to 9.02 from 9.01 to be eligible to go the rest of the way up to 11, allowing us to run into a nice issue I’m thinking many others are running into.

    As far as 11 is concerned the big difference is it is the first version where the Cisco Agent Desktop (CAD) is not an option as it has been replaced by the new web-based Finesse client for Agents and Supervisors. For this reason many Voice Admins are choosing to take the leap this year to 10.5 instead as it gives you the option of Cisco Agent Desktop/Cisco Supervisor Desktop (CSD) or Finesse. The problem? These MSI installed client applications are not Windows 10 compatible. In our case it wasn’t a big deal as the applications were already installed when we did an in place upgrade of many of our agent’s desktops to Windows 10, but attempting to do an installation would error out saying you were running an unsupported operating system.

    *DISCLAIMER: While for us this worked just fine I’m sure it is unsupported and may lead to TAC giving you issues on support calls. Use at your own discretion.

    Fixing the MSI with Orca

    Luckily there is a way around this to allow the installers to run even allow for automated installation. Orca is one of the tools available within the Windows SDK Components download and it allows you to modify the parameters for Windows MSI packages and either include those changes directly into the MSI or to create a transform file (MST) so that the changes can be … Go Read More →

    Fun with the vNIC Shuffle with Cisco UCS

      By | | , , ,
    Leave a comment

    Here at This Old Datacenter we’ve recently made the migration to using Cisco UCS for our production compute resources. UCS offers a great number of opportunity for system administrators, both in deployment as well as on going maintenance, making updating the physical as manageable as we virtualization admins are getting used to with the virtualized layer of the DC. Of course like any other deployment there is always going to be that one “oh yeah, that” moment. In my case after I had my servers up I realized I needed another virtual NIC, or vNIC in UCS world. This shouldn’t be a big deal because a big part of what UCS does for you is it abstracts the hardware configuration away from the actual hardware.

    For those more familiar with standard server infrastructure, instead of having any number of physical NIC in the back of the host for specific uses (iSCSI, VM traffic, specialized networking, etc) you have a smaller number of connections as part of the Fabric Interconnect to the blade chassis that are logically split to provide networking to the individual blades. These Fabric Interconnects (FI) not only have multiple very high-speed connections (10 or 40 GbE) but each chassis typically will have multiple FI to provide redundancy throughout the design. All this being said, here’s a very basic design utilizing a UCS Mini setup with Nexus 3000 switches and a copper connected storage array:

    ucs-design

    So are you starting to thing this is a UCS geeksplainer? No, no my good person, this is actually … Go Read More →

    Getting the Ball Rolling with #vDM30in30

      By | | , ,
    Leave a comment

    Ahh, that time of year when geeks pull that long forgotten blog site out of the closet, dust it of and make promises of love and content: #vDM30in30. If you aren’t familiar with the idea, vDM30in30 is short for Virtual Design Master 30 blog posts in 30 days, an idea championed by Eric Wright of discoposse fame to get bloggers out there to work their way through regular generation of content. As you can see from this site new content is pretty rare so something like this is a welcome excuse to focus and get some stuff out there. vDM30in30 runs through the month of November and the best way to follow along with the content is to track the hashtag on twitter.

    So What’s the Plan?

    I’m a planner by nature so if I don’t at least have a general idea this isn’t going to work at all. The good news is I’ve got quite a few posts that I’ve been meaning to work on for some time so I’m going to be cleaning out my closet this week and get those out there. So the full schedule is going to look like this:

    • Week of Nov 1: random posts I’ve never quite finished but need to be released
    • Week of Nov 7: focus on all the new hotness coming from Veeam Software
    • Week of Nov 14: VMware’s upcoming vSphere 6.5 release
    • Week of Nov 21: randomness about community, career and navel gazing in general

    I’m really looking forward to participating this year as I do believe that a lot of growth comes from successfully forming out thoughts and putting them down. Hope you find some of this hopeful, if there is anything you’d like to see in the space feel free to comment.

    Lots of new stuff coming from Veeam

      By | | ,
    1 Comment

    Veeam had what they called “THEIR BIGGEST EVENT EVER” and while it at times did seem to be really heavy on the sales for the sake of sales pitch, there was a lot of stuff to legitimately be excited about for those of us who use their products. From the features coming in Veeam Backup & Replication in version 9.5 in a couple of months through the first new feature of next year’s version 10 all in total there were 5 major announcements here today that those of us using the product can make use of. In this post I’m going to run briefly through these and in the coming months will provide some deeper insights when possible.

    Veeam Backup & Replication / Veeam ONE 9.5 (October 2016)

    • Nimble Storage Integration- Nimble with be the next vendor after EMC, NetApp and HP storage systems that will allow Veeam to interact at the array level, allowing for backups from snapshot. If you are a Nimble customer (like me) this is going to be some good stuff
    • Advanced usage of Windows Server 2016 ReFS- This is the real gravy here for anybody who is having to work with any kind of synthetic operations with their backup files. Through an integration Veeam has with Microsoft when ReFS is used to back your Veeam repositories your weekly rollups are going to take a heck of a lot less time and as well as see less storage consumption for long terms “weekly fulls”.  This is due to ReFS’ basic mechanism where file copies and moves never actually move data, it just moves the pointers. An example I’ve seen is on a 10 GB change rate backup the weekly full went from 35 minutes on NTFS to 5 minutes on ReFS. Now move that out … Go Read More →

    VMware Tools Security Bug and Finding which VMware Tools components are installed on all VMs

      By | | , ,
    Leave a comment

    Just a quick post related to today’s VMware security advisories. VMware released a pair of advisories today, CVE-2016-5330 and CVE-2016-5331 and while both are nasty their scopes are somewhat limited. The 5331 issue is only applicable if you are running vCenter or ESXi 6.0 or 6.0U1, Update 2 patches the bug. The 5330 is limited to Windows VMs, running VMware Tools, and have the option HGFS component installed. To find out if you are vulnerable here’s a Power-CLI script to get all your VMs and list the installed components. Props to Jason Shiplett for giving me some assistance on the code.

    While the output is still a little rough it will get you there. Alternatively if you are just using this script for the advisory listed you can change  where-object { $_.Name -match $componentPattern }  to  where-object { $_.Name -match "vmhgfs" } . This script is also available on GitHub.

    The Unofficial Official CiscoLive! US Gatherings Page

      By | |
    Leave a comment

    Here’s the list of all the outside of business hours events that I and others know of at CiscoLive 2016. If you know of others please DM or tweet me @k00laidIT and I’ll get them added.

     

    Saturday 7/9/2016
    Adventure to  National Atomic Testing Museum

    • 2 PM
    • 755 E Flamingo Rd, Las Vegas, NV 89119 (Map)
    • #clatomic 

    Sunday 7/10/2016
    #CLUS Sunday Tweetup

    • 5:30 PM
    • Social Media Central, Bayside Foyer, Mandalay Bay

    Monday 7/11/2016
    Veeam & Nimble Integration party at Cisco Live!

    Tuesday 7/12/2016
    SD-WAN Mixer with Packet Pushers’ Ethan Banks

    Meraki After Party

    Wednesday 7/13/2016
    Customer Appreciation Event