Fixing Domain Controller Boot in Veeam SureBackup Labs

We’ve been dealing with an issue for past few runs of our monthly SureBackup jobs where the Domain Controller boots into Safe Mode and stays there. This is no good because without the DC booting normally you have no DNS, no Global Catalog or any of the other Domain Controller goodness for the rest of your servers launching behind it in the lab. All of this seems to have come from a change in how domain controller recover is done in Veeam Backup and Replication 9.0, Update 2 as discussed in a post on the Veeam Forums. Further I can verify that if you call Veeam Support you get the same answer as outlined here but there is no public KB about the issue. There are a couple of ways to deal with this, either each time or permanently, and I’ll outline both in this post. The booting into Safe Mode is totally expected, as a recovered Domain Controller object should boot into Directory Services Restore mode the first time. What is missing though is that as long as you have the Domain Controller box checked for the VM in your application group setup then once booted Veeam should modify the boot setup and reboot the system before presenting it to you as a successful launch. This in part explains why when you check the Domain Controller box it lengthens the boot time allowed from 600 seconds to 1800 seconds by default. On the Fly Fix If you are like …

The Most Magical Time of Year: Influencer Program Selection Season!

Each year many of the major companies in the tech industry allow people to be nominated, by themselves or by others, to be recognized for the contributions to the community that surrounds that company’s products. These people are typically active on social media, in both online and in person forums and user groups and often will write blogs about their experiences with the products. In return for what is essentially free, grass-roots type marketing the companies will provide awardees any number of benefits; access to licenses for products for homelabbing as well as sometimes access to engineers, preferred experiences at conferences, NDA level information, etc but in some cases the biggest benefit is the recognition itself. As of today (November 10, 2016) two of the bigger and in my opinion one of the best programs are all open for nominations. Program Name Program Leader Nomination Link Cisco Champions Lauren Friedman Nomination Link VMware vExpert Corey Romero Nominations Accepted until 12/16 Veeam Vanguards Rick Vanover Nominations Accepted until 12/9 I’m honored to be both a vExpert and a Veeam Vanguard and like to think of myself as an honorary Cisco Champion (they can’t accept government employees) so I have some experience with each of these programs. Let’s take a look at all three. VMware vExpert may not necessarily be the oldest influencers program but it is probably the one socially active technical people know except possibly the Microsoft MVP program. In many ways vExpert is not only an honorary of its …

Setting Up External Access To A Veeam SureBackup Virtual Lab

Hey y’all, happy Friday! One of the things that seems to still really fly under the radar in regards to Veeam Backup & Replication is its SureBackup feature. This feature is designed to allow for automated testing via scripts of groups of your backups. An example would be if you have a critical web application. You can create an application group that includes both the database server and the web server and when the SureBackup job is run Veeam will connect a section of its backup repository to a specified ESXi host as a datastore and, start the VMs within a NAT protected segment of your vSphere infrastructure, run either the role based scripts included or custom ones you specify to ensure that the VMs are connecting to the applications correctly, and then when done shut the lab down and fire off an e-mail. That workflow is great an all but it only touches on the edge of the power of what SureBackup can do for you. In our environment not only do we have a mandate to provide backup tests that allow for end-user interaction, but we also use SureBackup for test bed type applications such as patch tests. An example of the latter here is when I was looking to upgrade our internal Windows-based CA to Server 2012 R2. I was able to launch the server in the lab, perform the upgrade and ensure that it behaved as expected WITHOUT ANY IMPACT ON PRODUCTION first and then tear down …

Fun with the vNIC Shuffle with Cisco UCS

Here at This Old Datacenter we’ve recently made the migration to using Cisco UCS for our production compute resources. UCS offers a great number of opportunity for system administrators, both in deployment as well as on going maintenance, making updating the physical as manageable as we virtualization admins are getting used to with the virtualized layer of the DC. Of course like any other deployment there is always going to be that one “oh yeah, that” moment. In my case after I had my servers up I realized I needed another virtual NIC, or vNIC in UCS world. This shouldn’t be a big deal because a big part of what UCS does for you is it abstracts the hardware configuration away from the actual hardware. For those more familiar with standard server infrastructure, instead of having any number of physical NIC in the back of the host for specific uses (iSCSI, VM traffic, specialized networking, etc) you have a smaller number of connections as part of the Fabric Interconnect to the blade chassis that are logically split to provide networking to the individual blades. These Fabric Interconnects (FI) not only have multiple very high-speed connections (10 or 40 GbE) but each chassis typically will have multiple FI to provide redundancy throughout the design. All this being said, here’s a very basic design utilizing a UCS Mini setup with Nexus 3000 switches and a copper connected storage array: So are you starting to thing this is a UCS geeksplainer? No, no …

Getting the Ball Rolling with #vDM30in30

Ahh, that time of year when geeks pull that long forgotten blog site out of the closet, dust it of and make promises of love and content: #vDM30in30. If you aren’t familiar with the idea, vDM30in30 is short for Virtual Design Master 30 blog posts in 30 days, an idea championed by Eric Wright of discoposse fame to get bloggers out there to work their way through regular generation of content. As you can see from this site new content is pretty rare so something like this is a welcome excuse to focus and get some stuff out there. vDM30in30 runs through the month of November and the best way to follow along with the content is to track the hashtag on twitter. So What’s the Plan? I’m a planner by nature so if I don’t at least have a general idea this isn’t going to work at all. The good news is I’ve got quite a few posts that I’ve been meaning to work on for some time so I’m going to be cleaning out my closet this week and get those out there. So the full schedule is going to look like this: Week of Nov 1: random posts I’ve never quite finished but need to be released Week of Nov 7: focus on all the new hotness coming from Veeam Software Week of Nov 14: VMware’s upcoming vSphere 6.5 release Week of Nov 21: randomness about community, career and navel gazing in general I’m really looking forward to …

Lots of new stuff coming from Veeam

Veeam had what they called “THEIR BIGGEST EVENT EVER” and while it at times did seem to be really heavy on the sales for the sake of sales pitch, there was a lot of stuff to legitimately be excited about for those of us who use their products. From the features coming in Veeam Backup & Replication in version 9.5 in a couple of months through the first new feature of next year’s version 10 all in total there were 5 major announcements here today that those of us using the product can make use of. In this post I’m going to run briefly through these and in the coming months will provide some deeper insights when possible. Veeam Backup & Replication / Veeam ONE 9.5 (October 2016) Nimble Storage Integration- Nimble with be the next vendor after EMC, NetApp and HP storage systems that will allow Veeam to interact at the array level, allowing for backups from snapshot. If you are a Nimble customer (like me) this is going to be some good stuff Advanced usage of Windows Server 2016 ReFS- This is the real gravy here for anybody who is having to work with any kind of synthetic operations with their backup files. Through an integration Veeam has with Microsoft when ReFS is used to back your Veeam repositories your weekly rollups are going to take a heck of a lot less time and as well as see less storage consumption for long terms “weekly fulls”.  This is due …

VMware Tools Security Bug and Finding which VMware Tools components are installed on all VMs

Just a quick post related to today’s VMware security advisories. VMware released a pair of advisories today, CVE-2016-5330 and CVE-2016-5331 and while both are nasty their scopes are somewhat limited. The 5331 issue is only applicable if you are running vCenter or ESXi 6.0 or 6.0U1, Update 2 patches the bug. The 5330 is limited to Windows VMs, running VMware Tools, and have the option HGFS component installed. To find out if you are vulnerable here’s a Power-CLI script to get all your VMs and list the installed components. Props to Jason Shiplett for giving me some assistance on the code.

While the output is still a little rough it will get you there. Alternatively if you are just using this script for the advisory listed you can change  where-object { $_.Name -match $componentPattern }  to  where-object { $_.Name -match "vmhgfs" } . This script is also available on GitHub.

A how-to on cold calling from the customer perspective

Now that I’m back from my second tech conference in less than two months I am fully into the cold call season and I am once again reminded why I keep meaning to buy a burner phone and setup a Gmail account before I register next year. It seems every time I get back I am destined to months of “I am so glad you expressed deep interest in our product and I’d love to tell you more about it” when the reality is “I am calling you because you weren’t nimble enough to lunge away from our team of booth people who are paid or retained based on as many scans they can get. Most often when I get these calls or e-mails I’ll give each company a courteous thanks but no thanks and after that the iDivert button gets worn out. The genesis of this post is two-fold. First a cold call this morning that was actually destined for my boss but when informed he wasn’t here went into telling how glad the person was that I had personally expressed interest in their product, WTF? This first event reminded me of a second, where a few months ago I was at a mixer preceding a vendor supplied training when I was approached by a bevy of 20 something Inside Sales Engineers and asked “what can I do to actually get you to listen?” From this I thought that just in case a young Padawan Sales Rep/Engineer happens to …

Veeam Backup Repository Best Practices Session Notes

After a couple days off I’m back to some promised VeeamON content. A nice problem that VeeamON had this year is the session choices were much more diverse and there were a lot more of them. Unfortunately this led to some overlap of some really great sessions. A friend of mine, Jaison Bailey of vBrisket fame and fortune, got tied up in another session and was unable to attend what I considered one of the best breakout sessions all week, Anton Gostev‘s Backup Repository Best Practices so he asked me to post my notes. For those not too familiar with Veeam repos they can essentially be any manner of addressable disk space, whether local, DAS, NAS, SAN or even cloud, but when you start taking performance into account you have to get much more specific. Gostev, who is the Product Manager for Backup & Replication, lines out the way to do it right. Anyway, here’s the notes including links to information when possible. Any notations I have are in bold and italicized. Don’t underestimate the importance of Performance Performance issues may impact RTOs Five Factors of choosing Storage Reliability Fast backups Fast restores DR from complete storage loss Lowest Cost Ultimate backup Architecture Fast, reliable primary storage for fastest backups, then backup copy to Secondary storage both onsite AND offsite Limit number of RP on primary, leverage cheap secondary Selectively create offsite copies to tape, dr site, or cloud Best Repo: Low End Any Windows or Linux Server Can also serve …

Presenting at VeeamON 2015: Design, Manage and Test Your Data Protection with Veeam Availabilty Suite

Last week I was presented with the honor of being invited to speak at Veeam Software‘s annual user conference, VeeamON. While this was not my first time doing so I was very happy with the end result this year, with 30-40 attendees and positive feedback both from people I knew beforehand as well as new acquaintances who attended. My session is what I like to think of as the 1-1000 MPH with Veeam, specifically targeting the SMB space but with lots of general guidelines for how to get your DR system up and running fast and as error-free as possible. Some of the things I do with Veeam buck the Best Practices guide but we have been able to maintain high levels of protection over many years without much interruption. The session starts with the basics of designing your DR plan, then designing your Veeam infrastructure components to suit your needs, followed by tips for the actual implementation and other tricks and gotchas I’ve run into over the years. Anyway due to the amount of information that was covered I promised attendee’s that I would put my slide deck out here for reference so here it is. If anybody has comments, questions or anything in between please feel free to reach out to me either through the comments here or on twitter. For attendees please keep an eye on your e-mail and the #VeeamON hashtag as the videos of all presentations should be made available in the coming weeks. This …