I was honored to be selected to present at VeeamON again in 2015 with the same general concept of session but really highlighting some of the work I’d been doing with Windows Server deduplication and Veeam SureBackup. This was a much nice crowd, with 50 or so people, and I felt it went much better. Again there is no video this time around but as an aside watching the replay at the time this is when I first realized just how much of an accent I have, literally had never realized it before.

So this was immediately following the vBrownBag session. Again, now second shot at “public speaking” but this time I’m in a room for 200, in the hangover slot at a tech conference (8 am the morning after the party), there were a total of 3 people in the session. Probably not the worst thing but one thing I’ve found over the years is that crowds are good, they loosen you up. Just goes to show that we definitely all start somewhere.

Unfortunately I don’t have the video of these early session so you are just going to have to trust me that it was rough.

Slidedeck: https://www.slideshare.net/JimJones147/veeamon-2014-vbrownbag-physical-server-backup-strategies

This was actually my first public speaking presentation ever. Supposed to be a 15 minute time slot I believe I finished in under 8 minutes. What aided to my discomfort was that the whole purpose of this was to show a method I was using to backup physical servers in a Veeam world, something there was no answer to at the time. That morning it leaked that Veeam was announcing what is now known as Veeam Agent for Windows and my friend Rick Vanover was following me on the same stage to announce it. Talk about your topic becoming irrelevant in record time…

This year I was honored with the ability to attend Veeam Software’s Vanguard Summit. Summit comprises a meeting of 60-70 of world’s best Veeam Engineers, Architects, and Partners along with their own Product Strategy and Management groups in Prague, Czech Republic.

Often in talking about a subject we’ll float the old cliché of the good, the bad and ugly of it and frankly there is nothing negative I can say about this event. Because of that in this series of posts I’m going to cover the good (the event itself), the awesome (the people and content), the beautiful (the city of Prague itself), and as a bonus, the technical. This post will be the second in the series, covering the awesome people and content that comprise Vanguard Summit.

Community

There are many vendor community programs out there, most of which are much bigger than the Vanguards, and while they may have some nice intrinsic benefits such as licenses or hoodies, nobody does it better than Rick Vanover and his team. This is my 5th year in the program but due to the rules of my previous employer I’ve never been able to participate in person. The addition of being able to interact with seventy of the smartest people was invigorating and enjoyable. While I was happy to meet and spend time with everyone it was great to get to catch up with community stalwarts and friends such as Matt Crape, Craig Dalrymple, Brad Jervis, Dean Lewis and Al Rasheed.

There is nothing like being in a room with your peers, most of which leave me in awe with their abilities, hearing about what comes next from a company and have your feedback and thoughts sought out. In the process the attendees start collaborating on ideas that end up shaping things for years to come. I’ve said this before, but if you think the value of any program like this is measured monetarily, you are doing it wrong. The true value is the information that not only passes from the vendor to the program members but between the members itself and this group is in my mind the best of the best.

Content

I remember back when I began as a Veeam customer there was the free FastSCP product and Veeam Backup & Replication. After nine years it is amazing to see how the product line has grown and this week made that very apparent, with 2.5 days jam packed with content on current and future products. In a later post I’m going to be covering a lot of what’s coming in v10 of Veeam Backup & Replication but we were also treated to updates on Backup for O365, VeeamONE, Veeam Agents for Windows and Linux, as well as Veeam Availability Orchestrator. This doesn’t include the things we aren’t allowed to discuss but frankly are pretty exciting.

One of the ways that this content is vastly different from other vendor briefings you may have attended or seen maybe at a TechFieldDay (who Veeam will once again be attending for TFD #20) event or the like, besides the great Product Strategy staff many of these sessions were led or included the product management team for all of their products. The highlight to me of this even is the final session on Tuesday where we had Anton Gostev, Alec King, and others from the product management/development team in an “Ask Me Anything” style session where we were given the opportunity to ask all we wanted and as best I could tell were given no “BS” answers. In short the content was pretty amazingly done and consisted of far more access to information than I’ve seen from any other vendor.

Conclusion

Much like my last post rather than bore you with more of my opinions I’ll leave you with some pictures from the event.

This slideshow requires JavaScript.

This year I was honored with the ability to attend Veeam Software’s Vanguard Summit. Summit comprises a meeting of 60-70 of world’s best Veeam Engineers, Architects, and Partners along with their own Product Strategy and Management groups in Prague, Czech Republic. As one who had never been to Europe and has long been an advocate of Veeam’s products Vanguard Summit had all the makings of an awesome event and it never came close to letting me down.

Often in talking about a subject we’ll float the old cliché of the good, the bad and ugly of it and frankly there is nothing negative I can say about this event. Because of that in this series of posts I’m going to cover the good (the event itself), the awesome (the people and content), the beautiful (the city of Prague itself), and as a bonus, the technical.

The Good

As anybody who has ever been to VeeamON or even a Veeam Party at partner conferences such as VMworld will tell you, the company knows how to throw an event. That holds true especially when they do small group events such as Vanguard Summit. The Summit itself consists of two and a half days starting on Tuesday of technical content that we’ll discuss later, but the event, like the Vanguard program itself, is as much about its community as it is the content. Two full days before the first session many if not most of us arrived in Prague to allow those coming long distances to get acclimated prior to the event starting. That meant that as a group we had most of the day Sunday and Monday to hang together as a group, get reacquainted and go sightseeing.

Things got into high gear Monday night as we all gathered at a rooftop bar and restaurant for dinner and “responsible enjoyment” just as sunset was approaching. The views, the food, the people were all pretty magical. Afterwards many of us gathered back in the hotel bar for conversation before getting ready for the content portion to start the next day.

After a full day Tuesday of technical content; some current, much it forward it is a lucky thing that Tuesday was a free night. After the traditional Vanguard toast lead by Craig Dalrymple the free evening allowed us an opportunity to further gather and then get a little extra rest after all we’d processed that day.

Wednesday we got right back to it followed by Vanguard Summit’s signature evening which this year included us taking over the Starpromen Brewery Visitors Center for a good bit more “responsible enjoyment.” This event included some great local foods, a brewery tour and a photo booth all the while getting the chance to relax and get to know the people behind the products you probably use everyday for your backup and replication needs.

The final day of the even is actually just a half day of content, allowing for some final gatherings and sight seeing. In these sessions the content was extremely valuable as it included deep dives into shiny new technology coming in v10 but also included the ability for us to directly provide feedback on the event itself. If you haven’t figured it out yet my own feedback was nothing short of glowing as the event was amazingly planned by Aubrey Galen of Veeam’s events team and the Product Strategy team who among many other things are responsible for the Vanguard program itself. For all of this I owe them a hearty thank you for a wonderful time.

Conclusion

Rather than write in this space I’m just going to load you with some images from the week. Until next time!

This slideshow requires JavaScript.

Here in the US Thanksgiving Day traditionally falls on the fourth Thursday of November. While it is one of my favorite holidays today is a day of thankfulness for me as I’ve been honored to be named a Veeam Vanguard for 2018. I’ve been fortunate enough to have been a part of the group since its inception and it is one of my highest honors. Thanks as always to Rick, Kirsten, Dmitry, Andrew, Niels, Anthony, Michael, Melissa and Danny for keeping the Vanguards the best of its kind around.

To those who have also been renewed into the program please accept a heartfelt congratulations as you’ve earned it through your involvement and I look forward to trolling right along with you for another year.

While the e-mails have just been sent so there aren’t any statistics yet I see quite a few new members who are quite deserving popping up on twitter. Some I know already and other I look forward to getting to know. One of the really nice thing about the Vannies is we are a small group so everybody pretty much gets to know everybody. If you are looking for success in this group please don’t be shy, come be social and share the knowledge you have.

Are you just learning about the program or didn’t make the cut this year? If you are active with Veeam join the conversation in the forums, on Twitter, on Reddit, any of the various Slack communities, or your own blog and it will come. It doesn’t matter where you join, it just matters that you do.

Finally to dear, sweet Vanny Vanguard. We all miss you, please come home. 😉

Backup, among other things, is very good at creating multiple copies of giant buckets of data that don’t change much and tend to sit for long periods of time. Since we are in modern times, we have a number of technologies to deal with this problem, one of which is called deduplication with quite a few implementations of it. Microsoft has had server-based storage versions since Windows 2008 R2 that has gotten better with each release, but as any technology still has its pitfalls to be mindful of. In this post I’m going to look a very specific use case of Windows server deduplication, using it as the storage beneath your Veeam Backup and Replication repositories, covering some basic tips to keep your data healthy and performance optimized.

What is Deduplication Anyway?

For those that don’t work with it much imagine you had a copy of War and Peace stored as a Word document with an approximate file size 1 MB. Each day for 30 days you go into the document and change 100 KB worth of the text in the document and save it as a new file on the same volume. With a basic file system like NTFS this would result in you having 31 MB tied up in the storage of these files, the original and then the full file size of each additional copy.

Now let’s look at the same scenario on a volume with deduplication enabled. The basic idea of deduplication replaces identical blocks of data with very small pointers back to a common copy of the data. In this case after 30 days instead of having 31 MB of data sitting on disk you would approximately 4 MB; the original 1 MB plus just the 100 KB of incremental updates. As far as the user experience goes, the user just sees the 31 files they expect to see and they open like they normally would.

So that’s great when you are talking about a 1 MB file but what if we are talking about file storage in the virtualization world, one where we talking about terabytes of data multi gigabyte changes daily? If you think about the basic layout of a computer’s disk it is very similar to our working copy of War and Peace, a base system that rarely changes, things we add that then sit forever, and then a comparatively few things we change throughout the course of our day. This is why for virtual machine disk files and backup files deduplication works great as long as you set it up correctly and maintain it.

Jim’s Basic Rules of Windows Server Deduplication for Backup Repositories

I have repeated these a few times as I’ve honed them over the years. If you feel like you’ve read or heard this before its been part of my VeeamON presentations in both 2014 and 2015 as well as part of blog posts both here and on 4sysops.com. In any case here are the basics on care and feeding your deduplicated repositories.

1. Format the Volume Correctly. Doing large-scale deduplication is not something that should be done without getting it right from the start. Because when we talk about backup files, or virtual disks in general for that matter, we are talking about large files we always want to format the volume through the command line so we can put some modifiers in there. The two attributes we really want to look at is /L and /A:64k. The /L  is an NTFS only attribute which overrides the default (small) size of the file record. The /A controls the allocation unit size, setting the block size. So for a given partition R: your format string may look like this:

   format R: /L /A:64k /V:BackupRepo1

2. Control File Size As Best You Can. Windows Server 2012 R2 Deduplication came with some pretty stringent recommendations when it came to maximum file size and using deduplication, 1 TB. With traditional backup files blowing past that is extremely easy to do when you have all of your VMDKs rolled into a single backup file even after compression. While I have violated that recommendation in the past without issue I’ve also heard many horror stories of people who found themselves with corrupted data due to this. Your best bet is to be sure to enable Per-VM  backup chains on your Backup Repository (Backup Infrastructure> Backup Repositories> [REPONAME] > Repository> Advanced).
3. Schedule and Verify Weekly Defragmentation. While by default Windows schedules weekly defragmentation jobs on all volumes these days the one and only time I came close to getting burnt but using dedupe was when said job was silently failing every week and the fragmentation became too much. I found out because my backup job began failing due to corrupted backup chain, but after a few passes of defragmenting the drive it was able to continue without error and test restores all worked correctly. For this reason I do recommend having the weekly job but make sure that it is actually happening.
4. Enable Storage-Level Corruption Guard. Now that all of these things are done we should be good, but a system left untested can never be relied upon. With Veeam Backup & Replication v9 we now have the added tool on our backup jobs of being able to do periodic backup corruption checks. When you are doing anything even remotely risky like this it doesn’t hurt to make sure this is turned on and working. To enable this go to the Maintenance tab of the Advanced Storage settings of your job and check the top box. If you have a shorter retention time frame you may want to consider setting this to weekly.
5. Modify Deduplication Schedule To Allow for Synthetic Operations. Finally the last recommendation has to do more with performance than with integrity of data. If you are going to be doing weekly synthetic fulls I’ve found performance is greatly decreased if you leave the default file age before deduplication setting (3 or 5 days depending on version of Windows) enabled. This is because in order to do the operation it has to reinflate each of the files before doing the operation. Instead set the deduplication age to 8 days to allow for the files to already be done processing before they were deduplicated.  For more information on how to enable deduplication as well as how to modify this setting see my blog over on 4sysops.com.

Well with that you now know all I know about deduplicating VBR repositories with Windows Server. Although there is currently a bug in the wild with Server 2016 deduplication, with a fix available, the latest version of Windows Server shows a lot of promise in its storage deduplication abilities. Among other things it pushes the file size limit up and does quite a bit to increase performance and stability.

It has been a great day here because today I learned that I have once again been awarded acceptance into the excellent Veeam Vanguard program, my third time. This program, above any others that I am or have been involved with takes a more personal approach to creating a group of awardees who not only deserve anything good they get out of it but give back just as much to the community itself. In only its 3rd year the group has grown; from 31 the first year, 50(ish) the second, to a total of 62 this year. There are 21 new awardees in that 62 number so there really isn’t a rubber stamp to stay included, it is legitimately awarded each year. The group has grown each year but as you can see not by the leaps and bounds others have, and for good reason. There is no way this experience could be had with a giant community.

At this point in the post I would typically tell you a bit about what the Vanguard program is and isn’t but honestly, Veeam’s own Dmitry Kniazev really put it best in a couple recent posts, “Veeam Vanguard Part 1: WTH Is This?” and “Veeam Vanguard Part 2: What It’s Not.”  What I will add is that as nice as some of the perks are, as DK says in the Part 1 post the true perk is the intangibles; a vibrant community full of some of the smartest, most passionate people in the industry and in many cases access right to the people approving and disapproving changes to their software. These are the thing that made me sweat approval time.

Once again I would give a giant thank you to Veeam Software and especially the whole Vanguard crew. This includes Rick Vanover, Clint Wyckoff, Michael White, Michael Cade, Anthony Spiteri, Kirsten Stoner, Dmitry Kniazev, Andrew Zhelezko and finally Doug Hazelman. Without these people it wouldn’t be nearly as nice.

We’ve been dealing with an issue for past few runs of our monthly SureBackup jobs where the Domain Controller boots into Safe Mode and stays there. This is no good because without the DC booting normally you have no DNS, no Global Catalog or any of the other Domain Controller goodness for the rest of your servers launching behind it in the lab. All of this seems to have come from a change in how domain controller recover is done in Veeam Backup and Replication 9.0, Update 2 as discussed in a post on the Veeam Forums. Further I can verify that if you call Veeam Support you get the same answer as outlined here but there is no public KB about the issue. There are a couple of ways to deal with this, either each time or permanently, and I’ll outline both in this post.

The booting into Safe Mode is totally expected, as a recovered Domain Controller object should boot into Directory Services Restore mode the first time. What is missing though is that as long as you have the Domain Controller box checked for the VM in your application group setup then once booted Veeam should modify the boot setup and reboot the system before presenting it to you as a successful launch. This in part explains why when you check the Domain Controller box it lengthens the boot time allowed from 600 seconds to 1800 seconds by default.

On the Fly Fix

If you are like me and already have the lab up and need to get it fixed without tearing it back down you simply need to clear the Safe Boot bit and reboot from Remote Console. I prefer to

1. Make a Remote Console connection to the  lab booted VM and login
2. Go to Start, Run and type “msconfig”
3. Click on the Boot tab and uncheck the “Safe boot” box. You may notice that Active Directory repair option is selected
4. Hit Ok and select to Restart

Alternatively if you are command inclined a method is available via Veeam KB article 1277  where you just run these commands

bcdedit /deletevalue safeboot
shutdown -t 01 -r

it will reboot itself into normal operation. Just to be clear, either of these fixes are temporary. If you tear down the lab and start it back to the same point in time you will experience the same issue.

The Permanent Fix

The problem with either of the above methods is that while they will get you going on a lab that is already running about 50% of the time I find that once I have my DC up and running well I have to reboot all the other VMs in the lab to fix dependency issues. By the time I’m done with that I could have just relaunched the whole thing. To permanently fix the root issue is you can revert the way DCs are handled by creating a single registry entry as shown below on the production copy of each Domain Controller you run in the lab.

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"UseGranularBcdRestore"=dword:00000000

Once you have this key in place on your production VM you won’t have any issues with it going forward as long as the labs you launch are from backups made after that change is put in use. My understanding is this is a known issue and will eventually be fixed but at least as of 9.5 RTM it is not.

Each year many of the major companies in the tech industry allow people to be nominated, by themselves or by others, to be recognized for the contributions to the community that surrounds that company’s products. These people are typically active on social media, in both online and in person forums and user groups and often will write blogs about their experiences with the products. In return for what is essentially free, grass-roots type marketing the companies will provide awardees any number of benefits; access to licenses for products for homelabbing as well as sometimes access to engineers, preferred experiences at conferences, NDA level information, etc but in some cases the biggest benefit is the recognition itself.

As of today (November 10, 2016) two of the bigger and in my opinion one of the best programs are all open for nominations.

I’m honored to be both a vExpert and a Veeam Vanguard and like to think of myself as an honorary Cisco Champion (they can’t accept government employees) so I have some experience with each of these programs. Let’s take a look at all three.

VMware vExpert may not necessarily be the oldest influencers program but it is probably the one socially active technical people know except possibly the Microsoft MVP program. In many ways vExpert is not only an honorary of its own right but a launch pad towards acceptance into other programs. vExperts are as far as I know the largest such group with around 1500 members world-wide, it also boasts some really good benefits not only from VMware but from other companies in the virtualization ecosphere. There are many webinars and meet and greets throughout the calendar year which are either vExpert only or vExpert preferred and the vExpert party at VMworld is well-known as one of the best. The distinction I make most about vExpert is that while it is for and by VMware, some years much of the educational focus is on the ecosphere and community that surrounds it.

The vExpert program offers 4 paths to membership. The one most are in is the Evangelist path. These may be customers, partners or VMware employees themselves, but they are people speaking the good word of VMware. There are also specific paths for Partners and Customers but I don’t know that I’ve ever met anyone who was awarded in those tracks. Finally if you have achieved the highest level of VMware certification, VCDX, you automatically are awarded vExpert status.

Cisco Champions contrasts from vExpert most because it is a self-contained program with all the educational opportunities and benefits being from Cisco Systems itself. With the Champions there aren’t so many of the freebies with the notable exception of some nice perks if you attend CiscoLive, but what they do offer is exposure of your personal brand. Between the weekly Cisco Champions Radio podcast and the regularly featured blogs on Cisco’s website if you are working to make a name for yourself in the industry for whatever reason it is a very good program for that. Further Cisco gives you access to developers and program  managers within the company so that you can not only gain greater understanding of the products but in many cases have the opportunity to weigh in on technology decisions during the development process.

Cisco breaks their program down into business segments in regards to your qualification for the program with tracks in Collaboration, Data Center, Enterprise Networks, IoT, and Security. If you have expertise in any of these by all means apply.
In my mind I’m saving the best for last. The Veeam Vanguard program opened its nominations up today for its 3rd year and I’ve been honored to have awarded each year (so far). It is by far the most exclusive; there are currently only 50 members worldwide and I believe the philosophy is to keep it on the small side with only people who truly understand what the company is about. There are a lot of swag type benefits to the Vanguard to be sure, most noticeably something really special that revolves around  their VeeamON conference (NOLA this year baby!), but to be honest what I most get out of the program is the distributed brain of not only the Veeam employees affiliated with the group but the group itself. On a daily basis it seems sometimes somebody’s technology issues, Veeam related or not, are being sorted out through Vanguard communication methods. Long story short, in the Vanguard program they simply take care of you and I’m happy to call all of them not just my peers but friends.

Because Veeam is a much tighter set of products than the other two there aren’t any official tracks within the program. That said they are very good about selecting members who affiliate themselves with each of the hypervisor companies they support, VMware’s vSphere and Microsoft’s Hyper-V. This diversity is part of what makes the discussions between us so good.

Conclusion

Over the course of the past week I’ve heard various people talking about strategies regarding getting awarded to any number of these. I’m not going to do this one so I can focus on that one and so forth, and honestly all I can recommend to you if you are interested in applying to any of them is look at where your focus is or where you focus should be and apply. There is no thing that says “you belong to too many programs” or anything like that; if you feel you are qualified for any of these or any other by all means go apply. The name of the game is to grow your involvement with the technology community, regardless of what type of technology it is.