So yeah, I’m heading to an in-person tech conference next week. Not just any tech conference but arguably my favorite, VeeamON. The event is being held at the Aria Resort in Las Vegas May 16-19 for the in person aspect but like so many of conferences in the post-COVID era it will be a hybrid event with a free of charge, online aspect as well. If you have not registered for either in person or online there is still plenty of time to get involved.

It’s been since August of 2019 since I last did any event of real size and while I am absolutely excited to be able to hear about some wonderful technology and be back around friends, colleagues and fellow Vanguards, I’d be remiss to not acknowledge a healthy amount of hesitance about being around large groups of people again. That said at this point I’ve done everything that I can possibly do to protect myself and those around me from all the health concerns so it’s time to get at it.

On a personal note this time around is going to be a bit different for me as well as it is going to be the first event I’ve attended as part of a partner organization as opposed to just being there as a customer. In the lead up there is quite a bit of preparation involved when attending this way but a great deal to be interested in as well.

What’s Going On?

So a look at the full agenda tells you that Veeam Backup & Replication v12 is going to be a major focus of the conference this time around and for good reason; v12 from everything I’ve seen so far is going to be a MASSIVE release in terms of under the cover infrastructure. I’ve been working with the beta since release and honestly I’m rarely leaving the backup infrastructure tab because there’s so much different there. Layer in improvements to related products such as the Veeam Agents and the hyper scaler cloud backup products and I’ll be busy long after the event watching recordings.

While V12 will be a major focus it won’t be the only one. Support for backing up SaaS application will be there as well with further improvements to Veeam Backup for Microsoft 365 and the upcoming release of Veeam Backup for Salesforce.

I’m personally going to have a couple of speaking sessions myself. The first will be with Josh Liebster as we discuss why “All Good Things Start at 11:11” Monday at 6pm on the partner stage in the expo hall. This quick session will be for my employer iland, an 11:11 Systems company and will talk about how we can help you with your disaster recover and cloud workload needs.

The second session I will be involved in will be with Mr. Liebster again as well as Sean Smith from VMware Tuesday at 10:15am titled “VCPP, VMware Cloud Director and Veeam — Bringing It All Together as a Full Stack.” In this session Sean will be talking about VMware Cloud Director and the overall VCPP program of which iland is an “all-in” member. Josh and I will talk about how iland takes what VMware provides and turns it into an award winning set of products.

All in all I’m expecting it to be an excellent as always. If you are going to be there please reach out to me through twitter and I’d love to meet up!

In my last post I worked through quite a few things I’ve learned recently about interacting with S3 Compatible storage via the CLI. Now that we know how to do all that fun stuff it’s time to put it into action with a significant Service Provider/Disaster Recovery slant. Starting with this post I’m going to highlight how to get started with some common use cases of object storage in Backup/DR scenarios. In this we’re going to look at a fairly mature use case, with it backing Veeam Backup for Office (now Microsoft) 365.

Veeam Backup for Microsoft 365 v6, which was recently showcased at Cloud Field Day 12, has been leveraging object as a way to make it’s storage consumption more manageable since version 4. Object also provides a couple more advantages in relation to VBM, namely an increase in data compression as well as a method to enable encryption of the data. With the upcoming v6 release they will also support the offload of backups to AWS Glacier for a secondary copy of this data.

VBM exposes its use of object storage under the Object Storage Repositories section of Backup Infrastructure but it consumes it as a step of the Backup Repository configuration itself, which is nested within a given Backup Proxy. I personally like to at a minimum start with scaling out repositories by workload (Exchange, OneDrive, Sharepoint, and Teams) as each data type has a different footprint. When you really need to scale out VBM, say anything north of 5000 users in a single organization, you will want to use that a starting point for how you break down and customize the proxy servers.

Let’s start by going to the backup proxy server, in this case the VBM server itself, and create folder structure for our desired Backup Repositories.

Now that we have folders let’s go create some corresponding buckets to back them. We’ll do this via the AWS S3 CLI as I showed in my last post. At this point VBM does not support advanced object features such as Immutability so no need to be fancy and use the s3api, but I just prefer the command structure.

Ok so now we have folder and buckets, time to hop in to Veeam. First we need to add our object credentials to the server. This is a simple setup and most likely you will only need one set of credentials for all your buckets. Because in this example I will be consuming iland Secure Cloud Object Storage I need to choose the “S3 Compatible access key” under the “Add…” button in Cloud Credential Manager (menu> Cloud Credentials). These should be the access key and secret provided to you by your service provider.

Now we need to go to Backup Infrastructure > Object Storage Repositories to add our various buckets. Start by right clicking and choose “Add Object Storage.”

Now simply repeat the process above for any and all buckets you need for this task.

Now that we have all our object buckets added we need to pair these up with our on premises repository folders. It’s worth noting that the on-prem repo is a bit misleading, no backup data as long as you use the defaults will ever live locally in that repository. Rather it will hold a metadata file in the form of a single jetDB file that service as pointers to the objects that is the actual data. For this reason the storage consumption here is really really low and shouldn’t be part of your design constraints.

Under Backup Infrastructure > Backup Repositories we’re going to click “Add Repository..” and let the wizard guide us.

One note on that final step above. Often organization will take the “Keep Forever” option that is allowed here and I will say I highly advise against this. You should specify a retention policy that is agreed upon with your business/organization stakeholders as keeping any backup data longer than needed may have unintended consequences should a legal situation arise; data the organization believes to be long since gone is now discoverable through these backups.

Also worth noting item-level retention is great if you are using a service provider that does not charge you on egress fees because it gives you more granular control in terms of retention. If you use a hyperscaler such as Amazon S3 you may find this option will drive your AWS bill up because of a much higher load on egress each time the job runs.

Once you’ve got one added again, rinse and repeat for any other repositories you need to add.

Finally the only step left to do is create jobs targeting our newly created repositories. This is going to have way more variables based on your organization size, retention needs, and other factors than I can truly do justice in the space of this blog post but I will show how to create a simple, entire organization, single workload job.

You can start the process under Organizations > Your Organization > Add to backup job…

Once again you’d want to repeat the above steps for all your different workload types but that’s it! If we do a s3 ls on our s3://premlab-ilandproduct-vbm365-exch/Veeam/Backup365/ilandproduct-vbm365-exch/ full path we’ll see a full folder structure where it’s working with the backup data, proving that we’re doing what we tried to do!

In conclusion I went way into depth of what is needed here but in practice it isn’t that difficult considering the benefits you gain by using object storage for Veeam Backup for Microsoft365. These benefits include large scale storage, encryption and better data compression. Hope you find this helpful and check back soon for more!

I was honored to be selected to present at VeeamON again in 2015 with the same general concept of session but really highlighting some of the work I’d been doing with Windows Server deduplication and Veeam SureBackup. This was a much nice crowd, with 50 or so people, and I felt it went much better. Again there is no video this time around but as an aside watching the replay at the time this is when I first realized just how much of an accent I have, literally had never realized it before.

So this was immediately following the vBrownBag session. Again, now second shot at “public speaking” but this time I’m in a room for 200, in the hangover slot at a tech conference (8 am the morning after the party), there were a total of 3 people in the session. Probably not the worst thing but one thing I’ve found over the years is that crowds are good, they loosen you up. Just goes to show that we definitely all start somewhere.

Unfortunately I don’t have the video of these early session so you are just going to have to trust me that it was rough.

Slidedeck: https://www.slideshare.net/JimJones147/veeamon-2014-vbrownbag-physical-server-backup-strategies

This was actually my first public speaking presentation ever. Supposed to be a 15 minute time slot I believe I finished in under 8 minutes. What aided to my discomfort was that the whole purpose of this was to show a method I was using to backup physical servers in a Veeam world, something there was no answer to at the time. That morning it leaked that Veeam was announcing what is now known as Veeam Agent for Windows and my friend Rick Vanover was following me on the same stage to announce it. Talk about your topic becoming irrelevant in record time…

This year I was honored with the ability to attend Veeam Software’s Vanguard Summit. Summit comprises a meeting of 60-70 of world’s best Veeam Engineers, Architects, and Partners along with their own Product Strategy and Management groups in Prague, Czech Republic.

Often in talking about a subject we’ll float the old cliché of the good, the bad and ugly of it and frankly there is nothing negative I can say about this event. Because of that in this series of posts I’m going to cover the good (the event itself), the awesome (the people and content), the beautiful (the city of Prague itself), and as a bonus, the technical. This post will be the second in the series, covering the awesome people and content that comprise Vanguard Summit.

Community

There are many vendor community programs out there, most of which are much bigger than the Vanguards, and while they may have some nice intrinsic benefits such as licenses or hoodies, nobody does it better than Rick Vanover and his team. This is my 5th year in the program but due to the rules of my previous employer I’ve never been able to participate in person. The addition of being able to interact with seventy of the smartest people was invigorating and enjoyable. While I was happy to meet and spend time with everyone it was great to get to catch up with community stalwarts and friends such as Matt Crape, Craig Dalrymple, Brad Jervis, Dean Lewis and Al Rasheed.

There is nothing like being in a room with your peers, most of which leave me in awe with their abilities, hearing about what comes next from a company and have your feedback and thoughts sought out. In the process the attendees start collaborating on ideas that end up shaping things for years to come. I’ve said this before, but if you think the value of any program like this is measured monetarily, you are doing it wrong. The true value is the information that not only passes from the vendor to the program members but between the members itself and this group is in my mind the best of the best.

Content

I remember back when I began as a Veeam customer there was the free FastSCP product and Veeam Backup & Replication. After nine years it is amazing to see how the product line has grown and this week made that very apparent, with 2.5 days jam packed with content on current and future products. In a later post I’m going to be covering a lot of what’s coming in v10 of Veeam Backup & Replication but we were also treated to updates on Backup for O365, VeeamONE, Veeam Agents for Windows and Linux, as well as Veeam Availability Orchestrator. This doesn’t include the things we aren’t allowed to discuss but frankly are pretty exciting.

One of the ways that this content is vastly different from other vendor briefings you may have attended or seen maybe at a TechFieldDay (who Veeam will once again be attending for TFD #20) event or the like, besides the great Product Strategy staff many of these sessions were led or included the product management team for all of their products. The highlight to me of this even is the final session on Tuesday where we had Anton Gostev, Alec King, and others from the product management/development team in an “Ask Me Anything” style session where we were given the opportunity to ask all we wanted and as best I could tell were given no “BS” answers. In short the content was pretty amazingly done and consisted of far more access to information than I’ve seen from any other vendor.

Conclusion

Much like my last post rather than bore you with more of my opinions I’ll leave you with some pictures from the event.

This slideshow requires JavaScript.

This year I was honored with the ability to attend Veeam Software’s Vanguard Summit. Summit comprises a meeting of 60-70 of world’s best Veeam Engineers, Architects, and Partners along with their own Product Strategy and Management groups in Prague, Czech Republic. As one who had never been to Europe and has long been an advocate of Veeam’s products Vanguard Summit had all the makings of an awesome event and it never came close to letting me down.

Often in talking about a subject we’ll float the old cliché of the good, the bad and ugly of it and frankly there is nothing negative I can say about this event. Because of that in this series of posts I’m going to cover the good (the event itself), the awesome (the people and content), the beautiful (the city of Prague itself), and as a bonus, the technical.

The Good

As anybody who has ever been to VeeamON or even a Veeam Party at partner conferences such as VMworld will tell you, the company knows how to throw an event. That holds true especially when they do small group events such as Vanguard Summit. The Summit itself consists of two and a half days starting on Tuesday of technical content that we’ll discuss later, but the event, like the Vanguard program itself, is as much about its community as it is the content. Two full days before the first session many if not most of us arrived in Prague to allow those coming long distances to get acclimated prior to the event starting. That meant that as a group we had most of the day Sunday and Monday to hang together as a group, get reacquainted and go sightseeing.

Things got into high gear Monday night as we all gathered at a rooftop bar and restaurant for dinner and “responsible enjoyment” just as sunset was approaching. The views, the food, the people were all pretty magical. Afterwards many of us gathered back in the hotel bar for conversation before getting ready for the content portion to start the next day.

After a full day Tuesday of technical content; some current, much it forward it is a lucky thing that Tuesday was a free night. After the traditional Vanguard toast lead by Craig Dalrymple the free evening allowed us an opportunity to further gather and then get a little extra rest after all we’d processed that day.

Wednesday we got right back to it followed by Vanguard Summit’s signature evening which this year included us taking over the Starpromen Brewery Visitors Center for a good bit more “responsible enjoyment.” This event included some great local foods, a brewery tour and a photo booth all the while getting the chance to relax and get to know the people behind the products you probably use everyday for your backup and replication needs.

The final day of the even is actually just a half day of content, allowing for some final gatherings and sight seeing. In these sessions the content was extremely valuable as it included deep dives into shiny new technology coming in v10 but also included the ability for us to directly provide feedback on the event itself. If you haven’t figured it out yet my own feedback was nothing short of glowing as the event was amazingly planned by Aubrey Galen of Veeam’s events team and the Product Strategy team who among many other things are responsible for the Vanguard program itself. For all of this I owe them a hearty thank you for a wonderful time.

Conclusion

Rather than write in this space I’m just going to load you with some images from the week. Until next time!

This slideshow requires JavaScript.

Here in the US Thanksgiving Day traditionally falls on the fourth Thursday of November. While it is one of my favorite holidays today is a day of thankfulness for me as I’ve been honored to be named a Veeam Vanguard for 2018. I’ve been fortunate enough to have been a part of the group since its inception and it is one of my highest honors. Thanks as always to Rick, Kirsten, Dmitry, Andrew, Niels, Anthony, Michael, Melissa and Danny for keeping the Vanguards the best of its kind around.

To those who have also been renewed into the program please accept a heartfelt congratulations as you’ve earned it through your involvement and I look forward to trolling right along with you for another year.

While the e-mails have just been sent so there aren’t any statistics yet I see quite a few new members who are quite deserving popping up on twitter. Some I know already and other I look forward to getting to know. One of the really nice thing about the Vannies is we are a small group so everybody pretty much gets to know everybody. If you are looking for success in this group please don’t be shy, come be social and share the knowledge you have.

Are you just learning about the program or didn’t make the cut this year? If you are active with Veeam join the conversation in the forums, on Twitter, on Reddit, any of the various Slack communities, or your own blog and it will come. It doesn’t matter where you join, it just matters that you do.

Finally to dear, sweet Vanny Vanguard. We all miss you, please come home. 😉

Backup, among other things, is very good at creating multiple copies of giant buckets of data that don’t change much and tend to sit for long periods of time. Since we are in modern times, we have a number of technologies to deal with this problem, one of which is called deduplication with quite a few implementations of it. Microsoft has had server-based storage versions since Windows 2008 R2 that has gotten better with each release, but as any technology still has its pitfalls to be mindful of. In this post I’m going to look a very specific use case of Windows server deduplication, using it as the storage beneath your Veeam Backup and Replication repositories, covering some basic tips to keep your data healthy and performance optimized.

What is Deduplication Anyway?

For those that don’t work with it much imagine you had a copy of War and Peace stored as a Word document with an approximate file size 1 MB. Each day for 30 days you go into the document and change 100 KB worth of the text in the document and save it as a new file on the same volume. With a basic file system like NTFS this would result in you having 31 MB tied up in the storage of these files, the original and then the full file size of each additional copy.

Now let’s look at the same scenario on a volume with deduplication enabled. The basic idea of deduplication replaces identical blocks of data with very small pointers back to a common copy of the data. In this case after 30 days instead of having 31 MB of data sitting on disk you would approximately 4 MB; the original 1 MB plus just the 100 KB of incremental updates. As far as the user experience goes, the user just sees the 31 files they expect to see and they open like they normally would.

So that’s great when you are talking about a 1 MB file but what if we are talking about file storage in the virtualization world, one where we talking about terabytes of data multi gigabyte changes daily? If you think about the basic layout of a computer’s disk it is very similar to our working copy of War and Peace, a base system that rarely changes, things we add that then sit forever, and then a comparatively few things we change throughout the course of our day. This is why for virtual machine disk files and backup files deduplication works great as long as you set it up correctly and maintain it.

Jim’s Basic Rules of Windows Server Deduplication for Backup Repositories

I have repeated these a few times as I’ve honed them over the years. If you feel like you’ve read or heard this before its been part of my VeeamON presentations in both 2014 and 2015 as well as part of blog posts both here and on 4sysops.com. In any case here are the basics on care and feeding your deduplicated repositories.

1. Format the Volume Correctly. Doing large-scale deduplication is not something that should be done without getting it right from the start. Because when we talk about backup files, or virtual disks in general for that matter, we are talking about large files we always want to format the volume through the command line so we can put some modifiers in there. The two attributes we really want to look at is /L and /A:64k. The /L  is an NTFS only attribute which overrides the default (small) size of the file record. The /A controls the allocation unit size, setting the block size. So for a given partition R: your format string may look like this:

   format R: /L /A:64k /V:BackupRepo1

2. Control File Size As Best You Can. Windows Server 2012 R2 Deduplication came with some pretty stringent recommendations when it came to maximum file size and using deduplication, 1 TB. With traditional backup files blowing past that is extremely easy to do when you have all of your VMDKs rolled into a single backup file even after compression. While I have violated that recommendation in the past without issue I’ve also heard many horror stories of people who found themselves with corrupted data due to this. Your best bet is to be sure to enable Per-VM  backup chains on your Backup Repository (Backup Infrastructure> Backup Repositories> [REPONAME] > Repository> Advanced).
3. Schedule and Verify Weekly Defragmentation. While by default Windows schedules weekly defragmentation jobs on all volumes these days the one and only time I came close to getting burnt but using dedupe was when said job was silently failing every week and the fragmentation became too much. I found out because my backup job began failing due to corrupted backup chain, but after a few passes of defragmenting the drive it was able to continue without error and test restores all worked correctly. For this reason I do recommend having the weekly job but make sure that it is actually happening.
4. Enable Storage-Level Corruption Guard. Now that all of these things are done we should be good, but a system left untested can never be relied upon. With Veeam Backup & Replication v9 we now have the added tool on our backup jobs of being able to do periodic backup corruption checks. When you are doing anything even remotely risky like this it doesn’t hurt to make sure this is turned on and working. To enable this go to the Maintenance tab of the Advanced Storage settings of your job and check the top box. If you have a shorter retention time frame you may want to consider setting this to weekly.
5. Modify Deduplication Schedule To Allow for Synthetic Operations. Finally the last recommendation has to do more with performance than with integrity of data. If you are going to be doing weekly synthetic fulls I’ve found performance is greatly decreased if you leave the default file age before deduplication setting (3 or 5 days depending on version of Windows) enabled. This is because in order to do the operation it has to reinflate each of the files before doing the operation. Instead set the deduplication age to 8 days to allow for the files to already be done processing before they were deduplicated.  For more information on how to enable deduplication as well as how to modify this setting see my blog over on 4sysops.com.

Well with that you now know all I know about deduplicating VBR repositories with Windows Server. Although there is currently a bug in the wild with Server 2016 deduplication, with a fix available, the latest version of Windows Server shows a lot of promise in its storage deduplication abilities. Among other things it pushes the file size limit up and does quite a bit to increase performance and stability.

It has been a great day here because today I learned that I have once again been awarded acceptance into the excellent Veeam Vanguard program, my third time. This program, above any others that I am or have been involved with takes a more personal approach to creating a group of awardees who not only deserve anything good they get out of it but give back just as much to the community itself. In only its 3rd year the group has grown; from 31 the first year, 50(ish) the second, to a total of 62 this year. There are 21 new awardees in that 62 number so there really isn’t a rubber stamp to stay included, it is legitimately awarded each year. The group has grown each year but as you can see not by the leaps and bounds others have, and for good reason. There is no way this experience could be had with a giant community.

At this point in the post I would typically tell you a bit about what the Vanguard program is and isn’t but honestly, Veeam’s own Dmitry Kniazev really put it best in a couple recent posts, “Veeam Vanguard Part 1: WTH Is This?” and “Veeam Vanguard Part 2: What It’s Not.”  What I will add is that as nice as some of the perks are, as DK says in the Part 1 post the true perk is the intangibles; a vibrant community full of some of the smartest, most passionate people in the industry and in many cases access right to the people approving and disapproving changes to their software. These are the thing that made me sweat approval time.

Once again I would give a giant thank you to Veeam Software and especially the whole Vanguard crew. This includes Rick Vanover, Clint Wyckoff, Michael White, Michael Cade, Anthony Spiteri, Kirsten Stoner, Dmitry Kniazev, Andrew Zhelezko and finally Doug Hazelman. Without these people it wouldn’t be nearly as nice.