Today’s the day, the beginning of Veeam Software’s annual conference, VeeamON for 2023! Just like the Blues Brothers sang before me I’m Goin’ Down to Miami to learn, connect, and this year speak at the Fountainebleau in Miami, Florida and I can’t be more excited. This year Veeam has made a marked return to the community focus of the event with a significant number of the nearly 50 in-person sessions having at least one community speaker.

What is VeeamON?

If the event is new to you this is the seventh edition of the event and marks the return to Miami where it was held in 2019. The conference itself is two days, Tuesday and Wednesday May 23-24 but with training and partner events stretches Saturday through Wednesday. To kick the weekend off attendees can choose to take a discounted, forward looking version of the Veeam Certified Engineer (VMCE) training which I highly recommend if you are new to the software or are looking to grow your skills and add credentials to your resume. I’ve been a VMCE since 2015 and it’s been a very well considered certification.

Today, Monday is both the partner day, where those of us in the Veeam Cloud Service Provider (VCSP) world gather as well as a welcome reception for all attendees in the evening. It’s worth noting there will also be a Kubernetes and Kasten K10 Workshop running through the afternoon. The next two days will be jam packed with many sessions that cover everything from getting started with their entire suite of products through exceptional deep dives in to how to make your backup and more importantly recovery needs go seamlessly. The event will wrap up with another one of Veeam’s legendary parties right on the resort’s grounds.

Throughout the event Veeam’s many partners such as my employer, 11:11 Systems, will be available in the Tech Festival area. I highly recommend spending time in this area as Veeam is a very ecosystem focused software company and there any number of capabilities that partners make possible that you may not even be considering yet but should.

Sessions You Say?

Did I mention sessions and community speakers earlier? Many of these sessions are being given by fellow Veeam Vanguards and wider Veeam 100, a group that I’m privileged to be a member of and in the spirit of full disclosure is taking care of our travel, lodging and conference attendance. There are quite a number that I’m going to make sure I check out myself this year that are both technical and career focused, these include:

• You’re Hit by a Ransomware Attack, What’s Next? (Julien Mousqueton, Christopher Glemot, and Eric Machabert)
• 10 Years Working with Veeam: How it Helped Shape Our Careers (Ian Sanderson, Dean Lewis, and Matt Crape)
• A-Z Automation Journey with Veeam’s Automation Desk (Maurice Kevenaar and Joe Houghes)
• Kubernetes for the Virtualization Admin (Dean Lewis and Michael Cade)
• Veeam Replication: The do’s and don’ts of BC/DR (Eric Kubla)
• There is a Hole in my Bucket! How to Plug Security Leaks (Nicolas Serrecchia and Olivier Rossi)
• Veeam Service Provider Console v7: What’s New (Tim Hudson and Ivan Kochemasov)

I am also going to be a part of a couple of sessions, both on Tuesday afternoon. Starting things off will be “Architecting Veeam Backup for Microsoft365 at Scale” with my friend and fellow Vanguard Falko Banaszak. This session will be a very technical brain dump from both Falko and myself on many lessons learned on how to make this protection work and work well when your are talking about tens of thousands of users.

Immediately following that will be “Lessons From the Field: Modernizing and Enhancing Your BaaS & DRaaS Infrastructure,” a joint session with Tim Hudson of the Veeam VCSP team and Ivan Kochemasov of Veeam’s Product Management group. Through my position at 11:11 Systems I work with these gentlemen constantly and we will be talking about how to make a good Partner to Veeam relationship great.

Wrapping up I expect it to be a great event as always. Veeam this year in particular seems to have picked up the mantle created by VMworld events where it very much so feels community focused and allows for learning through collaboration. If you will be at the event I would love for you to reach out so we can meet up. I can be found in most places social media @k00laidIT.

As we continue our journey into exploring new products and features that Veeam has released for 2023 it’s well worth the time to look at what Veeam Backup for AWS (VBA) is looking like these days. If you haven’t been using this software yet with this version 6 release it has deepened its relationships with other Veeam products, namely Veeam Cloud Connect powered by Backup and Replication (VBR) and Veeam Service Provider Console (VSPC). This allows VCSPs to supply a true managed BaaS service without the customer needing their own VBR and/or VBA components.

In the first releases of Veeam Backup for AWS installation and licensing were exclusively done via the Amazon AWS Marketplace; customer goes and looks for it, installs it and then pays AWS a fixed fee for usage, all the while being required to manage their own backups. The more modern releases have allowed integration into a customer installed Veeam Backup and Replication, allowing for deployment and management via VBR’s console and allowing it to inherit Veeam Universal Licenses (VUL) via that installation.  That is much better but still requires customer own and manage their backups and licensing for the most part, something that I’ve heard from numerous customers and potential customers that are fully into hyperscale cloud for their infrastructure they are hesitant to do. 

Now that this combination of releases has come out, VBR v12, VSPC v7 and VBA v6, we have the ability to do something more like the common Shared Responsibility Model that cloud first customers are accustomed to; the customer owns the production workloads and data while the service provider owns the infrastructure that powers the backup service, with both capable of managing the backup jobs and copies as needed. 

With each release not only has Veeam added to the deployment and management methods they have also added features. With this most recent release when coupled with the aaS model you have the capability to backup workloads through the following AWS services: 

What Can It Do 

• EC2 compute instances 
• RDS databases 
• EFS file systems
• VPC configuration

All of these backups will land into immutable, encrypted, and secured AWS S3 storage with the capability to provide a second equally secured copy into a separate region and/or a separate AWS account. Further as part of your data protection policy creation you can also schedule AWS snapshots for the local availability zone (AZ) with replicas of snapshots then able to be sent to different regions and/or accounts. This is similar (but not quite) like VM replication. These replicas then show in inventory in the DR AZ and can be failed over to as needed. All in all you can protect the majority of services that most “cloud first” customers need and be able to backup, copy or replicate to a very promising number of locations. 

How Does This Thing Work 

For those who are used to how Veeam does backup job creation for traditional workloads this is quite a bit different in that it is policy based. Either you the customer or 11:11 Systems for a managed service create policy definitions for your workloads via Console. These policies define things like where to land backups, how copies should be handled, if snapshots are created and then replicated and schedules. Those policies can then be applied to workloads that are discovered via an AWS key pair that you create with defined IAM policies and then add to Veeam Backup for AWS via Console. Monitoring and management then also be provided by Console so in the end you have a modern, Shared Responsibility Model style BaaS capability with a single UI  for the customer. 

Conclusion 

Backup for public cloud workloads is that thing that we as IT professionals truly need to care about. These are still running workloads that are critical to your business and just as we’d all shudder at thinking “snapshots are backups” in traditional virtualized environments you shouldn’t exclusively rely on that in the cloud. The combination of Veeam Backup for AWS, as well as Azure and GCP, and service provider style services provided by 11:11 Systems will allow customers to in effect have their clouds and back them up too. 

I feel like I could write, produce, direct and star in a saga based in a galaxy far, far away with the amount of time I have spent talking about the combination of Microsoft365, backup applications that support it such as Veeam Backup for Microsoft365, and the recent Microsoft Teams Graph Export API. Yet here I am again because Microsoft has decided to change course with how this capability will be handled without telling anyone.

Recently I started to hear of support cases where users were doing the needful by filling out Microsoft’s form and then after the prescribed 5-7 business days attempting to enable support for Teams Chat backup available in VB365 versions 6a and 7 only to be presented with the error “To call this API the app must be associated with an Azure subscription, see https://aka.ms/teams-api-payment-requirements for details.”

For those who have been dealing with the Teams Graph Export metered API it’s a telltale sign that for whatever reason the Azure subscription ID associated with the organization modern application used to do backups was not available. Upon looking at the form I noticed that it no longer requires a subscription ID meaning that, once again, Microsoft has changed the process and not told anyone, leaving backup vendors as well as their partners scrambling to make sure our customers are properly suited. This has been well covered at this point by Ian Sanderson in his blog “Protecting Microsoft Teams Channel Chat Data: Are You Prepared?”

Upon trying to enable this protection for myself and as a matter of testing I’ve found that while the instructions are correct when using Azure Cloudshell it doesn’t work if you’ve just installed the Azure CLI on a windows system.  This is due to a bug mentioned deep in the recesses of github that essentially says that cloud shell is bash based so you have to use different escape sequences to make it work. Further the Microsoft guide leaves out the step of creating a resource group to bind your appId and Subscription ID together. For all of those reasons I felt it necessary to create a follow up to Ian’s post to show you how to enable all this via Azure CLI on a Windows Server.

How-To Time!

0. If you haven’t already install Azure CLI from the download or via Chocolatey “choco install -y azure-cli”

1. Login or have the account owner login to their Azure account with “az login –use-device-code”. That will have the effect of creating a code for you to login to Microsoft with much like when you register an organization with Veeam Backup for Microsoft365. Once processed it will output a list of your active subscriptions. The id field there should be your subscription ID you will need later.

2. Next you will need to register the Microsoft.GraphServices namespace as shown in the Microsoft document. This is general using “az provider register --namespace Microsoft.GraphServices”

3. Next you need to create a resource group to be the relationship will live within. This is done, assuming you want to create your resources in the Eastern US region of Azure, by using “az group create -l eastus -n myResourceGroup” If you need another region you can pull a list of those available with “az account list-locations“

4. Finally, we now must link the application ID (which you should have from the confirmation email from Microsoft) and the subscription ID from your login above and create a resource. This is where the formatting matters so simply select the all-caps text below to replace
az resource create --resource-group koolaidInfoBilling --name myResourceGroup --resource-type Microsoft.GraphServices/accounts --properties "{\`"appId\`": \`"INSERTAPPIDHERE\`"}" --location Global --subscription “INSERTSUBSCRIPTIONIDHERE”

That should return a block of JSON that looks like this:

{
"extendedLocation": null,
"id": "/subscriptions/3047753f-7bb6-47ff-802b-55358facde54/resourceGroups/myResourceGroup/providers/Microsoft.GraphServices/accounts/myGraphAppBilling",
"identity": null,
"kind": null,
"location": "Global",
"managedBy": null,
"name": "myGraphAppBilling",
"plan": null,
"properties": {
"appId": "MYAPPLICATIONID",
"billingPlanId": "MYNEWBILLINGPLANID",
"provisioningState": "Succeeded"
},
"resourceGroup": "myResourceGroup",
"sku": null,
"systemData": {
"createdAt": "2023-03-27T17:16:29.8280947Z",
"createdByType": "User",
"lastModifiedAt": "2023-03-27T17:16:29.8280947Z",
"lastModifiedByType": "User"
},
"tags": null,
"type": "microsoft.graphservices/accounts"
}

Once that provisioningState shows as succeeded you should be able to go run your Teams Chat enabled VB365 job again and find some success, at least until Microsoft decides to mess with it again. I will add personally that they are truly doing a disservice to their ecosystem; their partners don’t really seem to have a good idea of when/if changes are going to happen and those of us downstream, both Service Providers and Customers, are put into a perpetual state of trying to catch up to poorly considered implementations.

I’ve just completed watching the Veeam v12 Data Platform Launch Event keynote and one theme I feel is getting a lot of chatter is the new “Direct to object storage” capabilities of the release. What this means is that as I create backups of my production data the first copy that is made is written to object storage. I’ve written quite a bit about v12 and object storage before, both in how to setup and consume it in the performance tier of Scale Out Backup Repositories as well as comparing copy mode object storage and Veeam Cloud Connect Backup, but this is not that. Direct to object storage is the idea that you write your first backup copy of production data to object storage, rather than as a secondary copy.

Anton Gostev himself highlighted this as a favorite feature and outlined that there are limited uses cases where it should be used. As a long time Veeam administrator and of late an architect of their largest Service Provider partner I have some thoughts into what those use cases should and should not be that I’d like to share.

The Do’s

✅ On Premises Backup Repository

The first and in my opinion most exciting use case for direct to object storage capabilities is as a replacement for your old block based on premises backup repositories. Object storage has numerous benefits over traditional storage servers using NTFS, ReFS or even XFS. These include true object-lock backed immutability, real scalability in that you simply continue to add nodes to the cluster as needed and API first, automation driven provisioning.  All these things can combine to a scalable, secure, and durable on premises backup storage location.

✅ Distributed Workforce Agent Backups

As we find ourselves in the world of remote work in 2023 systems administrators have more concerns than in the past about protecting those laptops that our remote workers are using as critical company data further escapes the datacenter. By combining Veeam Backup and Replication v12, Veeam Service Provider Console v7 and public object storage services such as 11:11 Systems’ offering we can create a robust solution for managing these backups securely and at scale with self-service capabilities included.

✅ NAS Backups

As much as cloud-based storage services such as OneDrive, Google Drive and Dropbox have become prevalent in modern IT there are still a lot of traditional Network Attached Storage file servers around. The problem is these are usually measured in 100s of TBs and backups don’t scale well in any case. In this case I believe it’s appropriate to back these up via the NAS backup feature of VBR directly to immutable cloud-based object storage. Recoverability is in terms of files as opposed to needing to write Virtual Machines to a hypervisor so far more appropriate.

The Don’ts

Now that we’ve covered what are good use cases let’s talk about some things that should not be considered from a best practices point of view but are technically possible.

❌ Cloud-Based Object Storage for First Backup

There are going to be people who begin wanting to get rid of the on-premises backups all together and replace them with low-cost cloud object storage. While the immutability and ease of scale are tempting, don’t. Just don’t. Recovery from this for any backup platform would be painful at best, unsuccessful at worst. Further you lose so much of the power of Veeam Backup & Replication by doing this with capabilities such as Instant Recovery and SureBackup instantly going away.

❌ Remote Office/Branch Office (ROBO) Server Backups

One of the use cases specifically called out by Gostev for direct to object storage was the ROBO server. While I respect him very much, I do have to disagree with him here. As someone who’s had to restore this scenario in anger the pressure will be on in most if not all these situations to get that server back up and running as fast as possible, probably in your production datacenter with some creative networking setup, to get that remote location back up and in business as soon as possible. By and large direct to cloud-based backups for virtual machines or server grade agents should not in my opinion be considered as you quickly become limited in recoverability choices. If you cannot either back these up first to a small NAS unit at the ROBO or to storage available in your production location, then VCC-B may be interesting as a first write backup. While definitely not preferred at least things like Instant Recovery to Cloud Director can be an option to get these workloads available sooner.

Conclusion

While writing backups from Veeam Backup & Replication v12 directly to object storage is an exciting innovation to the platform it is a capability that should be given a great deal of consideration before putting into production grade use. Features such as immutability and scalability will drive adoption what you are using it for is far more important than the hype.

With the upcoming version 12 release of Veeam Backup & Replication the platform will start to support object storage as a primary location for backups. This is not a new idea, it feels like we’ve been talking about this forever, but it is a radical shift for the company and frankly one that for reasons many in the Veeam Certified Service Provider community have been dreading. While I believe object storage has its place, even within a VCSP, it should not in general practice be a replacement for Veeam Cloud Connect as VCC-B and VCC-R offer capabilities and enhancements that object storage services all by themselves will never be able to replicate.

What is Veeam Cloud Connect?

It’s appropriate to level set on the technologies at hand first. Veeam Cloud Connect was first announced at VeeamON 2014 and became available through partners shortly thereafter. There are two sides to today’s Cloud Connect Service: Veeam Cloud Connect Backup (VCC-B) and Veeam Cloud Connect Replication (VCC-R).

Veeam Cloud Connect Backup allows for a service provider to provision a slice of cloud storage and securely provision that a customer as a tenant. That storage can have further enhancements such as Immutability if on Linux repositories or with v12 object storage and Insider Protection, an out-of-band temporary storage location for deleted cloud backups as part of a ransomware mitigation strategy. Once the tenant is provisioned on the SP side a customer only needs to enter 3 pieces of information to add that storage as a repository or repositories to their backup infrastructure. Afterwards they can simply target that repo for backup copy or in some situations even direct backup jobs.

Veeam Cloud Connect Replication is the same general concept as VCC-B, but just with VMs being stored directly in IaaS. Your Service Provider would provision quotas in a VMware Cloud Director or vCenter environment in which powered off copies of the source VMs are copied. These replicas maintain a shorter maximum number of restore points which are stored as snapshots on the replica.  In conjunction with either a VPN connection or the Veeam Network Extension Appliance you can extend your on-prem environment into your replicated IaaS one and run those systems remotely for several reasons.

Object Storage Review

Veeam has been slowly building support for object storage over a few releases now. In short order the progression has been:

• 9.5u4 (2019): Support for the ability to archive or dehydrate older restore points to object storage, known as move mode. This functionality is powered by the Scale Out Backup Repository (SOBR) construct in VBR, defining performance tier (on-prem block storage) and cloud or archive tier (object storage). With this support cloud tier can only support a single bucket per SOBR which can cause scaling issues.
  
• 10.0 (2020): Copy mode is added which allows that same SOBR to make an immediate copy of any restore point that hits the performance tier to be immediately copied to the cloud tier. This is commonly seen as the capability that best competes with the VCC-B functionality. Support for Immutability and mounting backups regarding cloud tier happens in this release as well.
• 11.0 (2021): Addition of support for Google Cloud Platform (GCP) Storage.
• 12.0  (2023): Host of improvements to object storage support.
  • Object in the performance tier with multiple buckets supported as extents.
  • Support for multiple buckets of the same type in the cloud tier.
  • Reconfiguration of the bucket folder structure to allow for optimized performance especially around API calls.

In all there has been a steady line of improvement and innovation towards object storage making it the first-class citizen it will be with v12’s release.

Verdict

All that begs the question, should I as a Veeam Backup & Replication administrator be migrating my offsite backups from Cloud Connect Backup to object storage? In my opinion, probably not. The compelling use case today regarding offsite backups is the same that’s been around since v10; if you are only sending a copy of backups offsite for the purpose of compliance, checking the box if you will, then object storage may be a good fit for you. It is exceptional in how it supports Object Lock (Immutability). Further it is more efficient than anything else to date in how it writes data which makes it well suited for on-premises storage if available, but for cloud copy usage the capabilities quickly fall off.

In the end my biggest reason why I would still choose VCC-B today is the restore scenarios. With Object Storage unless you are utilizing a VCSP like 11:11 Systems you aren’t going to be able to have the object storage close enough to a VMware native compute to facilitate timely restore. Your other options are the opposite ends of the spectrum; low cost providers who have no compute capabilities and may have extra charges that may apply to your organization depending on deletion.Data egress or API calls along with hyperscalers that do have compute adjacent but are a completely different architecture to vSphere are other concerns. These scenarios require conversion of backups to instances and rely on your IT staff having the skillset to securely create an environment for these workloads to run from.

When you consider the tiered approach to Disaster Recovery, pairing a broad spectrum copy of backups to VCC-B with Tier 0 or 1 targeted VCC-R you gain 2 things by sticking to VCC-B:

1. Support for seeding replicas from VCC Backups
2. A defined DRaaS environment pre-created for your organization that in the event of on-prem not being available that you can boot replicas quickly and less quickly begin restoring backups into.

While not necessarily as important as restorability one other major difference to be aware of is the ability to control how much of your on-prem backups you keep off site. Keep in mind that copy mode works on the principle that you want to keep everything you have on-site in an offsite location, so if you want to have 30 days on prem and 7,14,60, or 90 days offsite there really isn’t a path besides Grandfather-Father-Son (GFS) to get there with just copy mode. Further it’s going to be everything that targets the given repository, if you have VMs you don’t necessarily want to keep off prem (think of like a door control system that does you no good anywhere but in the physical location) your only option is to create separate repositories and jobs. With VCC-B and backup copy all these things can be done in a very granular manner.

Finally, you have the issue of expertise. While many of us might be conversant in object storage and how it works you may not necessarily know the ins and outs of bucket policy, ACLs, versioning, etc. It’s a technology you do need to know but when it comes to backups and especially the need to restore them in anger using Veeam Cloud Connect not only gets you an easy path to restore. VCSP employs experts in Veeam software and access to this expertise with best practices for Veeam backups and replicas can be critical, ensuring a more seamless restoration of services when the pressure is already on.

Conclusion

While the promise and the capabilities of object storage are exciting as of the v12 release Veeam Cloud Connect Backup offers many more options when it comes to the restore and management of your backup data and for that reason should still be the go-to for offsite backup copies. If you are wanting to begin leveraging the technology an on-premises object storage platform for your local backups is a far better solution and can give you an out of the box upgrade to what you are using today.

In the time of ransomware and and cybersecurity attacks being in the news almost daily things like our disaster recovery applications have to start taking those protections very seriously. Often the easiest rung to reach in terms of good application security is having Multi-Factor Authentication (MFA) required for logins. For most MFA is old hat but for those unfamiliar modern MFA can take various forms including:

• SMS One Time Passcode (OTP): codes that are pushed from the website to you via SMS text messages. Better than nothing but easily spoofable.
• OTP Apps: apps such as Google Authenticator, Authy or 1password that provide the codes based on scanning a QR code typically to setup then using your mobile device to generate the code
• Push MFA: apps such as Duo or Gmail that when you authenticate it triggers a push notification to a particular instance of a mobile app that is also authenticated

With Veeam’s upcoming v12 release of Veeam Backup and Replication they are now supporting OTP App MFA in their console application. This takes a little bit of setup and in my particular case is a little quirky to setup but it does definitely work. Let’s walk through how to put this in effect.

Enable MFA in VBR v12

1. Once you have v12 Console installed, either the local or remote version, open it and navigate to Users and Roles

2. MFA in VBR is not supported with any use of groups. By default members of the local Adminstrators group are given the Veeam Backup Administrator role so we need to start by adding your own login first. You can do this with the Add… button.

If you weren’t already aware you’ll notice that there are number of roles that can be selected. More information about what each of these can do is available in the helpcenter documentation.

Bug: One other thing I’ll add here is that I noticed a quirk in some of our setups. If you use the browse method and if for some reason the NetBIOS domain is not just the portion before the first dot in the FQDN domain (for example prem for prem.lab.internal) then by default the User or group in the blank above is filled in incorrectly. In a few of our test environments we use the full FQDN without the dots as the NetBIOS name and it led to some issues that were easily fixed by just typing the correct domain in the blank. This feedback has been shared and I imagine it will be fixed before GA.

2a. If by chance you do not remove the groups before enabling MFA and hitting OK you will be presented with an error. Again, simply remove the groups to get past this.

3. It is important to understand that once you protect an account with MFA you will not be able to use it with automation methods such as PowerShell. To allow for this you will need to create an automation specific user, preferably with a very robust and often changed password, and set it in VBR as a service account to disable MFA. Complete adding your needed users and check the “Require two-factor authentication for interactive logon” to complete the server setup portion.

4. Once you setup your accounts in Veeam Console you will need to close out and relaunch to get into the MFA registration wizard. You would do this as you normally would, either using Windows session authentication or filling in the username and password.

5. Once you hit Connect you will get the standard MFA QR code for registration. Simply open the app of your choice and add by scanning the QR code then supply an active confirmation code to complete setup.

And with that you have now enabled 2 factor authentication for your Veeam Backup and Replication Users. You can potentially increase this further by not giving permissions to the user’s Windows logon account but instead doing a secondary, application specific account making them type in a username and password in step 4 above. With that you would have to authenticate to Windows, authenticate to Veeam and then provide an MFA code. That would all just depend on your organization’s security needs.

Conclusion

At the end of the day security practices should be a part of everything we do as IT and Disaster Recovery administrators. Little things like requiring MFA for our critical backups add up to a well design, layered security model.

As we come upon the end of the year, we are also getting closer and closer to Veeam’s latest release of their flagship product, Veeam Backup & Replication v12. Here at 11:11 Systems we have been testing quite a bit with the beta code as it has been released because there are quite a few features to really be excited about here. For me personally the biggest is being able to use object storage as a first hit repository and to be able to scale them out.

If you are not yet familiar with object storage now is the time. Object storage provides a good deal of native capabilities we’ve never had when using block-based storage (like volumes and partitions). These include:

• Immutability
• Versioning
• Consumption based pricing model

What this means to your backup plans is that you will have more capabilities, while being more agile in pricing.

Flavors of Object Storage

There are several ways to purchase and consume object storage. The hyperscale cloud names you are probably most familiar with: Amazon’s AWS (Amazon Web Services) S3 and Microsoft’s Azure Blob. But these providers can have several potentially unexpected costs around data egress, API call limits, and in some cases how long you store the data. As we all know with IT and disaster recovery, a good deal of our decisions come down to cost and budgeting, so having unexpected bursts can be a challenge.

With 11:11 Object Storage, you get access to a robust, performant object storage service that while competitively priced is flat rated with nothing hidden. Our service and others that use the AWS S3 API structure are commonly called S3 compatible. This means that they support the same calls and automation capabilities as S3 but may have differing architecture.

Finally, there are also a few physical device manufacturers such as Pure Storage, Cloudian and Object First that will sell you arrays for on-premises object storage. With the 3-2-1 backup model it may be worth considering an on-prem option for your first hit of backups before copying to the cloud.

Optimizing for Performance

One consideration as we begin to look at object storage for backup data is that depending on the platform the service is built on, there may be performance considerations as buckets scale in number of objects. In previous versions of VBR only a single bucket was supported for the capacity or archive tier. In this latest version, multiple buckets will be supported per tier, a move that is anticipated to enhance performance.  At 11:11 Systems,  we have put a great deal of work into overcoming limitations both in object storage itself but also how Veeam software integrates with it to the point that we consider it as performant if not more so than most other offerings in this space. This includes taking an active role in the open source Ceph project that powers our service.

As I began working on testing this capability, I realized that I was spending a lot of time creating buckets, creating repositories, and then ultimately creating SOBRs full of repositories. In speaking with my friend Joe Houghes we realized we were both doing the same things and so we collaborated on scripting this capability, the final product of which can be found in this GitHub repository. If you do choose to use this, please provide feedback through GitHub.

While the code is great, the takeaway you should consider here is that Object Storage as the repository portion of your VBR Infrastructure is coming. With that great capability comes great consideration and you should look to work with experts to assist you with these changes. You should also begin to educate yourself on the key capabilities and differences.

In the past few years Immutability, the concept of making data unable to be deleted or modified, has been a core tenant of disaster recovery design and has driven the rise of Object Storage for backup purposes. To this point in the Veeam ecosphere immutability has been contained to traditional Infrastructure backups, virtual machines, and cloud native instances, but there has been a great appetite for this capability to apply to SaaS backups as well.

With the upcoming v7 release of Veeam’s Backup for Microsoft365 (VB365) the product now has a method of allowing true, verifiable immutability or object-lock on backup sets. In their interpretation of how to make this capability happen Veeam has incorporated a long known best practice for all things backup, adhering to the 3-2-1 backup methodology where there are 2 copies of the backup. This happens because immutability can only occur on the secondary copy of the backup set, leaving the primary set on unlocked, performant object storage to ensure that recovery operations are capabilities in a optimized manner.

So What’s Changed?

Starting with VB365 v6 support for backup copies have been available but only in very limited situations. First the backups must land in Azure Blob or Amazon S3 for the first copy, second the copies could only go to related secondary locations. For example, backups residing in AWS S3 could only be copied to S3, S3 Infrequent Access or AWS Glacier storage. Further while these backups could be encrypted, they could not have the object-lock attribute, the object storage attribute that powers immutability, set.

With VB365 v7 both statements have changed. First backup copies can now land on Azure, S3 or any S3 compatible backup storage that supports the necessary functions of the S3 API. Further those backups can now optionally have the object-lock attribute set at the object repository level, with the immutability expiration automatically set to the duration of the retention period. In other words, unlike Veeam Backup and Replication where retention points are set at the job level and immutability is set at the repository level, with VB365 both are set together at the repository level.

VB365 Immutability in Action

1. Add primary object storage and backup repository. On this repository is when you want to set your long-term retention as needed.  Repeat the process until you have repositories to support your organization’s backup needs in accordance with best practices for maximum supported objects per repository.

2. Add your secondary object storage and VBO repositories. These should arguably be setup in a different object storage environment as your primary copy but still well connected. Because you will be writing data that is set to object-lock for the retention period of the repository you should consider the shortest retention period here in accordance with your organization’s data policy and standards.

3. Create backup jobs targeting your primary backup repositories. These will automatically copy to the secondary repository and become immutable as it is written.

Conclusion

This capability is going to be a game changer for everyone who uses VB365 to protect their M365 workloads. While this capability is only supported for VB365 data residing on object storage there’s an argument to be made that if you haven’t already moved or reseeded that data to object by now you should be working on it. Customers will be able to maintain a secondary copy of their backup sets on disparate systems to ensure the durability of their backups. If you can’t tell I’m actually very excited by this capability and look forward to using it in production!

Unless you’ve had your head stuck in the sand you probably have heard at this point that Microsoft has been trying numerous times to kill the Basic Authentication method for application integration into M365. While they tried and then delayed quite a few times the latest date of October 1, 2022 seems to be sticking so you do need to prepare for this. 

As someone who fortunately/unfortunately has to think far too much about data protection of things like M365 this is something that has been a big part of my day to day conversations as I tend to work a good deal with this in regards to Veeam’s Backup for Microsoft365 product. 

Let’s start with the good news: Working with what  in Veeam vocabulary is “Modern Only” authentication is probably the easiest and most robust way to let  Veeam organization configuration interact with your Microsoft365 organization. No more need to manually create and register Azure applications, no need to go through a laundry list of Exchange, Sharepoint and Graph API permissions, all of that is automatically created and managed through the registration process. Further modern application authentication also allows for MFA authentication at registration with any administrator level account in your organization without Veeam ever needing to record or manage those accounts. So all that is great but there are some drawbacks as well.

Unfortunately there are still a few things where Microsoft hasn’t achieved feature parity with the old method of authentication, most notably support for protecting Exchange Public Folders. This and other limitations of the API only method are outlined in Veeam’s handy KB3146.

In my day job at iland cloud we’ve been able to extend support for Modern Authentication to our own console that leverages the Veeam VB365 APIs. It’s relatively easy to and in short you

1. Choose to Update MS Credentials from the Action menu of your M365 Organization in the iland console.
2. Choose Modern Application Authentication Only, select the data types you wish to protect and then provide a name for your application that will be created in Azure Active Directory.
3. Login to Microsoft’s device authentication portal using your M365 administrator credentials and the provided device code.
4. Confirm that you are authenticating with Azure CLI.
5. Close the device authentication portal window when prompted and hit submit in the iland console.

This isn’t some magical black box, messing with your organization’s security without any ability to see what is happening. If you navigate to Azure Active Directory and Enterprise Applications you will find your created application there and you can review and document as needed any permissions and roles added. Understand that future builds/versions of Veeam Backup for M365 may require new permissions (the new Graph Teams Export API coming in 6a immediately comes to mind) so this will not be a completely static list but you will need to authenticate to the application any time changes will need to be made, we have no ability or rights to do this on our own.

And that’s it! Once you authenticate the created application’s token is saved to the underlying VB365 server and your jobs will continue working as they always have. This will also be the methodology for any restores you need to perform to M365 going forward, giving you an additional level of security in that any restores will require not only a user with administrative rights but also have Multi-Factor Authentication enabled as well. Enjoy and happy data protecting!

Microsoft has recently created documentation about changes coming to the Microsoft Graph API relating to Teams chat. In short the current method of backing up Teams chat will be decommissioned and it will be replaced by what is called the Teams Export API. This API will have fees associated with its use and those fees will be assessed directly by Microsoft against the M365 account in a burst method. 

In response to that Veeam has released guidance as to how this will be handled with their Veeam Backup for Microsoft365 product on which our Secure Backup for Microsoft365 product is based. This will take effect on the Veeam side from version 6a forward. It is anticipated that 6a will be released July/August 2022 time frame. It is unknown at this time when iland will be installing this to our VB365 environments.

While I’m writing this post with Veeam Backup for Microsoft365 in mind because they’ve released KB 4322, it’s important to note that this isn’t just a problem for Veeam, any vendor doing chat level backup for M365 is going to be running into this.

Costs

Microsoft has outlined a number of models in their document for how costs will be incurred for consuming this API. Due to Microsoft’s not considering data protection a Security & Compliance use case Veeam and any other M365 backup vendor will need to use the “Model B” as outlined here. This model provides access to backup channel based chat messages across the platform. Veeam does not at this time support backup of 1:1 or 1:many direct messages so this use case is out of scope.

When you get down to it what any Microsoft 365 customer are going to care about is cost. Once your organization begins being processed by the new API you charges will be paid through your organization’s Microsoft account on a per message rate directly to Microsoft. It is known that the cost per message will be $0.00075 per message and it is anticipated at this time that messages being sent to a channel will be considered “Group” data and as such will only incur cost on a per message basis. If at some point in the future Veeam supports protecting direct 1:1 and 1:many chats then these will be charged as per message x # of users who receive it.

So, When?!?

Tentatively I’m hearing that this should take full effect and the cutoff of the old method will begin on October 1, 2022 just like Basic Authentication depreciation. As for when costs will begin being incurred for the “Model B” API hits starting on July 5, 2022. At some point between those 2 VB365 6a will be released.

It is worth noting that regardless of if you have opted in or not Veeam will not be able to enable the Teams Chat option for you upon upgrade of 6a because it requires new permissions. They will notify you upon install if it detects you were backing up Teams Chat before so that you can investigate what you need to do.

Microsoft 365 Customer Responsibility

With this new model Customers will be required to “opt-in” as described in Veeam KB 4322 by requesting access to the Graph Export API. It is anticipated that from the beginning of requesting access to the API and then Microsoft granting it may take take up to 2 weeks.

Further once this process has been completed customers will need to access their organization in VB365 and re-authenticate to the organization to enable Teams Chat support and grant the new permissions necessary. As Microsoft is also beginning to depreciate the Basic Authentication method in the same time frame you may have to do this anyway.

Finally as mentioned above once and if a customer chooses to backup Teams chat the customer will need to access their jobs and select to backup Chats on a per job, per channel basis. Any existing jobs prior to the upgrade will have this new, granular Chats option available per channel but disabled, having the effect of turning off Teams Chat backup. This option was not available before as Chat was just a portion of the overall service. This is actually a positive in that they will have more granular ability to choose when and where to enable chat backup to minimize costs.

Conclusion

In the end this stinks horribly and Microsoft has made some changes that I’m sure are going to have their customers in a bind and thinking about options. Beginning with not considering data protection to be a Security and Compliance need (thus excluding partners from considering the Model A option) and carrying through the “I’ve got to go fill out an Office Forms doc and in a couple of weeks somebody will grant me access” workflow all of this screams “I don’t really care about you.” For those of us in a Service Provider or large enterprise space this hasn’t exactly been a long runway of potentially large scale impact on the cost of our services.