As we continue our journey into exploring new products and features that Veeam has released for 2023 it’s well worth the time to look at what Veeam Backup for AWS (VBA) is looking like these days. If you haven’t been using this software yet with this version 6 release it has deepened its relationships with other Veeam products, namely Veeam Cloud Connect powered by Backup and Replication (VBR) and Veeam Service Provider Console (VSPC). This allows VCSPs to supply a true managed BaaS service without the customer needing their own VBR and/or VBA components.
In the first releases of Veeam Backup for AWS installation and licensing were exclusively done via the Amazon AWS Marketplace; customer goes and looks for it, installs it and then pays AWS a fixed fee for usage, all the while being required to manage their own backups. The more modern releases have allowed integration into a customer installed Veeam Backup and Replication, allowing for deployment and management via VBR’s console and allowing it to inherit Veeam Universal Licenses (VUL) via that installation. That is much better but still requires customer own and manage their backups and licensing for the most part, something that I’ve heard from numerous customers and potential customers that are fully into hyperscale cloud for their infrastructure they are hesitant to do.
Now that this combination of releases has come out, VBR v12, VSPC v7 and VBA v6, we have the ability to do something more like the common Shared Responsibility Model that cloud first customers are accustomed to; the customer owns the production workloads and data while the service provider owns the infrastructure that powers the backup service, with both capable of managing the backup jobs and copies as needed.
With each release not only has Veeam added to the deployment and management methods they have also added features. With this most recent release when coupled with the aaS model you have the capability to backup workloads through the following AWS services:
What Can It Do
EC2 compute instances
RDS databases
EFS file systems
VPC configuration
All of these backups will land into immutable, encrypted, and secured AWS S3 storage with the capability to provide a second equally secured copy into a separate region and/or a separate AWS account. Further as part of your data protection policy creation you can also schedule AWS snapshots for the local availability zone (AZ) with replicas of snapshots then able to be sent to different regions and/or accounts. This is similar (but not quite) like VM replication. These replicas then show in inventory in the DR AZ and can be failed over to as needed. All in all you can protect the majority of services that most “cloud first” customers need and be able to backup, copy or replicate to a very promising number of locations.
How Does This Thing Work
For those who are used to how Veeam does backup job creation for traditional workloads this is quite a bit different in that it is policy based. Either you the customer or 11:11 Systems for a managed service create policy definitions for your workloads via Console. These policies define things like where to land backups, how copies should be handled, if snapshots are created and then replicated and schedules. Those policies can then be applied to workloads that are discovered via an AWS key pair that you create with defined IAM policies and then add to Veeam Backup for AWS via Console. Monitoring and management then also be provided by Console so in the end you have a modern, Shared Responsibility Model style BaaS capability with a single UI for the customer.
Conclusion
Backup for public cloud workloads is that thing that we as IT professionals truly need to care about. These are still running workloads that are critical to your business and just as we’d all shudder at thinking “snapshots are backups” in traditional virtualized environments you shouldn’t exclusively rely on that in the cloud. The combination of Veeam Backup for AWS, as well as Azure and GCP, and service provider style services provided by 11:11 Systems will allow customers to in effect have their clouds and back them up too.
I feel like I could write, produce, direct and star in a saga based in a galaxy far, far away with the amount of time I have spent talking about the combination of Microsoft365, backup applications that support it such as Veeam Backup for Microsoft365, and the recent Microsoft Teams Graph Export API. Yet here I am again because Microsoft has decided to change course with how this capability will be handled without telling anyone.
Recently I started to hear of support cases where users were doing the needful by filling out Microsoft’s form and then after the prescribed 5-7 business days attempting to enable support for Teams Chat backup available in VB365 versions 6a and 7 only to be presented with the error “To call this API the app must be associated with an Azure subscription, see https://aka.ms/teams-api-payment-requirements for details.”
For those who have been dealing with the Teams Graph Export metered API it’s a telltale sign that for whatever reason the Azure subscription ID associated with the organization modern application used to do backups was not available. Upon looking at the form I noticed that it no longer requires a subscription ID meaning that, once again, Microsoft has changed the process and not told anyone, leaving backup vendors as well as their partners scrambling to make sure our customers are properly suited. This has been well covered at this point by Ian Sanderson in his blog “Protecting Microsoft Teams Channel Chat Data: Are You Prepared?”
Upon trying to enable this protection for myself and as a matter of testing I’ve found that while the instructions are correct when using Azure Cloudshell it doesn’t work if you’ve just installed the Azure CLI on a windows system. This is due to a bug mentioned deep in the recesses of github that essentially says that cloud shell is bash based so you have to use different escape sequences to make it work. Further the Microsoft guide leaves out the step of creating a resource group to bind your appId and Subscription ID together. For all of those reasons I felt it necessary to create a follow up to Ian’s post to show you how to enable all this via Azure CLI on a Windows Server.
How-To Time!
0. If you haven’t already install Azure CLI from the download or via Chocolatey “choco install -y azure-cli”
1. Login or have the account owner login to their Azure account with “az login –use-device-code”. That will have the effect of creating a code for you to login to Microsoft with much like when you register an organization with Veeam Backup for Microsoft365. Once processed it will output a list of your active subscriptions. The id field there should be your subscription ID you will need later.
2. Next you will need to register the Microsoft.GraphServices namespace as shown in the Microsoft document. This is general using “az provider register --namespace Microsoft.GraphServices”
3. Next you need to create a resource group to be the relationship will live within. This is done, assuming you want to create your resources in the Eastern US region of Azure, by using “az group create -l eastus -n myResourceGroup” If you need another region you can pull a list of those available with “az account list-locations“
4. Finally, we now must link the application ID (which you should have from the confirmation email from Microsoft) and the subscription ID from your login above and create a resource. This is where the formatting matters so simply select the all-caps text below to replace az resource create --resource-group koolaidInfoBilling --name myResourceGroup --resource-type Microsoft.GraphServices/accounts --properties "{\`"appId\`": \`"INSERTAPPIDHERE\`"}" --location Global --subscription “INSERTSUBSCRIPTIONIDHERE”
That should return a block of JSON that looks like this:
Once that provisioningState shows as succeeded you should be able to go run your Teams Chat enabled VB365 job again and find some success, at least until Microsoft decides to mess with it again. I will add personally that they are truly doing a disservice to their ecosystem; their partners don’t really seem to have a good idea of when/if changes are going to happen and those of us downstream, both Service Providers and Customers, are put into a perpetual state of trying to catch up to poorly considered implementations.
Unless you’ve had your head stuck in the sand you probably have heard at this point that Microsoft has been trying numerous times to kill the Basic Authentication method for application integration into M365. While they tried and then delayed quite a few times the latest date of October 1, 2022 seems to be sticking so you do need to prepare for this.
As someone who fortunately/unfortunately has to think far too much about data protection of things like M365 this is something that has been a big part of my day to day conversations as I tend to work a good deal with this in regards to Veeam’s Backup for Microsoft365 product.
Let’s start with the good news: Working with what in Veeam vocabulary is “Modern Only” authentication is probably the easiest and most robust way to let Veeam organization configuration interact with your Microsoft365 organization. No more need to manually create and register Azure applications, no need to go through a laundry list of Exchange, Sharepoint and Graph API permissions, all of that is automatically created and managed through the registration process. Further modern application authentication also allows for MFA authentication at registration with any administrator level account in your organization without Veeam ever needing to record or manage those accounts. So all that is great but there are some drawbacks as well.
Unfortunately there are still a few things where Microsoft hasn’t achieved feature parity with the old method of authentication, most notably support for protecting Exchange Public Folders. This and other limitations of the API only method are outlined in Veeam’s handy KB3146.
In my day job at iland cloud we’ve been able to extend support for Modern Authentication to our own console that leverages the Veeam VB365 APIs. It’s relatively easy to and in short you
Choose to Update MS Credentials from the Action menu of your M365 Organization in the iland console.
Choose Modern Application Authentication Only, select the data types you wish to protect and then provide a name for your application that will be created in Azure Active Directory.
Confirm that you are authenticating with Azure CLI.
Close the device authentication portal window when prompted and hit submit in the iland console.
Select Update MS Credentials from the Action menu of your M365 Organization in the iland console
Select that you want to use Modern Application Only, the data types you wish to protect, then finally supply a name you would like to use for the application you create in Azure Active Directory
Supply the code that is generated in the iland console screen to microsoft.com/devicelogin.
Confirm that you are attempting to authenticate via Azure CLI
Confirm authentication, close this window and then hit submit in the iland console.
This isn’t some magical black box, messing with your organization’s security without any ability to see what is happening. If you navigate to Azure Active Directory and Enterprise Applications you will find your created application there and you can review and document as needed any permissions and roles added. Understand that future builds/versions of Veeam Backup for M365 may require new permissions (the new Graph Teams Export API coming in 6a immediately comes to mind) so this will not be a completely static list but you will need to authenticate to the application any time changes will need to be made, we have no ability or rights to do this on our own.
And that’s it! Once you authenticate the created application’s token is saved to the underlying VB365 server and your jobs will continue working as they always have. This will also be the methodology for any restores you need to perform to M365 going forward, giving you an additional level of security in that any restores will require not only a user with administrative rights but also have Multi-Factor Authentication enabled as well. Enjoy and happy data protecting!
For those that haven’t been playing in the RHEL/CentOS/Fedora space for a while Red Hat finally got around to making CentOS a more “productized” Linux Distribution and replaced it with CentOS Stream. That then spawned a fewforked distributions to fill the gap, most notably Rocky Linux which recently released their version 9 to track the current cycle of RHEL.
Aside from the name pretty much everything in Rocky looks and feels the same as what you are used to with CentOS at this point, but because it is such a newly named Operating System VMware hasn’t really known how to deal with it until the 7.0u3c release. As you may be aware the 7.0u3 releases have had a bit of troubling history so many organizations have chosen to stay on the 7.0u2 for now so upgrading to get named release isn’t really going to work. This is quite the problem for those of us who are used to using the combination of VMware templates and Custom Specifications to rapidly deploy virtual machine instances.
Luckily there are a couple of options that VMware has finally documented in KB59557. The first of which is kind of janky, edit the /etc/redhat-release file and change the word “Rocky” to “CentOS.” Sure this works but anytime you run a system upgrade it’s going to change it back. The better option is to install and configure Canonical’s cloud-init. If you haven’t looked at cloud-init yet it’s definitely time as it allows you to be a bit more distribution agnostic in how you automatically provision virtual machines and cloud instances. As it is from the same maker Ubuntu has been supporting it and installs it by default for some time now so having support here is just nice.
So first thing’s first when you go to build your template source VM set the distribution to CentOS 8. This is as close as you are going to get until you upgrade to 7.0u3c or later. You then can build out your VM template as you normally would. Once the system is up you simply need to install cloud-init.
sudo yum makecache --refresh
yum -y cloud-init
Once installed it’s worth checking out /etc/cloud/cloud.cfg and ensure that the disable_vmware_customization parameter is set to false. This will allow VMware to utilize cloud-init if it is available.
Now that you have installed cloud-init you can shutdown the VM (after making sure the system is up to date of course) and convert it to a template. It is then just a matter of using your VMware Customization Specification under Policies and Profiles to supply your rules for how you will automate the deployment.
In my last post, Configuring Veeam Backup & Replication SOBR for Non-immutable Object Storage, I covered the basics of how to consume object storage in Veeam Backup & Replication (VBR). In general this is done through the concept of Scale-out Backup Repositories (SOBR). In this post we are going to build upon that land layer in object storage’s object-lock features which is commonly referred to in Veeam speak as Immutability.
First, let’s define immutability. What the backup/disaster recovery world thinks of as immutability is much like the old Write Once, Read Many (WORM) technology of the early 00’s, you can write to it but until it ages out it cannot be deleted or modified in any way. Veeam and other backup vendors definitely treat it this way but object-lock under the covers actually leverages the idea of versioning to make this happen. I can still delete an object through another client but the net effect is that a new version of that object is created with a delete marker attached to it. This means that if that were to occur you could simply restore to the previous version and it’s like it never happened.
With VBR once Immutability is enabled objects are written with Compliance mode retention for the duration of the protected period. It recognizes that the object bucket has object-lock enabled and then as it writes it each block is written with the policy directly rather than assuming a bucket level policy. If you attempt to delete a restore point that has not yet had its retention policy expire it won’t let you delete but instead gives you an error.
Setting up immutability with object storage in Veeam is the same as with non-immutability but with a few differences. This starts with how we create the bucket. In the last post we simply used the s3 mb command to create a bucket, but when you need to work with object-lock you need to use the s3api create-bucket command.
Once your bucket is created you will go about adding your backup repository as we’ve done previously but with one difference, when you get to the Bucket portion of the New Object Store Repository wizard you are going to check the box for “Make recent backups immutable” and set the number of days desired.
You now have an immutable object bucket that can be linked to a traditional repository in a SOBR. Once data is written (still the same modes) any item that is written is un-deleteable via the VBR server until the retention period expires. Finally if I examine any of the objects in the created bucket with the s3api get-object-retention command I can see that the object’s retention is set.
Veeam Backup & Replication (VBR) currently makes use of object storage through the concept of Scale-Out Backup Repositories, SOBR. A SOBR in VBR version 11 can contain any number of extents as the performance tier (made up of traditional repositories) and a single bucket for the capacity tier (object storage). The purpose of a SOBR from Veeam’s point of view is to allow for multiple on-premises repositories to be combined into a single logical repository to allow for large jobs to be supported and then be extended with cloud based object storage for further scalability and retention.
There are two general modes for object storage to be configured in a SOBR:
Copy Mode- any and all data that is written by Veeam to the performance tier extents will be copied to the object storage bucket
Move Mode- Only restore points that are aged out of a defined window will be evacuated to object storage or as a failure safeguard only when the performance tier extents reach a used capacity threshold. With Archive mode within the Veeam UI the restore points all still appear as being local but the local files only contain metadata that points to where the data chunks reside in the bucket. The process of this occurring in Veeam is referred to as dehydration.
In this post let’s demonstrate how to create the necessary buckets, how to create a SOBRs for both Copy and Move modes without object-lock (Immutability) enabled. If you haven’t read my previous post about how to configure aws cli to be used for object storage you may want to check that out first.
1. Create buckets that will back our Copy and Move mode SOBRs. In this example I am using AWS CLI with the s3 endpoint to make the buckets.
2. Now access your VBR server and start with adding the access key pair provided for the customer. You do this in Menu > Manage Cloud Credentials.
3. Click on Backup Infrastructure then right click on Backup Repositories, selecting Add Backup Repository.
4. Select Object Storage as type.
5. Select S3 Compatible as object storage type
6. Provide a name for your object repository and hit next.
7. For the Account Settings screen enter the endpoint, region and select your created credentials.
8. In the Bucket settings window click Browse and select your created bucket then click the Browse button beside the Folder blank and create a subfolder within your bucket with the “New Folder…” button. I’ll note here do NOT check the box for “Make recent backups immutable for…” here as the bucket we have created above does not support object-lock. Doing so will cause an error.
9. Click Apply.
10. Create or select from existing traditional, Direct Storage repository or repositories to be used in your SOBR. Note: You cannot choose the repository that your Configuration Backups are targeting.
11. Right click on Scale-out Backup Repositories and select “Add Scale-out Backup Repository…”
12. Name your new SOBR.
13. Click Add in your Performance Tier screen and select your repository or repositories desired. Hit Ok and then Next.
14. Leave Data Locality selected as the Placement Policy for most scenarios.
15. In the Capacity Tier section check to Extend capacity with object storage and then select the desired bucket.
16. (Optional but highly recommended): Check the Encrypt data uploaded to object storage and create an encryption password. Hit Apply.
17. This will have the effect of creating an exact copy of any backup jobs that target the SOBR both on premises and into the object store. To leverage Move mode rather than Copy you simply check the other box instead/in addition to and set the number of days you would like to keep on premises.
Now you simply need to target a job at your new SOBR to have it start working.
In conclusion let’s cover a bit about how we are going to see our data get written to object storage. For the Copy mode example it should start writing to data to the object store immediately upon completion of each run. In the case of leveraging Move mode you will see objects written after the run after the day specified to move archive. For example if you set it to move anything older than 7 days on-prem dehydration will occur after the run on day 8. These operations can be seen in the Storage Management section of the History tab in the VBR console.
Further if I recursively list my bucket via command line I can see lots of data now, data is good. 😉
% aws --endpoint=https://us-central-1a.object.ilandcloud.com --profile=premlab s3 ls s3://premlab-sobr-unlocked-copy/ --recursive
In my last post I worked through quite a few things I’ve learned recently about interacting with S3 Compatible storage via the CLI. Now that we know how to do all that fun stuff it’s time to put it into action with a significant Service Provider/Disaster Recovery slant. Starting with this post I’m going to highlight how to get started with some common use cases of object storage in Backup/DR scenarios. In this we’re going to look at a fairly mature use case, with it backing Veeam Backup for Office (now Microsoft) 365.
Veeam Backup for Microsoft 365 v6, which was recently showcased at Cloud Field Day 12, has been leveraging object as a way to make it’s storage consumption more manageable since version 4. Object also provides a couple more advantages in relation to VBM, namely an increase in data compression as well as a method to enable encryption of the data. With the upcoming v6 release they will also support the offload of backups to AWS Glacier for a secondary copy of this data.
VBM exposes its use of object storage under the Object Storage Repositories section of Backup Infrastructure but it consumes it as a step of the Backup Repository configuration itself, which is nested within a given Backup Proxy. I personally like to at a minimum start with scaling out repositories by workload (Exchange, OneDrive, Sharepoint, and Teams) as each data type has a different footprint. When you really need to scale out VBM, say anything north of 5000 users in a single organization, you will want to use that a starting point for how you break down and customize the proxy servers.
Let’s start by going to the backup proxy server, in this case the VBM server itself, and create folder structure for our desired Backup Repositories.
Now that we have folders let’s go create some corresponding buckets to back them. We’ll do this via the AWS S3 CLI as I showed in my last post. At this point VBM does not support advanced object features such as Immutability so no need to be fancy and use the s3api, but I just prefer the command structure.
Ok so now we have folder and buckets, time to hop in to Veeam. First we need to add our object credentials to the server. This is a simple setup and most likely you will only need one set of credentials for all your buckets. Because in this example I will be consuming iland Secure Cloud Object Storage I need to choose the “S3 Compatible access key” under the “Add…” button in Cloud Credential Manager (menu> Cloud Credentials). These should be the access key and secret provided to you by your service provider.
Now we need to go to Backup Infrastructure > Object Storage Repositories to add our various buckets. Start by right clicking and choose “Add Object Storage.”
1. Name your Object Repository2. Select S3 Compatible option3. Enter your endpoint URL, region, and select credentials4. Select your bucket from the dropdown menu5. Create a folder inside of your bucket for this repository’s data and hit finish
Now simply repeat the process above for any and all buckets you need for this task.
Now that we have all our object buckets added we need to pair these up with our on premises repository folders. It’s worth noting that the on-prem repo is a bit misleading, no backup data as long as you use the defaults will ever live locally in that repository. Rather it will hold a metadata file in the form of a single jetDB file that service as pointers to the objects that is the actual data. For this reason the storage consumption here is really really low and shouldn’t be part of your design constraints.
Under Backup Infrastructure > Backup Repositories we’re going to click “Add Repository..” and let the wizard guide us.
1. Name our repository2. Specify the hosting proxy server and the path to the folder you wish to use.3. If you don’t already have one created you can add an encryption secret to encrypt the data when specifying your object repository4. Specify the object storage repository and the encryption key to use. 5. Specify the retention period, retention level and hit finish.
One note on that final step above. Often organization will take the “Keep Forever” option that is allowed here and I will say I highly advise against this. You should specify a retention policy that is agreed upon with your business/organization stakeholders as keeping any backup data longer than needed may have unintended consequences should a legal situation arise; data the organization believes to be long since gone is now discoverable through these backups.
Also worth noting item-level retention is great if you are using a service provider that does not charge you on egress fees because it gives you more granular control in terms of retention. If you use a hyperscaler such as Amazon S3 you may find this option will drive your AWS bill up because of a much higher load on egress each time the job runs.
Once you’ve got one added again, rinse and repeat for any other repositories you need to add.
Finally the only step left to do is create jobs targeting our newly created repositories. This is going to have way more variables based on your organization size, retention needs, and other factors than I can truly do justice in the space of this blog post but I will show how to create a simple, entire organization, single workload job.
You can start the process under Organizations > Your Organization > Add to backup job…
1. Name your backup job2. Select Organization as your source3. Check the box for your desired organizaton4. Select your organization and click the edit button, allowing you to deselect all the workloads not in this job.5. Once edited you’ll see just the workload you want for your organization before hitting next6. I don’t have any exclusions for this job but you may have this need7. Select your desired proxy server and backup repository8. Finally do any customization needed to the run schedule.
Once again you’d want to repeat the above steps for all your different workload types but that’s it! If we do a s3 ls on our s3://premlab-ilandproduct-vbm365-exch/Veeam/Backup365/ilandproduct-vbm365-exch/ full path we’ll see a full folder structure where it’s working with the backup data, proving that we’re doing what we tried to do!
In conclusion I went way into depth of what is needed here but in practice it isn’t that difficult considering the benefits you gain by using object storage for Veeam Backup for Microsoft365. These benefits include large scale storage, encryption and better data compression. Hope you find this helpful and check back soon for more!
Recently a good portion of my day job has been focused on learning and providing support for s3 compatible object storage? What is s3 compatible you say? So while Amazon’s AWS may have created the s3 platform at its root today is an open framework of API calls and commands known as s3. While AWS s3 and its many iterations are the 5000 pound gorilla in the room many other organizations have created either competing cloud services or storage systems that can let you leverage the technology in your own environments.
So why am I focusing on this you may ask? Today we are seeing more and more enterprise/cloud technologies have reliance on object storage. Any time you even think of mentioning Kubernetes you are going to be consuming object. In the Disaster Recovery landscape we’ve had the capability for a few years now to provide our archive or secondary copies of data to object “buckets” as it is both traditionally cheaper than other cloud based file systems and provided a much larger feature set. With their upcoming v12 release Veeam is going to be providing the first iteration of their Backup & Replication product that can write directly to object storage with no need to have the first repository be a Windows or Linux file system.
To specifically focus on the VBR v12 use case many customers are going to choose to start dipping their toes into the idea of on-prem s3 compatible object storage. This can be as full featured as a Cloudian physical appliance or as open and flexible as a minIO or ceph based architecture. The point being that as Veeam and other enterprise tech’s needs for object storage matures your systems will be growing out of the decisions you make today so it’s a good time to start learning about the technology and how to do the basics of management from an agnostic point of view.
So please excuse the long windedness of post as I dive into the whys and the hows of s3 compatible object storage.
Why Object Then?
Before we go further it’s worth taking a minute to talk about the reasons why these technologies are looking to object storage over the traditional block (NTFS, ReFS, XFS, etc) options. Probably first and foremost it is designed to be a scale-out architecture. With block storage while you can do things like creating RAID arrays to allow you to join multiple disks you aren’t really going to make a RAID across multiple servers. So for the use case of backup rather than be limited by the idea of a single server or have to use external constructs such as Veeam’s SOBR to allow those to be stitched together, you can target an object storage gateway that then write to a much more scalable, much more tunable, infrastructure of storage servers underneath.
Beyond the scale-out you have a vast feature set. Things that we use every day such as file versioning, security ACLs, least privilege design and the concept of immutability are extremely important in designing a secure storage system in today’s world and most object storage systems are going to be able to provide these capabilities. Beyond this we can look at capabilities such as multi-region synchronization as a way to ensure that our data is secure and highly available.
Connecting to S3 or S3 Compatible Storage
So regardless of whatever client you are using you are going to need 4 basic pieces of information to connect to the service at all.
Endpoint: This will be a internet style https or http URL that defines the address to the gateway that your client will connect to.
Region: This defines the datacenter location within the service provider’s system that you will be storing data in. For example the default for AWS s3 is us-east-1 but can be any number of other locations based on your geography needs and the provider.
Access Key: this is one half of the credential set you will need to consume your service and is mostly akin to a username or if you are used to consuming Office 365 like the AppID
Secret Key: this is the other half and is essentially the generated password for the access key.
Regardless of the service you will be consuming all of those parts. With things that have native AWS integration you may not be prompted necessarily for the endpoint but be assured it’s being used.
To get started at connecting to a service and creating basic, no frills buckets you can look at some basic GUI clients such as CyberDuck for Windows or MacOS or WinSCP for Windows. Decent primers for using these can be found here and here.
Installing and Configuring AWS CLI Client
If you’ve ever used AWS S3 to create a bucket before you are probably used to go to the console website and pointy, clicky create bucket, set attributes, upload files, etc. As we talk more and more about s3 compatible storage that UI may or may not be there and if it is there it may be wildly different than what you use at AWS because it’s a different interpretation of the protocol’s uses. What is consistent, and in some cases or situation may be your only option, is consuming s3 via CLI or the API.
Probably the easiest and most common client for consuming s3 via CLI is the AWS cli. This can easily be installed via the package manager of your choice, but for quick and easy access:
Windows via Chocolatey
choco install -y awscli
MacOS via Brew
brew install awscli
Once you have it installed you are going to need to interact with 2 local files in your .aws directory on your user profile, config and credentials. You can get these created by using the aws configure command. Further aws cli supports the concept of profiles so you can create multiple connections and accounts. To get started with this you would simply use aws configure --profile obj-testwhere obj-test is whatever name you want to use. This will then walk through prompting you for 3 of those 4 pieces of information, access key, secret key and default region. Just as an FYI this command impacts 2 files within your user profile, regardless of OS, ~/.aws/config and ~/.aws/credentials. These are worth reviewing after you configure to become familiar with the format and security implications.
Getting Started with CLI
Now that we’ve got our CLI installed and authentication configured let’s take a look a few basic commands that will help you get started. As a reference here are the living command references you will be using
Awesome! We’ve got our first bucket in our repository. That’s cool but I want my bucket to be able to leverage this object lock capability Jim keeps going on about. To do that you use the same command but add the –object-lock-enabled-for-bucket parameter.
So yeah, good to go there. Next let’s dive into that s3api list-buckets command seen in the previous screenshot. Listing buckets is a good example for understanding that when you access s3 or s3 compatible storage you are really talking about 2 things; s3 protocol and the s3api. For listing buckets you can use either:
aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3 ls
While these are similar it’s worth noting the return will not be the same. The ls command will return data much like what it would in a standard Linux shell while s3api list-buckets will return JSON formatted data by default.
Enough About Buckets, Give Me Data
So buckets are great but the are nothing without data inside them. Let’s get to work writing objects.
Again writing data, especially if you are familar with the *nix methods to s3 can be very similar. I can use s3 cp or mv to copy or mv data to my s3://test-bucket-2-locked/ bucket or any other I’ve created or between them.
Now that we’ve written a couple files let’s look at what we have. As you can see above once again you can do the same actions via both methods, it’s just the s3api way will consistently give you more information and more capability. Here’s what the api output would look like.
Take note of a few things here. While the s3 ls command gives more traditional file system output s3api refers to the objects with their entire “path” as the key. Essentially in object storage for our benefit it still has the concept of file and folder structure but it views each unique object as a single flat thing on the file system without a true tree. Key is also important because as we start to consider more advanced object storage capabilities such as object lock, encryption, etc. the key is often what you need to supply to complete the commands.
A Few Notes About Object Lock/Immutability
To round out this post let’s take a look at where we started as the why, immutability. Sure we’ve enabled object lock on a bucket before but what that really just does is enable versioning, it’s not enforcing anything. Before we get crazy with creating immutable objects its important to understand there are 2 modes of object lock:
Governance Mode – In Governance Mode users can write data and not be able to truly delete it as expected but there are roles and permissions that can be set (and are inherited by root) that will allow that to be overridden and allow data to be removed.
Compliance Mode – This is the more firm option where even the root account cannot remove data/versions and the retention period is hard set. Further once a retention date is set on a given object you cannot shorten it in any way, only extend it further.
Object Lock is actually done in one of two ways (or a mix of both); creating a policy and applying it to a bucket so that anything written to that bucket will assume that retention or by actually applying a retention period to an object itself either while writing the object or after the fact.
Let’s start with applying a basic policy to a bucket. In this situations for my test-bucket-2-locked bucket I’m going to enable compliance mode and then set retention to 21 days. A full breakdown of the formatting of the object-lock-configuration parameter and the options it provides can be found in the AWS documentation.
Cool, now to check that compliance we can simply use s3api get-object-lock-configuration instead against the bucket to check what we’ve done. I’ll note that for either the “put” above or the “get” below that there is no s3 endpoint equivalent, these are some of the more advanced features I’ve been going on about.
Ok, so we’ve applied a baseline policy of compliance and retention of 21 days to our bucket and confirmed that it’s set. Now let’s look at the objects within. You can view a particular object’s retention with the s3api get-object-retention command. As we are dealing with advanced features at the object level you will need to capture the key for an object to test. If you’ll remember we found those using the s3api list-objects command.
So as you can see we have both mode and retention date set on the individual object. What if we wanted this particular object to have a different retention period than the bucket itself? Let’s now use the s3api put-object-retention option to work and try to set that down to 14 days instead. While we use a general purpose number of days when creating the policy when we set object level retention it’s done by modifying the actual date stamp so we’ll simply pick a day 14 days from today.
Doh! Remember what we said about compliance mode? That you could make the retention shorter than what was previously set? We are running into that here and can see that it in fact works! Instead let’s try this again and set it to 22 days.
As you can see now not only did we not get an error but when you check your retention it is now showing for defined timestamp so it definitely worked.
This feels like a good time to note that object locking is not the same as deletion protection. If I create an object lock enabled bucket and upload some objects to it, setting the object retention flag with the right info along the way, I am still going to be able to use a basic delete command to delete that file. In fact if I use CyberDuck or WinSCP to connect to my test bucket I can right click on any object there and successfully choose delete. What is happening under the covers is that a new version of that object is spawned, one with the delete marker applied to it. For standard clients that will appear that the data is gone but in reality it’s still there, it just needs to be restored to the previous version. In practice most of the UIs you are going to use to consume s3 compatible storage such as Veeam or developed consoles will recognize what is going on under the covers and essentially “block” you from executing the delete, but feel secure that as long as you have object lock enabled and the data is written with a retention date the data has not actually gone away and can be recovered.
All of this is a somewhat long winded answer to the question “How S3 Object Lock works” which Amazon has thoughtfully well answered in this post. I recommend you give it a read.
Conclusion
In the end you are most likely NOT going to need to know via the command line how to do all the above steps. More likely you will be using some form of a UI, be it Veeam Backup and Replication, AWS console or that of your service provider, but it is very good to know how to do these things especially if you are considering on-premises Object Storage as we move into this next evolution of IT and BCDR. Learning and testing the above is a relatively low cost consideration as most object services are literally pennies per GB, possibly with the egress data charges depending on your provider (hey AWS…), but it’s money well spent to get a better understanding.
I received an email yesterday that the fast track program for VMCE 2021 is available now through December 21, 2021. So what is this? According to the e-mail and discussion with Rasmus Haslund of Veeam Fast Track is designed to be a self service resource to allow existing VMCEs v9, 2020 or people who took the v10 course to upgrade the certification to the latest version with access to study materials and a test voucher all for just a bit more than the voucher itself.
According to the email the program will provide you in total with the following:
The latest Veeam Availability Suite v11 Configuration and Management (VASCM) courseware
13 days access to VASCM Labs for practicing and exam preparation
VMCE 2021 Exam Specification Guide
Access to the ‘Haslund Knowledge Check’
Exam preparation videos
VMCE Exam Voucher to take the exam at Pearson Vue
I will share that in the past (I’ve been a VMCE on versions 8 and 9 and recently renewed to the 2020 release) I’ve sworn by Rasmus’ always excellent practice exams so their inclusion here is noteworthy. While they are included these seem to remain a community resource provided by Rasmus so the value is more in the course materials and the videos, but still worth calling out.
If you are not currently 2021 certified and wish to be able to do an upgrade in the future without taking a course you will need to do so to keep current versions. If you are a standard end customer it’s true, your certification never expires but if you are in the partner space like me you unfortunately have to always be within the past 2 versions. In any case this is a pretty good deal for a recertification prep package
To purchase the fast track package you will need to log into the website you’ve been given access to your Veeam training materials at in the past, veeam.lochoice.com and click on whatever is the latest version of the VMCE materials you have available. Once there you will see a “Buy VMCE Fast Track to v11” button. Once clicked it’s as simple as providing a credit card and you are off and running.
Dealing with Microsoft API permissions has long been one of the hard parts of working extensively with Veeam Backup for Office 365. They tend to change fairly often and with little notice, leaving Veeam and other backup service providers for Microsoft365 in the enviable spot of needing to chase these permissions with their applications.
Veeam recently released version 5 of their M365 backup product and while the requirements are mostly the same permissions wise as they were in version 4 I’ve found there are a couple that potentially v4 was able to work without having in place due to other legacy setups that now don’t work in v5. My specific issue results in any Exchange Online jobs returning with the following error:
Processing mailbox XXXX failed with error: The remote server returned an error: (401) Unauthorized.
Fixing this requires adding the “full_access_as_app” permission but getting to that point is a bit more complicated for those not familiar with azure AD apps. This post will show you how to get to your existing Veeam Backup for Office 365 Azure app registration and verify that all of the necessary permissions have been added.
Navigate to Azure Active Directory and then app registrations
Click on your current app registration used for Veeam Backup for O365
Select API permissions. Once there the desired state for Veeam Backup for Office 365 currently is
Microsoft Graph
Directory.Read.All
Group.Read.All
Sites.ReadWrite.All
TeamSettings.ReadWrite.All
Office 365 Exchange Online
full_access_as_app
Sharepoint
Sites.FullControll.All
User.Read.All
If you have performed the v5 upgrade and your backups are currently failing the bolded permissions above may likely be missing and you will need to add them. When done adding your API permissions window should look like this
To add permissions you will need to
click the “+ Add a permission” button
click the tile for the necessary API group (Graph, Office 365 Exchange Online)
* For the Office 365 Exchange Online direct access has been deprecated by Microsoft. In that case you will need to choose the “API my organization uses and search for “Office 365 Exchange” to find it. This looks like
Choose “Application permissions”
Check the box for the permission you are requesting and click “Add permissions”
After adding permissions you will need to click the “Grand admin consent for <your organization>” button to complete adding the permissions for your organization.
It is worth noting that for future reference the documentation for Veeam required permissions for Veeam Backup for Office 365 modern authentication with legacy protocols allowed is located at https://helpcenter.veeam.com/docs/vbo365/guide/ad_app_permissions_legacy.html?ver=50. As these permissions are changed by Microsoft from time to time it is worth notating this link.
Hi there and welcome to koolaid.info! My name is Jim Jones, a Geek of Many Hats living in West Virginia.
This site was created for the purpose of being a locker full of all the handy things I’ve learned over the years, know I’m going to need again and know I’ll forget. It’s morphed a bit over the years as all things do but still that’s the main purpose. If you’d like to know more about me check out any of the social links at the top left of the site, I’m pretty much an open book.
If you’ve found this page I hope you find it’s contents helpful. Finally, anything written here are solely my views and do not reflect those of my employer.
You must be logged in to post a comment.