Recently a good portion of my day job has been focused on learning and providing support for s3 compatible object storage? What is s3 compatible you say? So while Amazon’s AWS may have created the s3 platform at its root today is an open framework of API calls and commands known as s3. While AWS s3 and its many iterations are the 5000 pound gorilla in the room many other organizations have created either competing cloud services or storage systems that can let you leverage the technology in your own environments.

So why am I focusing on this you may ask? Today we are seeing more and more enterprise/cloud technologies have reliance on object storage. Any time you even think of mentioning Kubernetes you are going to be consuming object. In the Disaster Recovery landscape we’ve had the capability for a few years now to provide our archive or secondary copies of data to object “buckets” as it is both traditionally cheaper than other cloud based file systems and provided a much larger feature set. With their upcoming v12 release Veeam is going to be providing the first iteration of their Backup & Replication product that can write directly to object storage with no need to have the first repository be a Windows or Linux file system.

To specifically focus on the VBR v12 use case many customers are going to choose to start dipping their toes into the idea of on-prem s3 compatible object storage. This can be as full featured as a Cloudian physical appliance or as open and flexible as a minIO or ceph based architecture. The point being that as Veeam and other enterprise tech’s needs for object storage matures your systems will be growing out of the decisions you make today so it’s a good time to start learning about the technology and how to do the basics of management from an agnostic point of view.

So please excuse the long windedness of post as I dive into the whys and the hows of s3 compatible object storage.

Why Object Then?

Before we go further it’s worth taking a minute to talk about the reasons why these technologies are looking to object storage over the traditional block (NTFS, ReFS, XFS, etc) options. Probably first and foremost it is designed to be a scale-out architecture. With block storage while you can do things like creating RAID arrays to allow you to join multiple disks you aren’t really going to make a RAID across multiple servers. So for the use case of backup rather than be limited by the idea of a single server or have to use external constructs such as Veeam’s SOBR to allow those to be stitched together, you can target an object storage gateway that then write to a much more scalable, much more tunable, infrastructure of storage servers underneath.

Beyond the scale-out you have a vast feature set. Things that we use every day such as file versioning, security ACLs, least privilege design and the concept of immutability are extremely important in designing a secure storage system in today’s world and most object storage systems are going to be able to provide these capabilities. Beyond this we can look at capabilities such as multi-region synchronization as a way to ensure that our data is secure and highly available.

Connecting to S3 or S3 Compatible Storage

So regardless of whatever client you are using you are going to need 4 basic pieces of information to connect to the service at all.

• Endpoint: This will be a internet style https or http URL that defines the address to the gateway that your client will connect to.
• Region: This defines the datacenter location within the service provider’s system that you will be storing data in. For example the default for AWS s3 is us-east-1 but can be any number of other locations based on your geography needs and the provider.
• Access Key: this is one half of the credential set you will need to consume your service and is mostly akin to a username or if you are used to consuming Office 365 like the AppID
• Secret Key: this is the other half and is essentially the generated password for the access key.

Regardless of the service you will be consuming all of those parts. With things that have native AWS integration you may not be prompted necessarily for the endpoint but be assured it’s being used.

To get started at connecting to a service and creating basic, no frills buckets you can look at some basic GUI clients such as CyberDuck for Windows or MacOS or WinSCP for Windows. Decent primers for using these can be found here and here.

Installing and Configuring AWS CLI Client

If you’ve ever used AWS S3 to create a bucket before you are probably used to go to the console website and pointy, clicky create bucket, set attributes, upload files, etc. As we talk more and more about s3 compatible storage that UI may or may not be there and if it is there it may be wildly different than what you use at AWS because it’s a different interpretation of the protocol’s uses. What is consistent, and in some cases or situation may be your only option, is consuming s3 via CLI or the API.

Probably the easiest and most common client for consuming s3 via CLI is the AWS cli. This can easily be installed via the package manager of your choice, but for quick and easy access:

Windows via Chocolatey

choco install -y awscli

MacOS via Brew

brew install awscli

Once you have it installed you are going to need to interact with 2 local files in your .aws directory on your user profile, config and credentials. You can get these created by using the aws configure command. Further aws cli supports the concept of profiles so you can create multiple connections and accounts. To get started with this you would simply use aws configure --profile obj-test where obj-test is whatever name you want to use. This will then walk through prompting you for 3 of those 4 pieces of information, access key, secret key and default region. Just as an FYI this command impacts 2 files within your user profile, regardless of OS, ~/.aws/config and ~/.aws/credentials. These are worth reviewing after you configure to become familiar with the format and security implications.

Getting Started with CLI

Now that we’ve got our CLI installed and authentication configured let’s take a look a few basic commands that will help you get started. As a reference here are the living command references you will be using

• s3 endpoint reference s3 — AWS CLI Command Reference
• s3api endpoint reference: s3api — AWS CLI Command Reference

A good first step will be creating a basic bucket. You can do this leveraging the s3api create-bucket command:

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api create-bucket --bucket test-bucket-1

Awesome! We’ve got our first bucket in our repository. That’s cool but I want my bucket to be able to leverage this object lock capability Jim keeps going on about. To do that you use the same command but add the –object-lock-enabled-for-bucket parameter.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api create-bucket --bucket test-bucket-2-locked --object-lock-enabled-for-bucket

Cool, let’s go and check our work with the s3api get-object-lock-configuration command:

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api get-object-lock-configuration --bucket test-bucket-2-locked

So yeah, good to go there. Next let’s dive into that s3api list-buckets command seen in the previous screenshot. Listing buckets is a good example for understanding that when you access s3 or s3 compatible storage you are really talking about 2 things; s3 protocol and the s3api. For listing buckets you can use either:

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api list-buckets

or

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3 ls

While these are similar it’s worth noting the return will not be the same. The ls command will return data much like what it would in a standard Linux shell while s3api list-buckets will return JSON formatted data by default.

Enough About Buckets, Give Me Data

So buckets are great but the are nothing without data inside them. Let’s get to work writing objects.

Again writing data, especially if you are familar with the *nix methods to s3 can be very similar. I can use s3 cp or mv to copy or mv data to my s3://test-bucket-2-locked/ bucket or any other I’ve created or between them.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3 cp ./IMG_4881.jpg s3://test-bucket-2-locked/

If I’m not doing this by hand but rather building into automation I can also write data via the s3api.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api put-object --bucket test-bucket-2-locked --body ./aws-configure.jpg --key folder1/aws-config.jpg

Now that we’ve written a couple files let’s look at what we have. As you can see above once again you can do the same actions via both methods, it’s just the s3api way will consistently give you more information and more capability. Here’s what the api output would look like.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api list-objects --bucket test-bucket-2-locked

Take note of a few things here. While the s3 ls command gives more traditional file system output s3api refers to the objects with their entire “path” as the key. Essentially in object storage for our benefit it still has the concept of file and folder structure but it views each unique object as a single flat thing on the file system without a true tree. Key is also important because as we start to consider more advanced object storage capabilities such as object lock, encryption, etc. the key is often what you need to supply to complete the commands.

A Few Notes About Object Lock/Immutability

To round out this post let’s take a look at where we started as the why, immutability. Sure we’ve enabled object lock on a bucket before but what that really just does is enable versioning, it’s not enforcing anything. Before we get crazy with creating immutable objects its important to understand there are 2 modes of object lock:

• Governance Mode – In Governance Mode users can write data and not be able to truly delete it as expected but there are roles and permissions that can be set (and are inherited by root) that will allow that to be overridden and allow data to be removed.
• Compliance Mode – This is the more firm option where even the root account cannot remove data/versions and the retention period is hard set. Further once a retention date is set on a given object you cannot shorten it in any way, only extend it further.

Object Lock is actually done in one of two ways (or a mix of both); creating a policy and applying it to a bucket so that anything written to that bucket will assume that retention or by actually applying a retention period to an object itself either while writing the object or after the fact.

Let’s start with applying a basic policy to a bucket. In this situations for my test-bucket-2-locked bucket I’m going to enable compliance mode and then set retention to 21 days. A full breakdown of the formatting of the object-lock-configuration parameter and the options it provides can be found in the AWS documentation.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api put-object-lock-configuration --bucket test-bucket-2-locked --object-lock-configuration '{ "ObjectLockEnabled": "Enabled", "Rule": { "DefaultRetention": { "Mode": "COMPLIANCE", "Days": 21 }}}'

Cool, now to check that compliance we can simply use s3api get-object-lock-configuration instead against the bucket to check what we’ve done. I’ll note that for either the “put” above or the “get” below that there is no s3 endpoint equivalent, these are some of the more advanced features I’ve been going on about.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api get-object-lock-configuration --bucket test-bucket-2-locked

Ok, so we’ve applied a baseline policy of compliance and retention of 21 days to our bucket and confirmed that it’s set. Now let’s look at the objects within. You can view a particular object’s retention with the s3api get-object-retention command. As we are dealing with advanced features at the object level you will need to capture the key for an object to test. If you’ll remember we found those using the s3api list-objects command.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api get-object-retention --bucket test-bucket-2-locked --key "folder1/aws-config.jpg"

So as you can see we have both mode and retention date set on the individual object. What if we wanted this particular object to have a different retention period than the bucket itself? Let’s now use the s3api put-object-retention option to work and try to set that down to 14 days instead. While we use a general purpose number of days when creating the policy when we set object level retention it’s done by modifying the actual date stamp so we’ll simply pick a day 14 days from today.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api put-object-retention --bucket test-bucket-2-locked --key "folder1/aws-config.jpg" --retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2022-02-16T00:00:00" }'

Doh! Remember what we said about compliance mode? That you could make the retention shorter than what was previously set? We are running into that here and can see that it in fact works! Instead let’s try this again and set it to 22 days.

aws --profile jimtest --endpoint-url=https://us-central-1a.object.ilandcloud.com s3api put-object-retention --bucket test-bucket-2-locked --key "folder1/aws-config.jpg" --retention '{ "Mode": "GOVERNANCE", "RetainUntilDate": "2022-02-24T00:00:00" }'

As you can see now not only did we not get an error but when you check your retention it is now showing for defined timestamp so it definitely worked.

This feels like a good time to note that object locking is not the same as deletion protection. If I create an object lock enabled bucket and upload some objects to it, setting the object retention flag with the right info along the way, I am still going to be able to use a basic delete command to delete that file. In fact if I use CyberDuck or WinSCP to connect to my test bucket I can right click on any object there and successfully choose delete. What is happening under the covers is that a new version of that object is spawned, one with the delete marker applied to it. For standard clients that will appear that the data is gone but in reality it’s still there, it just needs to be restored to the previous version. In practice most of the UIs you are going to use to consume s3 compatible storage such as Veeam or developed consoles will recognize what is going on under the covers and essentially “block” you from executing the delete, but feel secure that as long as you have object lock enabled and the data is written with a retention date the data has not actually gone away and can be recovered.

All of this is a somewhat long winded answer to the question “How S3 Object Lock works” which Amazon has thoughtfully well answered in this post. I recommend you give it a read.

Conclusion

In the end you are most likely NOT going to need to know via the command line how to do all the above steps. More likely you will be using some form of a UI, be it Veeam Backup and Replication, AWS console or that of your service provider, but it is very good to know how to do these things especially if you are considering on-premises Object Storage as we move into this next evolution of IT and BCDR. Learning and testing the above is a relatively low cost consideration as most object services are literally pennies per GB, possibly with the egress data charges depending on your provider (hey AWS…), but it’s money well spent to get a better understanding.

I am honored to have been chosen as a finalist for the 2021 IT Blog Awards selected by Cisco Systems. This is my first time with such and needless to say I’m a little excited about it. As a finalist koolaid.info will be put onto the ballot for all those who wish to vote (Vote Here!), with separate categories for blogs and podcasts. If you wish to vote you may do so now until voting closes on Friday, February 18th.

I recently attended Cloud Field Day 12 put on by Gesalt IT. While there were many great presentations and some really exciting technology discussed I’d like to highlight what I considered the highlight of the week, the Big Memory Cloud announcement by MemVerge.

MemVerge has some very big ideas around taking the “software defined” concept to memory, allowing advanced optimizations and flexibility of instance memory usage. In the past the company has launched Memory Machine products related to making the most of Intel Optane memory capabilities, but the new Big Memory Cloud product set is built all around allowing you to move stateful applications between clouds and instances.

The example given is around the idea of spot public cloud instances. For those unfamiliar with the concept the idea is to use excess capacity in a given region during non-peak times, allowing the provider to sell those instances for a much lower cost but subject to needing instances to be torn down quickly as capacity is needed. For that reason spot instances work great for stateless workloads where they can easily be torn down in one place and started in another without issue, but it doesn’t work well for stateful applications or more to MemVerge’s point big, CPU and memory intensive work loads like AI, graphics rendering, research that may need to run for days or weeks to complete a single process. MemVerge Big Memory Cloud is designed to solve that problem by allowing you to capture the running state of an application at any time, taking a snapshot, and then transferred to another running instance seamlessly.

The technical concept of this is around what they call an AppCapsule. An AppCapsule is essentially a snapshot of the DRAM memory of one instance that can then be applied via the software to another instance of the same workload elsewhere. I think the concept has dreams of being a “multi-cloud vMotion” where a workload is running in cloud A and then is “moved” to cloud B. Today it is more along the lines of “multi-cloud Fault Tolerance” where you have to have 2 running copies of the workload, 1 in each cloud and the active version of the workload is moved from one to another (or even to a third instance).

While you can see that this is currently very young it is still very very cool. At the moment the supported workloads for the underlying Memory Machine technology are pretty limited, mostly related to databases and caching systems but with this new announcement I look for that to grow. I can see this as a big winner for the IaaS and BCDR services industry, allowing for hard to move workloads to now be on the table. Of course those are down the road type things that might come through partnerships but exciting all the same.

I have been fortunate to be selected as a delegate for Cloud Field Day 12 next week, November 3-5, 2021. This will be the first Tech Field Day event being done with in person attendees since the beginning of the pandemic and I’m happy to say I am on my way to San Jose, CA to be in attendance. While I am definitely excited about being able to travel about this I am most excited about the exciting slate of vendors that will be presenting on all things Cloudy. Thus far the delegate panel appears to be very well attended and includes a number of friends and acquaintances such as Nico Stein and Nathan Bennett.

Disclosure

In the interests of full disclosure it is worth noting that Gesalt IT, the company behind Tech Field Day, is covering the costs of my expenses for this trip up to and including travel, lodging and meals while on site. That said I have had no stipulations or requests on commentary from either Gesalt IT or presenting sponsors. In the week prior to the event I was invited to be a guest on the On-Premise IT podcast discussion the premise “The Cloud Is Finally Ready for the Enterprise.”

Schedule

All presentations will be able to be viewed online live at the base event link https://techfieldday.com/event/cfd12/ and if you are following along the other delegates and I will be happy to pass along questions that are sent via twitter with the #CFD12 tag. In PST times the presentations will be on this schedule:

Wednesday Nov. 3:
8:00 to 9:30 – Prosimo

10:30 to 12:30 – Juniper Networks

Thursday Nov. 4:
8:00 to 10:00 – Ondat (formerly StorageOS)

11:00 to 12:30 – Red Hat

13:30 to 15:30 – MemVerge

Friday Nov. 5:
8:00 to 10:00 – Veeam Software

11:00 to 12:30 – Yotascale

I plan to post my notes on each of the above in relatively raw form as the event proceeds so watch this site for more information.

I recently had a successful attempt at the latest version of the Veeam Certified Architect (VMCA) certification exam and I’m happy to have that one done. I found this newest version of the exam to be much more approachable than the rap on the last (and first) version of the exam. I wanted to take a minute to give some thoughts about the credential and pointers about how I prepared.

Training Requirements

Unfortunately one thing that still survives in the latest version of the Veeam certification programs, the VMCE and VMCA, is a hard course requirement for each level of certification. This means that if you want or need to achieve both levels of certification you are going to need to take 2 courses and 2 exams. Further, each are now “year versioned” exams, with versioning done on an annual basis. When it comes to renewal each exam will need to be independently renewed. As long as you pass each every year you will not be required to retake the class to upgrade but if you miss a year you will need to retake the course.

I wholeheartedly disagree with this approach and consider it especially burdensome on the certified person. In my mind it is understandable to have ONE course requirement but not for both, I struggle to think of another vendor certification program that does this. Even more if you feel you need to do annual recertification, which I’m not wild about but can understand with the number of new features each release brings, doing the top level exam should recertify for both. It’s been explained to me the rationale for this is because the exams are testing for different skill sets but the VMCE is still listed on the website as a prerequisite for the VMCA so i believe it to be a bit too much. At the end of the day this whole setup screams money grab for a company that should be well past the point of needing it.

That said many of you like me may have employer requirements to maintain the credential so this is for you.

VMCE vs VMCA

While you would think that for a skill set as siloed as Veeam core backup platform, Backup & Replication, that there would be two exams worth of content to cover but these really are targeted for different levels of IT Professionals. The VMCE exam really wants you to know and understand the Veeam Availability Suite of products, requiring memorization of knowledge about the various components and how they all fit together, think of this as your stereotypical memorization exam. While both exams are multiple choice exams with the VMCE the questions are all against a core set of knowledge, if you can memorize you’ve got this

The VMCA on the other hand is very light on memorization but very heavy on thinking through how you would scale out the core products to work in a very large, distributed scale. Really here the focus on looking at a potential customer scenario and requirements and determining what you need to build or suggest to give them successful outcomes.

My VMCA Back Story

I was lucky enough that through the Veeam Vanguard program that I was able to take the course for the VMCA free of charge both in 2017 and again in 2021 while the courses were in beta status. Oh what a difference four years makes. In 2017 I was at that time well versed with what VBR could do but at the time I was a Systems Administrator for essentially a SMB, protecting 4 hosts and 60 VMs in a single location. While we had requirements they weren’t exactly stressing the product’s most basic capabilities. When I took the course the first time I’m not ashamed to say that it intimidated me to the point where I didn’t even consider sitting the exam because so much of it was not in line with my day to day work.

Fast forward to 2021 and not only has the course been retooled to be more approachable but I am now in an architecture role where I am working with Veeam at scale every day so a good deal of it made more sense to me. I say all this to point out that I wish I would have taken the exam before because I wasn’t as far away as I thought I was and even if that is the role you are in now if you want to do more this is something you can do, you just have to think differently about it.

The Exam Methodology

The entire exam is based on a single scenario, that in theory is stylized off of a very real Veeam customer design request. There are more than one of these so if you have to retake it won’t be the same so don’t bother with trying to brain dump this. In any case the scenario is broken up into a number of tabs and will always be present on the left side of your screen as you take the exam so you can refer back to it as needed. Even with that I will say that I do very much so recommend taking 15-20 minutes at the beginning of your exam and read through the ENTIRE scenario so you at least know where to look for information and understanding he basics of what is being asked for.

Once you get through the scenario there will be a number of multiple choice questions that all relate to the scenario, but one thing I will share from the Exam Guide is that none of the questions will build upon other questions, the all independently are asking you to provide an answer directly back against the scenario. This is nice in that it won’t create a cascading problem.

Preparation

As I stated above I was lucky enough to be able to take the course while in beta status so my impressions of the course itself may not be in line with what is currently being put out there. That said the core idea of the class is very good, that it teaches you the Veeam architect way of thinking through a design based around customer requirements. This is especially on point because the Subject Matter Experts for the course itself were the Global Solutions Architect group within Veeam, some of the most knowledgeable people I know on the subject. The course walks you through what they consider the six stages of the solution lifecycle, which in turn make up the six sections of your exam, with each being tested against.

Further the course focuses on the four basic design principles;

1. Simplicity
2. Security
3. Cost/Benefit
4. Flexibility

All of these will be well covered in the course and in the Exam Guide that will be a part of your course materials. The guide itself is only 5 pages I think but it is jammed packed with information like the above that will really assist you so definitely give it a read through.

Once you get past the point of understanding both the life cycle and the design tenants there is a requirement to really know how to design the various Veeam components for use at scale and for this I highly recommend you consider a full read through of the Veeam Best Practices guide. This again is content created and managed by the Veeam Solutions Architecture group and is exceptional for understanding how you need to consider things both for the scope of this exam but also for right sizing your environment.

Conclusion

In the end if you can conceptually think about designing a BCDR plan based on Veeam solutions at a large scale, understand the lifecycle of that plan and the given needs of a customer, and are familiar with the best practices for deploying such systems this exam is very doable.

I received an email yesterday that the fast track program for VMCE 2021 is available now through December 21, 2021. So what is this? According to the e-mail and discussion with Rasmus Haslund of Veeam Fast Track is designed to be a self service resource to allow existing VMCEs v9, 2020 or people who took the v10 course to upgrade the certification to the latest version with access to study materials and a test voucher all for just a bit more than the voucher itself.

According to the email the program will provide you in total with the following:

• The latest Veeam Availability Suite v11 Configuration and Management (VASCM) courseware
• 13 days access to VASCM Labs for practicing and exam preparation
• VMCE 2021 Exam Specification Guide
• Access to the ‘Haslund Knowledge Check’
• Exam preparation videos
• VMCE Exam Voucher to take the exam at Pearson Vue

I will share that in the past (I’ve been a VMCE on versions 8 and 9 and recently renewed to the 2020 release) I’ve sworn by Rasmus’ always excellent practice exams so their inclusion here is noteworthy. While they are included these seem to remain a community resource provided by Rasmus so the value is more in the course materials and the videos, but still worth calling out.

If you are not currently 2021 certified and wish to be able to do an upgrade in the future without taking a course you will need to do so to keep current versions. If you are a standard end customer it’s true, your certification never expires but if you are in the partner space like me you unfortunately have to always be within the past 2 versions. In any case this is a pretty good deal for a recertification prep package

To purchase the fast track package you will need to log into the website you’ve been given access to your Veeam training materials at in the past, veeam.lochoice.com and click on whatever is the latest version of the VMCE materials you have available. Once there you will see a “Buy VMCE Fast Track to v11” button. Once clicked it’s as simple as providing a credit card and you are off and running.

If there is any part of being an IT professional that I actively dislike it is the need to know the intricacies of how various vendors license their software products. As bad as it was when I was a “Geek of Many Hats” back when I used to work in the SMB space, believe me or not its even worse in the Service Provider space. This is because when I typically have to get involved with licensing questions or tasks it’s because something has gone sideways for a customer organization or they are wanting to do “creative” things with their licensing dollars. I get it, I really do, I used to be there and you have to get as much out of every budget dollar you possibly can because with “Enterprise IT” at any scale those licensing spends usually equate to efficencies or compliance check boxes, not products sold on a linear line like it is in other scenarios.

That said today I have the joy of trying to clean up after a human error situation with Veeam Backup for O365. Now while many of you may know and/or love this product in a one off scenario where the only organization you are protecting is your own VBO is also considered a Service Provider product where the software at scale such as at iland. While it is good for this, it’s a whole different world when you need to architect solutions where many organizations will share the same backup infrastructure and then be billed for just their use. In this case we have a situation where a customer has licensed a very small subset of their much larger overall Office365 organization for backup, but through any number of ways a job was created that captured the entire organization. This resulted in the given server pulling almost twice as many licensed users as the license file allows, so yeah, good times.

Now when talking about Veeam licensing in regards to how it is determined that a given user should be allotted a licensed seat it is based off of a) backup data existing on disk (or in objects) and b) actively being a part of backup jobs. Both of these situations need to be cleared up before you can begin to actually remove licenses from users. Unfortunately aside from actually deleting the jobs (which does not have an option to delete the data under it) very little of this process can be done in the UI, it has to be done via Powershell. That said, here’s the basic process for purging all data related to a given organization. If you need to be more fine tuned that that let me know and I’ll write that up as well.

This tutorial assumes that you are not sharing named repositories between tenant organizations. If you are doing that PLEASE CONSIDER THIS A GOOD TIME TO RETHINK THAT DECISION. Let me just say from experience it’s bad man, it’s bad. If you need to do this with a shared repository I would recommend you give the fine folks at Veeam support a call and have them assist you.

1. Remove the jobs associated to a given organization. This should be simple enough as selecting the organization in VBO, then selecting all jobs within, right click delete.
2. Next we need to ensure that all data associated with the given organization is being purged. Luckily Niels Englen has a handy script up on VeeamHub called VBO-ClearRepo.ps1 that will take a given repo name and purge all data from it. You should be able to take this and just feed it all the repositories that are relevant over and over again and it will purge the data.
3. Finally we need to go through and verify that all the licenses have been removed for a given organization. If the organization is of any size this is most likely not been cleaned out and will necessitate you manually doing so. Luckily it can be done in a relatively easy manner with this:

Specify the organization to set scope
$org = Get-VBOOrganization -Name "myawesomeorg.onmicrosoft.com"

#Get a count of how many mailboxes are involved before
Get-VBOLicensedUser -organization $org | measure

#Purge all users for our given organization
Get-VBOLicensedUser -organization $org | Remove-VBOLicensedUser

In my experiences this last cmdlet can take quite a while to run; for larger organizations I’m seeing a run rate of 10 licenses per minute but your mileage may vary. There is most likely a faster way to do it but would probably involve hacking on the Config DB something that should never be tried alone but that’s up to your scenario.

Veeam’s Backup and Replication version 11 has brought a number of enhancements to it’s support of Linux-based Infrastructure components as has been covered quite well In Anthony Spiteri’s posts on the subject. In my mind and experience one of the biggest upgrades here that is flying well below the service is the addition of a legitimate Transport service for Linux repositories. While we are used to this type of setup for Windows servers in our VBR Infrastructure, where a service is installed on the repository and it handles the communication path between other components (proxies and cloud gateways) to the repository on Linux servers this has been handled by per-connection SSH tunnels. So for a heavily loaded repository in a Service Provider environment it was not uncommon to have hundreds if not thousands of concurrent SSH sessions between the cloud gateways and the repositories through which the veeamagent utility would actually be doing the work.

With v11 Veeam has finally brought Linux server support up to par with its own transport service, making scalability much better, but unfortunately this is not a standard upgrade task. While any new managed Linux servers will have the service installed by default any existing servers will show as “Up to Date” and but still be using SSH as it’s transport method. The best way to discover if it is in fact installed is to leverage the Get-VBRPhysicalHost cmdlet and if the components parameter is empty then it hasn’t been installed. For example to narrow it down I use the following to discover per VBR server:

Get-VBRPhysicalHost | Where-Object {($_.OsType -eq "Linux") -and ($_.Components.Name -notcontains "Transport")}

Installing the Linux Transport Service

Now that you’ve discovered where you still need to do this the pointy-clicky method to get the service installed is pretty straight forward:

1. Right click on each server needed under Managed Servers in Backup Infrastructure
2. Choose Properties to launch the Server Settings Wizard
3. Click Next, Next…Finish until it completes.

While this is great and all when you are in a large scale environment nobody wants to do that process tens or hundreds of times over and over again. For this let’s look at a way to automate the installation at scale using powershell.

Using our existing list created above we can then feed that through set-vbrLinux -server $server.name -force to make the transport service actually install. Depending on how many you are doing at once this may take a bit to complete as they are done asynchronous but still better than wearing out your mouse clicking finger. The full script to use is

$servers = Get-VBRServer -Type Linux
foreach ($server in $servers){
    # Only update Managed Server if Transport service is not installed
    if (Get-VBRPhysicalHost | Where-Object {($_.Name -eq $server.Name) -and ($_.OsType -eq "Linux") -and ($_.Components.Name -notcontains "Transport")}){
        $server | Set-VBRLinux -Force
    }    
}

Much thanks to Chris Arceneaux for the assist on getting the script working. You can find his blog at arsano.ninja.

Dealing with Microsoft API permissions has long been one of the hard parts of working extensively with Veeam Backup for Office 365. They tend to change fairly often and with little notice, leaving Veeam and other backup service providers for Microsoft365 in the enviable spot of needing to chase these permissions with their applications.

Veeam recently released version 5 of their M365 backup product and while the requirements are mostly the same permissions wise as they were in version 4 I’ve found there are a couple that potentially v4 was able to work without having in place due to other legacy setups that now don’t work in v5. My specific issue results in any Exchange Online jobs returning with the following error:

Processing mailbox XXXX failed with error: The remote server returned an error: (401) Unauthorized.

Fixing this requires adding the “full_access_as_app” permission but getting to that point is a bit more complicated for those not familiar with azure AD apps. This post will show you how to get to your existing Veeam Backup for Office 365 Azure app registration and verify that all of the necessary permissions have been added.

1. Connect to https://portal.azure.com and login with your global administrator permissions
2. Navigate to Azure Active Directory and then app registrations
3. Click on your current app registration used for Veeam Backup for O365
4. Select API permissions. Once there the desired state for Veeam Backup for Office 365 currently is
   • • Microsoft Graph
       • Directory.Read.All
       • Group.Read.All
       • Sites.ReadWrite.All
       • TeamSettings.ReadWrite.All
     • Office 365 Exchange Online
       • full_access_as_app
     • Sharepoint
       • Sites.FullControll.All
       • User.Read.All
5. If you have performed the v5 upgrade and your backups are currently failing the bolded permissions above may likely be missing and you will need to add them. When done adding your API permissions window should look like this
   
   
6. To add permissions you will need to
   1. click the “+ Add a permission” button
   2. click the tile for the necessary API group (Graph, Office 365 Exchange Online)
      * For the Office 365 Exchange Online direct access has been deprecated by Microsoft. In that case you will need to choose the “API my organization uses and search for “Office 365 Exchange” to find it. This looks like
      
      
   3. Choose “Application permissions”
   4. Check the box for the permission you are requesting and click “Add permissions”
      
      
7. After adding permissions you will need to click the “Grand admin consent for <your organization>” button to complete adding the permissions for your organization.

It is worth noting that for future reference the documentation for Veeam required permissions for Veeam Backup for Office 365 modern authentication with legacy protocols allowed is located at https://helpcenter.veeam.com/docs/vbo365/guide/ad_app_permissions_legacy.html?ver=50. As these permissions are changed by Microsoft from time to time it is worth notating this link.

I recently had the opportunity to read through Anthony Spiteri’s excellent post on the enhancements to the Linux Proxy in the most recent release of it’s flagship Backup and Replication product. In the course of reading it I wondered how close to feature parity the Linux variant is to the Windows version and while the gap is rapidly closing I found that we aren’t quite there yet. Thanks to Rick Vanover and Michael Cade for coming through with some answers I’ve created the following table.

In short when it comes to the actual moving of data the vast majority of VBR users will be able to leverage Linux proxies without an issue, although this is a good reason to not be collapsing all the proxy types into a single server. While by and large this is still a fine idea there are situations where you should not. If you are doing the next step of Guest Processing you may need to take the extra step of specifying either the VBR server itself or another Windows server to act as the Guest Interaction Proxy but that should be doable for most without adding additional infrastructure.

One final note regarding proxies that is relevant to the v11 release it is considered best practice to never share the CDP proxy role with any thing else, it should be a dedicated server.

* Linux backup proxies that use virtual appliance (Hot-Add) transport mode do not support the VM copy scenario.
** Backup from Storage Snapshot with Linux Proxies is support on iSCSI and Fibre Channel targets only.

Source: Backup and Replication v11 User Guide