If there is any part of being an IT professional that I actively dislike it is the need to know the intricacies of how various vendors license their software products. As bad as it was when I was a “Geek of Many Hats” back when I used to work in the SMB space, believe me or not its even worse in the Service Provider space. This is because when I typically have to get involved with licensing questions or tasks it’s because something has gone sideways for a customer organization or they are wanting to do “creative” things with their licensing dollars. I get it, I really do, I used to be there and you have to get as much out of every budget dollar you possibly can because with “Enterprise IT” at any scale those licensing spends usually equate to efficencies or compliance check boxes, not products sold on a linear line like it is in other scenarios.

That said today I have the joy of trying to clean up after a human error situation with Veeam Backup for O365. Now while many of you may know and/or love this product in a one off scenario where the only organization you are protecting is your own VBO is also considered a Service Provider product where the software at scale such as at iland. While it is good for this, it’s a whole different world when you need to architect solutions where many organizations will share the same backup infrastructure and then be billed for just their use. In this case we have a situation where a customer has licensed a very small subset of their much larger overall Office365 organization for backup, but through any number of ways a job was created that captured the entire organization. This resulted in the given server pulling almost twice as many licensed users as the license file allows, so yeah, good times.

Now when talking about Veeam licensing in regards to how it is determined that a given user should be allotted a licensed seat it is based off of a) backup data existing on disk (or in objects) and b) actively being a part of backup jobs. Both of these situations need to be cleared up before you can begin to actually remove licenses from users. Unfortunately aside from actually deleting the jobs (which does not have an option to delete the data under it) very little of this process can be done in the UI, it has to be done via Powershell. That said, here’s the basic process for purging all data related to a given organization. If you need to be more fine tuned that that let me know and I’ll write that up as well.

This tutorial assumes that you are not sharing named repositories between tenant organizations. If you are doing that PLEASE CONSIDER THIS A GOOD TIME TO RETHINK THAT DECISION. Let me just say from experience it’s bad man, it’s bad. If you need to do this with a shared repository I would recommend you give the fine folks at Veeam support a call and have them assist you.

1. Remove the jobs associated to a given organization. This should be simple enough as selecting the organization in VBO, then selecting all jobs within, right click delete.
2. Next we need to ensure that all data associated with the given organization is being purged. Luckily Niels Englen has a handy script up on VeeamHub called VBO-ClearRepo.ps1 that will take a given repo name and purge all data from it. You should be able to take this and just feed it all the repositories that are relevant over and over again and it will purge the data.
3. Finally we need to go through and verify that all the licenses have been removed for a given organization. If the organization is of any size this is most likely not been cleaned out and will necessitate you manually doing so. Luckily it can be done in a relatively easy manner with this:

Specify the organization to set scope
$org = Get-VBOOrganization -Name "myawesomeorg.onmicrosoft.com"

#Get a count of how many mailboxes are involved before
Get-VBOLicensedUser -organization $org | measure

#Purge all users for our given organization
Get-VBOLicensedUser -organization $org | Remove-VBOLicensedUser

In my experiences this last cmdlet can take quite a while to run; for larger organizations I’m seeing a run rate of 10 licenses per minute but your mileage may vary. There is most likely a faster way to do it but would probably involve hacking on the Config DB something that should never be tried alone but that’s up to your scenario.

Veeam’s Backup and Replication version 11 has brought a number of enhancements to it’s support of Linux-based Infrastructure components as has been covered quite well In Anthony Spiteri’s posts on the subject. In my mind and experience one of the biggest upgrades here that is flying well below the service is the addition of a legitimate Transport service for Linux repositories. While we are used to this type of setup for Windows servers in our VBR Infrastructure, where a service is installed on the repository and it handles the communication path between other components (proxies and cloud gateways) to the repository on Linux servers this has been handled by per-connection SSH tunnels. So for a heavily loaded repository in a Service Provider environment it was not uncommon to have hundreds if not thousands of concurrent SSH sessions between the cloud gateways and the repositories through which the veeamagent utility would actually be doing the work.

With v11 Veeam has finally brought Linux server support up to par with its own transport service, making scalability much better, but unfortunately this is not a standard upgrade task. While any new managed Linux servers will have the service installed by default any existing servers will show as “Up to Date” and but still be using SSH as it’s transport method. The best way to discover if it is in fact installed is to leverage the Get-VBRPhysicalHost cmdlet and if the components parameter is empty then it hasn’t been installed. For example to narrow it down I use the following to discover per VBR server:

Get-VBRPhysicalHost | Where-Object {($_.OsType -eq "Linux") -and ($_.Components.Name -notcontains "Transport")}

Installing the Linux Transport Service

Now that you’ve discovered where you still need to do this the pointy-clicky method to get the service installed is pretty straight forward:

1. Right click on each server needed under Managed Servers in Backup Infrastructure
2. Choose Properties to launch the Server Settings Wizard
3. Click Next, Next…Finish until it completes.

While this is great and all when you are in a large scale environment nobody wants to do that process tens or hundreds of times over and over again. For this let’s look at a way to automate the installation at scale using powershell.

Using our existing list created above we can then feed that through set-vbrLinux -server $server.name -force to make the transport service actually install. Depending on how many you are doing at once this may take a bit to complete as they are done asynchronous but still better than wearing out your mouse clicking finger. The full script to use is

$servers = Get-VBRServer -Type Linux
foreach ($server in $servers){
    # Only update Managed Server if Transport service is not installed
    if (Get-VBRPhysicalHost | Where-Object {($_.Name -eq $server.Name) -and ($_.OsType -eq "Linux") -and ($_.Components.Name -notcontains "Transport")}){
        $server | Set-VBRLinux -Force
    }    
}

Much thanks to Chris Arceneaux for the assist on getting the script working. You can find his blog at arsano.ninja.

Dealing with Microsoft API permissions has long been one of the hard parts of working extensively with Veeam Backup for Office 365. They tend to change fairly often and with little notice, leaving Veeam and other backup service providers for Microsoft365 in the enviable spot of needing to chase these permissions with their applications.

Veeam recently released version 5 of their M365 backup product and while the requirements are mostly the same permissions wise as they were in version 4 I’ve found there are a couple that potentially v4 was able to work without having in place due to other legacy setups that now don’t work in v5. My specific issue results in any Exchange Online jobs returning with the following error:

Processing mailbox XXXX failed with error: The remote server returned an error: (401) Unauthorized.

Fixing this requires adding the “full_access_as_app” permission but getting to that point is a bit more complicated for those not familiar with azure AD apps. This post will show you how to get to your existing Veeam Backup for Office 365 Azure app registration and verify that all of the necessary permissions have been added.

1. Connect to https://portal.azure.com and login with your global administrator permissions
2. Navigate to Azure Active Directory and then app registrations
3. Click on your current app registration used for Veeam Backup for O365
4. Select API permissions. Once there the desired state for Veeam Backup for Office 365 currently is
   • • Microsoft Graph
       • Directory.Read.All
       • Group.Read.All
       • Sites.ReadWrite.All
       • TeamSettings.ReadWrite.All
     • Office 365 Exchange Online
       • full_access_as_app
     • Sharepoint
       • Sites.FullControll.All
       • User.Read.All
5. If you have performed the v5 upgrade and your backups are currently failing the bolded permissions above may likely be missing and you will need to add them. When done adding your API permissions window should look like this
   
   
6. To add permissions you will need to
   1. click the “+ Add a permission” button
   2. click the tile for the necessary API group (Graph, Office 365 Exchange Online)
      * For the Office 365 Exchange Online direct access has been deprecated by Microsoft. In that case you will need to choose the “API my organization uses and search for “Office 365 Exchange” to find it. This looks like
      
      
   3. Choose “Application permissions”
   4. Check the box for the permission you are requesting and click “Add permissions”
      
      
7. After adding permissions you will need to click the “Grand admin consent for <your organization>” button to complete adding the permissions for your organization.

It is worth noting that for future reference the documentation for Veeam required permissions for Veeam Backup for Office 365 modern authentication with legacy protocols allowed is located at https://helpcenter.veeam.com/docs/vbo365/guide/ad_app_permissions_legacy.html?ver=50. As these permissions are changed by Microsoft from time to time it is worth notating this link.

I recently had the opportunity to read through Anthony Spiteri’s excellent post on the enhancements to the Linux Proxy in the most recent release of it’s flagship Backup and Replication product. In the course of reading it I wondered how close to feature parity the Linux variant is to the Windows version and while the gap is rapidly closing I found that we aren’t quite there yet. Thanks to Rick Vanover and Michael Cade for coming through with some answers I’ve created the following table.

In short when it comes to the actual moving of data the vast majority of VBR users will be able to leverage Linux proxies without an issue, although this is a good reason to not be collapsing all the proxy types into a single server. While by and large this is still a fine idea there are situations where you should not. If you are doing the next step of Guest Processing you may need to take the extra step of specifying either the VBR server itself or another Windows server to act as the Guest Interaction Proxy but that should be doable for most without adding additional infrastructure.

One final note regarding proxies that is relevant to the v11 release it is considered best practice to never share the CDP proxy role with any thing else, it should be a dedicated server.

* Linux backup proxies that use virtual appliance (Hot-Add) transport mode do not support the VM copy scenario.
** Backup from Storage Snapshot with Linux Proxies is support on iSCSI and Fibre Channel targets only.

Source: Backup and Replication v11 User Guide

Let me start by saying I’m fine.

Ok, I’ll be fine, sooner than later.

Last week I had what the doctors are calling a non-stemi Heart Attack. Essentially what that means is that while none of the arteries in my body are clogged (they look pretty great actually according to the doctor) I had a couple of vessels in the heart that are narrowed which caused the event. The treatment of such things is just medicinal and lifestyle. In total this was essentially my warning shot, that as a middle aged man it’s time to push back against all of the lifestyle choices that both the work from home lifestyle and this life that 2020 is provided led me to believe was acceptable.

I honestly had no intention of mentioning this outside of some core groups but as I get to the backside of this event I realized there were quite a few places where I made all the wrong choices and in the interest of education I’d like to share here.

Oops

To start with I began feeling bad Wednesday night after helping myself to a bottle of wine. I do not often drink wine so when I started to feel some discomfort that I can only describe as being very gassy in my chest I made the assumption that I was just having a negative reaction to the wine and decided to just “gut through it.” This was major mistake #1. In any situation where you are having chest pains, do not mess around because every second after an event begins can lead to more damage to your heart.

Fast forward to Thursday morning after a night of pacing, vomiting, weakness in my upper extremities and the inability to breathe when laying down. My wife at this point pointed me towards a list of signs of a heart attack and low and behold all of those were on the list. I talked to my Primary Care Physician because, well because I still was in denial, who told me I should go to the hospital so finally I decided to go. Here is where mistake #2 came in as I decided to just drive myself to the local hospital as I didn’t want to worry my daughter or my wife any more than I already had. While it worked out I’ve had no end of health care professionals point out to me how very dumb an idea this was. The likelihood that I could have had an event while driving and not only killed myself but potentially others was higher than should have been acceptable.

Another mistake is how I got here in the first place. For the past couple years due to life, work stressors, etc. I’ve let a lot of things go; my diet has been largely fried and full of comfort foods. I’ve mostly stopped exercising in any meaningful or consistent way. Considering not long ago I was running a 5k or so 4-5 days a week the fact that my Fitbit has averaged for the past year 3000 steps or less is pretty telling. Also I’ve been bad about letting the normal everyday stressors be treated like Earth shattering problems that I alone can solve. All of these are things I have the power and the ability to fix and need to do so.

Finally the last mistake, but a very fixable one still, that I’ve made is not personally allowing myself true relaxation and reflection time after this. Even though I was released without lined out restrictions I was told to take it easy and work towards getting back to a healthier lifestyle. In my standard way I decided that meant it was time to go right back into “kickass mode”, starting the first Monday after getting out of the hospital with an exit interview (surprise!) and then doing 30 minutes on the treadmill and another 30 doing a Boxing workout on my Nintendo Switch. Neither of those workout choices were a great idea and as thus I felt like crap again last night. You have to allow yourself time to recover, something I need to learn not only with this event but also with life in general.

Changes

Now for the positive gift from 2020. As I alluded to earlier my time at OffsiteDataSync has come to an end, literally the same day I had my heart attack. I will be starting the year in a new position that I am very much so looking forward to so stay tuned.

As mentioned I’m going to also be in for some lifestyle changes. For now I’m starting with this:

• Eating more healthy and tracking my diet and health again with MyFitnessPal. This has really worked for me in the past so time to do it again. If you are also a user and would like to link up for motivation’s sake you can find my profile at https://www.myfitnesspal.com/profile/k00laidit.
• Slowly making my way back to some meaningful daily exercise, mostly tracked by my Fitbit. I may get back to using Runkeeper at some point but I’m not there yet. If you’d like to be friends on Fitbit you can find me at https://www.fitbit.com/user/2XBFK6.
• Going to truly give the mindfulness thing a better shot. Thanks to hearing a great deal about it over the years from Alicia Preston I truly believe that a quick few minutes here and there through the day will help me relax more and be better prepared to meet the day than I have been lately.
• Fortunately there are a number of medications available that will help me to get and stay better. I’m hopeful that all of the above will make most of them be short term fixes as I make the underlying problems better.

I would like to close by thanking the many people who helped me during this scare. The staff at CAMC Teays Valley Emergency Room and CAMC Memorial’s Cardiac Wing. Friends too numerous to count who stayed in touch and kept me conversing when I was very much so stressing out. Family who came and brought food to help out our family in some troubling times. Finally I wish to thank my wife for the love and support that only she can give. We can get through all the things together Tracy, I love you.

If nothing else that will be a start. Here’s to taking the gifts of 2020 and making positives from them in 2021!

I have been in the Information Technology field for about 20 years at this point but there was a very distinct inflection point in 2013 when the social side of my career began. In a very quick series of events (at the same time actually) I attended my first technical conference (CiscoLive US), setup my twitter account, and met many of the people I now consider the thought leaders I turn to when it comes to deciding what new technologies are coming to be and how important they are going to be to me and my organization’s needs.

At this same time I was also introduced to Tech Field Day and it’s founder Stephen Foskett and I was pretty much instantly in awe of what they were all about; selecting panels of delegates, flying them to a common city where a number of companies would present either themselves as a company or their newest products over the course of a couple of days, all of which was then recorded and put on their YouTube channel. I as a member of the tech twitterati could tweet at delegates and often get questions answered, it was amazing to me then! I honestly can’t tell you how many solutions I’ve used TFD presentations as part of my research into if I should be looking to purchase their products and services but it’s more than an handful.

From the get go I’ve had a bucket list item of being on a delegate panel, but sadly my previous role had distinct rules against such things. Fast forward to now, new organization, new rules and here I am invited to be on the panel for Cloud Field Day 8. While this is a virtual event I am no less excited to be a part of the early group delegate panel. The experience so far has been great and I’m very appreciative of Stephen, Ben Gage and Mel Zura in helping to get me up to speed with how the event will go.

In this vBrownBag/VMTN session I was joined by Matt Crape and Jim Millard to discuss the realities of joining a influencer program such as vExpert or Veeam Vanguard.

In 2017 I was allowed not one but two slots at for vBrownBag at VMworld 2017 which was a big deal to me. The first of these was a solo presentation by me about how to be an active member of the vCommunity while living in a rural area without easy access to things like VMUGs and such.

I was honored to be selected to present at VeeamON again in 2015 with the same general concept of session but really highlighting some of the work I’d been doing with Windows Server deduplication and Veeam SureBackup. This was a much nice crowd, with 50 or so people, and I felt it went much better. Again there is no video this time around but as an aside watching the replay at the time this is when I first realized just how much of an accent I have, literally had never realized it before.

So this was immediately following the vBrownBag session. Again, now second shot at “public speaking” but this time I’m in a room for 200, in the hangover slot at a tech conference (8 am the morning after the party), there were a total of 3 people in the session. Probably not the worst thing but one thing I’ve found over the years is that crowds are good, they loosen you up. Just goes to show that we definitely all start somewhere.

Unfortunately I don’t have the video of these early session so you are just going to have to trust me that it was rough.