Setting Up External Access To A Veeam SureBackup Virtual Lab

Hey y’all, happy Friday! One of the things that seems to still really fly under the radar in regards to Veeam Backup & Replication is its SureBackup feature. This feature is designed to allow for automated testing via scripts of groups of your backups. An example would be if you have a critical web application. You can create an application group that includes both the database server and the web server and when the SureBackup job is run Veeam will connect a section of its backup repository to a specified ESXi host as a datastore and, start the VMs within a NAT protected segment of your vSphere infrastructure, run either the role based scripts included or custom ones you specify to ensure that the VMs are connecting to the applications correctly, and then when done shut the lab down and fire off an e-mail.

That workflow is great an all but it only touches on the edge of the power of what SureBackup can do for you. In our environment not only do we have a mandate to provide backup tests that allow for end-user interaction, but we also use SureBackup for test bed type applications such as patch tests. An example of the latter here is when I was looking to upgrade our internal Windows-based CA to Server 2012 R2. I was able to launch the server in the lab, perform the upgrade and ensure that it behaved as expected WITHOUT ANY IMPACT ON PRODUCTION first and then tear down the lab and it was like it never happened. Allowing the VMs to stay up and running after the job starts requires nothing more than checking a box in your job setup.

By default access to a running lab is fairly limited. When you launch a lab from your Veeam server a route to the NAT’d network is injected to the Veeam server itself to allow access, but that doesn’t help you all that much if you are wanting others to be able to interact; we need to expand that access outwards. This post is going to walk you through the networking setup for a Virtual Lab that can be accessed from whatever level of access you are looking for, in my case from anywhere within my production network.

Setting Up the Virtual Lab

 

The first step if you haven’t setup SureBackup in your environment at all is to set up your Virtual Lab.  The first of two parts here that are critical to this task is setting up the Proxy IP, which is the equivalent to your outside NAT address if you’ve ever worked on a firewall. This IP is going to essentially be the production network side of the Lab VM that is created when you setup a Veeam Virtual Lab.

1-set-nat-host

Next we need to set up an isolated network for each production port group you need to support. While I use many VLANs in my datacenter I try to keep the application groups I need to test on the same VLAN to make this setup simple, but it doesn’t need to be, you can support as many as you need. Simply hit add, browse out and find the production network port group you need to support, give the isolated network a name and specify a VLAN.

2a-setup-vlans

The last step of setting up the Virtual Lab in this regard is creating a virtual NIC to map to each of your isolated networks. So where I see a lot of people get tripped up with this is always make the proxy appliance IP address here map to the default gateway of the production network it is reflecting. If you don’t do that the launched lab VMs will never be able to talk outside of the lab. Second, in regard to the Masquerade IP try to aim for some consistency. Notice that in my production network I am using a Class B private address space but with a class C mask. By default this will throw off the automatic generation of the Masquerade IP and I’ve found it isn’t always consistent across multiple Virtual NIC setups.  If you setup multiple isolated networks above you need to repeat this process for each network. Once you are done with this you can complete your Lab Setup and hit Finish to have it build or rebuild the appliance.

2-create-nat-network

Tweaking the SureBackup Job

For the sake of brevity I’m assuming at this point that you’ve got your Application Groups setup without issue and are ready to proceed to fixing your SureBackup job to stay up and running. To do so on the Application Group screen All you have to do is check the “Keep the application group running after the job completes” box. That’s it. Really. Once you do that this lab will stay up and running until you right click on the job in the Veeam Backup & Replication Console and choose stop. I’ve been lobbying for year for a “stop after X hours” option but still haven’t got very far with that one, but really the concern there is more performance impact from doubling a part of your load since you are essentially running 2 copies of a segment of your datacenter. If you have plenty to burn it isn’t an issue.

3-keep-lab-up

Fixing the Routing

Now the final step is to either talk to your network guy or go yourself to where your VLAN routing is taking place and add a static route to the IP range of your inside the lab into the routing table through the Proxy Appliance’s IP. For the example we’ve been working through in this post our Proxy appliance has an IP of 172.16.3.42 and all of our Lab networks are within the 172.31.0.0/16 network. If you are using a IOS based Cisco switch to handle your VLAN routing the command would be

After that is done, from anywhere that route is accessible from you should now be able to pass whatever traffic inbound to the lab network addresses. So sticking with our example, for a production VM with the IP address 172.16.3.10, you would interact with the IP 172.31.3.10 in whatever way needed. Keep in mind this is for lack of a better word one way traffic. You can connect in to any of the hosts within the lab network but you can’t really have them reach directly out and have them interact on the production network.

4a-testing

One More Thing…

One final tip that I can give you on this if you are going to let others in to play in your labs is to have at least one workstation grade VM that you include in each of your Applications Groups with the software needed to test with loaded. This way you can enable RDP on that VM and they user can just double-click an icon and connect into the lab, running their tests from there. Otherwise if you have locally installed applications that need to connect to hosts that are now inside the lab you are either going to need to reconfigure the application with the corrected address or modify the user’s hosts file temporarily so that they connect to the right place, neither of which is particularly easy to manage. The other nice thing about a modern RDP session is you can cut and paste files in and out of it, which is handy if the user wants to run reports and the like.

4-connecting-into-the-lab

As an aside I’m contemplating doing a video run through of the setting up a SureBackup environment to be added to the blog next week. Would you find such a thing helpful? If so please let me know on twitter @k00laidIT.

Veeam Backup Repository Best Practices Session Notes

After a couple days off I’m back to some promised VeeamON content. A nice problem that VeeamON had this year is the session choices were much more diverse and there were a lot more of them. Unfortunately this led to some overlap of some really great sessions. A friend of mine, Jaison Bailey of vBrisket fame and fortune, got tied up in another session and was unable to attend what I considered one of the best breakout sessions all week, Anton Gostev‘s Backup Repository Best Practices so he asked me to post my notes.

For those not too familiar with Veeam repos they can essentially be any manner of addressable disk space, whether local, DAS, NAS, SAN or even cloud, but when you start taking performance into account you have to get much more specific. Gostev, who is the Product Manager for Backup & Replication, lines out the way to do it right.

Anyway, here’s the notes including links to information when possible. Any notations I have are in bold and italicized.

Don’t underestimate the importance of Performance

  • Performance issues may impact RTOs

Five Factors of choosing Storage

  • Reliability
  • Fast backups
  • Fast restores
  • DR from complete storage loss
  • Lowest Cost

Ultimate backup Architecture

  • Fast, reliable primary storage for fastest backups, then backup copy to Secondary storage both onsite AND offsite
  • Limit number of RP on primary, leverage cheap secondary
  • Selectively create offsite copies to tape, dr site, or cloud

Best Repo: Low End

  • Any Windows or Linux Server
    • Can also serve as backup /backup proxy server
  • Physical server storage options
    • Local Storage
    • DAS (JBOD)
    • SAN LUN
  • Virtual
    • iSCSI LUN connected to in guest Volume

Best Backup Repo: High End

Backup Repos to Avoid

  • Low-end NAS  & appliances
    • If stuck with it, use iSCSI instead of other protocols * Ran into this myself with a Qnap array as my secondary storage, not really even feasible to run anything I/O heavy on it
  • SMB (CIFS) network shares
    • Lots of issues with existing SMB clients
    • If share is backed up by server, add actual server instead
  • VMDK on VMFS *Nothing wrong with running a repo from a virtual machine, but don’t store backups within, instead connect an iSCSI LUN directly to the VM and format NTFS
    • Extra logic on the data path- more chances for data corruption
    • Dependent on vSphere being functional
  • Windows Server 2012 Deduplication (scalability) *I get his rationale, but honestly I live and die by 2012 R2 deduplication, it just takes more care and feeding than other options. See my session’s slides for notes on how I implement it.

Immediate Future: Technologies to keep in mind

  • Server 2016 Deduplication
    • Same deduplication, far greater performance and scale (64 TB files) *This really will be a big deal in this space, there is a lot of upside to a simple dedupe ability rolled into a Windows server
  • ReFS 2.0
    • Great fit for backup repos because it has built in data corruption protection
    • Veeam is currently working on some things with it

Raw Disk

  • Raid10 whenever you can (2x write penalty, but capacity suffers)
  • Raid5 4x write penalty, greater risks)
  • Raid6 severe performance overhead (6x write penalty
  • Lookup Maximum performance per spindle
  • A single job can only keep about 6-8 spindles busy- use multiple jobs if you have them to saturate
  • RAID volume
    • Stripe Size
      • Typical I/O for Veeam is 25k-512KB
      • Windows Server 2012 defaults to 64KB
      • At least 128 KB stripe size is highly recommended
        • Huge change for things like Synthetics, etc
    • RAID array
      • Fill as many drives as possible from the start to avoid expansion
      • Low-end sorage systems have significant performance problems
    • File System
      • NTFS (Best Option)
        • Larger block size does not affect performance, but it helps avoid excessive fragmentation so 64KB block size recommend
        • Format with /L to enable larger file records
        • 16 TB max file size limit before 2012 (now 256)
        • * Full string of best practices for format NTFS partition from CLI: Format <drive:> /L /Q /FS:NTFS /A:8192
      • ReFS not ready for prime time yet
      • Other
    • Backup Job Settings
      • Always a performance vs disk space choice
      • Reverse incremental backup mode is 3x I/O per block
      • Consider forever incremental instead
      • Evaluate transform performance
      • Repository load
        • Limit concurrent jobs to a reasonable amount
        • Use ingest rate throttling for cross-SAN backups

Dedupe Storage: Pains and Gains

  • Gains
    • True global dedupe
    • Lowest cost/ TB
  • Do not use deduplicating storage as your primary backup repository!
  • But if you must leverage vendor-specific integrations, use backup modes without full backup transformation, us active fulls instead of synthetics
  • If backup performance is still bad, consider VTL
  • 16TB+ backup storage optimization for 4MB blocks (new)
  • Parallel processing may impact  dedupe ratios

Secondary Storage Best Practices

  • Vendor-specific integrations can make performance better
  • Test Backup Copy retention processing performance. If too slow consider Active Full option of backup copy jobs (new in v9)
  • If already invested and stuck
    • Use as primary storage and leverage native replication to copy backups to DR

Backup Job Settings BP

Built-In deduplication

  • Keep ON for best performance (except lowest end devices) even if it isn’t going to help you with Per VM backup files
  • Compression
    • Instead of disabling keep Optimal enabled in job and use “decompress before storing- even locally
    • Dedupe-friendly isn’t very friendly any more (new)
      • Will hinder faster recovery in v9
  • Vendor recommendations are sometimes self-serving  to achieve higher dedupe ratios but negatively effect performance

Disk-based Storage Gotchas

  • Gostev loves tape
    • Cheaper
    • Reliable
    • Read-only
    • Customer Success is the biggie
    • Tape is dead
      • Amazon, Google & 50% of Veeam customers disagree
  • Storage-level corruption
    • RAID Controllers are your worst enemies
    • Firmware and software bugs are common, too
    • VT402 Data Corruption tomorrow at 1:30 for more
  • Ransomware  possible

The 2 Part of the 3-2-1 Rule

  • 3 copies, 2 different medias, 1 offsite
  • Completely different storage type!

Storage based replication

  • Betting exclusively on storage-based replication will cost you your job
  • Pros:
    • Fantastic performance
    • Efficient bandwidth utilization
  • Cons:
    • Replicates bad data too
    • Backups remain in a single fault domain

Backup Copy vs. Storage-Based Copy

  • Pros:
    • Breaks the data loop (isolated source and target storage)
    • Implicitly validates all source data during its operation
    • Includes backup files health check
  • Cons:
    • Higher load on backup storage

Make Tape out of drives

  • Low End:
    • Use rotated drives
    • Supported for both primary & backup copy jobs
  • Mid-range:
    • Keep an off-site copy off-prem (cloud)
  • High End:
    • Use hardware-based WORM solutions

Virtualize your Repository (SOBR)

  • simplify backup storage and backup job management
  • Reduce storage hardware spending by allowing disks to be fully utilized
  • Improve backup storage performance and scalability

 

Setting Up Endpoint Backup Access to Backup & Replication 8 Update 2 Repositories

A part of the Veeam Backup & Replication 8 Update 2 Release is the ability to allow users to target repositories specified in your Backup Infrastructure as targets for Endpoint Backup. While this is just one of many, many fixes and upgrades (hello vSphere 6!) in Update 2 this one is important for those looking to use Endpoint Backup in the enterprise as it allows for centralized storage and management and equally important is you also get e-mail notifications on these jobs.

Once the update is installed you’ll have to decide what repository or repositories will be available to Endpoint Backup and provide permissions for users to access them. By default every Backup Repository Denies Endpoint Backup access to everyone. To change this for one or more repositories you’ll need to:

  1. Access the Backup Repositories section under Backup Infrastructure, then right click a repository and choose “Permissions.”
  2. Once there you have three options for each repository in regards to Endpoint permissions; Deny to everyone (default), Allow to everyone, and Allow to the following users or groups only. This last option is the most granular and what I use, even if just to select a large group. In the example shown I’ve provided access to the Domain Admins group.
  3. You will also notice that I’ve chosen to encrypt any backups stored in the repository, a nice feature as well of Veeam Backup & Replication 8.

Also of note is that no user will be able to select a repository until they have access to it. In setting up the Endpoint Backup job when the Veeam server is specified you are given the option to supply credentials there so you may choose to use alternate credentials so that the end users themselves don’t actually have to have access to the destination.

Top New Features in Veeam Backup & Replication v8

We are now a couple of months out from the release of version 8 of Veeam Software’s flagship product Backup & Replication. Since then we’ve seen the first patch release a couple of weeks after, almost a Veeam tradition, and I’ve had it deployed and running for a while now. In that time I’ve found a lot to really like in the new version.

End to End Encryption

Backup & Replication now has the ability to encrypt your backup data from the moment it leaves your production storage system, through the LAN and WAN traffic and once it is at rest, either on disk or tape. This encryption is protected by password stored both with humans as well as within the Enterprise Manager database keeping you from losing backups. Finally the encryption does not change ratios for either compression or deduplication of the backup data.

Resource Conservation Improvements

Quite a few of the new Backup & Replication features are geared towards keeping your RPO goals from getting in the way of production efficiency. First and foremost is the availability of Backup I/O Control, a feature that will monitor the latency of your production storage system and if measured metrics climb above a user defined level will throttle backup operations to return systems to acceptable levels.

On the networking side if you have redundant or other none production WAN links you now have the ability to specify preferred networks for backup data, with failover to production if it isn’t available. Further the WAN Accelerator for site to site backup copy and replication has been improved to allow for up to 3x what was seen in v7.

Cloud Connect

Both of the above features make this one possible. With this new version brings a new partnership opportunity where VARs and other cloud storage service providers have the ability to directly act as a repository for your backup data. These providers can then allow you to spin these backups up as part of a second offering or as part of a package. With this the need to own, manage and maintain the hardware for a DR site becomes much lighter and I personally believe this will be a big deal for many in the SMB space.

New Veeam Explorers for Recovery

Veeam has been phasing out the use of the U-AIR wizards for item level restore for a while but with v8 we now have the release of the Explorers for Active Directory, Microsoft SQL Server and Exchange. The Active Directory one is particularly of note because it not only allows you to restore a deleted AD item but do so with the password intact.  Transaction log backup for SQL servers is also now supported allowing for point in time restore. The Exchange option has a few new features but I especially like the option of recovering hard-deleted items.

These are frankly just the tip of the iceberg when it comes to the new features. For more on what’s new I recommend you checkout the What’s New documents for both Backup & Replication as well as for VeeamONE, Veeam’s virtualization infrastructure monitoring package.

 

Getting Started with Veeam Backup & Replication v7

Come one, come all virtualization geeks, the latest installment of Veeam‘s excellent Backup & Replication suite has arrived.  As noted in lots of places, v7 boasts a boatload of new and new-to-them features that the community has been requesting for some time.  Among these are a few that I am quite excited about as they should in theory make my job as an admin easier; built in WAN acceleration, support for tape libraries, a vSphere Web Client Plugin, and the ability to create backup copy jobs to support your basic Grandfather-Father-Son backup strategy without external help. Among the biggies are:

  • Built in WAN acceleration * – will be great for me, I’ll only need to take one backup of each VM a night now (didn’t like the rsync or xcopy methods).
  • Ability to take backups from storage snapshots * (as long as you have HP Storage devices)- According to Veeam, should be high performance, capable of near continuous data protection without impacting production performance
  • Plugin for the vSphere Web Client * – manage Veeam directly from within the vSphere Web Client
  • Self Service Recovery * – Let them eat cake!
  • Tape Library Support – Straight to tape from Veeam as long as it can directly see it.  This has been requested for a while
  • Virtual Labs for Hyper-V – Us VMware guys don’t get to have all the fun now, you can now sandbox and test backups in Hyper-V now too.
  • Parallel Processing of VMs and disks within VMs
  • Backup Copy Jobs – Built in ability to create a Grandfather-Father-Son policy on per VM and per Job basis.

* These items require the new Enterprise Plus licensing level.  While Veeam is currently giving existing customers free upgrades from Enterprise to Enterprise Plus, understand that taking the upgrade will make your support contract cost more.

There are a great deal of other new features, for more please take a look at their what’s new in v7 document.

I’ve got it installed myself and so far I am impressed.  The installation went very smoothly both on Windows Server 2008 and 2012, with a minor hiccup with the Enterprise Manager required components install requiring a reboot midway, Veeam didn’t know how to handle that so I had to cancel install, reboot, and then begin again. Along the way I learned that the Search Server (capability to search within your backup files for a give guest file) has now been built into the Enterprise Manager component, which is nice, especially if you remember to turn on the guest file system indexing setting in your jobs. 🙂

So What’s Missing?

While I am extremely happy with the obvious work that the guys at Veeam have put into this release, there are still things I wish they would get around to.  I would love to see some kind of capability in regards to physical servers, even if it is nothing more than file synchronization jobs.  Many if not most of us systems guys who manage a virtualized environment still have at least a couple physical boxes around that for one reason or another can’t or won’t be virtualized. In my case this includes a system that houses a 69 GB flat file database that is slow when virtualized no matter what we do as well as an assortment of SOHO domain controller/ file servers that because of their size and the number of people they support it doesn’t make sense to pay to setup them up virtually.  The other alternative is to manage some kind of “other” backup facility for these servers, which makes it a bit of a pain.

Further I see that the delete restore points of no longer managed VMs is still just a number of days thing, rather than having the option to turn it completely off. At no point should any backup software remove data from a backup chain without the backup admin expressly requesting the process to happen.

So What’s Next?Veeam Backup Infrastructure DiagramBecause of the capabilities the WAN Accelerator and Backup Copy Jobs now give me, I’m taking a look at completely restructuring the way that I manage my backups.  After reading documentation and working it out for myself the data flow should look something like shown to the right.  If you see any holes in what I’ve done please feel free to comment or let me know in other ways.

I’m also going to soon be working on moving the test environment to production, with the most noticeable change being the move my production backup infrastructure from Windows Storage Server 2008 to Windows Server 2012 Standard.  Why you may ask?  Server 2012 now include the ability to do volume level deduplication, something that when paired with Veeam’s already built in deduplication process should equal some pretty serious disk real estate savings.  As a test launch I’ve setup dedupe on a VM and copied approximately 250 GB of backup files over to it.  The result afterwards is Windows saved me about 10%, less than Veeam is claiming, but better than nothing.  I think when I throw some of the bigger jobs at it I will see that percentage go up.  Veeam has a good article with video about the process and I’ll have a blog on how to get Server 2012 deduplication up either here or over on 4sysops soon.