New in Veeam v11: Hardened Repository – Immutable backups [Part 1]

This post is about a great new feature in Veeam Backup&Replication v11: Hardened Repository. This means immutable backups. It will massively increase security on repositories. With immutability enabled, root access is needed to delete or change backup files. Read here how it (just) works and how it behaves.

This post is very detailed. Therefore I split it into two part:

• Part 1 (this post) concerned with these topics:
  • What is immutability is about
  • Requirements
  • Setup
  • How it works
  • Enhance Security
• Part 2 is about
  • Behavior

If you are just interested in highlights, jump to the end. There you can find a summary.

Disclaimer: All testings are made on the latest beta version of v11. Details are subject to change.

What is immutability is about

Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.

Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!

So Immutability is a further, very important feature for high secure backup repositories. Read more about securing Linux repositories later in this post.

Requirements

What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes. Important note: When using hardened repository, Linux server cannot be used as backup proxy! If you need additional proxy features, you need an additional server.

What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.

What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.
[Update: Veeam is planning to fully support fast cloning with Ubuntu LTS 18.04, RHEL/CentOS 8.2 or later, SLES 15 SP2 and Debian 10 as well]

Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.

Setup

Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.

Settings are easy to understand:

By turning on this feature, you can choose for how many days files should be immutable. At least for beta version: minimum is 7 days.

[update, 7.10.2021] read my post about extending immutability expiration date in v11 and v11a.

During the wizard you get a question including a call for action. Change the owner of the repository directory by running chown -R user:group /path

How it works

The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.

To see file attributes, including immutable flag, run: lsattr filename in Linux shell. Sample output see here:

Note, flags are removed from a whole backup chain, not just a single file.

The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:

Notice: this service has no connection to the network, so it cannot be compromised remotely!

Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.

Enhance Security

It makes perfect sense to harden the Linux system that runs the repository. Here are some additional considerations to secure Linux repository server:

• User for Veeam transport service does not need to be a privileged user. So do not use one.
• By default, root user in Ubuntu cannot be used for login. This should not be changed. Use sudo command instead to run privileged commands.
• To make it harder for hackers to login to OS, use certificate based authentication. This can be done by:
  1. Create director .ssh in users home director
  2. Running ssh-keygen
  3. Make public key to authorized key by running: cat id_rsa.pub >> authorized_keys
  4. Use the private key to move/copy to clients like putty. You can also use this authentication to create a Linux user in Veeam.
  5. Also consider to disable password based logins via ssh. More about this, read this post.
• Firewall should of course be enabled. Open just necessary ports. Transport service takes care, necessary port 6162 is opened.
  Use Veeam Ports Tool (after the release of v11) to check for needed ports.
• In fact, ssh is just necessary for deployment and upgrade. As soon this happened, sshd can be stopped. So disable ssh. This makes it really hard to access host OS remotely!
• Think about disabling IPMI interfaces, like HPE iLO or Dell iDRAC. Because this opens remote access to server console.
• You can also remove to elevate to root permission for the transport service user (su respectively sudo)
• Date/time is crucial (see detail in Part 2) on the Linux repository server. Therefore make sure to have the correct one. Use NTP to keep time current.

Summary

Hardened Repository is a new feature in Veeam Backup&Replication v11. With it, backups are immutable for a specified period of time. Without root access to the Linux server hosting the repository, backups cannot be deleted.

• Requirements are:
  • Veeam Backup&Replication v11.
  • Linux repository server – preferred: Ubuntu.
  • Backup chain: forward incremental with periodic synthetic/active fulls.
• When using XFS filesystem, reflink provides Fast Cloning benefits.

Linux Security

• Veeam transport agent is running with non-privileged user.
• ssh is not necessary and can be disabled.
• Just one network port open for communication: TCP/6162.

Notes

• New with v11 is also that transport service is permanently running. In previous versions, service just runs during running jobs.
• In v11 only image level backup are supported.
• To set immutable flag manually, run chattr +i filename. Remove flag by running chattr -i filename.
• How to analyse saved space with XFS reflink, you find here.
• Read my post about extending immutability expiration date in v11 and v11a.