New in Veeam v11: Hardened Repository – Immutable backups [Part 1]
This post is about a great new feature in Veeam Backup&Replication v11: Hardened Repository. This means immutable backups. It will massively increase security on repositories. With immutability enabled, root access is needed to delete or change backup files. Read here how it (just) works and how it behaves.
This post is very detailed. Therefore I split it into two part:
- Part 1 (this post) concerned with these topics:
- Part 2 is about
If you are just interested in highlights, jump to the end. There you can find a summary.
Disclaimer: All testings are made on the latest beta version of v11. Details are subject to change.
What is immutability is about
Immutability in this context means, a backup file cannot be changed or deleted without having root access within hosing Linux OS. So, even the backup administrator is not able to delete backups on such a repository.
Why is this important? Think about ransomware. This software is that smart these days, it is able to recognize backup systems. It can trigger tasks like deletion of backup files. But when they are immutable, this cannot be done!
So Immutability is a further, very important feature for high secure backup repositories. Read more about securing Linux repositories later in this post.
What is needed to get immutable backups? First of all: Veeam Backup&Replication v11. This is the first version supporting hardened repositories. Secondly a Linux server hosting repository volumes. Important note: When using hardened repository, Linux server cannot be used as backup proxy! If you need additional proxy features, you need an additional server.
What about the filesystem? Veeam is using immutable flag. So every filesystem supporting this flag can be used. These are pretty much all. Veeam supports reflink/Fast Clone on XFS. Because of this XFS is the recommended filesystem.
What about the distribution? At the moment of writing I had no information about this. I think this feature will not constrain the selection of supported Linux distributions. When using XFS, we get a fist choice: Ubuntu 20.04 LTS (long-term support). Because: (1) Ubuntu is supported by Veeam. (2) 20.04 uses kernel version of 5.4. This version seems to provide highest quality of reflink, tested by Veeam.
[Update: Veeam is planning to fully support fast cloning with Ubuntu LTS 18.04, RHEL/CentOS 8.2 or later, SLES 15 SP2 and Debian 10 as well]
Thirdly: Backup chains must be compatible with immutable files. What does this mean? Because files cannot be changed, the backup chain only can create new files without changing any of the existing. Only forward incremental with periodic synthetic or active fulls fulfill this requirement. For backup copy jobs, GFS settings are required.
Immutable backups are enabled on repository level. Either at creating the repository. Or for a existing repository. How to setup Linux as repository server I will covered in another post.
Settings are easy to understand:
By turning on this feature, you can choose for how many days files should be immutable. At least for beta version: minimum is 7 days.
[update, 7.10.2021] read my post about extending immutability expiration date in v11 and v11a.
During the wizard you get a question including a call for action. Change the owner of the repository directory by running
chown -R user:group /path
How it works
The beauty of this feature is the use of native filesystem features. In Linux each file can have an attribute i. When this is set, file cannot be changed or removed. When Veeam creates backup files, this flag is set. After the entered period of immutability, flag is removed and file can be deleted.
To see file attributes, including immutable flag, run:
lsattr filename in Linux shell. Sample output see here:
Note, flags are removed from a whole backup chain, not just a single file.
The question may arise how the flag is set in Linux. Because, when the specified Linux user gets privileged access to add or remove the flag, this could be used by a hacker to get access to these files as well. Right, BUT: flag is not set by this user. Instead it is set by root. This can be done by running a service with root access: veeamimmureposvc:
Notice: this service has no connection to the network, so it cannot be compromised remotely!
Apropos network: What ports are being used? Also new in v11 is that just one port is used to communicate with repository host: TCP/6162. During a backup other ports can be opened on demand.
It makes perfect sense to harden the Linux system that runs the repository. Here are some additional considerations to secure Linux repository server:
- User for Veeam transport service does not need to be a privileged user. So do not use one.
- By default, root user in Ubuntu cannot be used for login. This should not be changed. Use
sudocommand instead to run privileged commands.
- To make it harder for hackers to login to OS, use certificate based authentication. This can be done by:
- Create director
.sshin users home director
- Make public key to authorized key by running:
cat id_rsa.pub >> authorized_keys
- Use the private key to move/copy to clients like putty. You can also use this authentication to create a Linux user in Veeam.
- Also consider to disable password based logins via ssh. More about this, read this post.
- Create director
- Firewall should of course be enabled. Open just necessary ports. Transport service takes care, necessary port 6162 is opened.
Use Veeam Ports Tool (after the release of v11) to check for needed ports.
- In fact, ssh is just necessary for deployment and upgrade. As soon this happened, sshd can be stopped. So disable ssh. This makes it really hard to access host OS remotely!
- Think about disabling IPMI interfaces, like HPE iLO or Dell iDRAC. Because this opens remote access to server console.
- You can also remove to elevate to root permission for the transport service user (
- Date/time is crucial (see detail in Part 2) on the Linux repository server. Therefore make sure to have the correct one. Use NTP to keep time current.
Hardened Repository is a new feature in Veeam Backup&Replication v11. With it, backups are immutable for a specified period of time. Without root access to the Linux server hosting the repository, backups cannot be deleted.
- Requirements are:
- Veeam Backup&Replication v11.
- Linux repository server – preferred: Ubuntu.
- Backup chain: forward incremental with periodic synthetic/active fulls.
- When using XFS filesystem, reflink provides Fast Cloning benefits.
- Veeam transport agent is running with non-privileged user.
- ssh is not necessary and can be disabled.
- Just one network port open for communication: TCP/6162.
- New with v11 is also that transport service is permanently running. In previous versions, service just runs during running jobs.
- In v11 only image level backup are supported.
- To set immutable flag manually, run
chattr +i filename. Remove flag by running
chattr -i filename.
- How to analyse saved space with XFS reflink, you find here.
- Read my post about extending immutability expiration date in v11 and v11a.
I would love to know what system you set this up on. I need this environment when Veeam 11 roles out!
I would like raid and a ton of raided storage.
Sorry for my late response! I used a fully virtualized environment. Nothing for production!
Hy , I have a question … if I configure 7 day of retention in linux hardened repository… made a backup job … after one day I change the policy with a reconfiguration of the linux repository .. the vbk files saved before the second change continue to have the immutability sign signed at 7 days or changed to the second setting ?
Sorry for the late response!
When you higher immutable interval after backup ran, Veeam will also higher the immutability to the new interval of all files within the active backup chain – after the next backup ran. No other restore point will be changed. If you shorten the interval, just new backup files will be created using the new interval. Existing files will not be changed.
I hope I have expressed myself clearly?
you wrote, that private/public-key authentication can also be used in Veeam; but it is not useable with “Single-use credentials for hardened repository”, so public key is saved somewhere in veeam, correct? What will be “more safe”? 🙂
private/public key can be used for Linux authentication generally in Veeam. Single-use credentials is especially for Hardened Repository. My recommendation is to go with Single-use credentials for Hardened Repos, it is more safe.
That was also my impression… thanks for clarification.
When using a hardened repository/linux repository, the server itself is shown under “Files”; so a hacker could browse through the filesystem of immuteable storage. Could that be a security issue? (I know, that when a hacked already gained access to the backupserver, the disaster already happened)
What does you mean by “shown under files”? Basically a hacker cannot delete any files in a Hardened Repo even he/she is administrator on the backup server. This is only possible with root-access within the Linux system that hosts the repository. This would be the disaster.
Within the VBR-Console under “Files” you can see all your Windows and Linux-Servers. There you can browse through the Linux filesystem. I’m not sure, if that could be a security issue. And currently I am not sure, which protocol (ssh, veeam transport) the browsing technology uses…
Sorry for the late answer! You are right, you can browse OS-files there. For communication Veeam transport service is used. Therefore (if implemented correctly) browsing is done by an un-privileged user. So it is not possible to delete any files this user has no right for. This includes all immutable files of course. But I agree with your concerns. Would be better, this browsing wouldn’t be available at all.
Hi and thanks for all the information. I am just not sure if using NTP is not a security problem. What if an attacker can change the NTP server’s time? This issue is considered a problem in other discussions and it is suggested that one uses e.g. GPS based time synch or the internal server’s HW clock. What do you think?
Using NTP for clock synchronization on Hardened repo server is a security risk. A hacker could change time on the NTP server or even could act as the NTP for the OS. You can use a GPS or radio controlled clock dongle for time synchronization if you want to keep time accurate to sub-second. Or – the solution I personally prefer – keep servers HW time running as it is. In my experience it is “accurate enough”. It is also the safest solution.