Freeing space in Veeam Hardened Repository – what works, what doesn’t

Veeam Hardened Repository makes it rather hard for a hacker to delete your company’s backup restore points. Therefore we love it! But what can we do for freeing space in Veeam Hardened Repository? This can be necessary if repository is almost full. I will show in this post what works and what doesn’t

That works

Manually delete restore points

When Repository isn’t that full, it can help to remove some large, old restore points. As for other repositories too, this must be done at the hosting server. For a Hardened Repository you need privileged access (sodu/su) to the system. Perform these steps to remove a single restore point:

1. Find the directory the restore point is located.
   This can be tricky in for a ScaleOut Backup Repository (SOBR). You can use the Copy path button in properties of backups.
   
   
   
2. Remove immutability-flag of the file.
   For this step you need root access. Fist check current state by running lsattr.
   
   
   
   Remove flag by running chattr -i. Don’t forget command sudo if necessary.
   
   
3. Delete file.
4. Rescan Repository.
   
   
   
5. Check properties of backups.
   
   
   
   Here you will see a red cross at the restore point(s) you deleted. Use the Forget function to remove the restore point in the database too.
   
   

Although this works well it is very time consuming for many files in many backups.

Manually delete complete backup sets

This option is very similar to the deletion of a singe or multiple restore points. Only difference is that you delete all restore points here! Therefore the steps are also similar:

1. Find the directory the restore points are located.
2. Remove immutability-flag of all files.
   Here you have to remove i-flag from each file in the directory. If there are that many files in the directory chattr -i * is not working, you can use a loop instead: for i in *; do sudo chattr -i $i; done. Do not forget to delete hidden lock file too: .veeam.nn.lock. Show it using ls .v*.
3. Delete files and/or directory.
4. Remove backup set from configuration.
   
   
   

That doesn’t work

Some of the following is just not working. Some others can be harmful to your environment. So please be very careful if you decide to play around with them!

Decrease immutability interval on Repository

When a restore point is created on a Hardened Repository, the information until which date the file is immutable is stored there too. The same applies to GFS restore points. These are immutable as long as they are stored – according to retention definition. When changing immutability interval at repository level, just new restore points will get new interval accordingly. No other restore point will be changed. Normally this will not help to free some space, because then you wish to delete old data.

So, changing the interval does not delete existing files earlier than they would have been deleted originally.

Reduce number of restore points in job definition

This would be the solution for non-Hardened Repositories – for example ReFS based datastores. With immutable backups it is not that easy. As already mentioned, restore points get set end of immutability when they are created. This date will not be altered when retention definition in job get changed. Furthermore immutable restore points will not be deleted by retention!

There is a way to increase and decrease immutable interval for specific restore points. But it is not possible to reduce the interval so that it is smaller than it was originally.

Reduce immutability interval and number of restore points

Isn’t this the solution for Hardened Repositories? Neither immutability nor restore point retention would avoid deletion. Sorry, no. Existing restore points stay immutable even you change retention or immutability settings afterwards.

Reduce GFS restore points in job definition

If you want to free space on a repository, you probably want to delete old data. So the oldest GFS restore points would be okay to get rid of. The idea to decrease the number of GFS restore points or even deselect some intervals (weekly, monthly, yearly) would work for non-Hardened Repositories. Again, saved immutability period for GFS restore points prevents early deletion.

Play with date/time on Hardened Repository

Now its getting dirty! Yes, with Hardened Repository, date and time on the Linux system is also used to check if restore points have reached the end of their immutability period. Therefore it is not possible to remove a whole backup set in the VBR console.

The idea could arise to turn date forward on the repository Linux OS. But it is not that simple. It is not just the Linux’s date and time that counts here. For restore point creation, VBR server safes data about immutability in database, but repository host uses its own date for saving metadata. So even with a date in the future, filename will show current date (comes from VBR) and immutability end-date as defined. But repository server shows different date.

In the screenshot, the date of the repository server is one year in the future.

What could work is if you change date of VBR server too. But is something I would strongly advise against! If you do not know what you are doing, you will get into a hell of a mess!

Notes

• Get your trial of Veeam Backup&Replication!
• Read more details of Hardened Repository in my blog post.
• Read more about behavior of Hardened Repository in my blog post.