New in Veeam v11: Hardened Repository - Immutable backups [Part 2]

New in Veeam v11: Hardened Repository – Immutable backups [Part 2]

Posted: November 24, 2020/Under: Veeam/By: vNote42

This post is about a new feature in Veeam Backup&Replication v11: Hardened Repository. This means immutable backups. It will massively increase security on repositories. Read here how it behaves.

This post is very detailed. Therefore I split it into two part:

Part 1 concerned with these topics:
Part 2 (this post) is about
- Behavior

Behavior

This section give answers to the following questions:

What happens, when…

… removing immutable restore points?
Task starts and ends up with errors. No restore point is removed.

Warning when trying to remove immutable files

… hacker changes date/time on Veeam server to remove files with next backup run?
Excellent question! But it will not work! Configured period of time is checked at Linux server, not at Veeam B&R Server!
[little detail: system time, not hwclock]
So a job running one year in the future will create new files, it cannot delete old ones.

… enabling feature at existing repository, not compatible with hardened repository?
Veeam checks if there are backup jobs configured with an incompatible backup chain. If found, error is shown:

… changing a backup job writing to immutable repository?
When backup chain is changed to an incompatible one, error is shown:

[Immutability repository requires configured forward incremental with periodic full backups job settings and without rollbacks.]

… days for immutability is higher than the number of restore points?
Restore points are kept to the end of the period, defined for immutability. Notice that there is an information message in job log:

… immutability turned on on repository containing restore points?
In this case immutable flag is set on all files, according immutability period.

… immutability turned off on repository containing immutable backups?
In this case, immutable flags are kept. That makes perfect sense. Otherwise a hacker with backup administrator permissions could disable hardened repository to delete backup files. Notice: new backups are not immutable after disabling the feature.

… trying to remove restore points in GUI on files with immutable flag manually removed?
Interestingly files will not be deleted! Probably because Veeam knows, files should be protected. There will be more details on that after the release of v11.

Cannot remove files even immutable flag is manually removed

… use immutable repository as target for copy job without GFS?
Backup copy jobs requires GFS when copied to immutable repository. If no GFS is set up, error appears.

[Immutability repository requires configured GFS for backup copy job.]

… trying to add a non-immutable repository on a Linux server hosting a immutable repository?

Currently mixing immutable with non-immutable repositories on the same Linux server is not supported. So you get an error:

[There is at least one another repository with enabled immutability on server. Concurrent using Linux server as immutable and mutable repository is not allowed.]

… deploy proxy role on a Linux server with hardened repository?

Does not work in v11. When hardened repository is enabled, server cannot be backup proxy. You get an error when trying:

[Specified Linux server is already used by immutable repository repository]

Notes

If you want to know the behavior of some other situations, leave a message I will try to test it.

14 responses to “New in Veeam v11: Hardened Repository – Immutable backups [Part 2]”

bg_ranken says:

January 4, 2021 at 10:51 pm

Are there any changes to the backup copy job structure in V11 that allow them to be used for these hardened repositories. Currently the only types of jobs that allow for incremental with active/synthetic full would be normal backup job, but ideally it would make sense to allow this same type of behavior on the backup copy jobs.

Reply
- vNote42 says:
  
  January 5, 2021 at 8:45 am
  
  Backup copy jobs are always forward incremental. The reason for GFS restore points will be to be able to just delete files without changing any of them. This would be necessary if you run forward incremental without any additional fulls.
  
  Reply
subhasis says:

February 19, 2021 at 9:38 am

Can we leverage S3 immutable storage along with the hardened local repositories for long term retention. The incremental snapshots will be persisted into hardened repo for a day or so then archive into S3 storage which is immutable for long term retention and has a storage level hard immutability.

If yes, when you so restore , the copies will be restored from S3 or from hardened repo

Reply
- vNote42 says:
  
  March 2, 2021 at 9:06 pm
  
  Subhasis, sorry for my late answer!
  I did not test hardened repository with capacity tier in a scale-out repo. But now, after the launch of v11, you can see in helpcenter how it works:
  
  https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110
  
  In short: Because immutable file cannot be deleted, Veeam is copying these files to the capacity tier instead of moving it. Only after immutable period is over, files can be deleted.
  
  Reply
Tyler Pittman says:

March 23, 2021 at 10:58 pm

Hi, I’m labbing this out, and while I can see that the VBKs and VIBs are immutable and can’t be deleted by Veeam’s service account, the VBM file is not immutable, which makes sense, as it needs to be updated every time a new backup is made. The VBM file defines your backup chain though – if the VBM file is lost or encrypted, aren’t your backups toast?

Reply
- vNote42 says:
  
  March 25, 2021 at 9:34 pm
  
  Hi Tyler! You are right, it makes perfect sense, VBM file is not marked as immutable.
  Don’t be afraid of your backup files, when VBM file is lost! You can still restore data from it. You can still use Veeam Extract Utility without VBM file. To get restore points back in VBR console you have to import it, rather then rescan the repository.
  
  Reply
Zed says:

March 26, 2021 at 3:07 pm

I tried with a NAS backup, but it doesn’t seem to work.
Files can be deleted
have you tried?

Reply
- vNote42 says:
  
  March 26, 2021 at 3:12 pm
  
  Zed, NAS backups are not supported for immutability:
  https://helpcenter.veeam.com/docs/backup/vsphere/hardened_repository.html?ver=110.
  Hardened Repository can be use, but flag will not be set. So what you see is by design.
  
  Reply
  - Zed says:
    
    March 26, 2021 at 5:06 pm
    
    It’s a shame !! NAS can be a good target for ransomware !
    Do you know why, I mean technically?
    I think it’s because of the incremental forever mode?
    
    Reply
    - vNote42 says:
      
      March 26, 2021 at 6:29 pm
      
      For sure it is because of the incremental structure of the backup files. You can also have no immutable backups with incremental forever VM jobs.
      
      Reply
Zane Allyn says:

April 23, 2021 at 10:27 pm

Hello ZED

Easy answer. This feature uses the Extended Attributes on Linux, and these are not supported on the NFS mounted file system.

Reply
- vNote42 says:
  
  April 24, 2021 at 11:03 am
  
  Hardened Repository does not use immutable flags on source-date, instead it uses it on Linux Repository. Therefore it does not matter, if NFS is able to flag files as immutable. You can also backup Windows machines on hardened repo. They do not support this flag at all.
  
  Reply
Philipp says:

January 5, 2022 at 2:39 pm

Hi. Great post about a great feature 😉
But how can be “monitored”, that the backup job itself was not modified by a hacker? Any idea?

Reply
- vNote42 says:
  
  January 5, 2022 at 2:50 pm
  
  Great question, Philipp!
  Currently I am preparing at least one posts about this. One option is to use VeeamONE, another is to use PowerShell.
  Regards
  Wolfgang
  
  Reply