Various backup failures in Veeam B&R

Posted: April 5, 2023/Under: Veeam/By: vNote42

Some time ago a customer was suffering from various backup failures in Veeam Backup & Replication. After some time – about two weeks – the backup seems to become more and more unstable. First there were isolated errors. As time went on, more and more errors occurred.

Errors that occur

Some of these errors were:

[Error: Only one usage of each socket address (protocol/netv/ork address/port) is normally permitted]

[Error: Backup server has lost connection to backup repository]

Especially the second error we saw quite often in this environment. The error can occur occasionally when repository – mainly at Windows repositories – is in extreme use in terms of disk IOs. We also suspected this here, but could not find any evidence.

Environment

Veeam Backup & Replication (VBR) v11 is running in this certain environment. There are various proxy hosts running Windows and Linux systems.

Troubleshooting

We have done a lot to find the cause of these problems. For a long time we were not very successful. Veeam support increased the memory allocation of VBR server. But after a few weeks, problems started again.

Fortunately, a colleague at the customer had a great idea. He looked at the open network connections on the VBR server. A huge amount of connections were still open. A quick script was implemented that logged the counter for open connections. This number was increasing over time. It reached the default Windows connection limit. Without knowing the reason, the customer increases the limit in Windows.

TCP/IP and NBT configuration parameters for Windows XP – Windows Client | Microsoft Learn

With this change, backup was running without problems for a few weeks longer.

Finally Veeam support found the root cause and provided a patched version of VeeamGuestHelperCtrl.exe. This binary was found by using this PowerShell command:

Get-NetTCPConnection | Group-Object -Property State, OwningProcess | Select -Property Count, Name, @{Name="ProcessName";Expression={(Get-Process -PID ($_.Name.Split(',')[-1].Trim(' '))).Name}}, Group | Sort Count -Descending | Select-Object -First 10

Long story short

If your backup is suffering from very strange errors which do not reveal a clear cause, check open network connections. If counters are high, open a Veeam case and ask for the patched version.