Why Backups Fail and How to Know Before You Need Them

By Chris Rosa, IT Guru

It is a well-documented fact that things can and do happen to data storage devices: they wear out, overheat, suffer from manufacturer defects, mechanical failures, go through disasters like fires or floods, burn out from a power surge or any number of possibilities.
Data storage is the life-blood of a company and if a data loss incident occurs, an organization could suffer severe financial and reputational damage and could even end up going out of business.
Regular data backups are a common task, and improvements in software and cloud storage over the years have made the process less painful. But what happens when a device fails and you find that the restored data from the backup doesn’t contain the critical files you need?
Financial databases, accounting records, shipping data, customer data, credit card information, intellectual property and any number of irreplaceable digital files may be lost forever—all because the backup was misconfigured, untested or unverified.

When Good Backups Go Bad

Having a good backup strategy plus the appropriate hardware and software is just part of a successful formula. There are too many variables that can prevent backups from completing and being available when needed. If you don’t test backups regularly, you may find that you’re not able to restore data when a data loss scenario occurs – even when you thought you were prepared.
Bad backups happen far more often than you’d think and there are a number of reasons why:

• Applications that do not work well with the backup software
• Improperly stored backup media, causing physical damage to the backups
• Overly complex backup applications with settings that don’t work as anticipated
• Hardware issues – all devices fail eventually, including backup devices
• If it’s a physical backup, what do you do when the storage is full?
• If it’s a cloud backup, who’s being notified of downtime, backup failures or API updates? Who has access?

Offline testing on a regular basis of critical backups will ensure that everything is backing up properly and can be properly restored, and is functional when a device failure occurs.

How to Test Backups

Use a Spare

Oftentimes, companies don’t check their backups because they don’t have spare servers or space to restore. Having idle hardware at the ready, even if it’s slower or less optimal, is a good idea to have in case you need to replace equipment when a server or storage failure happens.
This same spare equipment is perfect for testing a full restore of backup data. If no tests are conducted, there may be an issue with the backup that will prevent a proper restore and the company won’t be aware of the issue until it’s too late, leaving them open to the risk of losing it all when they need it most.
It’s important to have a full system, physical or virtualized, where you can restore a backup when your working system goes offline. In the case of a physical system, the spare server should be identical to the working server and be configured with the same version of the operating system and backup software whenever possible. The spare server used to test or restore a virtualized system does not need to match, but should have a greater capacity than the amount of data in the original system.
Running multiple servers requires that additional server setups also be available. While it may be difficult to justify the expense of additional servers and/or VM resources, it will save a ton of time when your working system goes offline, plus it’ll save on the expense of business downtime that would occur if parts or hardware need to be ordered.
Bonus: You can quickly transfer data to your offline backup servers if you ever suspect something is wrong with the hardware with one of your working servers.

Verify the Actual Files

As a data recovery service provider, the most common mistake we see from our customers is that they don’t actually verify that their backup data is functional. They may review backup log files that reveal that a backup function completed, or look at what folders or directories appear to have been backed up but that’s about all. Then, when the inevitable happens and they try to restore from the backup, some or all of the files needed turn out to be missing or corrupt.
Without opening the most important files and attempting to use them in some manner, there is no way to know if they may be partially copied, corrupt or otherwise unusable.
When checking your backups, it’s important to actually open files – at least the most important files – and make sure they work. It requires some extra time, but opening and testing the most important files can save a business in the event of system failure or other data loss occurrence.

How Often Should You Test Your Backups?

Each situation has its own threshold for when a backup should be made and tested. Some companies only have a few gigabytes of data added each month or year. Others have terabytes of data created that might need to be restored immediately if there is a failure.
How often backups should be made really depends on three criteria:

• The sensitivity of your data
• The amount of data
• How frequently your data changes

Make a schedule for a full restore test. Maybe it’s once a year, maybe it’s quarterly. Whatever you decide, put the date on the calendar, and do it.

3, 2, 1 Backup Strategy

No matter what type of device you use to back up your data or how often you test your backups, there is no way to guarantee against data loss. The best way to truly protect your data is to have multiple copies and test all of them. For this reason, we always recommend following the 3, 2, 1 rule:

• 3 copies of your data at all times – one working copy and two backups
• 2 copies on local devices
• 1 copy kept off-site or in the cloud

In a real-world scenario, this could look like one server that is in use, a second server that the first server is backed up to and a cloud solution to store a third complete copy.
If you are dealing with a large amount of data, many cloud backup companies such as Backblaze offer business class accounts where they send you physical storage to copy your data to and then send it back to their data center where it is migrated to their cloud.
For more information about how to find the best backup solution for your situation, read Best Backup Solutions.

Data Recovery as a Last Resort

Third-party data recovery service should always be included in a backup plan as a last resort. Just in case something goes wrong with your backup system, it’s a good idea to have already vetted and chosen a data recovery company that you trust with your business data. Have their phone number written into your plan.
This Cybersecurity Report explains why vetting a data recovery company is important for data security and provides a good checklist to help.