A Guide to Disk Imaging, Backup and Restore

March 19, 2012 by . 3 comments

So I finally get to write my own entry for the Super User Blog! Very excited to be here. I’ve been asked to write specifically because I’ve just started working as a developer for Macrium Reflect, a disk imaging solution for Windows-based PCs and super user features a lot of backup questions. So here goes, an as-close-to-canonical tour of disk imaging as I can write. I’ve tried not to be overly technical here, whilst giving you the knowledge to understand the essentials; however, it gets technical from time to time.

Why would I want disk imaging?

Ever lost all of your data? Our company founder did; that’s why he created the product we now work on. My personal backup strategy has always revolved around the idea of backing up copies of important files – I’ll back up /home or My Documents and I’ll be fine! Yes, to an extent I will not lose any of my data if I run that regularly; however, I have a lot of programs installed. An awful lot. Each with tweaked configurations. It’s fair to say that if my machine dies, I’ll spend a few weeks hunting through to find all those weird tweaks and config changes I’ve made.

The same was true for my current boss and as a contractor, the downtime really hurt. What he wanted was a simple way to be able to restore a working system from a backup and have everything – every program, every driver – work as it was. So, Macrium Reflect was born and a few years later it’s not quite so simple… although it does indeed restore disk images!

Don’t talk about backups, talk about recovery.

In a way this is going to be the theme of this blog entry. Joel has blogged on this topic, so we’ll do a quick executive summary: it doesn’t matter how painless, smooth and efficient your backups are, if recovering your data is not as easy, the time and frustration cost of an incident will be huge. You want easy recovery – or as easy as you can get.

What is a disk image, how does it work and how can I take one on a running system?

A disk image is exactly what it sounds like – a file storing an exact copy of your disk’s data. Actually, that’s not quite true: To understand this, we’re going to have to talk a little about file systems. A file system can be thought of as a filing system, a scheme for storing data on a disk. There are various ways and schemes for doing this, depending on the requirements of the end user. Now, the file system must not only write data to a disk, but also a certain amount of meta data such that when it comes to finding a particular file, the computer can. Actually, we can use this fact for an important purpose. The metadata allows us to pick out sectors on the disk which are in-use storing data we want to keep and ignore sectors marked as free. Which leads us to the first choice of the evening:

  • Active Sector Backups: This type of backup stores only the active sectors and filesystem metadata. When the image is written back to a disk, the data is written in the correct location on the disk – and the file system knows no different.
  • Full copies: These are bit-for-bit copies of the file system, which take up more space and are technically unnecessary. However, there is an important use-case for a full-copy backup: forensic examination of a disk.
Macrium Intelligent Sector Copy vs Full Disk

Macrium Intelligent Sector Copy vs Full Disk

So what constitutes a disk image really depends on how you’re backing it up – and which you choose depends on your storage capabilities, and trust in your software’s file system processing.

Now – live file systems. Well, the most obvious method for doing backups is to use the built-in Windows functionality, the Volume Shadow Copy Service. This is a Windows-provided snapshot system which provides a number of backup methods and is very flexible – see the msdn article.

It is, also, of course, possible to achieve this yourself. The technique for doing this is called copy-on-write; that is, any block that is written to is also duplicated to an extra copy first. When you come to read back the image, you can read the file system, or the store of the point-in-time data, depending on whether it has changed or not. Other operating systems, file systems etc. provide a variety of near-equivalent methods; the important point here being that it is safe to image a live hard disk by employing some fairly well known techniques.

Not always the full picture

We’ve alluded to the next topic with the copy on write idea above. If you take an image of a hard disk on the 1st of the month and then again on the 2nd of the month, you have two whole images – a bit of a waste of space if all that’s changed on that box is a few server logs. You might think, “Hey, we could just store the bits that have changed!” — You’d be right. There are two very similar strategies which use this “differential technique”. Both assume a full backup at some point in time.

  • Differential backups: These are the difference between the last full backup and the one just taken. The next time an image is taken, it is also the difference between the current state and the last full backup – so all images until the next full backup are the difference between that point in time and the full backup. To recover one of these backups, you need both the full backup and the differential backup. Clearly, these take less space than a full backup.
  • Incremental backups: These are like differential backups, but instead of always being the difference to the last full backup, they are the difference from the last incremental backup (or full backup, if none exists previously). These backups have the potential to take up the least space – but there’s a caveat – you need both the original full backup and each of the incremental backups taken to restore your system.

As you’d probably expect, performing either an incremental or differential backup is vastly quicker than a full backup – which can be a very important factor when you’re backing up every day.

Sounds easy! I’ll just go copy some partitions…

…wait right there. To recover your whole system and image your entire disk, you’re not just talking about partitions – you need to know about the partition scheme in use and any boot considerations this may entail.

In modern PCs, there are two types of partitioning scheme:

  • Master Boot Record disks, which can have at most four primary partitions (although this is worked around with an extended/logical partitions).
  • GUID Partitioning Scheme (GPT) disks. These can have (on Windows) 128 Partitions and are used with EFI systems for booting – they can not be booted from a MBR-based BIOS.

MBR is by far the most common scheme, but GPT/EFI systems are becoming more popular (and are used in modern Macs) as support now exists in 64-bit Windows systems.

Macrium Reflect showing MBR/GPT Disks

Macrium Reflect showing MBR/GPT Disks

But that’s not all. Older systems (XP and earlier – you probably have this if you have upgraded) use Cylinder, Head, Sector as a partition alignment scheme, whereas newer systems use a 1MB alignment scheme. It turns out a SSD design flaw means that CHS schemes may degrade disks far more quickly than the newer scheme. Your disk, when imaged, will be aligned as per the scheme it was created with (and upgrading Windows won’t have changed it, either!) so do make sure if deploying to an SSD you have software capable of re-aligning your partitions. More here.

Live rescue media schmedia, why do I want that?

Having backup software that runs whilst you work is great – it doesn’t interrupt you. But if your hard disk fails, you’re not going to be able to boot your system; when you pop that fresh disk in, it’ll be empty and you won’t be able to boot. Being able to boot and recover your system from live media is absolutely essential.

There are some considerations to bear in mind here. You need rescue media capable of recovering your system; ideally, you’ll have tested it. Rescue media generally comes in two flavours: Linux, and Windows Preinstallation Environment (“Windows Lite”). If you’re running some esoteric hardware for which one of these environments does not have drivers, you really do need to know beforehand; some software (such as ours!) can incorporate drivers into PE images.

Another consideration is access to your images. If you intend to use a network, you’ll need to check you can from this media; you’ll also need to check any external storage you wish to mount works.

So now we’re getting around thinking more about recovery, let’s do some more thinking.

Fixing bootloader errors and deploying to new hardware

So, now we’re all backed up with a great imaging scheme. We’ve got rescue media just in case and we know it works with our kit. Great! Then something goes wrong – no problem! We can handle this!

Except that our motherboard has failed and it’s 3 years old. They’re not available anywhere you can see and you can’t spend enough time at work on ebay to hunt down that spare one somebody’s selling and you can’t afford it now the bids are getting large anyway. Oh dear – this image we’ve got isn’t a perfect, working backup.

There is a way around this – we call it redeploy: the ability to insert drivers into a new install before you boot it. Our specific system, I am reliably informed, will also update the HAL drivers and CPU type when you move to a new system.

Another consideration is the ability to repair Windows bootloader settings on redeploy. An example I have worked on recently: On systems running PGP Whole Desktop Encryption, we can take images, but they are unencrypted (more on that in a second). We can then restore them as needed – except the PGP bootloader remained in the MBR (I’d only imaged the partitions I wanted, not the whole disk). Being able to fix problems in booting is something you may well need.

Disk Encryption

So, I’ve briefly touched disk encryption – now here’s how it works. Either your controller (hard disk) handles the crypto and issues a bios interrupt on boot to allow a password to be inserted (or token), in which case it is totally invisible to software, or software handles the crypto by inserting a driver just above the block layer stuff and below the file system. The effect is the same: when the system is running, imaging software will image the unencrypted data, as the encryption process is transparent to it. However, the live media will only be able to image the encrypted disk and may fail interpreting it. If you employ disk encryption, do investigate your compatibility options.

Extracting images, partial recovery

Restore Menu in WinPE Disk

Restore Menu in WinPE Disk

All this talk of recovering whole systems is great; however, there are times where you just want part of a system to be extracted – usually for user data. Or you might want to look at a specific image. Many backup programs offer you the ability to mount your recovery images and extract data from them; this is a good way to ensure your backup has worked – you can run

chkdsk X:

on drive X to verify the data is intact, on Windows systems.

So, that is all for a basic introduction to disk imaging and system backup. I’ll leave you with a quick backup checklist:

A recovery checklist

  • Recovery Hardware: can I recover to different hardware?
  • Recovery viability: can you afford to recover to when you need to, given your storage constraints for storing backups?
  • No-recoverable interval: how much data do you lose if you have to recover? How often are snapshots made?
  • Recovery testing: have you tested your recovery plan?
  • Recovery time: how long can you get back to the mission critical work of reading blogs?

Ninefingers is a software developer writing C, C++, stepping through crash dumps and disk images to create Macrium Reflect, the best disk imaging solution for Windows with a free edition that just works and paid editions with bar-none the best support in the industry. There are other products, but they all suck. Buy ours instead.

The last is the reason they don’t let me do marketing :(

Filed under Backup & Restore

3 Comments

Subscribe to comments with RSS.

  • Without doubt a very thorough account. Great to see somebody talk about something they really know a lot about and trying to articulate it for the ordinary person. I am very enthusiastic about the advantages of online backup but see the best backup of all – excuse me recovery (and you are quite right about that of course) would be the availability of a daily online backup of a system image. Clearly the size of an image file even for an average user could be quite daunting. Do you have any advice on how this could be done effectively? Or is this a question for post 2013?

    • ninefingers says:

      It’s ironic you should ask as my previous job was building a utility not unlike dropbox…

      Online backup is interesting and incredibly convenient, but it does have its challenges.

      The first is availability. If the backup software cannot access the storage from the rescue environment for whatever reason, be it because the environment is a proprietary one that needs special software to access or because of the more fundamental network connectivity issue, not being able to reliably get at your backup means you cannot restore when you need to.

      The second is data size, specifically the cost of getting that data over the network. When large streaming video or voice is played, UDP is the protocol used. The easiest way to explain this is to say the packets of data are fired at your browser several times just to make sure. There’s no effort to synchronise data at all – if one frame gets dropped well the video player will just work around that.

      You can’t do that with backup data. Fire it enough times to be sure it got there isn’t good enough unless the other end is verified, too. The alternative is to use a protocol that makes sure everything gets there – TCP. This has a performance overhead.

      So getting that data there is going to take a while in software terms. In hardware terms, you are at the mercy of the quality of connection between you and the other end. If you’re like me, it’s not great!

      Unfortunately, the bad news is that image sizes are going to get larger, not smaller, as disk sizes increase, so these factors are only going to get more relevant.

      The only way to mitigate this is to take a full backup first, then take incremental backups from there. That said, you still need to read the remote backup file. Even taking into account the fact that most home broadband connections are asynchronous and that these reads will happen much faster than the writes, that data still needs transferring and that still takes time.

      So right at this moment, I don’t think there’s an ideal easy solution to this problem. I’m aware some imaging products offer cloud based backup solutions and have probably looked at these issues before – you could always investigate. If your connection is good enough, you won’t notice the performance isssues (they apply to local gigabit networks too, but they’re so fast relatively speaking you get good performance from them).

      That said, I think to an extent computing might be moving to a solution that does solve the problem. Many devices (tablets, smartphones etc) store data in the cloud in perpetuity. Even browsers can sync with the cloud. In these cases, the important data is backed up anyway. The user has very little locally to image – should they lose their device or it fail, they simply get another one or access their data through another device.

      Given you can do this on the PC too (browser sync, dropbox et al) maybe its fair to say that disk imaging has a niche for those who need entire systems backup, and cloud provides the general backup for those who want to back up user files and not whole PCs?

      I’m no business analyst. Only time will tell if the cloud fizzles out, but it’s interesting to think about!

  • user1147688 says:

    Thank you, that was a very nice write up!

    However, I cannot avoid thinking it it sound a like a promotional ad for Macrium. But that is ok, if you can live up to your promises. Having recently tried several different backup software, I am still not satisfied with what they offer. Why? Simply because they all try to do too many things. Thus they often are not able to doing any one thing intuitively (easy), properly, fast. They all try to satisfy as many different scenarios as possible, which for all practical reasons, they are unable to do. Instead you get a mediocre solution that is hard and time consuming to maintain. And it certainly isn’t free.

    What am I talking about? I suppose that in these times of cheap data storage, there is little need to physically try to recover a hard disk and even less need to compress whatever you have on it. That is, if you have made your regular backups! So what I believe most people are looking for, is a small and fast backup solution, that always works, and is easy to access. This is where almost all “professional” backup solutions fails. The typical scenario should be the following.

    Backup: (a) Select a disk or partition. (b) Make an image of it to an external HD/NAS, in the fastest way possible. Tell the user exactly what’s going on while it’s happening (i.e. time, files used and remaining). Provide sensible error, should they occur. There is no need for forensic cloning or other extra features, just copy what’s used. That’s it!

    Recovery: (c) Attach external HD/NAS (d) Use your systems file browser (e.g. Explorer in windows) browse to image you want to recover from. (e) Either recover whole disk/partition or just copy the files/directory you want to recover. (g) Done!

    The ONLY software I have seen so far, that is capable of doing anything near that, is a homemade and single executable, 352 KB (!) software called “Drive Snapshot” made by Tom Ehlert. The only thing missing from making this a huge success, is a slightly modernized UI and an explorer plugin like that used in EaseUS (~300+ MB), so that you can browse the files on the image.

    On the contrary, although your (Macrium) has a relatively easy to use UI, it is far from minimalistic and it is lacking in many fundamental respects, like: easy file browsing of backedup image, correctly estimating time remaining, providing human intelligible and clear error codes, and showing the user what is being done. Everyone hates the lying green flashing windows progress bar and Macronium have two of them!

  • Leave a comment

    Log in
    with Stack Exchange
    or