How do I place a bootable ISO on a USB drive?

(Question ID 66948)

This is a popular question with many similar or overlapping questions. Browsing the various linked and related questions I noticed the information on this subject is limited and fragmented. Most answers are simple software recommendations and do not touch upon any underlying issues. This article is meant to help fellow superusers appreciate the assumptions we take for granted and serve as a springboard for those that need to dig deeper to solve their problem.

Most people who are looking to place an ISO on a USB drive want to put any ISO on a USB drive and make the drive bootable (also known as making a live USB or installer USB from a binary image file). Examples include:

• Windows installer / Windows recovery
• Some other rescue media needed to restore a backup (e.g. Macrium reflect)
• Linux live image, possibly persistent
• A full OS install onto a USB
• does not include useless ISOs, e.g. non-data like those filled with music or video.

This is not necessarily the same as:

• putting an ISO on a bootable USB drive
• copying an ISO file or its filesystem contents onto a formatted USB drive

The problem we solve with today’s post is: “Given an ISO and a non-bootable USB drive, how do I create a bootable USB?”

What is an ISO anyway?

In the broadest sense it is a binary image (a.k.a. carbon copy, snapshot or raw data backup) of the storage sectors of a block-device such as a DVD, USB drive, micro-sd card or HDD. The ISO format has no standard (ironically), so there is no pre-defined header (if any), nor any restriction on what filesystems may be contained. These image files are not related to graphical/visual image files like JPEGs. But in common usage an ISO is expected to consist of a single standard filesystem, like the ISO 9660 filesystem. Here is another good definition (from MagicISO website): 

Going by the more restrictive definition, an “ISO” is created by copying an entire disc, from sector 0 to the end, into a file. […] it isn’t possible to store anything but a single data track in this fashion. Audio tracks, mixed-mode discs, CD+G, multisession, and other fancy formats can’t be represented. [this specifically refers to discs with an ISO 9660 filesystem, as a convention]

In case you’re wondering about tracks, they’re basically partitions–so obviously a single filesystem can not have partitions. Of course an image can contain multiple tracks/partitions, but if we refer to such an image as an ISO we are using the loose definition (and probably making some developer angry). So an ISO can be any data image, but by convention we mean the entire user data* from an optical disk, which is commonly formatted with a single filesystem. So your ISO will probably conform to this more restrictive definition.

The name originated from CD images that commonly used ISO 9660 to store data. Nowadays most uses assume an ISO refers to an image from an optical disk like CD or DVD (even though in DVD the UDF filesystem is more common) . Even tools that can create an ISO from a HDD folder probably implicitly convert to an optical media filesystem like ISO 9660. For this reason it may be better for software to use a different extension like “.IMG” to denote non-optical image files.

*With CDs a raw sector is 2352 bytes, but only 2048 bytes are used for data in formatted storage. The rest is used for error correction related stuff. This is why you might see corrupted files if you extract a true raw image from a CD and try to mount it–files are injected with junk bits. DVDs don’t make this distinction, their sectors only have 2048 bytes of user data. So in the case of CDs with filesystems an ISO is normally not a true sector-by-sector carbon copy.

These days it’s not altogether uncommon to see messages about “making an ISO” of an audio CD, which makes no sense at all. (MagicISO FAQ)

Aha, a copy ‘sectors’ from a ‘block device’…

A block device is just jargon for storage media, or ‘drive’ if you like. From a programmer’s point of view storage is a one dimensional array (row/line) of bits. But at the hardware level you can’t address each bit individually. The smallest ‘word’ size for reading and writing is called the sector size. Filesystems are implemented to improve aspects like speed or security (e.g. we’d like for files to have names instead of a sector number like 0x0000008AF1). It has been a while since I attempted graphics:

Figure 1 Block Device: The smallest addressable unit in a block device is called a sector. CDs and DVDs typically have a sector size of 2048 bytes. Filesystems (managed by the OS) may group multiple sectors into blocks. E.g. 4 sectors in the physical addressing space may constitute 1 block address in the logical (filesystem) addressing space–see image. The same data may occupy different sectors depending on the filesystem type. In the case of ISO 9660 a block is normally also 2048 bytes (but this value isn’t demanded by the standard).

So files occupy at least 1 sector. Images (ISOs in the loose definition) are created by copying all the bits in all the sectors. In case the data occupies less than the whole sector (e.g small files) the other bits are still copied instead of being zeroed out, because the filesystem may require them. On a sidenote, in my opinion file based images (like WIM used in Windows backups) do not fit the definition of “image”. But that is a discussion for another article.

The size of a sector depends on the hardware. Careful when burning an image your software does not make any wrong assumptions. For example copying every two original sectors into one new one might mess up the filesystem.

Beginning in late 2009, accelerating in 2010 and hitting mainstream in 2011, hard drive companies are migrating away from the legacy sector size of 512 bytes to a larger, more efficient sector size of 4096 bytes, generally referred to as 4K sectors, and now referred to as the Advanced Format by IDEMA (The International Disk Drive Equipment and Materials Association). (seagate)

You can manage disk images with tools like these.

Tools that can create an ISO from optical media are common. Most disk burning software will also have this functionality. Though it seems much harder to find software that will make a true image of non-optical media (like a HDD or USB drive). Most backup software (Acronis, Paragon, Norton, etc.)  will do this, but they may use proprietary formats and/or compression. 

It is also possible to image individual partitions and even files on a hard drive. Mounting or file exploring software will need to recognize the filesystem before it becomes usable. Ok so, enough about images, how do we use some ISO with a flash drive?

*A sidenote for Windows users: If you want to quickly get an ISO from your DVD or explore the files of one, 7zip might be all you need. If you want to make an ISO from folders you’ll need to try something more advanced like imgburn–it lets you add/load a custom bootsector. Surprisingly few utilities support making ISOs from folders. Note that only adding a bootsector will not re-create the original ISO if hidden settings were lost.

Bootable vs Non-bootable

The question as it is now specifically refers to bootable ISOs as opposed to non-bootable ISOs. Is there such a distinction, and if so, what’s the difference? Well, bootable ISOs contain ‘invisible’ data called  boot-sectors. Now this is where it gets messy because many people don’t know what kind of firmware they have (BIOS or UEFI) and it determines the bootcode requirements.

UEFI

UEFI systems don’t need this invisible data to boot, provided your files include .efi files in the right place on your USB drive. The drive should also be marked active and be formatted with the FAT32 filesystem. Simple. This is why some cases of copy and pasting the ISO contents just work, while others do the same steps and fail miserably on their BIOS systems. In certain cases the USB drive has an MBR instead of a GPT and the UEFI goes into BIOS mode without trying to look for the .efi files. To fix this bad behavior either change a setting in the UEFI or change the drive to a GPT type partitioning.

BIOS

So technically an ISO with efi files might be bootable, but I will reserve the term “bootable ISO” for those images with invisible boot files. For the BIOS case you will need this invisible data (i.e. a bootable ISO) unless your USB drive was previously formatted with compatible boot code, in which case you can just replace the visible filesystem contents from the ISO–copy paste. Additionally, in some rare cases the BIOS might be smart enough to boot without a sector like UEFI.

There are normally two sectors involved in loading the operating system, the MBR and the VBR. This implies a BIOS system can’t boot from a GPT style drive. To get this invisible data from your ISO to your USB you need to burn the image, not just copy the filesystem (visible) files. If you don’t know if your ISO is bootable, burn it and test it, or maybe check the first sector.

“ The location and size of the boot sector (perhaps corresponding to a logical disk sector) is specified by the design of the computing platform.”

— Wikipedia (Boot_sector)

So it’s a few Kb somewhere that the CPU loads to do the bootstrapping process. You can’t access this data with a file explorer, because it is not part of the filesystem proper. A good ISO image editor should let you modify all the bytes, possibly overwriting them with custom sub-images.

Now the basic theory is out of the way, perhaps a better question is: what about all those free tools that take an ISO and give back a bootable USB? I suppose some tools may add the invisible stuff for you if you have a non-bootable ISO. Or they might choose between MBR and GPT for you. Or you might be having trouble finding software you like that burns your ISO to your drive. They just make life a bit easier. But they might not consider all the variables I’ve mentioned.

The most important thing to remember is a USB drive can be made bootable independently of the data. The following general steps can be taken to create our live/installer USB:

1. If you don’t know if your ISO is bootable, burn and test it. If that fails then wipe and:
2. Make USB bootable by adding bootsector(s), formatting may wipe the drive. A common tool is bootsect.exe. This is needed for BIOS systems. This process (particularly the VBR) is often OS specific, but many non-windows solutions use GRUB or Syslinux (link). There are many tutorials on the web about how to install one of these bootsectors. But before you get stuck in, invest some more time into finding a working image file, most developers distribute them. 
3. Add compatible files (e.g. linux distro, windows installer, rescue program). You can just copy the files contained by the ISO to your USB drive. Depending on your target operating system your folder structure may need to be in a specific format. In some cases you are allowed to copy the whole ISO.

No-one wants to burn a DVD twice, even if they could. But once you have a bootable USB drive you can simply copy the data or folders to it. As long as your invisible boot-sector is valid for whatever you want to use you can just swap files. I believe Vista and up use compatible boot code for BIOS and UEFI. An alternative route is to modify the ISO directly and then burn it all at once. 

 Though we agreed to follow the ISO convention, ISO hybrids exist. They already contain the MBR code the flash drive needs while also keeping the 9660 filesystem intact. Rarely are USB-only images distributed (unfortunately).

The above shows how a possible hybrid ISO can host both types of bootsector (optical and HDD). Only the green area is viewable to a typical user when mounting the partition.

There are many tools like the official Microsoft tool (probably uses bootsect), Rufus, YUMI (a.k.a pendrivelinux) and Unetbootin. I’m not sure how these programs do everything, but I believe most software will require a bootable ISO to generate the bootable USB sector. Perhaps they translate the boot volume descriptor (optical disk filesystem) into an MBR compatible with whatever filesystem they use to format the drive (probably FAT32, or maybe they leave it in 9660).

So what if you have a bootable ISO, but it has become unbootable?

Normally non-bootable ISOs are useless for the purposes of live/installer USBs (e.g. a DVD video ISO)–unless you somehow accidentally end up with a non-bootable ISO that should have been bootable.

For example, if you take a bootable DVD you can create an ISO from it both with or without the bootable property, depending on how you do it. In fact you can save any file or folder in ISO format, because there is no standard–it is just a filesystem snapshot. It is the settings used in the ISO creation process that determine if the produced ISO is bootable or not.

People often end up with non-bootable ISOs when they try to customize their ISO or create an ISO from a folder. The bootable property is lost when extracting an ISO and then repacking it without setting ‘make bootable’ options. Be sure to copy your bootsector.bin from the original ISO.

Now, let’s say you have a folder with files you want to turn into a bootable USB. You can either manually make the USB bootable, but this may be tricky for some images/systems. Or you can try and turn the folder into a bootable ISO, which you then supply to some one-click tool to create the live/installer USB (you’d need to find/create a bootsector first–and no guarantee it will work). Lastly there may be a niche tool that converts your customized folder directly to a working official ISO (like WAIK or RT se7en Lite for Windows images).

Note the lack of 2 options: there doesn’t seem to be software that accepts a folder directly, probably due to the lack of bootsector info, though it should work for modern UEFI systems. And there doesn’t seem to be a tool that can make a USB bootable for a given image distribution (i.e. just do step 2 in a few clicks). I’d be happy to hear information to the contrary!

Conclusion

Many one-click tools do require an ISO (preferably a bootable one) to generate a bootable USB. Just remember that an ISO may be more than just a file archive, and the underlying filesystems may vary. Get the right ISO and any burning software will copy all the sectors to your USB drive, it shouldn’t be hard. And don’t be worried if the extension is something like .img or .bin, in many cases an image is an image!

I have been using the Windows 8 Developer Preview for about 5 months or so and in order to install it on my PC, I had to create a rather small partition on my primary hard drive consisting of only about 16GB. Thankfully, a clean install of Windows 8 will fit comfortably within that space and still provide enough room for some basic utilities and drivers (of course, I do have other secondary partitions that I use for installing applications).

During the installation of a particular application, the free disk space on the partition dropped to about 28MB. One of the users over in the Root Access chat room suggested that I try filling the disk until it is completely full and then blogging about what happens – hence this article. I was a bit reluctant at first since I didn’t want the hassle of reinstalling anything if it became corrupted somehow, but since I have up to date backups of everything, I decided to proceed anyway.

more »

After reading a review of the Drobo FS, I became obsessed with network attached storage (NAS). I realised that a NAS device would neatly solve a couple of long-standing problems I hadn’t got around to fixing: data backup and data organisation.

This post will explain how I picked the hardware and software for my NAS.

more »

This is the third part of a three part article, for parts two and three go to Part 1 and Part 2

Continuing our journey into the world of missing hard drive space I hope to impress on you one of the worst offenders for making your hard drive space disappear:

Programs Themselves

One of the absolute major problems cruft building up on your computer is using the programs you install.

Everything from Internet Explorer to Google Earth to Picasa, each and every one of them wants to be fast at what they do and every one of them uses your disk space to store their temporary files.

Some of these programs are quite careful and elegant about how much space they use, keeping only what they immediately need for speed on your disk or deleting everything over a certain amount.

Other programs, well, don’t. Other programs store every last piece of data they got their grubby mits on. I’m looking at you Google Chrome, you and your ever increasing and seemingly unlimited disk cache. I don’t care if I have a huge hard drive and you think allocating $SOMERANDOMNUMBER percent of it for your cache is acceptable, you can dang well work to a limit like every other sane piece of software. (note: when I wrote this my Chrome cache was sitting pretty at 1.5GiB and showing no signs of remorse for its actions)

more »

This is the second part of a three part article, for parts one and three go to Part 1 and Part 3

Continuing our journey into the world of missing hard drive space this post will focus on things you can change, but shouldn’t.

In this post we’ll be focusing on some of the shadier characters:

• The page file
• WinSxS
• Program Installers

These are the kind of things that seem to be taking up a rather worrying amount of hard drive space but do actually provide some benefit or can break things if removed. You can get rid of them, but at best you’ll get ignored by tech support if you tell them what you did…

more »

This is the first part of a three part article, for parts two and three go to Part 2 and Part 3

A common problem with Windows is that no-one really knows how much space you need before you install it. Microsoft themselves recommend a minimum of 16GB of hard disk space before trying to install Windows, but that doesn’t take into account any of the growth when you install applications or any of the other things that go on “behind the scenes”

Some people dread the Unexpected free disk space disappearance

On my Windows 7 PC, the free disk space has gone down by 1GB even though I haven’t downloaded or installed any new files and I haven’t downloaded any updates or other things? What could have made the disk space go down for apparently no reason? Is this the result of some sort of a spy program that is undetectable?

The problem is that in Windows there are so many reasons as to why disk space suddenly disappears and when you are using an SSD where you are paying a premium for every last gigabyte it can be quite concerning. Some of the key culprits are the following items:

• System Restore,
• Page & Hibernation files,
• Windows Update,
• Recycle Bin
• WinSxS
• Program Installers, and last but by no means least
• Programs themselves.

This is nowhere near an exhaustive list and by no means rules out a malware infection but these items tend to account for the sudden and rather annoying disappearance of several gigabytes of precious free space.

Due to the sheer quantity of things that can be taking up space I’m going to split this post into three posts, “System options you can change”, “System Options you should leave alone” and “Clearing up program data.”

more »

I’m willing to bet you haven’t heard of Libraries.  No, I’m not talking about a library of dusty books, rather the Libraries feature in Windows 7.  If you have heard of Libraries, I’m sure you don’t know much about it or what it does.  Or even how to use it!

Libraries is probably one of the most underhyped, misunderstood,and ignored features of Windows 7.  With Libraries, you can keep all of your scattered data files from many different locations, and bring them together in one folder, all without duplicating data and wasting your precious hard drive or solid state drive space. It is one of the best built-in tools you can use to organize your data.  Don’t believe me? Let’s take a look. First, to access Libraries, either click the pinned Windows Explorer icon on the taskbar or type ‘Libraries’ in the Windows 7 Start Menu search box.

more »

I have seen a large number of questions on Super User recently all around the same topic of Linux and Unix file permissions. For example:

unix file permissions

File permissions in UNIX are frequently specified as an octal number. Why is octal the preferred base for this purpose?

… and …

Linux file permissions

In a world of Windows where file permissions can be granted on a per-user basis, Linux and Unix permissions seem to be very hokey and restricted.

Well, let me tell you, they’re not.  For such a seemingly basic arrangement they are an incredibly powerful tool.

Most people forget that to do anything even remotely fancy with permissions in Linux you really have to couple them with groups.  And you really can do some fancy things with them!

more »