Why would I want disk imaging?

Ever lost all of your data? Our company founder did; that’s why he created the product we now work on. My personal backup strategy has always revolved around the idea of backing up copies of important files – I’ll back up /home or My Documents and I’ll be fine! Yes, to an extent I will not lose any of my data if I run that regularly; however, I have a lot of programs installed. An awful lot. Each with tweaked configurations. It’s fair to say that if my machine dies, I’ll spend a few weeks hunting through to find all those weird tweaks and config changes I’ve made.

The same was true for my current boss and as a contractor, the downtime really hurt. What he wanted was a simple way to be able to restore a working system from a backup and have everything – every program, every driver – work as it was. So, Macrium Reflect was born and a few years later it’s not quite so simple… although it does indeed restore disk images!

Don’t talk about backups, talk about recovery.

In a way this is going to be the theme of this blog entry. Joel has blogged on this topic, so we’ll do a quick executive summary: it doesn’t matter how painless, smooth and efficient your backups are, if recovering your data is not as easy, the time and frustration cost of an incident will be huge. You want easy recovery – or as easy as you can get.

What is a disk image, how does it work and how can I take one on a running system?

A disk image is exactly what it sounds like – a file storing an exact copy of your disk’s data. Actually, that’s not quite true: To understand this, we’re going to have to talk a little about file systems. A file system can be thought of as a filing system, a scheme for storing data on a disk. There are various ways and schemes for doing this, depending on the requirements of the end user. Now, the file system must not only write data to a disk, but also a certain amount of meta data such that when it comes to finding a particular file, the computer can. Actually, we can use this fact for an important purpose. The metadata allows us to pick out sectors on the disk which are in-use storing data we want to keep and ignore sectors marked as free. Which leads us to the first choice of the evening:

• Active Sector Backups: This type of backup stores only the active sectors and filesystem metadata. When the image is written back to a disk, the data is written in the correct location on the disk – and the file system knows no different.
• Full copies: These are bit-for-bit copies of the file system, which take up more space and are technically unnecessary. However, there is an important use-case for a full-copy backup: forensic examination of a disk.

Macrium Intelligent Sector Copy vs Full Disk

So what constitutes a disk image really depends on how you’re backing it up – and which you choose depends on your storage capabilities, and trust in your software’s file system processing.

Now – live file systems. Well, the most obvious method for doing backups is to use the built-in Windows functionality, the Volume Shadow Copy Service. This is a Windows-provided snapshot system which provides a number of backup methods and is very flexible – see the msdn article.

It is, also, of course, possible to achieve this yourself. The technique for doing this is called copy-on-write; that is, any block that is written to is also duplicated to an extra copy first. When you come to read back the image, you can read the file system, or the store of the point-in-time data, depending on whether it has changed or not. Other operating systems, file systems etc. provide a variety of near-equivalent methods; the important point here being that it is safe to image a live hard disk by employing some fairly well known techniques.

Not always the full picture

We’ve alluded to the next topic with the copy on write idea above. If you take an image of a hard disk on the 1st of the month and then again on the 2nd of the month, you have two whole images – a bit of a waste of space if all that’s changed on that box is a few server logs. You might think, “Hey, we could just store the bits that have changed!” — You’d be right. There are two very similar strategies which use this “differential technique”. Both assume a full backup at some point in time.

• Differential backups: These are the difference between the last full backup and the one just taken. The next time an image is taken, it is also the difference between the current state and the last full backup – so all images until the next full backup are the difference between that point in time and the full backup. To recover one of these backups, you need both the full backup and the differential backup. Clearly, these take less space than a full backup.

• Incremental backups: These are like differential backups, but instead of always being the difference to the last full backup, they are the difference from the last incremental backup (or full backup, if none exists previously). These backups have the potential to take up the least space – but there’s a caveat – you need both the original full backup and each of the incremental backups taken to restore your system.

As you’d probably expect, performing either an incremental or differential backup is vastly quicker than a full backup – which can be a very important factor when you’re backing up every day.

Sounds easy! I’ll just go copy some partitions…

…wait right there. To recover your whole system and image your entire disk, you’re not just talking about partitions – you need to know about the partition scheme in use and any boot considerations this may entail.

In modern PCs, there are two types of partitioning scheme:

• Master Boot Record disks, which can have at most four primary partitions (although this is worked around with an extended/logical partitions).
• GUID Partitioning Scheme (GPT) disks. These can have (on Windows) 128 Partitions and are used with EFI systems for booting – they can not be booted from a MBR-based BIOS.

MBR is by far the most common scheme, but GPT/EFI systems are becoming more popular (and are used in modern Macs) as support now exists in 64-bit Windows systems.

Macrium Reflect showing MBR/GPT Disks

But that’s not all. Older systems (XP and earlier – you probably have this if you have upgraded) use Cylinder, Head, Sector as a partition alignment scheme, whereas newer systems use a 1MB alignment scheme. It turns out a SSD design flaw means that CHS schemes may degrade disks far more quickly than the newer scheme. Your disk, when imaged, will be aligned as per the scheme it was created with (and upgrading Windows won’t have changed it, either!) so do make sure if deploying to an SSD you have software capable of re-aligning your partitions. More here.

Live rescue media schmedia, why do I want that?

Having backup software that runs whilst you work is great – it doesn’t interrupt you. But if your hard disk fails, you’re not going to be able to boot your system; when you pop that fresh disk in, it’ll be empty and you won’t be able to boot. Being able to boot and recover your system from live media is absolutely essential.

There are some considerations to bear in mind here. You need rescue media capable of recovering your system; ideally, you’ll have tested it. Rescue media generally comes in two flavours: Linux, and Windows Preinstallation Environment (“Windows Lite”). If you’re running some esoteric hardware for which one of these environments does not have drivers, you really do need to know beforehand; some software (such as ours!) can incorporate drivers into PE images.

Another consideration is access to your images. If you intend to use a network, you’ll need to check you can from this media; you’ll also need to check any external storage you wish to mount works.

So now we’re getting around thinking more about recovery, let’s do some more thinking.

Fixing bootloader errors and deploying to new hardware

So, now we’re all backed up with a great imaging scheme. We’ve got rescue media just in case and we know it works with our kit. Great! Then something goes wrong – no problem! We can handle this!

Except that our motherboard has failed and it’s 3 years old. They’re not available anywhere you can see and you can’t spend enough time at work on ebay to hunt down that spare one somebody’s selling and you can’t afford it now the bids are getting large anyway. Oh dear – this image we’ve got isn’t a perfect, working backup.

There is a way around this – we call it redeploy: the ability to insert drivers into a new install before you boot it. Our specific system, I am reliably informed, will also update the HAL drivers and CPU type when you move to a new system.

Another consideration is the ability to repair Windows bootloader settings on redeploy. An example I have worked on recently: On systems running PGP Whole Desktop Encryption, we can take images, but they are unencrypted (more on that in a second). We can then restore them as needed – except the PGP bootloader remained in the MBR (I’d only imaged the partitions I wanted, not the whole disk). Being able to fix problems in booting is something you may well need.

Disk Encryption

So, I’ve briefly touched disk encryption – now here’s how it works. Either your controller (hard disk) handles the crypto and issues a bios interrupt on boot to allow a password to be inserted (or token), in which case it is totally invisible to software, or software handles the crypto by inserting a driver just above the block layer stuff and below the file system. The effect is the same: when the system is running, imaging software will image the unencrypted data, as the encryption process is transparent to it. However, the live media will only be able to image the encrypted disk and may fail interpreting it. If you employ disk encryption, do investigate your compatibility options.

Extracting images, partial recovery

Restore Menu in WinPE Disk

All this talk of recovering whole systems is great; however, there are times where you just want part of a system to be extracted – usually for user data. Or you might want to look at a specific image. Many backup programs offer you the ability to mount your recovery images and extract data from them; this is a good way to ensure your backup has worked – you can run

chkdsk X:

on drive X to verify the data is intact, on Windows systems.

So, that is all for a basic introduction to disk imaging and system backup. I’ll leave you with a quick backup checklist:

A recovery checklist

• Recovery Hardware: can I recover to different hardware?
• Recovery viability: can you afford to recover to when you need to, given your storage constraints for storing backups?
• No-recoverable interval: how much data do you lose if you have to recover? How often are snapshots made?
• Recovery testing: have you tested your recovery plan?
• Recovery time: how long can you get back to the mission critical work of reading blogs?

Ninefingers is a software developer writing C, C++, stepping through crash dumps and disk images to create Macrium Reflect, the best disk imaging solution for Windows with a free edition that just works and paid editions with bar-none the best support in the industry. There are other products, but they all suck. Buy ours instead.

_{The last is the reason they don’t let me do marketing}

• Windows 8 is the name of Microsoft’s new version of the operating system.
• Windows 8 sports Metro, an attempt to introduce touch interactions as a first class citizen in Windows without crippling mouse and keyboard interaction.
• Windows 8 is different. If you’re stuck in 2001 with Windows XP, or GNOME 2 and/or KDE 3, obviously you also aren’t going to like Windows 8. Also I have a few dancing bunnies for you to look at.

This is the most important thing to realize: Windows 8 is different. Different in huge, important ways: It marks the transition to a brave new world in ways similar to what Windows 95 did. To do so, it abandons UI conventions that have been around us since then. The Start button? Gone. The Start menu? Gone. Pressing Start to shut down the computer? No more. Menus (and ribbons)? A thing of the past. Titlebars? Oh, please.

Brave new world

More often than not, it is different in positive ways. However, it is still not awesome. I know it’s a prerelease version, but we also know how many iterations it is taking Canonical to get Unity just right. Introduced two years ago in Ubuntu 10.10 (whoops!), it is still being tweaked in 12.04. This stuff takes time and Microsoft will need more of it to reach the right level of polish.

It is not for lack of trying. The Windows team has put lots of thoughts into the redesign, and it shows their efforts on the Building Windows 8 blog. The thought process for the Start screen, for example, is explained here, and here, and here, and here. The blog is choke-full of insight, data and information porn: Most of it is worth the read.

You might be looking at the interface with disgust. Why do I need such a huge bloated interface?! This is a mouse I’m using! Those icons are way too big! Actually, I think the icons are just the right dimension for your mouse; it is actually faster to click your way around the system because large click targets are easier to click and harder to miss. And by taking benefit of your whole screen, it lists more than the usual 10 entries.

Science. It works, bitches.

I’ve spent three days with the operating system now, and I’m using Metro IE 10 to write this very article. This is what I’ve discovered and what I think.

Setup

First things first: You need to download Windows 8. The fastest way, I found out, is to keep your download managers at bay and actually use the Windows 8 installer itself. It doesn’t download the .iso file directly (which saves space), but it can build it for you if you need it. It can also install the system to a USB storage device (I don’t know if this means you can boot from it and have Windows 8, or if this means you can boot from it and install Windows 8 to your hard drive.)

Oh yes, because you do want to install Windows 8 to your hard drive. If you don’t want to mess with partitions you might want to take the VHD route (thanks, Ian). Still you’re not giving Windows 8 a fair chance until it has direct access to your graphics card. That rules virtual machines out, just like it happens with Unity. Also that means you won’t be giving Windows 8 its chance until you download and install the relevant drivers, just in case you’re using more obscure graphics card makers such as little-known NVIDIA. (Seriously Microsoft, it’s 2012, please do get drivers right for $deity’s sake!)

Windows 8 will actually not work in some virtualization tools such as Windows Virtual PC.

The Windows 8 installer will offer to upgrade your copy of Windows to Windows 8 Customer Preview. Let me be perfectly clear. Do not update your copy of Windows to Windows 8 Customer Preview. You can’t go from Windows 8 Customer Preview to Windows 8 retail (you know, the one that costs money). You also can’t go from Windows 8 Customer Preview back to what you were using. Windows 8 Customer Preview is time-bombed and will become unusable a few months after the release of Windows 8. Also, Windows 8 has still a few rough spots as we’ll see shortly.

So, what I’ve done is:

1. Repartition the disk using your favourite ways (I used good ol’ gparted from an Ubuntu Live CD – leave the area you want Windows 8 to install itself to unpartitioned).
2. Download and run the Windows 8 installer from here.
3. Let it check for compatibility problems and make sure it doesn’t complain about devices or drivers you need or use. (This will take a while.)
4. Let it download Windows 8.
5. Tell it to build an ISO image out of it. (This will take a shorter while.)
6. Burn that ISO to a DVD. Note down the product key. (Why?!)
7. Boot off that DVD.
8. Click through the setup process.
9. Let it reboot and groan at the obligatory betta fish. (Get it? Get it? Betta? Beta? Huh? Get it?)
10. Wait for Windows to boot and discover the first pain point of Windows 8.

The bootloader

Windows 8 has got a fancy new bootloader. Nice stuff huh?

On the left, GRUB 2. On the right, the bootloader found in Windows 8 CP. I've got no doubts on which one I'd rather use.

It’s a pity it takes roughly as much time to load as the entirety of Windows 8 does. No, really.

It is not terribly surprising. After all, you need to to do more work to get high resolution graphics, a mouse cursor and other niceties. The time cost though exceeds the gains in functionality. Yes, it’s much more friendly to use. Yes, it should also take roughly a hundreth of the time it does take to load.

The lock screen

Conversely, Windows 8 does boot fast once you get past the bootloader. The time from boot to lock screen is quite short, just like you’d expect from a freshly installed copy of Windows after all. By the way, yes, I did say lock screen – that’s what you meet when you turn Windows 8 on. Rather then being just a password input box, it’s a fullscreen picture of whatever you want, with the time and a few indicators on top of the password input box.

And here’s where the purists will begin to moan. Why do I have to go through the lock screen? The answer is you don’t. Just press Enter and you can enter your password. It would’ve been sweeter if you could just type it in but… oh, well. You can also dismiss the lock screen using the mouse by “dragging it up off the screen.” Really, all it takes is a quick flick of the hand. It’s barely noticeable and not a pain at all.

The Start screen

We’ve all seen the start screen for Windows 8 (those who haven’t can just scroll back to the top of the article…) and we’ve all groaned at its general direction. I’ve used it extensively for three days and I can tell you: I’m not looking back.

The main feature is the invisible one. Want a program? Any program? It’s just like in Windows Vista and Windows 7: press the Windows key on your keyboard and just start typing its name, then press enter. It’s precisely the same but much faster actually. On Windows 7 the start menu always takes a few seconds to actually come up, and what I type in this period of time gets lost. The start screen doesn’t have this delay. It might be because I’m comparing a one year old install of Win7 with a three days old install of Win8, but boy is it a pleasure to use.

Fundamental software at your fingertips. (Generous amounts of whitespace cropped.)

If you have your hand on the mouse, you can also summon the mouse screen by clicking on the start button. Just kidding, there’s no start button. Simply throw your mouse in the very left bottom hand corner of your screen and click when the ‘Start’ box appears. If that’s what you’ve been doing so far then you can continue doing it! If you’ve been aiming for the center of the button you’ll have to just keep going a little more (and learn that this very gesture will still work in all versions of Windows since 95!)

But there’s more to it than that.

The application switcher

The application switcher in all its vertical glory.

Always thought Win-Tab was a waste of time? So have I! That flip view thingadongdong really never added anything to the ball. You’ll be pleased to know that the flip view has been replaced by a left-side application switcher thuoght mostly for finger and mouse input. If you’re a keyboard user, things remain mostly the same in Windows 8: Win-Tab’bing is still a waste of time. Just keep Alt-Tab’bing instead.

What has changed however is that you can now switch between Metro applications, the desktop and the Start screen with just the mouse.

With the mouse, move to the top left hand corner. You’ll see a thumbnail of what of the above (metro application, desktop and start screen) you’ve used last. You can just click to restore it. If that’s not what you want, from that top left hand corner move your mouse straight down. This gives you the Win-Tab interface, only mouse operable.

Dragging windows around

You can do more than just clicking to these thumbnails – you can also drag them in place. If you drag the application on the center of the screen it’ll replace what you’re working on. If you drag it to the left or to the right, it will snap in place and only take roughly one fifth of the screen. (This is as much multitasking as people using Windows on ARM get to have, by the way.) Drag them to the very bottom to unload them. Unlike the iOS application bar, this does actually unload the program from memory. A smooth animation makes the result of your gesture clear.

This kind of dragging doesn’t have to start from the application switcher. The very top of the screen is also dedicated to this purpose: Fling your mouse up then start dragging. This kind of window management is kind of limited, but still pretty awesome.

So, we’ve seen what the top left and bottom left corners do, and we’ve also looked at the top and left edges. Time to look at the right edge!

The charms bar

Charms bar.

This is where synergy happens, at least in theory. Here you have another way to access the start screen, the search feature (programs shouldn’t have a button to expose search features), the share feature (so far it’s mostly for emailing people), the devices feature (so far it’s mostly for printing) and the settings feature (also another things programs shouldn’t have a button for). It also tells the time.

The settings (Win-I) are trimmed down to the essential, so if you’re a GNOME 2/3 kind of person you’ll be used to this tradeoff between simplicity and tweaking. In particular, it’s from here that you can do things such as connecting to wireless networks, changing volume, luminosity, shutting the computer down or reaching the Metro interface to change system settings.

All in all it’s a pretty important piece of functionality in dire need of more compatible programs and drivers. Even without these, you’ll be using it regularly though.

Multiple screens support

The charms bar combined with the multiple screens support is also my biggest pain point in Windows 8. The reason is that all of Metro happens on the very edges of your primary screen, from the hot corners to the hot edges to the actual metro applications themselves. They are trapped in the primary screen, so plugging more screens in exposes your desktop and (ugh) the betta fish.

Now, in my setup (two screens side by side) this doesn’t work well in a few ways:

• The charms bar is triggered by the right edge of the primary screen… but the mouse can get past it, right into my secondary screen! I’ve never been able to activate the charms bar with the mouse. Luckily there’s a keyboard shortcut: Win-C.
• You might be pleased to know that now each screen gets its own taskbar! Only the primary monitor’s taskbar, however, gets the traybar and the clock. Curses.
• Traditional programs by default launch in the primary screen. Metro applications will then give way to the desktop to show your program.

The way I had organized things was to have my laptop screen with just metro – one fifth email and four fifths idling in chat through Metro IE10. The secondary monitor, which isn’t wide enough to support snapped applications, would be left for traditional programs such as Chrome or Team Fortress 2.

Unfortunately this means that every time I launch these programs, Metro IE10 goes in the background and I have to manually restore it. If the gesture wasn’t so easy to do I would’ve given up on actually using Metro. It is though, so I keep on.

All in all it is sad to see Windows, probably the only operating system that got multiple screens right, screw this so badly. Again, though, it will take Microsoft years to really make Metro shine. Multiple screen support remains vastly superior to anything I’ve ever seen delivered by Linux, but that’s not a high bar to meet unfortunately.

If you’ve found all this edge and corner talk confusing, perhaps you will find the Ars Technica coverage more readable.

Metro IE10

The idea behind Metro IE10 is awesome. The whole damn screen is dedicated to the webpage. There’s no chrome. Zero. Nada. It’s all webpage. If you want chrome, you right click on an empty part of the page. This gives you the list of open tabs at the top (with thumbnails, a la Opera 10 but actually useful) and the address bar at the bottom.

So much room for activities!

It’s just a pity that it’s still, yknow, Microsoft effin’ Internet Explorer. IE10 is nicer than IE9, but that’s not saying much. It often flickers, it often fails to render properly and worryingly is unable to play most YouTube videos in any way shape or form.

See, Metro IE 10 is plugin-free. This means no Flash. If you were looking forward to a Windows tablet because, hey, Windows does Flash, then you’re in for a disappointment. This makes the only browser in Windows for ARM (and the only Metro browser so far) strictly worse than Safari. Yes, Flash for Android was pretty unusable, but one expects more from a device with flippin’ Windows on it.

(Sure, YouTube does support playing some videos in HTML 5, but even many of those never load at all. All you get is the throbber endlessly spinning.)

It gets worse. Say you get fed up with IE10 — if you’re willing to give it any chance at all, which I doubt. This means you install Chrome, for example. When you do, a very nice and new popup asks you what browser you want to use: Internet Explorer or Chrome. Pick Chrome and Metro IE10 goes away.

I’m not kidding. If your default browser is not Internet Explorer 10, you don’t get to have Internet Explorer 10 at all in Metro. This boggles my mind. Why would I want to switch to the desktop when opening a link from Windows Mail? I like Metro, I like the design, I like the immersion, but if you want to keep it it’s the IE10 way or the highway.

Is it unreasonable to want to use IE10 for lightweight browsing in the Metro environment and Chrome on the desktop for when you want to do more serious stuff such as, yknow, watching cats ramming table legs at full speed on YouTube?

Things get worse when you realize that Microsoft is actually cheating when it comes to IE10. This post by Mozilla has more details.

Unlike standard Metro apps, IE runs in the desktop configuration (medium integrity parent process, multiple low integrity children). [...] As a medium integrity process, IE10 can do anything it pleases and it does not require the broker to do file access.

I am disappoint. If you’re going to build a pretty restrictive environment such as Metro, least you could do is show off how the restrictions actually aren’t so bad and not cheat. Oh well.

WHOOPS! Redditor clubdirthill reports that this is actually not cheating, but a design decision I was previously unaware of. Brian Bondy, Firefox Platform Engineer, explains, third party browsers can also be built in the hybrid metro-desktop way IE10 (a so-called “Metro style enabled desktop browser”). The metro side of the equation however is only available for your one default browser. If that browser has no metro side to it… tough.

Multitasking, or the almost lack thereof

With Metro, you can do multitasking. You can indeed run two things at a time. One snapped and one on the remainder on the screen.

Metro (whoops!) applications in the background won’t run. If you’re using Metro IE10, none but the foreground tab is actually active. There is a list of exceptions to this, starting from traditional desktop apps to apps uploading, downloading, printing, synchronizing, playing music in the background. But that’s it. For an example of what this entails, go look at this article about Microsoft’s very own Skype client on Windows Phone 7: I don’t foresee things getting any easier on Windows Phone 8.

One has to wonder however: why accept tradeoffs to multitasking on my desktop computer? It is built for this kind of load! I have no answer here. This is quite a disappointing turn of events where programs can do less with more. I can only hope this will be addressed in Windows 8+1.

SmartScreen gets serious

This new feature in Windows 8 would’ve been easy to overlook in the short period of time I’ve used Windows for, if it wasn’t for the release of Mari0. I’ve been waiting for this one for a long time, so I mindlessly downloaded it, extracted it and ran it, when this happened.

You get the second screen by clicking on 'More info'

Windows SmartScreen prevented an unrecognized app from starting. Running this app might put your PC at risk.

SmartScreen is the IE9 feature that maintains a reputation system for downloads and publishers. If not enough people have downloaded a given link and the publisher doesn’t have good reputation of his own, SmartScreen kicks in. With Mari0 being a fresh release with no publisher information, it’s no surprise this happpened.

I do recall leaving it enabled from the Setup process, but I forgot that, as the Windows 8 blog explains, SmartScreen checks now also happen from Windows itself. (What if there is no internet connection?). I guess this is a better focused protection measure than Apple’s Gatekeeper (and a comparatively more gentle one) but I’m kind of surprised there is one to begin with. The dialogue really is designed to look like a dead end: how will apps without enough reputation to actually build reputation to begin with?

Localization: more than just translation

Another rough spot is definitely localization. I don’t mean the translation of strings, though, but the support for multiple cultures, as Microsoft calls them. Here’s a few examples:

• IE10 is spellchecking this post in Italian because I’ve used an Italian keyboard layout to write all of this. Due to its simplification it supposes the language you set is the one you will always use… That idea is neat but ultimately misguided.
• All the XBox applications plus video are not available in this region. What is this, Hulu? Please.
• The inbuilt finance application, Bing Finance, fails to acknowledge the existence of anything outside the US borders.
• The calendar application in week and day view fails to account for the fact that Italy does not use am/pm, so the hour labels read “10 11 12 1 2″ instead of “10 11 12 13 14″. It doesn’t even care to look at the regional settings I’ve configured in the Control Panel.
• The Weather app defaults to showing temperature in Fahrenheit instead of Celsius. This can be changed by going to the Settings charm (go to the top right corner and click the Settings wheel or press Win+I), then click on settings and tick Celcius. It again doesn’t look at my regional settings and is hidden quite well…

These are not translation issues – it’s not a simple matter of changing strings, there’s no right-to-left thingamajiggery involved, no Unicode shenanigans required. These are disheartening endemic, systematic issues through Metro.

And now for something completely different. Being an avid gamer, I’ll spend a few moments to discuss videogames and Windows 8.

Stuff that matters: XBOX LIVE™ Solitaire

From left to right, XBOX Live Solitaire, Metro Internet Explorer 10, Visual Studio 11 Express Beta and a clock-less, tray-less sysbar

Finally we get to the good stuff. What people really care about in the newer versions of Microsoft: Solitaire and Minesweeper. Unfortunately, the Customer Preview version of Windows 8 does not include the latter. Solitaire however delivers through XBOX Live integration, oh my! ESRB hasn’t yet rated the game, however.

Klondike purists will be delighted to know that XBOX Live Solitaire defaults to the Deal Three cards mode. People who would rather have a more casual exprerience will find the more comfortable “Deal One” option under the settings charm. More importantly, the game works even in snapped mode. It is however kind of uncomfortable to play with in this unusual arrangement, detracting from the kind of concentration required to do well.

I have doubts that Microsoft can also make the expert version of Minesweeper fit well in the snapped view, but we’ll need to wait until release to put this important open question about Windows 8 to rest.

(actual) Videogames and Metro

My hats, let me show you them

Okay, enough fooling about, let’s get back to business. Of videogames. Serious business, with serious money at stake. Windows 8 Customer Preview includes a demo of “Pinball FX2″ (also with the ‘XBOX Live’ monicker), the only thing that comes close to a real 3D accelerated game running natively in Metro. How well do the two things integrate?

The basic mouse interactions remain active: The top edge allows for closing and snapping, although the snapped view merely states “Game paused.” The game however has its own inbuilt settings menus, the Metro Way be damned. The game is theoretically playable with just the mouse, but the mouse controls involve clicking on the flippers. Hint: It’s terrible.

All in all the one application available at launch is underwhelming and fails to set the tone for a good, Metro videogame. Comparatively, Wordament makes a much better job of this. If only I wasn’t so terrible at it.

Traditional videogames

There’s not really much to report here. Videogames run just like they used to, with no added bells and whistles. Full screen games effectively disable metro (since they launch on the primary desktop and ignore all the hot edges and corners), which is a letdown.

I remain wanting for a better integration between videogames and the operating system. Even Ubuntu can display OSD messages on top of the two or three videogames that do run natively on the damned thing!

I’m just kidding I’m just kidding. Please don’t murder me in my sleep. Also, the videogame parenthesis is now closed and we resume the normal schedule as IE10 starts really struggling under the weight of the three throusand words I’ve written so far. Time to save drafts compulsively.

Your Microsoft Account

The Settings app in all its Metrosity. (The regular control panel is still there!)

I don’t fully understand the consequences of binding your computer and your user account to your Hotmail/Messenger/MSDN/Microsoft credentials, other than the fact you can do so. This is for lack of paranoia: I don’t know what happens when I don’t do this.

What you do want to make sure before you install Windows 8, however, is that you have sensible values for your first and last names. I don’t: my name is “b”, my surname is “p”. Now my home folder is C:\Users\b\, I’m “b p” on the login screen and “b” on the start screen and there’s no direct way to change this. Less than awesome.

Other inbuilt software

• Store. It is weird to give a footnote to the Windows Store (long last!) but the paltry selection of application just doesn’t give me a lot to talk about. The one thing I will point out is that applications do not update automatically. It was probably too much of a Google Apps thing to do.
• Maps. It’s just pointless. Broken in snapped view. Just uninstall it.
• Mail and Calendar. The first desktop email and calendar clients I’ve managed to use and keep using in a very long time. Google integration just works (if you have two-factor authentication, make sure you use an application-specific password) and so do the apps. The email client has no text-only feature and a pretty ridiculous “Sent from my Windows 8 PC” signature; forgiveable. See Ars Technica’s coverage for more.
• Weather. If you’ve seen the keynote, you’ve probably seen this app with beautiful animated backgrounds that aren’t there in Windows 8 Customer Preview. Relies on Weather Underground and AccuWeather for forecasts, a much better choice than Apple’s (lol, Yahoo!).
• People. Just like on Windows Phone 7, this is the app that would be responsible for integrating your social networks. I’ve only got Twitter (it also does Facebook and LinkedIn) and, as far as Twitter clients go, People is kind of clumsy and buggy. There’s no way to directly tweet, but you can send replies to people.
• Messaging. This app would be responsible for MSN and Facebook chat. It would do both better if the multitasking model in Metro was a bit more forgiving. Oh well. See Ars Technica’s coverage about People and Messagging.
• Reader. A spartane PDF reader with note taking and highlighting capabilities. To open a new PDF you close Reader then open it again. Obviously.
• Camera and Photos. Nothing extraordinary here. Open camera, see yourself, click on image, take picture. Open photos, look at photos, then delete them. Ars Technica goes in further detail.
• Task Manager. So much better than Windows 7′s. Read all about it here. Also why is the “Microsoft spell checking facility” taking such a toll on my system? It’s just 3,405 words I’ve got here… oh God finally no more red underlined words. Phew!
• XBox Live Companion. Some kind of remote for your XBox. I don’t have one so I can’t say much about it.
• XBox Live Games. Some kind of Steam-like client with games and friends and achievements and stuff that “doesn’t support logins from my region”. Blech.
• 

  This appears when you use the multimedia keys. I wish it also popped up on track change, however.

  Music and Video. They, uh, play music and video from your libraries. Not much to say here except they also don’t let me log in (so I refer you again to the relevant Ars Technica coverage). The playlist in Music doesn’t seem to work in the snapped view if you use “Play All” from the Explorer ribbon, another (barely noticeable, really) can of worms of its own. Background playback of music works as advertised; when active, the volume overlay gains track information with pause, back and forward controls similar to what happens in Unity.
• Team Fortress 2. Just kidding, it isn’t inbuilt. It should’ve been though!

Third party apps

The Store has precious few applications currently available and not all bring all the potential of Metro to the table, as we’ve already seen with the Pinball app. In some ways this however also because of limitations of the common controls built into Windows. In Start and most inbuilt programs, you can scroll horizontally through the menu with vertical mousewheel scrolling (it actually makes sense). No such thing happens however in the default controls, which you can see from the templates in Visual Studio and the Vimeo app. If the building blocks are unpolished, unpolished apps is what you’re likely to make out of them. Pity.

No new apps have joined the starting roster of 65 programs (also counting the inbuilt apps listed above, mind you).  Wordament is fun, Soluto is ready to rock (athough only for limited management of others’ computers).

tl:dr; Rough around the edges

When all’s said and done, Windows 8 delivers a completely fresh way to dick around with your computer. It remains to be seen whether or not you can also actually get things done with Metro. That’s where third-party apps will need to kick in, from the up and coming Office 15 to the official clients for all kind of online services that aren’t yet on board. There’s also a few rough spots and most of them will probably require more work than it is feasible to put in so late in the release cycle: we’ll be lucky to find solutions in a service pack. Especially the multitasking limits seem concerning: we’ll see how many programs will be “playing background music” just for the purpose of running in the background.

For now however, this is all I have to say on the topic. I encourage you to give this Customer Preview a fair try and make up your own opinion of the operating system. It costs you nothing. If you still will think it sucks, at least you’ll know why it sucks firsthand, rather than regurgitating what all the other cool kids on the block say.

As for what I’m going to do with this install of Windows 8? Right now I think I’ll play some TF2, I’m almost sure it runs better here. We’ll see about the rest.

A rather famous question on Stack Overflow asked for tips on how to be more productive with VIM – and he got one of the best answers I’ve ever seen. Not a list of tips but a working, detailed explanation from Stack Overflow user Jim Dennis. It’s so good, I am copying his answer’s raw source here, running it through the Markdown parser and copied it here.

So, without further ado, over to Jim’ amazing answer.

Your problem with Vim is that you don’t grok vi.

You mention cutting with yy and complain that you almost never want to cut whole lines. In fact programmers, editing source code, very often want to work on whole lines, ranges of lines and blocks of code. However, yy is only one of many way to yank text into the anonymous copy buffer (or “register” as it’s called in vi).

The “Zen” of vi is that you’re speaking a language. The initial y is a verb. The statement yy is a synonym for y_. The y is doubled up to make it easier to type, since it is such a common operation.

This can also be expressed as dd P (delete the current line and paste a copy back into place; leaving a copy in the anonymous register as a side effect). The y and d “verbs” take any movement as their “subject.” Thus yW is “yank from here (the cursor) to the end of the current/next (big) word” and y'a is “yank from here to the line containing the mark named ‘a‘.”

If you only understand basic up, down, left, and right cursor movements then vi will be no more productive than a copy of “notepad” for you. (Okay, you’ll still have syntax highlighting and the ability to handle files larger than a piddling ~45KB or so; but work with me here).

vi has 26 “marks” and 26 “registers.” A mark is set to any cursor location using the m command. Each mark is designated by a single lower case letter. Thus ma sets the ‘a‘ mark to the current location, and mz sets the ‘z‘ mark. You can move to the line containing a mark using the ' (single quote) command. Thus 'a moves to the beginning of the line containing the ‘a‘ mark. You can move to the precise location of any mark using the ` (backquote) command. Thus `z will move directly to the exact location of the ‘z‘ mark.

Because these are “movements” they can also be used as subjects for other “statements.”

So, one way to cut an arbitrary selection of text would be to drop a mark (I usually use ‘a‘ as my “first” mark, ‘z‘ as my next mark, ‘b‘ as another, and ‘e‘ as yet another (I don’t recall ever having interactively used more than four marks in 15 years of using vi; one creates one’s own conventions regarding how marks and registers are used by macros that don’t disturb one’s interactive context). Then we go to the other end of our desired text; we can start at either end, it doesn’t matter. Then we can simply use d`a to cut or y`a to copy. Thus the whole process has a 5 keystrokes overhead (six if we started in “insert” mode and needed to Esc out command mode). Once we’ve cut or copied then pasting in a copy is a single keystroke: p.

I say that this is one way to cut or copy text. However, it is only one of many. Frequently we can more succinctly describe the range of text without moving our cursor around and dropping a mark. For example if I’m in a paragraph of text I can use { and } movements to the beginning or end of the paragraph respectively. So, to move a paragraph of text I cut it using { d} (3 keystrokes). (If I happen to already be on the first or last line of the paragraph I can then simply use d} or d{ respectively.

The notion of “paragraph” defaults to something which is usually intuitively reasonable. Thus it often works for code as well as prose.

Frequently we know some pattern (regular expression) that marks one end or the other of the text in which we’re interested. Searching forwards or backwards are movements in vi. Thus they can also be used as “subjects” in our “statements.” So I can use d/foo to cut from the current line to the next line containing the string “foo” and y?bar to copy from the current line to the most recent (previous) line containing “bar.” If I don’t want whole lines I can still use the search movements (as statements of their own), drop my mark(s) and use the `x commands as described previously.

In addition to “verbs” and “subjects” vi also has “objects” (in the grammatical sense of the term). So far I’ve only described the use of the anonymous register. However, I can use any of the 26 “named” registers by prefixing the “object” reference with " (the double quote modifier). Thus if I use "add I’m cutting the current line into the ‘a‘ register and if I use "by/foo then I’m yanking a copy of the text from here to the next line containing “foo” into the ‘b‘ register. To paste from a register I simply prefix the paste with the same modifier sequence: "ap pastes a copy of the ‘a‘ register’s contents into the text after the cursor and "bP pastes a copy from ‘b‘ to before the current line.

This notion of “prefixes” also adds the analogs of grammatical “adjectives” and “adverbs’ to our text manipulation “language.” Most commands (verbs) and movement (verbs or objects, depending on context) can also take numeric prefixes. Thus 3J means “join the next three lines” and d5} means “delete from the current line through the end of the fifth paragraph down from here.”

This is all intermediate level vi. None of it is Vim specific and there are far more advanced tricks in vi if you’re ready to learn them. If you were to master just these intermediate concepts then you’d probably find that you rarely need to write any macros because the text manipulation language is sufficiently concise and expressive to do most things easily enough using the editor’s “native” language.

A sampling of more advanced tricks:

There are a number of : commands, most notably the :% s/foo/bar/g global substitution technique. (That’s not advanced but other : commands can be). The whole : set of commands was historically inherited by vi‘s previous incarnations as the ed (line editor) and later the ex (extended line editor) utilities. In fact vi is so named because it’s the visual interface to ex.

: commands normally operate over lines of text. ed and ex were written in an era when terminal screens were uncommon and many terminals were “teletype” (TTY) devices. So it was common to work from printed copies of the text, using commands through an extremely terse interface (common connection speeds were 110 baud, or, roughly, 11 characters per second — which is slower than a fast typist; lags were common on multi-user interactive sessions; additionally there was often some motivation to conserve paper).

So the syntax of most : commands includes an address or range of addresses (line number) followed by a command. Naturally one could use literal line numbers: :127,215 s/foo/bar to change the first occurrence of “foo” into “bar” on each line between 127 and 215. One could also use some abbreviations such as . or $ for current and last lines respectively. One could also use relative prefixes + and - to refer to offsets after or before the current line, respectively. Thus: :.,$j meaning “from the current line to the last line, join them all into one line”. :% is synonymous with :1,$ (all the lines).

The :... g and :... v commands bear some explanation as they are incredibly powerful. :... g is a prefix for “globally” applying a subsequent command to all lines which match a pattern (regular expression) while :... v applies such a command to all lines which do NOT match the given pattern (“v” from “conVerse”). As with other ex commands these can be prefixed by addressing/range references. Thus :.,+21g/foo/d means “delete any lines containing the string “foo” from the current one through the next 21 lines” while :.,$v/bar/d means “from here to the end of the file, delete any lines which DON’T contain the string “bar.”

It’s interesting that the common Unix command grep was actually inspired by this ex command (and is named after the way in which it was documented). The ex command :g/re/p (grep) was the way they documented how to “globally” “print” lines containing a “regular expression” (re). When ed and ex were used, the :p command was one of the first that anyone learned and often the first one used when editing any file. It was how you printed the current contents (usually just one page full at a time using :.,+25p or some such).

Note that :% g/.../d or (its reVerse/conVerse counterpart: :% v/.../d are the most common usage patterns. However there are couple of other ex commands which are worth remembering:

We can use m to move lines around, and j to join lines. For example if you have a list and you want to separate all the stuff matching (or conversely NOT matching some pattern) without deleting them, then you can use something like: :% g/foo/m$ … and all the “foo” lines will have been moved to the end of the file. (Note the other tip about using the end of your file as a scratch space). This will have preserved the relative order of all the “foo” lines while having extracted them from the rest of the list. (This would be equivalent to doing something like: 1G!GGmap!Ggrep foo<ENTER>1G:1,'a g/foo'/d (copy the file to its own tail, filter the tail through grep, and delete all the stuff from the head).

To join lines usually I can find a pattern for all the lines which need to be joined to their predecessor (all the lines which start with “^ ” rather than “^ * ” in some bullet list, for example). For that case I’d use: :% g/^ /-1j (for every matching line, go up one line and join them). (BTW: for bullet lists trying to search for the bullet lines and join to the next doesn’t work for a couple reasons … it can join one bullet line to another, and it won’t join any bullet line to all of its continuations; it’ll only work pairwise on the matches).

Almost needless to mention you can use our old friend s (substitute) with the g and v (global/converse-global) commands. Usually you don’t need to do so. However, consider some case where you want to perform a substitution only on lines matching some other pattern. Often you can use a complicated pattern with captures and use back references to preserve the portions of the lines that you DON’T want to change. However, it will often be easier to separate the match from the substitution: :% g/foo/s/bar/zzz/g — for every line containing “foo” substitute all “bar” with “zzz.” (Something like :% s/(.foo.)bar(.*)/\1zzz\2/g would only work for the cases those instances of “bar” which were PRECEDED by “foo” on the same line; it’s ungainly enough already, and would have to be mangled further to catch all the cases where “bar” preceded “foo”)

The point is that there are more than just p, s, and d lines in the ex command set.

The : addresses can also refer to marks. Thus you can use: :'a,'bg/foo/j to join any line containing the string foo to its subsequent line, if it lies between the lines between the ‘a‘ and ‘b‘ marks. (Yes, all of the preceding ex command examples can be limited to subsets of the file’s lines by prefixing with these sorts of addressing expressions).

That’s pretty obscure (I’ve only used something like that a few times in the last 15 years). However, I’ll freely admit that I’ve often done things iteratively and interactively that could probably have been done more efficiently if I’d taken the time to think out the correct incantation.

Another very useful vi or ex command is :r to read in the contents of another file. Thus: :r foo inserts the contents of the file named “foo” at the current line.

More powerful is the :r! command. This reads the results of a command. It’s the same as suspending the vi session, running a command, redirecting its output to a temporary file, resuming your vi session, and reading in the contents from the temp. file.

Even more powerful are the ! (bang) and :... ! (ex bang) commands. These also execute external commands and read the results into the current text. However, they also filter selections of our text through the command! Thus we can sort all the lines in our file using 1G!Gsort (G is the vi “goto” command; it defaults to going to the last line of the file, but can be prefixed by a line number, such as 1, the first line). This is equivalent to the ex variant :1,$!sort. Writers often use ! with the Unix fmt or fold utilities for reformatting or “word wrapping” selections of text. A very common macro is {!}fmt (reformat the current paragraph). Programmers sometimes use it to run their code, or just portions of it, through indent or other code reformatting tools.

Using the :r! and ! commands means that any external utility or filter can be treated as an extension of our editor. I have occasionally used these with scripts that pulled data from a database, or with wget or lynx commands that pulled data off a website, or ssh commands that pulled data from remote systems.

Another useful ex command is :so (short for :source). This reads the contents of a file as a series of commands. When you start vi it normally, implicitly, performs a :source on ~/.exinitrc file (and Vim usually does this on ~/.vimrc, naturally enough). The use of this is that you can change your editor profile on the fly by simply sourcing in a new set of macros, abbreviations, and editor settings. If you’re sneaky you can even use this as a trick for storing sequences of ex editing commands to apply to files on demand.

For example I have a seven line file (36 characters) which runs a file through wc, and inserts a C-style comment at the top of the file containing that word count data. I can apply that “macro” to a file by using a command like: vim +'so mymacro.ex' ./mytarget

(The + command line option to vi and Vim is normally used to start the editing session at a given line number. However it’s a little known fact that one can follow the + by any valid ex command/expression, such as a “source” command as I’ve done here; for a simple example I have scripts which invoke: vi +'/foo/d|wq!' ~/.ssh/known_hosts to remove an entry from my SSH known hosts file non-interactively while I’m re-imaging a set of servers).

Usually it’s far easier to write such “macros” using Perl, AWK, sed (which is, in fact, like grep a utility inspired by the ed command).

The @ command is probably the most obscure vi command. In occasionally teaching advanced systems administration courses for close to a decade I’ve met very few people who’ve ever used it. @ executes the contents of a register as if it were a vi or ex command.

Example: I often use: :r!locate ... to find some file on my system and read its name into my document. From there I delete any extraneous hits, leaving only the full path to the file I’m interested in. Rather than laboriously Tab-ing through each component of the path (or worse, if I happen to be stuck on a machine without Tab completion support in its copy of vi) I just use:

1. 0i:r (to turn the current line into a valid :r command),
2. "cdd (to delete the line into the “c” register) and
3. @c execute that command.

That’s only 10 keystrokes (and the expression "cdd @c is effectively a finger macro for me, so I can type it almost as quickly as any common six letter word).

A sobering thought

I’ve only scratched to surface of vi‘s power and none of what I’ve described here is even part of the “improvements” for which vim is named! All of what I’ve described here should work on any old copy of vi from 20 or 30 years ago.

There are people who have used considerably more of vi‘s power than I ever will.

The tools for the job

Now of course I could use fairly established tools such as Offline Windows Password & Registry Editor (also known as ntpasswd for you geeks out there) or Ophcrack. But for multiple reasons, those tools just don’t seem to cut it for me.

While they do their job brilliantly and have both saved me on numerous occasions, they have drawbacks. ntpasswd allows a user to reset the password of a user account by booting from a special boot disk. Although this is very useful, it leaves traces; my client has to reset his or her password and actual changes are made to the Windows setup.

Ophcrack does a fine job cracking Windows password hashes using so-called rainbow tables. Basically, Windows stores your user password in a hash, a seemingly arbitrary series of numbers and letters. Hashing is a way of storing passwords without having access to the plain password, which is much more secure than just storing the unencrypted password.

Now what is this hashing you speak of?

The below YouTube clip does an excellent job explaining how encrypted communication works and in the process it also shows how hashing works. Just substitute “The original colours” with “The user password” and you’re good to go. The part you’ll want to take a look at to understand hashing is from 0:40 to 1:05.

Ophcrack basically knows (in some cases, not all) which original colours (passwords) are mixed into the end colour (the hash). These colours and their end products are stored in a table, a rainbow table. Now this works well with passwords that are easy to guess but if the passwords are more complicated, Ophcrack will take way too long at guessing the password — or it won’t find it at all (at least not within our lifespan). If you would like to know more about cryptography, then I invite you to watch the entire YouTube video. It’s worth it.

Enter Kon Boot

Back to our password problem. This client’s really hard to get a hold of, won’t answer his phone and won’t reply to texts or email. Annoying… 

Until recently this meant having to wait and wait and wait, having the client’s machine just taking up space in the room. Then Kon Boot drew my attention. Where ntpasswd and Ophcrack help you login with a legitimate password, Kon Boot simply hooks onto Windows’ kernel and tells it to disregard the authentication of the password entered, correct or not. Doing this, it allows you to log into any local user account using any (or even a blank) password you enter into the password field. Windows will accept anything you input as the correct password, even insults. (No really, I tried “Windows sucks, Linux FTW!” as a password — it didn’t seem to upset Windows at all.)

Now I can hear you thinking, “It can’t possibly be that easy?”. Guess again, it’s probably even easier than you imagine it to be right now. Kon Boot can be run from a CD, USB drive or even a good old floppy disk. You boot into Kon Boot, Kon Boot will boot into Windows and that’s it.

Some popular BootCDs (like Hiren’s BootCD) even include the free version of Kon Boot saving you from having to create your own bootable disk image. Just burn the ISO to a blank DVD or put it on a USB drive following the clear instructions.

Kon Boot has a free version that handles virtually all 32-bit versions of Windows except Windows 7. If you do decide to pay for this awesome piece of software, you’ll get a version that adds Windows 7 and 64-bit support. Check out their website for pricing.

What Kon Boot won’t do

Now because Kon Boot doesn’t actually know the password to the account, it is unavoidably inhibited in performing certain actions. The most noteworthy are:

• Kon Boot does not allow you to change the password of the account used to log in. However, changing the password of other local accounts is possible.
• You can not use the private keys stored in the Windows certificates store. These certificates are used to decrypt encrypted email, websites and other connections.
• EFS (Encrypted File System) encrypted files can not be accessed because they are encrypted with the real password. An “access denied” message is displayed by the system.
• Kon-boot does not allow you to bypass the authentication on an Active Directory domain. Only locally stored accounts can be accessed. However, cached accounts seem to be accessible in certain cases.

Pretty scary stuff

Kon Boot seems to be the nightmare of any system administrator who wants to keep unwelcome guests out of their files and settings. Countermeasures can be taken though:

• Disable USB and Firewire booting in the BIOS.
• Change the boot order to make sure the computer won’t boot from any CD or DVD.
• Disable any other boot methods your BIOS may allow you don’t need (such as network boot).
• Encrypt your files by using e.g. EFS, Truecrypt, etc.

Other than that, there isn’t really much you can do about it. The bypass has been there since 2009 and it seems unlikely Microsoft will release a patch after three years. I have tried have Kon Boot on several of my PCs running different versions of Windows and Kon Boot allowed me to log into all of them like a charm.

Although Kon Boot seems like a pretty scary tool, I believe it can be put to good use by underpaid tech fix guys as ourselves. I’m sure I needn’t remind you; using Kon Boot to grant yourself access without permission is illegal and may get you in trouble.

Chances are you’ve probably heard of FFmpeg already. It’s a set of tools dedicated to decoding, encoding and transcoding video and audio. FFmpeg is based on the popular libavcodec and libavformat libraries that can be found in many other video conversion applications, like Handbrake.

So why would you need FFmpeg? Got a video in an obscure format that every other player couldn’t recognize? Transcode it with FFmpeg. Want to automate cutting video segments out of movies? Write a short batch script that uses FFmpeg. Where I work, I constantly have to encode and process video material, and I’d never want to go back to using a GUI tool for doing this.

This post is a follow-up on Video Conversion done right: Codecs and Software, where I discussed the various codecs and containers that you can find these days. For a quick overview, I’d suggest to read this one as well, because it covers some important basics.

Now, let’s dive into the more practical aspects.

Installation

FFmpeg is free and open source, and it’s cross-platform. You’ll be able to install it on your Windows PC as well as your Linux server. The only thing you need to be comfortable with is using your command line. As it’s actively developed, you should try to update your version from time to time. Sometimes, new features are added, and bugs are fixed. You won’t believe how many times updating FFmpeg solved encoding problems I had.

Windows

Compiling FFmpeg on Windows is not too easy. That’s why there are (semi-) automated builds of the latest version available online, most prominently the ones from Zeranoe.com. Download the latest build labeled as “static” for your 32 or 64 Bit system.

It will include ffmpeg.exe, which is the main tool we are going to use. And you’re done. That was easy, was it?

OS X

On OS X, you have the tools for building programs from source. It’s very easy to install FFmpeg if you use a package manager like Homebrew or MacPorts. Install one of them if you haven’t already, and then run either

brew install ffmpeg --with-ffplay 

sudo port install ffmpeg-devel 

Note that there’s also ffmpegX, but its bundled version always is a bit behind, and I wouldn’t recommend using it unless you have a good reason for doing so. The .app package contains a compiled version of the ffmpeg command line tool already, which you could theoretically use.

Linux

On many Linux distributions, FFmpeg comes in the default packages (e.g. on Ubuntu, where libav – an FFmpeg fork – is bundled). However, mostly this is a very old release version. Do yourself a favor and compile from source – it sounds harder than it actually is.

FFmpeg has a few dependencies, so therefore you’ll want to install encoders such as LAME for MP3 and x264 for videos as well. As I’ve been using Ubuntu almost exclusively, I found these Ubuntu Forums tutorials quite helpful: HOWTO: Install and use the latest FFmpeg and x264. Follow the steps there to get your version up and running. The process should be similar for other distributions.

Our first encoded Video

Now that we have FFmpeg working, we can start with our video or audio conversion. Note that this is always a very resource intensive task, especially with video. Expect some conversions to take a little while, for example when re-encoding whole movies. A quad-core CPU like the i7 can easily be saturated with a good encoder and performs very fast then.

Before you handle videos, you should know what the difference between “codec” and “container” (also, “format”) is, as I will be using these terms without explaining them any further. I know these can be easily mixed up and aren’t as self-explaining as you’d like them to be. To read more about this, see this Super User answer: What is a Codec (e.g. DivX?), and how does it differ from a File Format (e.g. MPG)?

Basic Syntax

Okay, now let’s start. The most basic form of an FFmpeg command goes like this – note the absence of an output option like -o:

ffmpeg -i input output 

ffmpeg -i audio.mp3 audio.wav
ffmpeg -i video.mp4 video.mkv
ffmpeg -i video.mp4 audio.wav

Specifying Video and Audio codecs

In most cases, you want a specific output codec, like h.264, or AAC audio. Today, there’s almost no reason not to encode to h.264, as it offers incredible quality at small file sizes. But which codecs do really work in FFmpeg? Luckily, it has a ton of built-in codecs and formats, which you can get a list of. The D and E stand for “decoding” and/or “encoding” capabilities, respectively.

ffmpeg -codecs
ffmpeg -formats 

ffmpeg -i input.wav -c:a libfaac output.mp4
ffmpeg -i input.avi -c:v libx264 -c:a libfaac output.mkv 

When to copy, when to encode?

Sometimes, you want to just copy the contents of a video and not re-encode. This is actually very critical when just cutting out portions from a file, or only changing containers (and not the codecs inside). In the example I gave above, we wanted to change MP4 to MKV.

The following command is wrong: It will re-encode your video. It will take forever, and the result will look bad.

ffmpeg -i input.mp4 output.mkv 

ffmpeg -i input.mp4 -c:v copy -c:a copy output.mkv 

If changing the container without encoding does not work for you, you can try the mkvmerge tool that mkvtoolnix offers. This will create an MKV file.

mkvmerge input.avi -o output.mkv

Advanced Options

If you’ve used some test videos or audio files for the above sequences, you might have seen that the quality wasn’t good enough to be useful. This is completely normal. Also, you will probably have constraints about bit rate, file size or target devices, such as when encoding a video for the PlayStation 3 or your Android phone.

Quality Settings

Quality comes first. What is “quality”, even? Generally, the more bits you can spend on a file, the better it will look or sound. This means, that for the same codec, larger file size (or larger bit rate) will equal in better quality. In most cases, that is.

If you want to set the bit rate, you can do that with the -b option, again specifying the audio or video part (just to resolve any ambiguities).

ffmpeg -i input.wav -b:a 192K out.mp3
ffmpeg -i input.avi -c:v libx264 -b:v 500K -c:a copy out.mp4 

In video, this means setting a certain value called Constant Rate Factor. For x264 (the h.264 encoder you should use), this is very easy:

ffmpeg -i source.mp4 -c:v libx264 -crf 19 out.mp4 

ffmpeg -i source.mp4 -c:v mpeg4 -qscale 3 out.mp4 

In general, the best bet is to just try and see for yourself what looks good. Take into account the result file size you want and how much quality you can trade in for smaller file sizes. It’s all up to you.

Cutting Video

Often, you want to just cut out a portion from a file. FFmpeg supports basic cutting with the -t and -ss options. The first one will specify the duration of the output, and the second one will specify the start point. For example, to get the first five seconds, starting from one minute and 30 seconds:

ffmpeg -ss 00:01:30 -i input.mov -c:v copy -c:a copy -t 5 output.mov 

Note that we’ve just copied the contents instead of re-encoding. Also, see how the -ss option is before the actual input? This will first seek to the point in the file, and then start to encode and cut. This is especially useful when you’re actually not just copying content, but encoding it, because it’ll speed up your conversion (although it’s less accurate).

Resizing

Got a High Definition 1080p video that you want to watch on your mobile phone? Scale it down first! FFmpeg supports software-based scaling when encoding with the -s option. The following example will do that:

ffmpeg -i hd-movie.mkv -c:v libx264 -s 854x480 -c:a copy out.mp4 

Other useful Examples

On Super User, I’m currently very active in the FFmpeg tag. You could browse the questions there, or see this list of questions and answers I’ve compiled. Of course, if you have any specific problem, feel free to ask it on the main site and we Super Users can help you out!

General issues:

• Resources To Use FFMPEG Effectively, including some general links and resources
• What do the numbers 240 and 360 mean when downloading video? How can I tell which video is more compressed?, which focuses on tradeoffs between bit rate, file size, video dimensions, et cetera.

Specific conversion problems:

• Convert from MOV to MP4 Container format, which talks about switching containers only
• How to convert any video to DVD (.vob) in good video quality?, about the limitations of retaining quality when converting
• What parameters should I be looking at to reduce the size of a .MOV file?, in depth explanation about how to use the Constant Rate Factor
• Use DivX settings to encode to mp4 with ffmpeg, more examples about CRF and how to set advanced x264 options like presets and profiles
• Convert any video to HD Video, about up- or downscaling video

During the installation of a particular application, the free disk space on the partition dropped to about 28MB. One of the users over in the Root Access chat room suggested that I try filling the disk until it is completely full and then blogging about what happens – hence this article. I was a bit reluctant at first since I didn’t want the hassle of reinstalling anything if it became corrupted somehow, but since I have up to date backups of everything, I decided to proceed anyway.

The first step was to run Disk Cleanup to make sure that the disk wasn’t full of a bunch of transient temporary files or something silly like that. Then the next step was to actually fill the disk. This sounds like an easy task, but Windows is pretty good at preventing the disk from becoming full in the first place. After filling the disk to within 500KB, I suddenly discover that Windows has managed to free 300MB on its own. This shocked me because I had just run Disk Cleanup and there was (as far as I was concerned) very little that Windows could remove. I did however receive a notification in the system tray:

I began filling up the remaining 300MB of disk space with more files and finally managed to get the disk to 0 bytes of free space for a brief moment:

As you can see, the drive is completely full. But not so fast – I had hardly finished doing this when Windows somehow (miraculously) managed to free up another 308MB of space on the drive. I was beginning to wonder what was getting deleted at this point since I had no idea what was on the disk that took up so much space yet was not needed and could be deleted. Still puzzled, I filled the disk up once again and began launching programs and doing things that I would normally do to see if anything was broken or wouldn’t work. The results were quite interesting.

The first application I tried, Notepad++, was completely unhelpful since it would happily report that it saved a text document – only to find out later that it had silently failed and created a completely blank document. Another text editor, PowerPad, was a little bit more helpful when trying to save a file when there was not enough room:

The error message does contain the root of the problem though the additional technical details aren’t really helpful to an end user. When trying to save a web page with Internet Explorer 10, I was pleasantly surprised to see the following:

This error message was simple but completely accurate – a very good example of what an application should do when it encounters such an error. I began to try a few other applications and all of them were able to at least start without crashing or running into an unrecoverable error. iTunes was able to load my library and play / stream music though it did warn me:

Some more experimentation revealed little other interesting results – applications worked fine until they attempted to write to the disk in which case they would present some form of error message (well, all except for Notepad++). However, I was surprised to discover that every time Windows tried to scavenge some free space, it never touched the recycle bin which had about 30.1MB of data in it. I would have thought that would be one of the first places Windows would look to free space.

After I was satisfied that my experiment was finished, I deleted all of the files I had used to fill the drive and I am sure I heard the PC breathe a sigh of relief as free disk space shot up to a more reasonable number.

In closing, I would like to share a rather curious message that popped up once I had finished with my experiment and was deleting files:

DNS simply stands for Domain Name System, which is a hierarchy of Name Servers that have the intent to translate host names into IP addresses on a global scale. A name server hosts and/or caches these translations. In the case where they are at least hosted, the name server is often called a “DNS Server”. If you gave the host name superuser.com to a DNS server, it would give you an IP back. In our case, that would be something like 64.34.119.12.

How does it work?

The following image by Nirlog.com explains this very well:

Can you give an explicit example?

Let me give you an example on resolving superuser.com in a non-cached way:

• The name server that I use is 192.168.1.1; this is my local router which caches DNS entries. I gave it a Hosts file to reroute malicious domains. Here is an example, although I am using a different one which I can’t find anymore. I might update this post later to include it though…

• The router has the name servers 8.8.4.4 and 8.8.8.8 set, known as Google Public DNS.

• Google Public DNS will contact authoritative root name servers (ARN), which are the top of the DNS hierarchy. Then Google DNS will contact the .com name servers, the IP of which it received from the ARNs. After that, Google DNS will contact the DNS server for superuser.com, the IP of which it received from .com.

A simple run of whois reveals:

NS1.SERVERFAULT.COM NS2.SERVERFAULT.COM NS3.SERVERFAULT.COM

The IPs for these name servers are known as they are registered at the .com name server. When you register a new domain, you always have to fill in the DNS servers that you use, along with their IP.

• Google DNS will contact the DNS servers of superuser.com to find the IP of superuser.com, after which we finally get the IP I mentioned above:

nslookup superuser.com NS1.SERVERFAULT.COM Server: NS1.SERVERFAULT.COM Address: 64.34.119.33

Name: superuser.com Address: 64.34.119.12

Where can I find more information on DNS?

I would suggest you to read through Domain Name System and Name Servers for more details.

The name server terms are also used in different contexts, read Spiff’s answer for more.

So, as there is more than one DNS server, can I pick a different one?

It is a common misconception that an ISP that is closer to you is always faster and more secure: A short distance does not make up speed on its own nor does it make up security. Compare using the old single network cables out of metal to the newer multiple glass fibre cables systems, used in connections like EuroDOCSIS 3.0.

Other than propagation delay, you also have a processing delay on the server; I’m pretty sure that the DNS server at an ISP is less powerful to a DNS server hosted by Google, hence it is completely possible or a server that’s at a much greater distance will actually be faster than your ISP’s DNS server.

Hosting your DNS router/server yourself might be the fastest, though it wouldn’t have all the DNS records stored, which would make it not usable to surf the internet!

namebench by Google can help you pick out a better DNS server:

Are you a power-user with 5 minutes to spare? Do you want a faster internet experience? Try out namebench. It hunts down the fastest DNS servers available for your computer to use. namebench runs a fair and thorough benchmark using your web browser history, tcpdump output, or standardized datasets in order to provide an individualized recommendation. namebench is completely free and does not modify your system in any way. This project began as a 20% project at Google. namebench runs on Mac OS X, Windows, and UNIX, and is available with a graphical user interface as well as a command-line interface.

Here is the most important screen you will see when using namebench:

See their website for more screens. There is also a command line version available.

As an alternative, there is also DNS-Benchmark by GRC.

Caution though, it could be that DNS servers result in slower browser experience in some cases, but as I haven’t seen scientific proof on that, I would take that with a grain of salt. Many people have used more popular DNS servers like Google Public DNS and OpenDNS and haven’t had a problem with it. They are extremely fast and secure and I don’t notice any negative speed differences when surfing myself (with Google DNS).

Please feel free to take the survey multiple times if you regularly use multiple operating systems. You can visit my survey on Google Docs here.

You can take a look at the final results in this Google Docs spreadsheet.

While this does not always work in every browser (eg. some versions of Firefox), it does work in most browsers like MSIE and Google Chrome. It really depends on the implementation of how the URL is parsed, Firefox seems to not go beyond our usual ways to type a URL.

This is all about how the URL is stored. Many of you know that you can also access Google through their IP, eg. http://173.194.65.99. Now let’s see how much data storage that IP requires. As one character is 1 byte for ASCII, it takes 13 bytes to store the IP address. Or with Unicode (UTF-16) you will need the double, 26 bytes. Another way to store the IP is by taking each number and storing that apart, resulting in unsigned octets from 0 to 255 which each take a single byte, so that totals out at 4 bytes.

Now, another thing one can do to the storage of an IP address is not having our program read these four apart and use a function to convert them to and from the different data types. This is why sometimes programmers use the long integer format (which is also 4 bytes) such that they can access the IP by accessing a single variable. This ranges from 0 to 4,294,967,295; which actually means that there are only 4,294,967,296 IPs available in IPv4. Internally, the bits of the four integers are shifted so that they don’t overlap and fit in a long integer.

unsigned long ipAddr = ( a << 24 ) | ( b << 16 ) | ( c << 8 ) | d;

So, those programmers that don’t waste lots of space by storing the IPs by strings have two alternative ways to store them in integer formats. And that’s why some browsers are also capable of interpreting the combined long integer format. So now you know what’s really happening under the hood if you ever come across such a weird link again…

But, how does one make such a link?

Easy, you can find Long IP converters online like Elf Qrin’s Long IP Converter which basically call one of these conversion functions for you. As I mentioned before this only does apply to the IP address, so if a hosting hosts multiple sites on an IP address (or doesn’t configure the IP address to respond) then it might not work fine. This is also why for example the IP of YouTube will respond with the website of Google…

We want you for WTFriday!

If you are interested in writing a blog post like this one for your community to read, then check this out. It doesn’t have to be as informative as this one, just about that strange WTF that has been buggering you lately; why it occured and how you resolved it, if possible.

The purpose of the Super User Community Blog is to highlight what you want to see.

We are always excited to bring new writers and editors to the Super User Community Blog; so, if you are interested in contributing, please let us know! There are various ways in which you can contribute — ranging from your own stories to product reviews, tips and beyond. Don’t be afraid that you don’t come up with an idea because we already have some ready for you, but you are always welcome to share your own ideas…

Writing a blog post is simple:

1. Register at Trello and leave us a message in the Super User Blog Editor Room with your Trello user name, we’ll set up access for you so that you can see our ideas and share your progress. Consider bookmarking both links for your convenience.
   
   
2. You can start writing your post while you wait, we can later import it from any format into the WordPress Dashboard. Keep us up to date on your progress, so we can give you access to our WordPress Dashboard after you have shown a first draft, as well as proof-read and schedule your final version.
   
   
3. The Blog Editor Room and Trello are our main communication points about the blog, feel free to share your progress in either. Remember that we are here to help you…
   
   
4. Have fun! Enjoy expressing yourself, as well as being part of the Super User Community.

From idea to draft to finished post.

Blog posts mainly develop out of ideas and questions, for some you might have to do some research. From that point on you can think up the different paragraphs you will write in a draft, then it’s a matter of writing and rewriting them. Just writing one paragraph after another might not cut it for some…

Add some nice pictures for those that are easily distracted, get at least two other editors to proof read the blog post and we’ll schedule it for you. Please note that we don’t publish blog posts immediately, but intend to spread out the posts such that we regularly have new content and they are published at an optimal time. We’ll do this for you.

Don’t have a fear of writing, you know you have been doing it before!