Shaping problems into solutions

Chris Walton recently posed the question:

How to troubleshoot a problem when one has no idea where to start?

where he is looking for general information on how to get started on troubleshooting a problem under intermittent hardware & software conditions, where you have the expertise to handle the causes but not the problem itself.

This is a very useful question which relevant to most of us, it serves as a good starting point when you don’t have a clue what you should next. That’s why we formulated a methodical problem solving approach which we will outline here for you…

Get a better idea.

You ain’t going to win a battle without sufficient field information.

A problem exists in a context. If we want to think how we can solve the problem, we should first think about the problem context to get a better idea. This will allow us to troubleshoot the problem in more accurately and into more detail, instead of trying irrelevant solutions…

Let’s go through steps that help you reach this:

1. Describe your problem

Describe your problem in detail so that you have a good idea, who knows it’s just once.

You could simply state out that you have a problem, but that results in this bad example and no specific solutions:

I am currently encountering a problem with my new machine. On a few occasions the machine has just frozen, how do I solve this problem?

Or… You could spend effort and describe your problem, which results in this good example and perhaps to a solution to the problem:

I am currently encountering a problem with my new Windows 7 machine. On a few occasions the machine has just frozen; not accepting keystrokes, mouse clicks, or anything except the power on/off switch. Invariably I have been merely browsing the web; I have had a few (<= 6 other applications) running. None of these applications are major; and represent a mix of commercial programs and open source programs, typically migrated from Unix of some variety.

My machine is a high powered laptop; is my main machine; is used for development and technical writing, communications – email, web, FTP, etc, and for photo editing and indexing. A rigorous and extensive suite of hardware test programs,including CPU tests, multiple memory tests, and tests on all other components are run on it at least monthly. Also run at least monthly are a full virus scan; a full spyware scan; a disk cleanup; and a disk defragmentation. The disk contains approximately 3*10^6 files; disk usage is 300Gb leaving 150Gb free. Memory is 8Gb.

While the machine can get slightly warm when I am running a full complement of major development tools, I have encountered the problem only when using the machine very lightly – web browsing plus Textpad plus Graphwiz plus a Firebird database plus a lightweight database browser (Flame Robin). In these circumstances even the fan is not slightly warm. I have made no changes to software, operating system or hardware over the period I have encountered the problem. There have been a number of automatic updates occur – Microsoft, Adobe, and Lenovo mostly but not exclusively

As you see, there is a lot more detail in this description which eliminates assumptions and allows us to see the problem in more detail.

2. Track back in time

Track back in time what you and your computer did before and during the problem.

Thinking about the things you did to your computer and the things that your computer reports to you can help to give you an idea of the possible cause(s) of your problem. Good places to look are logs and file modification times, think about new hardware you added or software/drivers you have installed/updated.

@JRobert‘s answer lines this out, it’s all about good logs and intuition – really.

• From day 1, keep track of everything you do to the system: app & OS updates, new installs, new or removed hardware or connections, the thunderstorm that “didn’t cause a problem”.
• When you first noticed the issue:
  • What had you been doing?
  • What else unusual happened recently?
  • What have you done differently recently?
  • From then on, keep aware of what you’re doing so the next time it happens, you have a better handle on what had just preceded it.
  • Snapshot the system logs.
• See if you can you reproduce it. Until you can reproduce it, you can’t find it.
• Start partitioning the system: safe mode vs. running live, new account vs. your regular account, different keyboard and mouse than your regular ones (esp. bluetooth vs. wired), does it happen within a few minutes of starting or waking vs. only after an hour more of running (think thermal).

So, for example, it’s monday and last friday you’ve tried out a new developer tool during the pause but you forgot to uninstall it after you thought it was bad.

3. Think of the possible causes

Think of possible causes because sometimes they might not be obvious.

In the before mentioned problem, freezing might be CPU related because it processes things. This could be a process, a driver or even malfunctioning hardware…

By thinking about this, we know that we need to look how the processes and drivers are functioning and if any of our hardware is bad.

4. Get more information

Get more information whenever you have no idea of what’s happening, this could range from Events, to SysInternals Tools, to Performance Analysis, to Debugging, and to any other tool.

Mark Russinovich’s Blog has a lot of cases which show you how to use part of the above tools to solve nasty problems, for those that want to get their hands more dirty there is also the Ntdebugging Blog.

On Super User, there are some questions of interest:

• How do I create a memory dump of my computer freeze or crash?
• How do I create a trace of a performance problem which I can later analyze?
• What files are accessed often?
• What are the essential tools you always have handy when attempting to fix someone’s PC problem?

5. Test your assumptions

Test your assumptions to be sure that your thoughts don’t filter the cause away:

For example, if you assume a CPU related problem in this example; we can easily check this which various tools from the most simple Task Manager to the full-blown XPerf Analysis Tool for Windows, and other tools for your specific OS.

Divide and conquer.

Because that’s how the military defeat their opponents even when outnumbered.

Eliminate the possible causes, or you’ll have a problem keeping track of the problem. This way, you will get closer and closer to the root cause of the problem, it allows you to solve the problem a lot easier.

For example, with hardware, disconnect and remove anything that you don’t need for fixing your problem. This way, you might disconnect the component causing the problem. And then it’s again a matter of inserting half the components in, checking if it reoccurs and repeat splitting till you have the bad component…

Testing something on another computer, if available, is also a good benefit towards solving the problem.

For example, with software, rebooting into safe mode, disabling start-up entries also helps. This also applies to enabling/disabling settings, trying the default configuration and so on…

Let’s put it to the test.

Here is the first paragraph we had earlier, Chris didn’t include more yet so this is what we had by the time of testing:

I am currently encountering a problem with my new machine. On a few occasions the machine has just frozen; not accepting keystrokes, mouseclicks, or anything except the power on/off switch. Invariably I have been merely browsing the web; I have had a few (<= 6 other applications) running. None of these applications are major; and represent a mix of commercial programs and open source programs, typically migrated from Unix of some variety.

1. That’s a proper description by itself, it doesn’t just happen once.
2. He knows what happened together with the problem, but hasn’t thought of things he or his computer did before the problem.  He, his event log and recently modified files/folders could tell more about it.
3. A possible cause is that it is most likely to be CPU related, because that component that processes things.More specifically this could be a process, a driver or failing hardware (perhaps temperature problems?).
4. We know it’s the processor, but we don’t know what causes it. Events don’t show this, Process Explorer could for example hang on Deferred Procedure Calls.So, as a next step, we let a trace analysis run with XPerf which we close after the hang has occured.Then, upon looking into the trace, we see that driver X is causing the problem!
5. No real assumptions are made. The processor assumption is checked above and handled by our Divide & Conquer approach…

This is where we start dividing to conquer the problem, we stop once the problem is resolved:

1. Problem with current version of the driver? Update the driver to the latest version.
2. Problem with newest versions of the driver? Get a new trace. Update the driver to an older version different from the initial.
3. Problem with the device? Configuration problem in the registry? Get a new trace. Reinstall and/or disable the device if possible.
4. Problem is random, is it the processor heating up? Check the processor temperature, replace fan if needed.
5. Problem is not the processor, are there other hardware and software influences? Remove hardware and disable software from running, to nail down third-party influence.
6. Problem is not in a removable part, it should be replaced. In the worst case, if all else fails, you need to go for a replacement.

So, as you can see… Getting new traces and removing hardware gives us more information, so we know where to look next.

And by now, you should be able to shape a computer problem into a solution with ease. 🙂