Tuesday, 23 October 2012

BSOD Debugging Part 2: Debugging

Check out BSOD Debugging: Part 1 if you missed it.  OK, so you have your Windows debugging environment set up and you have a dump file to analyse. Lets get on with it!

Here I will show you how I use the debugger to get to the bottom of a BSOD issue I was having.  Your BSOD issue(s) will be completely different, however the basics of using the Windows debugger and investigating the Windows debugger output will be the same.

Using Windows Debugging tools to find a BSOD root cause

Start up WinDbg, and choose File - Open Crash Dump...


Open your MEMORY.DMP file.
Wait whilst the the debugger to does it's thing:

If you have set up your debugging environment as described in part 1, you will be using the Microsoft symbols server.  This is confirmed by the debugger when a debug is under way, as highlighted in the screenshot above.

Symbol search path is: SRV*c:\Symbols*http://msdl.microsoft.com/download/symbols

OK so what can we see then?

Firstly the details of the BSOD, or BugCheck as the debugger calls it:

 BugCheck 9F, {3, 84abc530, 82b65ae0, 86116de0}

OK, a STOP 9F... What is that?

Looking at this handy list of BSOD codes (pcsupport.about.com) a STOP 9F is described as follows:

STOP Error 0x0000009F: DRIVER_POWER_STATE_FAILURE
STOP error 0x9F means that the driver is in an inconsistent or invalid power state. STOP code 0x0000009F may also display "DRIVER_POWER_STATE_FAILURE" on the same STOP message.

So some sort of driver issue when the system is changing power state.

Next the debugger is suggesting that the BSOD was probably caused by ntkrpamp:

Probably caused by : ntkrpamp

OK, whats a ntkrpamp?  Searching for ntkrpamp at processlibrary.com shows the following:

So ntkrpamp is the NT Kernel.  That's not a driver!  However it is the kernel's job to run drivers.  We are going to need to do some further digging with the debugger.

Luckily, the debugger makes it easy for us.  Simply click the blue text !analyze -v in the debugger results window to perform some further analysis.
Here is the output:

Right, now we are getting there. 

FAULTING_MODULE: 93d04000 dne2000
 and
DEFAULT_BUCKET_ID:  WIN7_DRIVER_FAULT

WIN7_DRIVER_FAULT that ties in with the BSOD being caused by a driver power state failure.  However, what is this FAULTING_MODULE dne2000 ?

Lets click on the  dne2000 link in the debugger window to find out:

Oh look. dne2000.sys, a driver located in C:\Windows\system32\DRIVERS folder.

Wa-hey! Root cause found: The driver dne2000.sys is having issues, causing the BSOD.

Fixing the Root Cause

So we know that there is an issue with the driver dne2000.sys that is causing the Windows BSOD. Lets get Google (other search engines are available) to take the strain from here on in.

Something like dne2000.sys bsod should do us nicely.

The very first google search result points us to this discussion thread on MS Technet Forums  Which in turn links us to this page discussing an available DNE update

Following the advice and completing the installation of the DNEUpdate and after a bit of extensive testing, my STOP 9F BSOD's are a thing of the past.  Job done!

Conclusion

As I said at the top of the post, your BSOD may - probably will - be completely different to mine.  However the basics of using the Windows debugger and investigating the Windows debugger output will be the same.  The take away from this post is as follows:
  1. How to use the Windows debugger to open a BSOD crash dump
  2. How to find out more about the BSOD here
  3. How to find out about Windows processes here
  4. How to use the Windows debugger !analyze -v function to find out more about the BSOD crash dump
  5. How to use other links offered by the Windows debugger to find out more, such as file names etc
  6. Finally, once you have an idea of BSOD the root cause, use a search engine to investigate further.  Chances are you are not the first person to suffer with the issue!
Happy debugging.

- Chris

Tuesday, 16 October 2012

BSOD Debugging Part 1: Setup

By now I'm sure we are all familiar with the humble Windows 'Blue Screen Of Death' (BSOD), displayed when windows crashes. No? Strange as they do tend to pop up in the most unlikely places.  Have a look at these examples (click for larger images):


Pic: Clefty



Absolutely anywhere!!!

So what can be done to get to the bottom of these BSOD's, how can Windows be fixed so as not to suffer another BSOD in the future?  Simple.  Use debugging tools for Windows.  Here is how.

Capturing a Memory Dump File

First off, you are going to need to capture a memory dump file from the BSOD. A complete memory dump records all the contents of system memory when the system stops unexpectedly. A complete memory dump may contain data from processes that were running when the memory dump was collected.  Here is how to configure Windows to dump it's memory to disk:
  • Windows 2000/2003/XP: Right-click the My Computer icon, click Properties and then click the Advanced tab. In the Startup and Recovery section, click the Settings... button.
  • Windows Vista/Windows 7/Windows 8: Right-click the Computer icon, click Properties and then click the Advanced system settings link on the left pane. Click the Advanced tab and in the Startup and Recovery section, click the Settings... button.
From the drop-down menu in the Write debugging information section, you can select one of the following options:
  • Small memory dump – 64 KB in size, records the most important information about the problem
  • Kernel memory dump – A complete record of system memory; creates files at the size of the operating memory
Here is my setting:

All you need to do now is wait for your system to crash again!

OK, so you've created your MEMORY.DMP dump file.  Now we need to examine it using the Microsoft Debugging Tools for Windows.

Installing Debugging Tools for Windows

First off, you do not need to install the debugging tools on the system that is having the BSOD problem. Use another system if possible.  The system will need to have around 200Mb free disk space  and be able to access the internet.

Microsoft, in their wisdom, have decided to bundle the debugging tools in with their Windows Software Development Kits (SDK).  But have no fear, we don't have to download the whole SDK!

But before we can get to the SDK, we first have to install MS .NET Framework 4.5. Grab the Web installer or Offline installer from here (Microsoft)

Once you have .NET 4.5 installed, you can go ahead and grab the Windows 8 SDK installer from here (Microsoft)

Launch the SDK installer, and choose "Install to this computer":

Choose whether to join CEIP:

Accept the license agreement. Now select ONLY "Debugging Tools for Windows" and click Install:

Once installation is complete, click Close to finish.

Using the Windows Debugging Tools

Windows Debugging tools will buried away in your start menu under "Windows Kits":

Don't worry if you only have X86 version installed.  This version will debug 64bit crash dumps without issue.  The program you are looking to run is WinDbg(x64) or WinDbg(x86)

Next you need to set the debugging symbol file path:

Whats a debugging symbol?  Here you go (Wikipedia). We need to set the debugger to use Microsoft's very own symbols server and cache them locally in C:\Symbols. This is entered as follows:
The text entered is:

SRV*c:\Symbols*http://msdl.microsoft.com/download/symbols

Click OK, then click File - Save Workspace to save.  Open Windows explorer and create a new folder in the root of C:\ called Symbols:

That is it for this part folks.

You have your MEMORY.DMP dump file and you have setup your Windows debugging environment.  Carry on to part 2: Debugging available here

- Chris

Wednesday, 10 October 2012

Clone Windows Installed Roles and Features

Mmmm a Roll with a Ham Feature...

Quite often I'm asked to build servers that need to have identical configurations in terms of the Windows roles and features installed.


Firstly, What are server roles, role services, and features?  Have a look at this article to understand roles and features and the difference between a Windows role and a Windows feature.


Export and Import of Windows Roles and Features

OK, so I've manually installed a selection of required roles and features on the first of my servers and that server is working perfectly.  How can I export a list of roles and features installed on the working server so that I can build the second server?

Powershell to the rescue!
This is where the powershell Server Manager Module comes into it's own.  It has three handy cmdlets, they are:
  • Add-WindowsFeature
  • Get-WindowsFeature
  • Remove-WindowsFeature
 Lets look at the Server Manager Module in action:

PS> Import-Module Servermanager
PS> Get-Command -Module Servermanager


So if we list all installable modules:

PS> Get-WindowsFeature


Lots and lots!  OK lets pair this down and see if we can list only those roles and feature installed.  We use this command:
PS> Get-WindowsFeature | ? { $_.Installed }


Nice.  Now we know which roles and features are installed. Handy for documentation or comparison.

HOW TO: Export Installed Roles and Features to File

Firstly you need to export that nice list obtained above into something structured that can be used by the other server to install the required roles and features.  For this we use the Export-Clixml cmdlet.  This will allow us to export the list of installed roles and features into an xml file. Here is the command with the xml creation:

PS> Get-WindowsFeature | ? { $_.Installed -AND $_.SubFeatures.Count -eq 0 } | Export-Clixml .\RnF.xml

The resulting xml file looks like this:


OK, lets copy this xml file over to our target server ready for the import.

 HOW TO: Install Roles and Features from File

Dead simple, using the Import-Clixml cmdlet:
 
PS> Import-Module Servermanager
PS> Import-Clixml .\RnF.xml | Add-WindowsFeature


Quick check in Server Manager GUI:


Looks Good. Job done!


So there you have it then. Two commands, one piece of xml, two servers running the exact same set of roles and features.

- Chris

Tuesday, 2 October 2012

New Release: Cisco Discovery for Windows v1.3


Yes, that is correct dear reader, your favorite cable tracing tool just got a version bump.

Whats new in this release?  Here you go:

Release 1.3 - 02 Oct 2012
  • Move GUI elements around to better handle longer switch names 
  • Saving CDP data will now also append to an existing file

The full change log is available here on github

Download the new v1.3 version here from github

New to CDP? Find out more here

- Chris

Newer Posts Older Posts Home