Debugging

When 'almost' isn't good enough.

Bugs can be evasive, ranging from multi-threading issues to nasty problems caused by interaction with 3rd party software. Having your top engineers spend dozens of hours hunting a bug might not be the best use of their time and often a fresh perspective from an expert is what it takes to figure out the problem and deliver a fix quickly to the customer.

We are fluent in assembly language and were actually involved in developing one of the world's best known system level debuggers, so we truly understand how to approach a problem. Combine that with the fact that we have analyzed thousands of crash dumps and can often pinpoint a fix with very minimal information (e.g. the offset of the crash) and the potential for time savings is clear. Next time a nasty bug bogs down your team, get in touch and let us help get it resolved quickly.

Our primary debugging environment is WinDbg and WinDbg Preview, and we are fluent with the full suite of kernel debugging extensions including bugcheck analysis, NTSTATUS decoding, NTSTATUS code lookup, pool tag inspection, and handle leak tracking. We routinely work with ETW (Event Tracing for Windows), Driver Verifier, and Application Verifier to reproduce intermittent failures. For crash dump work we use our custom built, internal tooling to accelerate triage. Memory leak detection, IRQL violations, and use-after-free errors in kernel-mode code are areas where our depth of experience consistently shortens time-to-fix.

We can work from crash dumps you supply without needing a live session at all. For time-critical production issues, such as a driver causing Blue Screens on customer systems, a hang that blocks a product launch, or a performance regression that showed up in a new Windows update, we offer priority turnaround. Contact us to discuss the situation and we will put the right resources on it immediately. For broader driver projects and development work, see our Windows driver development services.

What Kernel Debugging Involves

Kernel debugging is the practice of diagnosing failures and performance problems in code that runs inside the Windows kernel: device drivers, file system filters, network drivers, and security agents that have no user-mode equivalent. The discipline requires specialist expertise because the operating environment has none of the safeguards that make user-space debugging tractable. There is no process isolation, no memory protection between components, no standard library, and no tolerance for unhandled errors. A bug in a kernel driver can corrupt memory used by an unrelated part of the system, producing a crash whose call stack points nowhere near the actual cause. Live debugging requires a second machine connected via a debug cable or a virtual machine configured for kernel debugging; you cannot attach a debugger to the system that is crashing. These constraints demand experience with specific tools, specific workflows, and enough prior cases to recognize failure patterns quickly.

Debugging Services We Offer

• Crash dump analysis. Submit a minidump or full kernel dump and we will perform a complete analysis: call stack reconstruction, pool corruption detection, deadlock identification, and root cause isolation. In many cases a crash dump contains enough information to identify and fix the problem without a live session. We use internally developed tooling to accelerate triage of dump files.
• Live kernel debugging. For problems that require dynamic investigation, such as inspecting kernel data structures at runtime, setting conditional breakpoints on specific execution paths, or tracing IRP flow through a driver stack, we connect a kernel debugger to a test system and work through the problem interactively.
• BSOD root cause analysis. When a driver causes Blue Screen of Death errors on production or customer systems, we analyze the bugcheck code and parameters to identify what code triggered the crash and why, and we deliver a written root cause analysis, not just a patch.
• Race condition and deadlock diagnosis. Concurrency bugs in kernel code are timing-dependent and often do not reproduce under a debugger. We use Driver Verifier, ETW tracing, lock ordering analysis, and targeted instrumentation to find them without requiring a reliable reproduction on demand.
• Memory corruption and pool leak tracking. Kernel pool corruption produces crashes at a distance from the write that caused them. We use pool tagging, verifier options, and custom WinDbg extensions to identify the corrupting allocation and trace it back to the driver responsible.
• Performance profiling. When a driver introduces unacceptable latency or CPU overhead, we use ETW traces and Windows Performance Analyzer to locate the bottleneck and propose a targeted fix.

Common Scenarios

The most urgent situation clients bring to us is a production BSOD with no reliable reproduction: customer machines are crashing in the field, the crash cannot be reproduced in the lab, and only a minidump is available. Nearly as common is the intermittent crash that appears under load or on specific hardware, the kind that disappears the moment you slow down to attach a debugger. Slow memory leaks in non-paged pool are a third recurring problem: the driver gradually consumes kernel memory that is never released, the system degrades over hours or days, and a reboot is the only short-term fix. For any of these, our reference tools can help your team get started: the NTSTATUS code reference and the NTSTATUS decoder are useful first steps when interpreting unfamiliar error codes in crash analysis output.

What You Receive

At the conclusion of a debugging engagement you receive a written root cause analysis document describing what failed, why it failed at that point in the code, and what change is required to fix it. This document is written for the engineers on your team who will implement or review the fix. For crash-related work it includes the specific bugcheck code and parameters, an analysis of the faulting call stack, and our reasoning for the root cause conclusion. If the fix requires a code change, we can deliver it together with the analysis or leave implementation to your team. Contact us with the details of your situation and we will let you know what information we need to begin.

Frequently Asked Questions

Q: What debugging tools do you use?

Our primary tool is WinDbg, including kernel-mode live debugging and offline crash dump analysis. We also use IDA Pro, Driver Verifier, ETW tracing, Application Verifier, and custom kernel extensions for pool and handle leak tracking.

Q: Can you analyze Windows crash dumps remotely?

Yes. Send us your minidump or full kernel dump and we will analyze it remotely. In many cases a crash dump alone contains enough information to identify the root cause and propose a fix, without needing a live debug session on your hardware.

Q: What types of bugs do you specialize in?

We specialize in kernel-mode bugs that are notoriously hard to reproduce and fix: race conditions, IRQL violations, pool corruptions, deadlocks, use-after-free errors, and driver interaction issues caused by third-party software. We also handle user-mode crashes and hangs when they have a kernel-mode component.

Q: How quickly can you typically resolve a critical driver bug?

For bugs accompanied by a good crash dump or a reliable reproduction case, we can often identify the root cause within hours. A complete fix with testing typically follows within one to a few business days, depending on how deep the change needs to go. Get in touch to discuss your specific situation.

Our Services

• What our customers say about us?

  

  Read our customer testimonials to find out why our clients keep returning for their projects.

  View Testimonials