Recently at work we had some serious production environment issues. Our web application basically made the IIS application pool terminate unexpectedly. And we didn’t get a single stack trace or a line of code. The only thing that was logged inside of the Windows Event Viewer, was event id 1009, which basically told that the application pool serving our app terminated unexpectedly. And that was it. All active user sessions were wiped, and the users had to login again.
The error started after we did a major upgrade. A lot of our code had changed since the previous version, so we set up an new website and application pool inside IIS 6.0 for the new version, so we could upload the new application, and then turn off the old one, to get as seamless an upgrade as possible. It worked fine, and we were happy. Until a few hours later, when we saw our users were logged off.
At first we thought it was IIS settings we had done wrong, so we did a complete comparison with the old website and application pool to see if we forgot anything. There was no difference at all. After a while, we realized that the only thing that could make the IIS crash like that, was our own code. But with no clue of where the danger in our code were – we were lost.
After some time, we got WinDbg attached to the worker process of IIS, serving our application. We caught a few memory dumps, but those were not of the exception we were looking for.
Later I found an HttpModule for ASP.NET. Basically the .Net framework 2.0 has changed the way it handles exceptions from other threads in IIS. In ASP.NET 1.1, unhandled exceptions from asynchronous threads inside IIS were ignored. But in ASP.NET 2.0, those unhandled exceptions makes the IIS application pool crash.
Make the IIS crash yourself
So try to do this. Create a new ASP.NET 2.0 website running on your local IIS. You only need a single page, so you’re fine with the default.aspx Visual Studio creates for you. Add a button to the page, and create an event handler for the buttons OnClick event.
Add this to your code-behind:
The buttons click event handler uses the ThreadPool to execute the MakeIisCrash() method. This method instantiates a DataSet as null, and the sets a property. Since the DataSet is null, this throws a NullReferenceException which is not handled, as you can see.
Click the button, and see how IIS will crash.
You can sense that your computer is working harder – that is because IIS terminates the application pool. If you go and check the Windows Application log, you can see that is has added an error:
Now this doesn’t tell you much. Imagine if you had hundreds of thousands line of code – how would you find the cause of the error?
Inside Visual Studio, add a new class to the website – call it UnhandledExceptionModule, and apply this code:
#region Using using System; using System.Diagnostics; using System.Globalization; using System.IO; using System.Runtime.InteropServices; using System.Text; using System.Threading; using System.Web; #endregion///
Modify the httpModules section inside web.config to look lige this:
This is your new friend. It is a must for any ASP.NET 2.0 application running in production environment. This will catch your exception, and write the message and the stack trace to the Windows Application Event log, and now you can see what caused the crash:
So after we applied this HttpModule to our production environment, we get a nice entry in our Application event log from ASP.NET. It includes a stack trace, which makes us able to find the problem, and fix it!
Further reading regarding ASP.NET production environment issues:
- Tess Ferrandez – She has a lot of great posts about debugging ASP.NET using WinDbg
- David Wang – HOWTO: Understand and Diagnose an Application Pool Crash
- Advanced .NET debugging
Hope this will help someone.