¶How to cause a stack overflow in Notepad using DirectShow filters
To cause a stack overflow in Windows XP Notepad:
- Find and install two DirectShow audio or video playback effect filters, and set both of them to automatically activate in Windows Media Player.
- Start Notepad, and choose File > Open.
- Change the file filter to All Files.
- Hover over a video file.
Boom.
This crash occurs because the standard Windows Explorer-based file dialog tries to use DirectShow to determine video file specific information for the video file using DirectShow, and because of the lame way in which some DirectShow filters force themselves into the filter graph created by Windows Media Player, which as a side effect can screw up other programs that use DirectShow. If you install DirectShow filters that have such an "auto-enable on playback" option, please leave it off when using other audio/video programs that may use DirectShow, such as video capture applications.
Why this happens:
DirectShow constructs video filter graphs using an "intelligent" mechanism that attempts to find filters that will get a graph to a desired result. The Registry contains a list of video filters, along with their class (filter, compressor, capture source, etc.) and priority. The filter graph manager keeps adding the highest compatible filters to a graph until it finally has an output that a renderer can accept, finishing the graph.
Say you're trying to play a video file, for instance.
Well, it's an AVI file, so a File Reader and an AVI Splitter are inserted. The AVI Splitter outputs compressed video, which the Video Renderer can't handle, but both the AVI Decompressor and MJPEG Decompressor can. The MJPEG Decompressor has higher priority, so it gets inserted, and it now outputs RGB video that the Video Renderer can handle. And we're done.
The intelligent connect/render mechanism has the advantage that filters can be added to the Registry and they will automatically enable new connection paths that weren't possible before. The disadvantage is that some of the paths that the filter graph manager creates are a bit... suboptimal. Like using a Motion JPEG compressor followed by a Motion JPEG decompressor to convert between two different RGB formats.
How does this pertain to stack overflows? Well, the way that some filters insert themselves into the Windows Media Player filter graph is to advertise themselves as a renderer at maximum priority. This ensures that they are always inserted first into the graph, but then they still have an output pin that needs to be rendered. The filter graph manager then tries to insert another instance of that filter, but its query function returns false this time because the filter refuses to connect to an instance of the same class. The filter graph manager then tries the next renderer filter, which succeeds. The problem is that if two filters are installed that try to do this, the filter graph manager keeps inserting both of them into the graph alternatingly in an ever-increasing chain until the thread dies with a stack overflow due to excessive call depth.
In the case of VirtualDub, the two places you are likely to encounter this problem is either trying to open a video file or selecting a capture device in capture mode. Stack overflows are really nasty in that there is only ~4K of stack to use when the exception is thrown; VirtualDub's crash handler attempts to switch stacks ASAP by creating a new thread, and if you are lucky, the call stack in the Advanced view will show which DirectShow filters are responsible. If you're not, though, the application will simply vanish and you will have to guess. Usually it is undesirable for DSP filters to be active during a video capture, so it's best to disable all of them before capturing video.