¶InterVideo, you suck
I always have problems debugging DV capture in VirtualDub because I don't actually have a capture device that outputs it. Recently, I got the idea to insert the DV Video Encoder filter into the DirectShow filter graph, which would then create DV on the fly. So I set the magic registry key to use it as the transform filter, started a debug session in Visual Studio, and set the capture device.
VirtualDub vanished.
Did it crash? Nope, no exception dialog. Stack overflow? Nope. Failed exception unwind? Nope. The debugger just showed all threads in the process exiting with return code 2. That's not a failure code.
Long shot: Set a breakpoint on {,,kernel32}_ExitProcess@4... what do you know, breakpoint hit. WTF? Who's causing my process to exit?
The answer:
0:000> ~1kb
*** WARNING: Unable to verify checksum for C:\Program Files\InterVideo\Common\Bin\MP4VDEC.ax
*** ERROR: Symbol file could not be found. Defaulted to export symbols for C:\Program Files\InterVideo\Common\Bin\MP4VDEC.ax -
ChildEBP RetAddr Args to Child
016aeda4 01d6fb1f 00000002 01037e28 01037c30 kernel32!ExitProcess+0x1
WARNING: Stack unwind information not available. Following frames may be wrong.
016aede0 01d42be4 80000000 01d819b0 01037c30 MP4VDEC!DllCanUnloadNow+0x215af
00000000 00000000 00000000 00000000 00000000 MP4VDEC+0x2be4
Gee, thank you. Let's try a known good program, Microsoft GraphEdit. Insert the InterVideo MPEG-4 Video Decoder, everything's OK. Try it again with GraphEdit running under a debugger... oh, look, ExitProcess() got called. Hmm, I wonder....
0:005> bp kernel32!IsDebuggerPresent
0:005> g
ModLoad: 75f40000 75f51000 C:\WINDOWS\system32\devenum.dll
ModLoad: 77920000 77a13000 C:\WINDOWS\system32\setupapi.dll
ModLoad: 76c30000 76c5e000 C:\WINDOWS\system32\WINTRUST.dll
ModLoad: 77a80000 77b14000 C:\WINDOWS\system32\CRYPT32.dll
ModLoad: 77b20000 77b32000 C:\WINDOWS\system32\MSASN1.dll
ModLoad: 76c90000 76cb8000 C:\WINDOWS\system32\IMAGEHLP.dll
ModLoad: 736b0000 736b7000 C:\WINDOWS\system32\msdmo.dll
ModLoad: 10000000 10054000 C:\Program Files\InterVideo\Common\Bin\MP4VDEC.ax
Breakpoint 1 hit
eax=7c813093 ebx=00000000 ecx=7c919aeb edx=7c97c0d8 esi=008f14d8 edi=7c800000
eip=7c813093 esp=0006ec64 ebp=7c81cdda iopl=0 nv up ei pl nz na pe nc
cs=001b ss=0023 ds=0023 es=0023 fs=003b gs=0000 efl=00000206
kernel32!IsDebuggerPresent:
7c813093 64a118000000 mov eax,dword ptr fs:[00000018h] fs:003b:00000018=7ffdd000
What a surprise. Oh look, it tests the return value and conditionally calls ExitProcess(). How incredibly lame. This must have come out of "Software Protection For Dummies."
Essentially, the InterVideo decoder contains logic that detects debuggers and deliberately kills the application if one is found. Because this happens on load, it happens any time the decoder is queried, even for formats it doesn't support. I've run into this once before with the DivX decoder, but to the credit of the DivX guys, their decoder actually popped up a dialog saying that debuggers weren't allowed, and they turned it off after I complained. This just silently fires -- if you're lucky it exits silently, and if you're not, it causes some DLL cleanup code to crash the application in an utterly confusing manner.
Why'd I have this installed? Well, I think it came with some "InterVideo Home Theater" software that came with my Adaptec GameBridge. Which I will be uninstalling, not just because it interferes with my work, but also because, well, it sucks. As far as I can tell, you have to actually exit the program and run a separate config app to adjust the contrast.
In case you've been wondering why Windows-based systems are such a sea of instability, anti-social behavior like this is one reason. Drivers that are written for general-purpose APIs like DirectShow are guests in an application's process. I don't care how much developers want to "protect" their driver, they have no right to interfere with my ability to develop my software when I'm not even using their driver. Don't want your driver debugged? Don't load into my process and leave my program alone.
Don't support software developers that do lame things like this.