¶File dialogs break in capture mode with UScreenCapture driver

If you are using VirtualDub with the UScreenCapture driver to capture video from the desktop screen, you may run into problems with the Set Capture File option in VirtualDub not bringing up a file dialog, at least on Windows 7. This appears to be due a problem within v2.0.14 of UScreenCapture.ax which also reproduces with Microsoft GraphEdit. I am trying to contact the vendor for a resolution, but have not gotten a reply as of yet.

Technical details:

VirtualDub's file dialogs fail to come up because of an internal error within the Windows common file dialog. The specific failure path is GetSaveFileName() -> DialogBoxIndirectParam() -> CreateWindowEx(ComboBoxEx) -> SetWindowSubclass() -> GlobalAddAtom(). That in turn is caused by UScreenCapture's worker thread calling SetProcessWindowStation() with a handle to WinSta0 that does not have the WINSTA_ACCESSGLOBALATOMS access right. This only happens while the capture driver is running, but as it pretty fundamentally borks UI process state, it's not something I can easily work around safely.

Comments

This blog was originally open for comments when this entry was first posted, but was later closed and then removed due to spam and after a migration away from the original blog software. Unfortunately, it would have been a lot of work to reformat the comments to republish them. The author thanks everyone who posted comments and added to the discussion.