FixClassPaths

As a hardening measure, modern Windows applications can use SetDefaultDllDirectories to prevent DLL planting attacks. This can cause mis-registered COM servers that have otherwise been working fine to suddenly stop working. Here’s why:

SetDefaultDllDirectories tells the loader to only look in the specified directories for DLLs and their dependencies, typically the application’s own folder and system32. COM comes along and tries to load an InprocServer32 DLL and specifies the LOAD_WITH_ALTERED_SEARCH_PATH flag that tells the loader to look in the DLL’s directory for additional dependencies. So far, so good - the COM server DLL’s dependencies should indeed come from its directory and not the application directory. However, take a look at how COM servers are supposed to be registered. Despite the hilarious age of the article (“downloading a new component from an on-line service or receiving one from a friend on a floppy disk”), the point at the bottom is very important: The server must register the full path to the installation location of the DLL or EXE module for their respective InprocServer32, InprocHandler32, and LocalServer32 keys in the registry. If it does not, this is where things start to fall apart.

Looking at the MSDN documentation for LoadLibraryEx, a small note is mentioned next to LOAD_WITH_ALTERED_SEARCH_PATH: “If this value is used and lpFileName specifies a relative path, the behavior is undefined”. So if a COM server only registers a filename instead of a fully qualified path, we’re already in undefined behavior territory - for compatibility, it seems Windows still searches the default DLL directories and so the COM server is still found in PATH and loads successfully. However, if an application opts in to using SetDefaultDllDirectories, the loader behavior is drastically changed, and this becomes a hard error - ntdll!LdrpGetDllPath returns 0xC000000D (STATUS_INVALID_PARAMETER) prior to touching any DLLs on disk (so they won’t even show up in a ProcMon trace for example).

This problem has manifested in OBS Studio since version 32.2.0 where SetDefaultDllDirectories was added as a hardening measure. Since DirectShow relies heavily on COM servers for things like filters and property pages, any mis-registered server will silently fail to load. Unfortunately several 3rd party developers do not register their DirectShow components with fully qualified paths, including popular manufacturers like Elgato and their 4K60 Pro MK.2 SC0710.X64.AX driver.

The Fix #

Developers must ensure they register their COM servers with fully qualified paths, following MSDN documentation. Until then, I have released a tool which will enumerate all registered servers (both 32-bit and 64-bit) and fix up any without a fully-qualified path by searching for the file in the current PATH. Disclosure: This tool was largely developed with AI assistance.

Download: FixClassPaths.zip (30KB)

How To Use #

Run the included .exe as administrator (admin is required for modifying the registry). A list of all affected servers will be shown, along with the changes that the FixClassPaths would make. If everything looks good, press Y to accept the changes. A .reg file will be created with the old data as a backup if needed.

It is normal to see some keys that cannot be fixed - some Windows DLLs are in write-protected keys, and DLLs in the “Known DLLs” list are directly loaded instead of having their path searched, so will work even with non-qualified paths. Modifying the system registry is inherently risky; while the program has been tested to the best of my ability, use this at your own risk.