Windows: JNA fails to load libraries if SetDefaultDllDirectories() was used

[SyntevoAlex]

1. Version of JNA and related jars - any since 93ebb9f
2. Version and vendor of the java virtual machine - any
3. Operating system - any Windows
4. System architecture (CPU type, bitness of the JVM) - any

Example snippet:

public interface Dwmapi extends StdCallLibrary {
}

public interface Kernel32_2 extends StdCallLibrary {
	Pointer AddDllDirectory(WString NewDirectory);
	boolean SetDefaultDllDirectories(int DirectoryFlags);
}

public static void main(String[] args) {
	final Kernel32_2 kernel32 = Native.loadLibrary("kernel32", Kernel32_2.class);
	final int LOAD_LIBRARY_DEFAULT_DIRS = 0x00001000;

	// SetDefaultDllDirectories() is used to supply another DLL search path.
	// But it causes JNA to fail later.
	Path pluginsPath = Paths.get("plugins").toAbsolutePath();
	kernel32.SetDefaultDllDirectories(LOAD_LIBRARY_DEFAULT_DIRS);
	kernel32.AddDllDirectory(new WString(pluginsPath.toString()));

	// This is perfectly fine, but fails after SetDefaultDllDirectories().
	Native.load("dwmapi", Dwmapi.class);
}

Explanation:

1. Since 93ebb9f, JNA uses LOAD_WITH_ALTERED_SEARCH_PATH in LoadLibraryExW().
2. MSDN says: If this value is used and lpFileName specifies a relative path, the behavior is undefined
3. When library name is given like "dwmapi", this is a relative path.
4. Windows logs this error, but still tries to proceed - set WinDBG breakpoint to see for yourself: bp ntdll!LdrpLogRelativePathWithAlteredSearchError "du @rcx" (here, even the function's name is already a proof)
5. However, if SetDefaultDllDirectories() was also used, ntdll!LdrpGetDllPath considers this a fatal error and returns STATUS_INVALID_PARAMETER

TLDR: JNA uses LOAD_WITH_ALTERED_SEARCH_PATH and this is a bug to use it with non-absolute path. However, in most cases it still works, because Windows is kind to interlopers.

[SyntevoAlex]

The workaround is to set custom load options:

Map<String, Object> options = new HashMap<>();
options.put(Library.OPTION_OPEN_FLAGS, Integer.valueOf(0));
Native.load("dwmapi", Dwmapi.class, options);

I think that the solution would be to test if path is absolute and don't use LOAD_WITH_ALTERED_SEARCH_PATH if it's not.

I have not tested if using LOAD_WITH_ALTERED_SEARCH_PATH with a relative path like plugins\foo.dll will cause LoadLibrary() to search foo.dll's dependencies in plugins. If it does, then even though it's a bug, it still has some beneficial consequences.