Cannot call ENB as a proxy library from 3D Vision/3Dmigoto

  • Author
  • Message
Offline
Posts: 23
Joined: 23 Jan 2018, 02:53

Cannot call ENB as a proxy library from 3D Vision/3Dmigoto

When attempting to call the ENB d3d11.dll as a proxy library from another similar package (in my case, the Helixmod 3D Vision fixes) in Skyrim Special Edition, it always results in a recursive loop resulting in a stack crash.

Since we cannot see the ENB code, we can only guess at what's causing it, but here's a description of what appears to be happening based on an analysis of the logs from one of the 3Dmigoto guys:

"Our hooks are interlocked... we call to the original call we found, which is ENB, which then calls the current version, which is us, and then we call them, and ... so on."

"The looping of our log definitely demonstrates that somehow the hook is calling itself again in a recursive loop until the stack crashes. I don't think there is any way for our code to do that, because we are really strict on how we call through to whatever we see as the original. We LoadLibrary the proxy before we hook the calls, so we presumably allow him to hook first, then we see his hook as our 'original'."

There are some contexts where using this load order is preferable. As I consider ENB a terrific work that is indispensable, but feel the same about 3D Vision, it would be very appreciated if you could take a look and see if this analysis is right or not. The 3Dmigoto code is open source, and the coders there have advised to please feel free to take a look at it if you believe the problem lies there instead.

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

1) For which game?
2) When stereo is applied by Migoto, my mod will not work properly anyway, because it use crc of shaders and some other important factors from render pipeline, so this kind of connection is useless (Fallout 4 and Skyrim SE versions will fail badly). But ENBSeries have proxy feature which allow to load other libraries instead of system one, but i think many features will be broken anyway in stereo. I need to modify code for compatibility for certain game and certain version of Migoto. In general, i load d3d11.dll from system path (if proxy feature not enabled in enblocal.ini), so problem is some kind different, i guess.
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
Posts: 23
Joined: 23 Jan 2018, 02:53

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

1. Skyrim Special Edition.

2. I was initially surprised to hear you say this... when I load ENBSeries first, and call the 3D fixes from 3Dmigoto via enblocal.ini, they appear to work quite beautifully together, at least with Rudy ENB for Obsidian Weathers. But then it occurred to me that I'm only using the color corrections, sharpening and SSAO. Bloom for sure, and DOF possibly, weren't working right, but I don't care for those effects anyway so I didn't consider them much of a loss, and they hardly ever work right with 3D anyway even without the ENB. The color corrections, sharpening and SSAO-SSIL work brilliantly though. Oh, and the subsurface scattering appears to work quite nicely as well.

Honestly, my main reason for trying to reverse the load order was as a possible workaround for the following issue I posted last week, wherein the normally working load order (ENB calling 3DFix) doesn't work when called from MO2:

http://enbdev.com/enbseries/forum/viewt ... =21&t=5777

As you can see there, the MO2 developer is willing to take the full blame for the issue, but he did state at one point that he was short on time, and it struck me as plausible that addressing the issue of not being able to call the ENB from the 3DFix might at least theoretically manage to circumvent that issue as well.

Do let me assure you, if you're thinking that your ENB isn't worth installing if you're using 3Dmigoto/3D Fixes, that's definitely not the case. With just the features I listed that I'm using, the effect is still dramatic, and the two in combination are seriously spectacular.

So is it possible that, when the ENB is called as a proxy, that changing which d3d11.dll it returns to might make changing the load order work properly?

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

I only compute direct lighting intensity and ambient intensity (if remember now) by not using crc of each shader. But as i understand, Migoto modify all shaders via recompiling them, so their control summ is different very much. So i have no idea how my mod can work at all, many things corrupted and not linked to each other. Some shaders are triggers to some other things, like capturing depth or color data.
So is it possible that, when the ENB is called as a proxy, that changing which d3d11.dll it returns to might make changing the load order work properly?
I cant do any tests now. D3d11.dll from system folder is loaded on D3D11CreateDevice, D3D11CreateDeviceAndSwapChain, D3D11CoreCreateDevice, D3D11CoreCreateLayeredDevice, D3D11CoreRegisterLayers functions call (and they called by Migoto in this case). At same moment it loading d3dcompiler_46e.dll. Here is a source of loading library code:

Code: Select all

//
#include "Emain.h"
#include "ED3D11.h"


FUNC_D3DDisassemble		F_D3DDisassemble=D3DDisassemble;


class EProxyLibrary
{
public:
	BOOL		m_IsLoaded;
	BOOL		m_IsInit;//+++ not used yet
	HMODULE		m_ProxyLibrary;
	//IDirect3D9	*m_IDirect3D9;

	_D3D11CoreCreateDevice			pD3D11CoreCreateDevice;
	_D3D11CoreCreateLayeredDevice	pD3D11CoreCreateLayeredDevice;
	_D3D11CoreGetLayeredDeviceSize	pD3D11CoreGetLayeredDeviceSize;
	_D3D11CoreRegisterLayers		pD3D11CoreRegisterLayers;
	_D3D11CreateDevice				pD3D11CreateDevice;
	_D3D11CreateDeviceAndSwapChain	pD3D11CreateDeviceAndSwapChain;

	//functions
	EProxyLibrary()
	{
		m_IsLoaded=FALSE;
		m_IsInit=FALSE;
		m_ProxyLibrary=NULL;
		//m_IDirect3D9=NULL;
	}

	~EProxyLibrary()
	{
		m_IsLoaded=FALSE;
		m_IsInit=FALSE;
		m_ProxyLibrary=NULL;
		//m_IDirect3D9=NULL;
	}

	BOOL	Start(char *proxypath)
	{
		if (proxypath==NULL) return FALSE;

		m_IsLoaded=FALSE;
		m_IsInit=FALSE;
		m_ProxyLibrary=NULL;
		//m_IDirect3D9=NULL;

		m_ProxyLibrary=LoadLibrary(proxypath);
		if (!m_ProxyLibrary)
		{
			return FALSE;
		}
		m_IsLoaded=TRUE;

		this->pD3D11CoreCreateDevice=(_D3D11CoreCreateDevice)GetProcAddress(m_ProxyLibrary, "D3D11CoreCreateDevice");
		this->pD3D11CoreCreateLayeredDevice=(_D3D11CoreCreateLayeredDevice)GetProcAddress(m_ProxyLibrary, "D3D11CoreCreateLayeredDevice");
		this->pD3D11CoreGetLayeredDeviceSize=(_D3D11CoreGetLayeredDeviceSize)GetProcAddress(m_ProxyLibrary, "D3D11CoreGetLayeredDeviceSize");
		this->pD3D11CoreRegisterLayers=(_D3D11CoreRegisterLayers)GetProcAddress(m_ProxyLibrary, "D3D11CoreRegisterLayers");
		this->pD3D11CreateDevice=(_D3D11CreateDevice)GetProcAddress(m_ProxyLibrary, "D3D11CreateDevice");
		this->pD3D11CreateDeviceAndSwapChain=(_D3D11CreateDeviceAndSwapChain)GetProcAddress(m_ProxyLibrary, "D3D11CreateDeviceAndSwapChain");

		return TRUE;
	}


	void	End()
	{
		if (m_IsLoaded==TRUE)
		{
			if (m_IsInit==TRUE)
			{
				//if (m_IDirect3D9) m_IDirect3D9->Release();
				//m_IDirect3D9=NULL;
			}
			if (m_ProxyLibrary!=NULL)
			{
				FreeLibrary(m_ProxyLibrary);
				m_ProxyLibrary=NULL;
			}
		}
		m_IsLoaded=FALSE;
		m_IsInit=FALSE;
		m_ProxyLibrary=NULL;
		//m_IDirect3D9=NULL;
	}
};

EProxyLibrary	myProxyLibrary;
EProxyLibrary	mySystemProxyLibrary;//used when not correct third party proxy or it's path


BOOL	proxyloaded=FALSE;
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
BOOL	LoadProxyLibrary()
{
	ELinkFunctions();

	if (proxyloaded==TRUE) return TRUE;
	proxyloaded=TRUE;

	//load plugins
	{
		char	tempstr[512];
		memset(tempstr, 0, 512);
		//sprintf_s(tempstr, 510, "%s\\%s\\*.*", g_CurrentPath, _STR_CONFIG_EXTRAPATH);
		sprintf_s(tempstr, 510, "%s\\%s\\*.dllplugin", g_CurrentPath, _STR_CONFIG_EXTRAPATH);
		WIN32_FIND_DATA	fd;
		HANDLE	hFind=FindFirstFileA(tempstr, &fd);
		if (hFind!=INVALID_HANDLE_VALUE)
		{
			do
			{
				if (!(fd.dwFileAttributes & FILE_ATTRIBUTE_DIRECTORY))
				{
					char	tempp[512];
					memset(tempp, 0, 512);
					sprintf_s(tempp, 510, "%s\\%s\\%s", g_CurrentPath, _STR_CONFIG_EXTRAPATH, fd.cFileName);
					HMODULE	lib=LoadLibraryA(tempp);
					if (!lib)
					{
						lib=LoadLibraryA(tempp);
						sprintf_s(tempp, 510, "%s\\%s", _STR_CONFIG_EXTRAPATH, fd.cFileName);
					}
				}
			} while (FindNextFileA(hFind, &fd));
			FindClose(hFind);
		}
	}


	//this line is protected, added at end 1+4 symbols
	char	*STRING_SYSTEM_LIB="\\d3d11.dll";


	//make path first (required)
	char	tempdir[512];
	char	ConfigFilePath[512];
	memset(ConfigFilePath, 0, 512);
	memset(tempdir, 0, 512);
	sprintf_s(ConfigFilePath, 510, "%s%s", g_CurrentPath, _STR_CONFIG_FILE_NAME);


//	DownloadFile("http://enbdev.com/enbseries/fallout4patch.txt", "c:\\test.txt");

	//read config first to get file name of proxy
	BOOL	IsValid=TRUE;
	char	*category=NULL;
	char	*key=NULL;
	int		defaultdata=0;


#ifdef EDISABLEPROXY
	qq_EnableProxyLibrary=FALSE;
	qq_InitProxyFunctions=FALSE;
#endif

	memset(tempdir, 0, 512);

	//first try to start 3rd party proxy
	BOOL	proxystarted=FALSE;
	if ((IsValid==TRUE) && (qq_EnableProxyLibrary==TRUE))
	{
		proxystarted=myProxyLibrary.Start(qq_ProxyLibrary);
	}

	//the reason to load the system library
	BOOL	loadsystemlib=FALSE;
	if (proxystarted==FALSE) loadsystemlib=TRUE;
	if ((proxystarted==TRUE) && (qq_InitProxyFunctions==FALSE)) loadsystemlib=TRUE;
	//load system library proxy
	BOOL	systemproxystarted=FALSE;
	if (loadsystemlib==TRUE)
	{
		memset(tempdir, 0, 512);
		GetSystemDirectory(tempdir, 490);
		strcat(tempdir, STRING_SYSTEM_LIB);

		systemproxystarted=mySystemProxyLibrary.Start(tempdir);
	}

	//check if need to terminate
	if ((systemproxystarted==FALSE) && (proxystarted==FALSE))
	{
		return FALSE;
	}

	pD3D11CreateDevice=NULL;
	//connect to 3rd party proxy if required, note that no need functions pointers if
	//not connect to it.
	if ((proxystarted==TRUE) && (qq_InitProxyFunctions==TRUE))
	{
		pD3D11CoreCreateDevice=myProxyLibrary.pD3D11CoreCreateDevice;
		pD3D11CoreCreateLayeredDevice=myProxyLibrary.pD3D11CoreCreateLayeredDevice;
		pD3D11CoreGetLayeredDeviceSize=myProxyLibrary.pD3D11CoreGetLayeredDeviceSize;
		pD3D11CoreRegisterLayers=myProxyLibrary.pD3D11CoreRegisterLayers;
		pD3D11CreateDevice=myProxyLibrary.pD3D11CreateDevice;
		pD3D11CreateDeviceAndSwapChain=myProxyLibrary.pD3D11CreateDeviceAndSwapChain;
	}

	//otherwise, need to connect system proxy
	if (systemproxystarted==TRUE)
	{
		pD3D11CoreCreateDevice=mySystemProxyLibrary.pD3D11CoreCreateDevice;
		pD3D11CoreCreateLayeredDevice=mySystemProxyLibrary.pD3D11CoreCreateLayeredDevice;
		pD3D11CoreGetLayeredDeviceSize=mySystemProxyLibrary.pD3D11CoreGetLayeredDeviceSize;
		pD3D11CoreRegisterLayers=mySystemProxyLibrary.pD3D11CoreRegisterLayers;
		pD3D11CreateDevice=mySystemProxyLibrary.pD3D11CreateDevice;
		pD3D11CreateDeviceAndSwapChain=mySystemProxyLibrary.pD3D11CreateDeviceAndSwapChain;
	}

	//check if connected
	if (!pD3D11CreateDevice)
	{
		return FALSE;
	}

	return TRUE;
}



//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
void	UnloadProxyLibrary()
{
	EUnlinkFunctions();
	myProxyLibrary.End();
	mySystemProxyLibrary.End();
	return;
}



//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
//
//++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
BOOL	iscompilerinked=FALSE;
HMODULE	compilerlibrary=NULL;

void	ELinkFunctions()
{
	if (iscompilerinked==TRUE) return;

	//default functions
	F_D3DDisassemble=D3DDisassemble;

	//43 compiler have bug with no "_aofimmi" output and wrong "discard" type

	//d3dcompiler_46f.dll patched for more float precision %.9f
	//d3dcompiler_46e.dll patched for exp float precision %e
	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46f.dll");
	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46f");
	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46e.dll");
	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46e");

	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46.dll");
	if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_46");
	if (compilerlibrary==NULL)
	{
		#ifdef EDEBUG
		ELogString("Failed to load d3dcompiler_46.dll, trying different");
		ELogString(NULL);
		#endif
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_47.dll");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_47");
/*		//these are buggy
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_43.dll");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_43");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_42.dll");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_42");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_41.dll");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_41");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_40.dll");
		if (compilerlibrary==NULL) compilerlibrary=LoadLibraryA("d3dcompiler_40");*/
	}
	if (compilerlibrary!=NULL)
	{
		void	*m_func=NULL;
		m_func=(void*)GetProcAddress(compilerlibrary, "D3DDisassemble");
		if (m_func!=NULL)
		{
			F_D3DDisassemble=(HRESULT (WINAPI *)(LPCVOID, SIZE_T, UINT, LPCSTR, ID3DBlob**))m_func;
			iscompilerinked=TRUE;
		}
	} else
	{
		#ifdef EDEBUG
		ELogString("Failed to load d3dcompiler libraries");
		ELogString(NULL);
		#endif
	}
}



void	EUnlinkFunctions()
{
	if ((iscompilerinked==TRUE) && (compilerlibrary!=NULL))
	{
		FreeLibrary(compilerlibrary);
	}
	compilerlibrary=FALSE;
	iscompilerinked=FALSE;
}
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
Posts: 23
Joined: 23 Jan 2018, 02:53

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

Just wanted to say thanks VERY much for posting that. Can't ask for anything more at the moment. It's being reviewed and when we figure out what's needed to sort out the issue, we'll let you know here. Thanks again!

Offline
Posts: 3
Joined: 03 Feb 2018, 16:22

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

Thought I'd drop by to clarify a couple of things, and also report on the bug in our code.


You are right that when both ENB and 3Dmigoto are both active that it can and will break our 3D shaders. We do modify the shaders to fix broken 3D effect. However, we only do a fairly small subset of shaders, as small as we can get away with.

But, when there is a conflict, like we fix a shader for the sky because in 3D it is drawn too close, and ENB also fixes it to improve the visual quality, then it will be broken in one or the other because the hash would no longer match.

This will vary wildly depending upon the game. Sometimes we fix a lot of shaders, sometimes not. Mostly we'd expect conflicts in shadows, since we both are fixing them. Still, this is why Qwinn can get away with running both at once though- he's only using shaders that do not conflict.

If ENB is named d3d11.dll and 3Dmigoto is the proxy, then they both load and work at the same time. This is not the scenario that Qwinn needs though because of constraints in the loader he needs.


I've found and fixed a bug in our code for proxy loading. We setup a hook on LoadLibrary and force any calls to that back to the game exe folder. Some games and the nvidia driver try to override the normal dll search chain, and we override that to be sure to be able wrap the game.

This broke our proxy loading mechanism though. In proxy load, we load ENB first, then do GetProcAddress on the interesting pieces like CreateDeviceAndSwapChain. With our hook on LoadLibrary though, when the game calls CreateDeviceAndSwapChain, that then gets intercepted by the 3Dmigoto wrapper on the call, we do our thing, then call through to the proxy- which is ENB. ENB then calls what it thinks is the System32 version, but... had the wrong DLL after it's LoadLibrary, because 3Dmigoto blocked the access to System32. So, ENB calls 3Dmigoto's version again, which calls ENB, which calls 3Dmigoto, and so on recursively until we get to boom.

I've been testing in Fallout 4 using both 3Dmigoto and ENB at once to verify behaviour. I've fixed it to not force the LoadLibrary back to the game folder, as an option in our d3dx.ini file. This is not always required, and having an option fixes this problem. Some games it is required like WatchDogs and AlienIsolation, so it's not a complete answer, but gives us some flexibility.


This now works for 3Dmigoto to be named d3d11.dll, and to load ENB as the proxy. I can tell that the proxy is working because I can see the requests for d3dcompiler_46e.dll, which are unique to ENB.

However, I'm not seeing the ENB overlay in this setup, even though it's clear the code is being called.

So, I wanted to see if there is anything obvious that would block ENB in this setup? Anything like noticing the shaders are patched and deciding not to run? If there is a log mechanism, I can take a look with that. Any suggestions here would be most welcome.

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

Hope i understood everything well. Locking occurs because ENBSeries loading system library inside CreateDeviceAndSwapChain function call, right? If so, you can just add new condition to hooked LoadLibrary by Migoto, which is activated right before CreateDeviceAndSwapChain call to parse valid d3d11.dll to ENBSeries. After its done, restore to previous state of hooked LoadLibrary. Not 100% sure about very prehistoric versions of my mod (like 0.074), but most likely i never called LoadLibrary in dllmain, every version should be the same (except maybe d3d8 to d3d9 convertor). Probably this already done, but just in case...

As remember, in all more or less recent versions i exporting NvOptimusEnablement, so it can be used for easy detection if it's ENBSeries or some other library (d3dcompiler_46e.dll sometime ignored by users and not copied to game folder).
However, I'm not seeing the ENB overlay in this setup, even though it's clear the code is being called.
Do you mean startup message of ENBSeries or editor? Start message in dx11 version use its own shader and font texture to draw strings, so maybe its shaders wrongly processed/patched by some reason. Or device states are modified (in such case better try to set something default). But if its editor do not work, then something inside AntTweak library code i suppose. Btw, if timer functions are hooked by Migoto, startup message may not occur.
So, I wanted to see if there is anything obvious that would block ENB in this setup? Anything like noticing the shaders are patched and deciding not to run? If there is a log mechanism, I can take a look with that. Any suggestions here would be most welcome.
viewtopic.php?f=29&t=496&p=4266#p4266 Check out first post attachment in that topic, made a logged build for Fallout 4, which generate enbseries.log in game folder, but be careful it outputs almost all wrapped functions, so very slow. There is also log from Fallout4 terminated after first frame on startup to compare.
Why mod can fail. First of all, if Present is hooked with no return, of course before this function ENBSeries draw messages, fps, editor, read input, some clearing. When shaders not compiled by d3dcompiler library (if its hooked). When objects failed to create because of some weird initialization parameters of d3d, like no back buffer resolution which required as global variable for my own render targets. When device state is modified by "crapware" and not restored back, of course it just visual errors, mod almost always continue to work. Dont remember any other scenario when startup message do not occur.
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
Posts: 23
Joined: 23 Jan 2018, 02:53

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

b03b has made some fixes to the code, I tested it, everything is working beautifully now, even running the proxies through MO2 is working fine.

Thanks tons for your help on this, Boris! Donation sent :)

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

Thanks, but i prefer perfect fix, not just temporary solution, so ill keep an eye on this topic.
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
Posts: 23
Joined: 23 Jan 2018, 02:53

Re: Cannot call ENB as a proxy library from 3D Vision/3Dmigo

Far as I can tell, bo3b's solution isn't a workaround, it's an actual fix. I'll check with him about that tho.
Post Reply