New users, please be careful with antispam system, for you allowed image hosts are imgur.com, imageshack.us, tinypic.com
It is currently 25 Jul 2014, 08:57

All times are UTC





Post new topic Reply to topic  [ 406 posts ]  Go to page 1, 2, 3, 4, 5 ... 41  Next
Author Message
PostPosted: 16 Jan 2014, 21:20 
Offline
User avatar

Joined: 11 Sep 2013, 22:34
Posts: 56
Image

Disclaimer

This patch will require your first born and a pint of your blood.

What bad can happen when Skyrim uses more memory without tripping over itself?
Fried Hardware maybe? If anything bad does happen then it is your problem since I already got your firstborn and I do not care. This information is provided as is. I am sharing it because I am hoping a large part of the community will benefit from it. Take it or leave it.

Be sensible about memory use and modding done right in general - refer to S.T.E.P. and Skyrim Revisited. If you want to go beyond the default limits of the game you need to have a stable foundation. Modding requires dedication and patience and you have to read a lot.

The information below helps my(!) Skyrim with higher ugrids and many mods. For the love of the holy cow make sure to only use well documented and tested ini settings or spend some time testing them thoroughly yourself. If you can not properly verify cause and effect all you do is wasting time. No need to mess with unknown, untested, unproven settings or mindlessly copy and paste made up things that have no effect.

tl;dr just give me the setting

For additional memory add this to the .ini
[NotPlacebo].
GiveFirstBornToSheson=1

Yes. Really.




Updates

2014, February 9

2014, January 29
  • Added MemoryBlocksLog 1.2 SKSE plugin for simple logging. No functional changes. Public release on Nexus
  • Removed all old versions of MemoryBlocksLog. Download latest version from Nexus

2014, January 28
  • Added MemoryBlocksLog 1.1 SKSE plugin for simple logging

2014, January 27
  • Added MemoryBlocksLog 1.0 SKSE plugin for simple logging

2014, January 24
  • SKSE says it is better to use Visual Studio 2008 than 2010

2014, January 20
  • Added How much memory to pre-allocate section
  • Added Recommended section and note about precache killer

2014, January 19
  • Added a link to more detailed post from ZerOxShadows how to compile

2014, January 18
  • Added a note about the need of Stable uGridsToLoad in requirements
  • Added hint to run updates after installing a 4 year old program from MS

Time for Skyrim Memory Patch 3.0

It has been a while since the first memory patch that fixed 4GB LAA that made it into the official game. Thanks to Boris we have the second memory patch ENBoost taking care of textures not wasting main memory anymore. In this naming tradition this third memory patch will allow Skyrim to use that available space right from the start to fix ILS (infinite loading screen) without side effects and other memory related CTD (crashes to desktop) or freezes. After many hours of peeking and poking it turns out there is a simple fix and one has to wonder why this is not something that is implemented as an option or parameter for the PC version of the game.

How it works

When tesv.exe is started it allocates two 256MB blocks of memory. When the first block gets full, the engine will allocate more blocks. This can cause the known troubles. Thankfully, by telling the engine to request a bigger block from the start it magically makes use of it without any further ado. This isn't the case with the second block. Thankfully again, the second block does not fill up as quickly and once it is full the engine does not trip over itself when allocating more blocks.

To make the engine allocate a larger block of memory it needs to be patched. This needs to happen very early in the process. This can not be done by a SKSE plugin because they are executed much later. However, SKSE loader already patches tesv.exe before the game starts and since SKSE sources are provided adding the patch is straight forward.

Because of the nature of this patch I will only provide instructions how to do this.
I contacted Boris about this and he is doing his own tests and investigations. If this can be part of ENB I do not know. Best not to waste his time with silly questions :)

Will this help your ILS, CTD or freezing?

You have a 64bit OS and lots of main memory. You have decent hardware. You already use ENBoost and do not run out of VRAM. You went through all your mods, used tes5edit to clean them and use BOSS and what not. You searched and read lots of discussions and guides and spend hours over hours fixing and optimizing everything.

If you use safety load this fix will most likely help you to replace it and not have any side effects.

You can get VMMap and check if the first memory block is fully used. It is a yellow private data block that at first has 2 childs. Once all its memory is commited it only has one child and both the size and committed column showing the same number 262,144K.
Image
This is not problematic in itself, but this condition raises the chance for ILS, CTD and freezing and this fix will most likely help avoiding any of that.

If you have "random" CTD you need to try to make them reproducible.
For example, typically the committed number grows when crossing cell borders but maybe also shrink when busy cells are unloaded. If it reaches 262,144K troubles may start. Use tb for toggleborder in skyrim console to show cell borders and if you freeze or CTD right on a yellow line this fix will most likely help.

SKSE can write minidumps that can be analyzed in what part of the program the game crashed and why. Even if this doesn't mean anything to you, you can check if the crash is the same every time. If this crash data would be supplied whenever people talk about CTD it could help tremendously in finding related discussions to a particularly crash condition and over time eliminate a lot of guess work.

If you happen to come across
Code:
FAULTING_IP: TESV+bd832
004bd832 0f298660910000 movaps xmmword ptr [esi+9160h],xmm0

this fix will help.

Basically, if you ever see that tesv.exe committed all of the memory block with the single child it is very likely this fix will help. It may even improve performance when zoning and also give you a homecooked meal once a day.

Magic

Here is a well documented patch that you can put into SKSE steam_loader main.cpp at the end of
Code:
void InstallHook(void * retaddr, UInt32 hookSrc)

just before
Code:
FlushInstructionCache(GetCurrentProcess(), NULL, 0);

There may be a better way into implementing this, but hey I didn't write SKSE. I simply hack my(!) stuff into it for my(!) own needs.
I will happily update anything if someone more knowledgable provides feedback about it.

Code:
   UInt32   enableMemPatches = 0;
   if(GetConfigOption_UInt32("NotPlacebo", "GiveFirstBornToSheson", &enableMemPatches))
   {
      if(enableMemPatches)
      {
         _MESSAGE("Sheson took your first born in exchange for more memory");

         //These patch addresses only apply to tesv.exe 1.9.32.0.8
         //do not use this on any other version

         //1st block, request 512MB (0x0300) instead of 256MB (0x0200)
         //if too small cause of ILS or CTD when to much cell+buffer data does not fit anymore and engine allocates more memory.
         //tesv.exe:00687E73   sbb     edi, edi
         //tesv.exe:00687E75   push    0
         //tesv.exe:00687E77   and     edi, 0x0FFFFE00
         //tesv.exe:00687E7D   push    0x158
         //tesv.exe:00687E82   mov     ecx, offset 0x01B418B0
         //tesv.exe:00687E87   add     edi, 0x200   <-----
         //tesv.exe:00687E8D   call    0x00A48D60
         //tesv.exe:00687E92   test    eax, eax
         //tesv.exe:00687E94   jz      short 0x00687EB1
         //tesv.exe:00687E96   push    1
         //tesv.exe:00687E98   push    0
         //tesv.exe:00687E9A   push    offset 0x010CD3C8
         //tesv.exe:00687E9F   push    0x5500000
         //tesv.exe:00687EA4   shl     edi, 0x14
         //tesv.exe:00687EA7   push    edi
         //tesv.exe:00687EA8   mov     ecx, eax
         //tesv.exe:00687EAA   call    0x00A4DCF0
         SafeWrite32(0x00687e87+2, 0x00000300);

         //2nd block, request default 256MB (0x10000000), typically seems big enough
         //changing this value is not recommended because side effects are not yet properly tested
         //when 1st buffer is set over 512MB and there are a lot of mods this needs to be raised as well or tesv.exe crashes at start
         //however there are no commits over 256MB, hard coded max limits later in the code
         //tesv.exe:00A4E6B1   mov      ebp, dword ptr 0x0106B268
         //tesv.exe:00A4E6B7   push   4
         //tesv.exe:00A4E6B9   push   0x2000
         //tesv.exe:00A4E6BE   push   0x10000000   <-----
         //tesv.exe:00A4E6C3   mov      [ebx+0x8D08]
         //tesv.exe:00A4E6C9   mov      [ebx+0x8D0C]
         //tesv.exe:00A4E6CF   mov      [ebx+0x8D10]
         //tesv.exe:00A4E6D5   push   edi
         //tesv.exe:00A4E6D6   mov      [ebx+0x8D14]
         //tesv.exe:00A4E6DC   call   ebp
         //SafeWrite32(0x00a4e6be+1, 0x10000000);

         //change movaps to movups (slower on old CPUs) in case memory is not 16byte aligned,
         //there are most likely more, but when 1st buffer large enough this is not problem anymore
      
         //this one is executed once per new cell load
         //tesv.exe:004BD82F   xorps   xmm0, xmm0
         //tesv.exe:004BD832   movaps   xmmword ptr [esi+0x9160], xmm0
         SafeWrite8(0x004bd832+1, 0x11);
      }
   }


Compiling your own skse_steam_loader.dll

For the love of the holy cow, please do not ask me for help with this. Check out this post from ZerOxShadows for a bit more detail on the process.

  • Get SKSE sources, doh
  • Get a free Visual Studio 2008, get service packs and run windows updates to make sure this has all the updates
  • Open SKSE src/skse/skse.sln
  • add patch to steam_loader/main.cpp
  • in top menu change dropdown from Debug to Release
  • in left pane right click steam_loader, build
  • you will find new skse_steam_loader.dll in src/ske/Release
  • if Skyrim does not start just put original skse_steam_loader.dll back

Requirements

  • 64bit OS - 32bit may work but untested, probably should use /userva switch, but if you are that desperate good luck
  • a decent amount of RAM - more than 4GB
  • a suitable graphics card with decent amount of VRAM - depending on texture sizes
  • ENBoost - need to free up main memory

Recommended

  • Stable uGridsToLoad - if you want to test with higher uGrids. You may not really need it, but it fixes a recursion bug that could potentially cause CTD with default uGrids as well.
  • You may still need a precache killer to load all the assets. Remember this fix is not making the game use more memory it just tells it how much memory for a certain block to anticipate.

Verify it works

Remember to add this to skse.ini

[NotPlacebo]
GiveFirstBornToSheson=1

If you look at the code you can see how this becomes a real setting ;)

Check skse_steam_loader.log for the message about your offspring.

Compare VMMap

Before: two 256MB allocations.
Image

After: one 512MB and one 256MB allocation.
Image

Disable Safety Load and test if ILS are fixed. Test with any old saves you have, travel back and forth Solstheim.
You can still use Safety Load, but if you need it, it basically means the 1st memory block is not large enough for your setup.

Do speed runs, you should already know about tgm and player.setav speedmult

Use Skyrim Performance Monitor to keep an eye on VRAM

How much memory to pre-allocate

There is no need to pre-allocate a much larger block than the game will actually need. In fact doing so can have a negative impact, because other parts of the game need memory as well and the more is pre-allocated to this specific block the less is available to the other processes.

Remember, we are not telling the game to use more memory, we are telling it to pre-allocate the first block in one big continuous chunk instead of the default small chunk and later using additional smaller chunks somewhere else.

If you look at this screenshot of VMMap you can see that from the 524,288K of the first block 94,464K are not used. It is not much in this example, but it also means there is no use pre-allocating any more than this.

When testing for the highest needed value of your setup, you need to find the busy spots with lots of cell data. Windhelm Harbour comes to mind, the Rift or Solitude Harbor. It is typically the exterior spots where you had ILS. You can lower your value with a buffer in mind.

Any Megabyte size should work, but only experience will show if there needs to be a multiple of something.

Only raise the second block if the game doesn't start. It is a workaround at the moment and should only really be needed when testing limits with high uGrids or many mods.

Download MemoryBlocksLog. This is a SKSE plugin that writes the sizes of the two memory blocks to a logfile whenever more memory is committed to them. Check the readme.txt in the zip for more info.

Important Notes - Pay Attention

It is only recommended to raise the first block to 512MB and leave the second block alone. Like with all new mods this needs to be tested. Extensively.
This fix should be suitable for a lot of setups, provided there is enough main memory, a decent graphics card with plenty of VRAM and some common modding sense.
This will simply raise the limit of when the engine will trip over itself again when overloaded. If VMMap shows the same number in both size and committed column the engine is in trouble.
Raising uGrids can have effects on quests and game play, depending on installed mods.
It is your responsibility to find out what is safe for your setup.

Desired Feedback

If this fixes something I didn't describe yet.

If this fix works for you on exotic hardware or 32bit OS.

Please be detailed and thorough in any report.

Skyrim.ini

For the love of the holy cow, please get rid of all the nonsense settings. For starters you can check the tesv.exe for its content with notepad.
All settings are there to read. saveini does not dump everything it seems or there are strings that are not used.
Whatever wierd things you copied and pasted in your futile attempts to make higher uGrids stable, get rid of them now.

[Grass]
iGrassCellRadius=X
A little known setting, make this half uGrids if you want to raise fGrassStartFadeDistance in prefs.ini


Legal?

Use this info for whatever you like. This will stop working if for some strange reason a new version of tesv.exe comes out. Fat chance.

Please link back to this original post on ENB forum instead of copy and pasting.

If you use this in your loader, patch, mod or whatever I would be honoured if you keep the setting name. As an alternative make a dancing holy cow animation for one of MMOxReview videos. Please give credit if you use this.

Do not repost MemoryBlocksLog anywhere please.

The Sky is the Limit

Yeah, so I can set 768MB for both and raise uGrids to 17/19 with 100s mods and textures packs etc. The only practical use is pretty? screenshots. Papyrus is pretty much dead at this point. Test what is possible if you like. You understand the consequences. It will fry your hardware, mess up your save game and you will never be able to have kids again.
It is your responsibility to find out what is safe for your setup.

Rhetorical

Can we filter out the crap from the Internet please?
Can you please stop mindlessly repeating stuff that you read somewhere else as fact and thus adding to the crap on the Internet?
So SKSE can write minidumps and nobody uses that information to share and discuss?
You do not have a certain ini and need to ask about it?
So much time and resources wasted to answer the same basic questions over and over again. RTFM, RTFA?

If you need help with anything explained here I can not help you.
Please help yourself or patiently wait until someone spoon feeds you.

Did I mention that it is your responsibility to find out what is safe for your setup.


Credits

Thanks to Boris for his continuous work on ENB, ENBoost and having a Forum
Thanks to the SKSE team for SKSE and its sources
Thanks to who explains and document things for other to learn from it
Thanks to who can help themselves with the information provided.

Image
Nord Cattle - HD cows

Holy Cow I fixed Skyrim - Sheson

==============================================================

_________________
Sheson
Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes
Dynamic Distant Objects LOD - Finally I can see my house from here


Last edited by sheson on 09 Feb 2014, 05:06, edited 28 times in total.

Top
 Profile  
 
Tomoko
PostPosted: 16 Jan 2014, 22:36 
Offline

Joined: 04 Mar 2012, 09:57
Posts: 24
Just gave it a quick read but this is very interesting. Thanks alot for sharing and I'll report back if it worked and helped. :)


Top
 Profile  
 
PostPosted: 17 Jan 2014, 00:17 
Offline

Joined: 08 Nov 2012, 15:24
Posts: 18
Have you mentioned this to the authors of SKSE? If it isn't better implemented through some other method they could likely add it as an official feature, as they did with the higher-quality face overlays.


Top
 Profile  
 
PostPosted: 17 Jan 2014, 00:26 
Offline
User avatar

Joined: 11 Sep 2013, 22:34
Posts: 56
roxahris wrote:
Have you mentioned this to the authors of SKSE? If it isn't better implemented through some other method they could likely add it as an official feature, as they did with the higher-quality face overlays.


I send an email to the SKSE team. Time will tell if this patch is stable enough across all platforms etc. and if they feel it is safe enough to implement in their code.

_________________
Sheson
Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes
Dynamic Distant Objects LOD - Finally I can see my house from here


Top
 Profile  
 
PostPosted: 17 Jan 2014, 20:56 
Offline

Joined: 04 Mar 2012, 09:57
Posts: 24
You weren't joking about the noplacebo entry in the ini. :D

After some testing I must say it works for me. I could avoid reproducable CTDs when crossing cellborders while using ugrids 7 and 9.

Then I tested and raised ugrids ingame via console (stable ugridstoload is installed) up to 15. Worked. Changing it back down to 9 from 15 gave me a black screen. :D

Will test a second .dll with both blocks set to 762. I'll report back.

/EDIT
Ok tried with a second dll, but with 1024. Works perfectly. 8-)

Image
uGrids 15


Top
 Profile  
 
PostPosted: 17 Jan 2014, 23:40 
Offline
User avatar

Joined: 11 Sep 2013, 22:34
Posts: 56
sinister.sinner wrote:
Then I tested and raised ugrids ingame via console (stable ugridstoload is installed) up to 15. Worked. Changing it back down to 9 from 15 gave me a black screen. :D


Changing uGrids ingame is of course a bit risky :)
Would be nice if you could report OS version amount of main memory.

_________________
Sheson
Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes
Dynamic Distant Objects LOD - Finally I can see my house from here


Top
 Profile  
 
PostPosted: 18 Jan 2014, 00:25 
Offline

Joined: 04 Mar 2012, 09:57
Posts: 24
Windows 7 64bit
16 GB DDR3 RAM

Figured out changing uGrids ingame is less of a problem if you disable alle mapmarkers first.


Top
 Profile  
 
PostPosted: 18 Jan 2014, 05:20 
Offline

Joined: 18 Jan 2014, 04:27
Posts: 29
I can also report that this appears to work perfectly for me so far in my testing, I will even add a little story to show what I have tried.

I have a clean save at Alvors house in Riverwood (straight after the intro to there, have done the meeting with Alvor and Hadvar, using just the main game, the 3 DLC and the Official High Res), i exit the house and run back to the standing stones, fighting 2 wolves on the way, opening my inventory randomly in combat and during the trip, once at the stones i turn on no clipping float above the stones and look around.

If I have any mod problems I can load this save and it will always crash somewhere on this route, usually after the wolves on the path up the road just before the embershard turn off, sometime i will get crashes opening my inventory or looting the wolves. If it crashes i can start the game up and get it to reliably crash same place every time, i would then have to figure out which mod or mods to enable or disable, last time it was strangely Moss Rocks, even though it had been running fine with it before.

I have been using an ENB no problem (Somber Lut Sepia), however one common thing I had noticed was that after these crashes Skyrim Performance Monitor showed that game memory peaked at a similar point each time (roughly 1024), and this seemed to be the crash point, so as long as i enabled and disabled the correct combination of mods to keep the game peaking over this point I could play.

After reading the OP through, more than once I may add, since I really like to understand what is happening before I try this sort of things and as he mentioned there are a lot of suggested changes (ini edits) that are undocumented and do nothing, and following a few of the tests, like the mini dump and VMMap, I felt confident that conclusions drawn were solid.

My issues were occurring without any uGrid changes and minimal ini edits, just what is needed for the ENB to run and that is it, I created 2 versions of the dll as per the OP's instructions, one with the default settings shown above (512 and 256) and one with a little higher start up (768 and 512), in both cases I can now run my full mod list without any game play issues, I am no longer able to get my replicable crashes, in fact after over 20 test runs I am yet to see any crash, I also see that the performance monitor now shows higher peak memory use, I am also able to switch to higher uGrid values but am yet to really test it, 7 seemed stable however.

Between the two versions of the DLL there seems to be no difference in peak use, which is to be expected, however I have seen some variance in stutter during cell border crosses, I will be honest I am unsure if it is really a difference or more of a case of i am seeing what i want to see.

In regards to my system, I am running Windows 8.1 x64, on an i5 3570K at 3.4GHz, with 16Gb RAM and a Gainward Geforce Phantom GTX780 GLH with 3Gb VRAM, the game is installed on an SSD, I run Driggers Texture Pack Combiner with a mixture of 1k and 2k textures, In note to my hardware, my CPU and GPU temps and usage numbers are no different than i would normally see.

I hope my wall of text will be helpful to others in deciding whether this is worth a try, I am still in the process of testing these DLL's and am yet to give it a test over a longer period of time, I am also yet to test without pre-cache killer for character creation and safety load, one change at a time here, but initial testing is positive :D


Top
 Profile  
 
PostPosted: 18 Jan 2014, 10:19 
Offline

Joined: 18 Jan 2014, 04:27
Posts: 29
Further testing so far shows no need for the safetyload after several hours of play, some load screens feel long (could just seem that way) but it has not frozen.

Also I removed pre-cache killer, and although a momentary freeze whilst it loads character creation once loaded it is really smooth and runs just as if you had no mods installed for character creation, I am running with Apachii Hair plus the Male and Female packs, Emo Hair and Oblivion hair.

On the performance front, I must admit i am floored, i did have one or two minor issues early on that turned out to be mod incompatibilities, once they were resolved everything was good and I am yet to have a crash or issue with the game, other than one or two places where i get a minor stutter over a cell border the game is playing without fault, I am still running with uGrids 5 at this point.

My next set of tests will increase the number of mods i am using and then try and up the uGrids.


Top
 Profile  
 
PostPosted: 18 Jan 2014, 15:12 
Offline
User avatar

Joined: 11 Sep 2013, 22:34
Posts: 56
Daetarek wrote:
Also I removed pre-cache killer, and although a momentary freeze whilst it loads character creation once loaded it is really smooth and runs just as if you had no mods installed for character creation, I am running with Apachii Hair plus the Male and Female packs, Emo Hair and Oblivion hair.


Thank you for your detailed and analytical feedback. Much appreciated. I was expecting this, but I just never had a testcase myself. This is great news.

I would expect hitching to due loading data be the same as before, just that it would not crash the engine anymore in certain conditions. It should be slightly faster since more memory space is already reserved now for operations that use that first block.

Higher uGrids is an easy way to tests limits, since it exponentially increases the engine need for buffers. Care is needed for real game play, since it is agreed on that too high a value will mess with quests and what not. However it will give an idea about the possible headroom. You will discover that many actors will slow down and limit scripting a lot ;)

_________________
Sheson
Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes
Dynamic Distant Objects LOD - Finally I can see my house from here


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 406 posts ]  Go to page 1, 2, 3, 4, 5 ... 41  Next

All times are UTC


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group