 |
Link to complete pk4? |
|
|
Trying to get as close to apples/apples as possible. 1 - Difficulty level? 2 - 32- or 64-bit TDM? 3 - Just start the mission and stand at the starting point, doing nothing? |
|
|
Ok so fired it up and just let it run (whithout moving) and the memory skyrocketed fairly quickly, started at 400mb and crashed at 2.1GB. But the fault module was different to last time. 1 - Difficulty level - none was a dmap/map 2 - 32- or 64-bit TDM? - 32bit 3 - Just start the mission and stand at the starting point, doing nothing? - yes Problem signature: Problem Event Name: APPCRASH Application Name: TheDarkMod.exe Application Version: 1.0.0.1 Application Timestamp: 5aa93125 Fault Module Name: nvoglv32.DLL Fault Module Version: 23.21.13.9101 Fault Module Timestamp: 5a9068c5 Exception Code: c0000005 Exception Offset: 00af55f3 OS Version: 6.1.7601.2.1.0.256.48 Locale ID: 2057 Additional Information 1: 0a9e Additional Information 2: 0a9e372d3b4ad19135b953a78882e789 Additional Information 3: 0a9e Additional Information 4: 0a9e372d3b4ad19135b953a78882e789 |
|
|
Got answers to my questions? |
|
|
# Full PK4 - https://mega.nz/#!PY0WTbKA!-ZW9qMtvePjh8Xj0FmjnALTKwBu2eFZTbyVla8KdT-s # Dumpfile for above crash - https://mega.nz/#!bN8DXLaa!Un7zvmBhbCELlBAAj4YiheOKedSMQGL1k5-x3bsoCYI Going to uninstall the nvidia driver, run driver sweaper and try both the x86 and x64 version and see if and when I get a CTD. edit my post above. |
|
|
Also, does it crash when you run on 2.05? |
|
|
Two things about the pk4: 1 - It's missing the startingmap.txt file, which causes it to crash at the start. 2 - The maps/assassins.script file should be named maps/city.script ??? |
|
|
Also, could use a copy of your Darkmod.cfg file to replicate your cvar settings. |
|
|
Watching memory use now. Appears to be rising around 4k/sec (or so). |
|
|
location separator 59 doesn't touch a portal. |
|
|
Initial run ... Set up the debugger to catch the crash. Started the mission. Memory started around 370mB. 30 minutes later it had climbed to 440mB. At the 60 minute mark, it was still sitting at 440mB. No crash. Ended the test. This was with my cfg file. Need to rerun the test with yours. |
|
|
# Attached autoexec and darkmod config files. # Added the missing startmap and readme files, and rename assassins.script to city.script # Fixed the VP intersecting seperator 59. # Did a test of 2.06x64 for 30 mins went upto 480MB # Doing a test of 2.06x86 atm, will report back. |
 |
I didn't see anything anomalous in your settings but you're another one who inexplicably has seta tdm_lg_split "0" (should be 1). Also, if I recall correctly, 2.06 allows you to have Soft Shadows without requiring Index Buffers which kills perf on Nvidia so I think you can set: seta r_useIndexBuffers "0" which I believe is the default. |
|
|
I don't remember setting those so will change both as per above. Back to testing with 2.06x86, its been open for an hour and the memory footprint is sitting at 2.235GB. I will leave it running but expect to crash at some point soon though. My system specs btw - Core i7 4790k nVidia 1070 - 391.01 Sb Xfi - latest driver Win 7 x64 sp1 16GB of DDR3-1333 |
|
|
To summarise I uninstalled the 391.01 nvidia driver, ran DDU (Display Driver Uninstaller), rebooted reinstall the driver (clean install) Ok so after nearly 2hrs it(2.06-x86) had risen to 2.36GB, I then tried to move around and it jumped to 2.458GB. The game is very laggy at this point. And then moved again for a little while and it then crashed with a malloc error. "Malloc failure for 1048576" Im gonna loadup BCD and see if that causes the same memory leak/usage. |
|
|
Corrected the cvars NB mentioned. Using BD's cfg files. Started at 360mB. 404mB after 30 minutes. 411mB after 60 minutes. No sign of a mem leak. |
|
|
Hmmm, wtf is causing the leak on my side then I wonder. Have you tried walking around the map? I ran BCD for 2 hrs (stood at the start location) and it never went above 400MB Am gonna try and run the mission from a .PK4 and not a custom game shortcut as atm thats the only thing I can think of being differ between Gman and my system. I will also run it on the laptop as that has a differ GPU in-case its an nvidia thing. |
|
|
Ok so used a .PK4 instead of the unpacked mission and normal TDM shortcut - # Win8.1/AMD-FX4800/GTX770/4GB, after 30mins ram was at 670MB, will do a longer test tomoz. # Win7sp1/Corei5-5200u/HD5500/8GB, after 60 mins the ram was at 640MB # Win7sp1/Corei7-4790k/GTX1070/16GB, after after 20mins ram was at 460MB, will do a longer test tomoz. Why would an archived(.pk4) version of the mission not use anywhere near the amount of ram compared to the uncompressed version of the same mission!!?? |
|
|
Archiving does squeeze large text files like the map file or the mtr files considerably but I have another suspicion about this. When the mission folder is open you might have long paths like darkmod\fms\my_mission\dds\textures\darkmod\stone\cobblestone\my_custom_stone in which the new Video encoder algorithm might be searching for pk4's to decompress. Depending on how that works, it might get into some recursion loop when the second "darkmod" folder entry is listed in the path... Now, since we have a workaround of packing the mission prior to testing it... can this be a 2.07 issue or is this critical for 2.06? |
|
|
And in answer to Gmans 2.05 question, TDM would run for far longer(hours) before I would see the memory usage hit 2GB. But I will do a side-by-side test of 2.05/2.06 x86 and chart the ram useage. |
 |
Did not manage to reproduce it on a first try. Things for me to try later: 1) use 32-bit version 2) use attached cfg-file 3) use 2.06 beta build instead of SVN. UPDATE: Tried to run 32-bit build of the most recent 2.06 beta --- still no leak (process explorer usage grows very slowly, and growth speed reduces over time --- still about 400 MB after 25 minutes). |
|
|
Did a test with the .PK4 above under 2.05 - Ater 34mins ram was at 375MB, but as soon as I started moving around (walking through the map, 10 seconds) the ram jumped to 452MB. Then after several minutes or running around the ram use increase calmed down after 580mb mark. I will do the same test with 2.06x86 after dinner. @Grayman, stgatilov - have you guys tried running around for a few minutes to see what your ram useage jumps to ..? |
|
|
I ran around for about a half hour. Mem usage went up to 1.6gB, but committed memory rose in turn to around 3gB. I assume that if memory used ever caught up to mem committed, it would crash. |
|
|
@Commited verses used, yes I think that's the case. On the testing front I let 2.06 x86 run for 3 hrs whith the player stood at the start spot and the ram rose to 2.01GB. So I imagine if I ran around the ram useage would hit the limiiter far sooner and crash. So that's gonna be my nest test and x86 and x64 version of the current build of 2.06 which i will do tomoz. |
|
|
Maybe I am not patient enough, but I cannot get anything looking like a leak. In my case the reserved virtual memory grows from 2 GB to 2.3 GB, which isn't critical. Also, the term "memory leak" is usually used when allocated memory never get deallocated. I have tried to run around, then quit mission, then start it again (without closing TDM), then run around some more. After the second running time, the memory usage was not greater than after the first time. This probably indicates that the additional memory was deallocated when I quitted mission. Then it is not a "memory leak". It may even be normal memory consumption of the mission. It is still not clear why it is so large then. The ugly workaround is to run 64-bit version. Then it won't crash at least. |
|
|
I am fairly sure I got significantly increased ram usage on the x64 version also, and will be testing that today. Have changed the title to reflect stgatilov's comment. Can I ask what OS & hardware your all running? |
|
|
Loading the map in debugger takes a very long time due to too many small memory allocations while loading LWO models. I have added a personal TODO to optimize memory handling in lwo loading (I think it is very inefficient there) after 2.06. But I think it should not cause leaks (all this memory must be gone by the time you start playing). Anyway, bikerdude, are you sure you really need 100 MB of custom architectural LWO models? Did we ever had a mission with so many custom LWO stuff? It is not that I discourage anyone, I'm just trying to guess what can eat so much memory compared to other missions. Do you have any custom LOD models in this map by the way? UPDATE: Ok, another reason for slow loading is that it has a lot of AAS/EAS. This is magnified by the fact that EAS is not quite efficient and uses STL, so it gets very slow when you run game from debugger. Another thing to look into later. P.S. Windows 10 x64, Ryzen 1600, 16 GB Ram. |
|
|
I had to export a bunch of brushwork to .LWO models to cut down on the amount of brushes in the map. I started with 32k which just cause TDM to crash on loading. No, because the map is all custom built. But I might have to look at LOD if some of the worst perf spots. So ran 2.06 x64 for 2hrs while stood at the start location and the ram hit 1.2GB, and within 1-2mins it jumped upto 1.4gb. I ran around for another 10-15 mins and it climbed to 2.5GB. Then stood still in the magic ship for a another 15mins and the ram useage just kept going up at roughly 20-1000kb per second but as soon as I went out into the garden it stopped climbing at such a high rate - sitting at 4.42GB So I went back inside the shop and stood at various spot to see where the ram useage was highest, and it was only when I looked down at the little furnace (witch has a moving torchflame) and that the ram skyrocketed again (@20-1000kb per second to start with after a minute of standing in the spot the ram useage increased at 500KB-1MB per second) Why the hell would a moving light cause such a massive memory usage? I am going to see if the same location does the same in 2.05 |
|
|
Doing a little test to see if its my larg map or 2.06 eating ram, I have exported the above little shop with the player looking at the moving light from the little furnace. Started at 90MB, within 10mins it rose to 150MB, gonna leave it running while I have some dinner. |
|
|
What is the light entity that seems to be causing the problem? |
|
|
I decided to look at memory usage in general. I'll write memory consumption in format (Current working set / Reserved virtual memory) after each major loading step. Note that this is Release x64 launched from debugger. Debugger increases memory consumption, as does the 64-bit mode. 0. Before SpawnMapEntities: (309M / 876M). 1. SpawnMapEntities/MapPopulate finished: (1272M / 1922M). This is the first big uprise in memory consumption. Models is one of the factors here (0004756), but I'm not sure how big it is. 2. SetupEAS finished: (2131M / 2750M). This takes a lot of time and highly increases memory consumption. 3. idImageManager::EndLevelLoad finished: (2039M / 4020M). It loads all the textures into OpenGL driver. I suppose it does not increase working set because OpenGL driver explicitly avoids caching the texture data on CPU side (using non-caching memory allows to interact faster with GPU, as far as I remember). 4. After Sys_SetPhysicalWorkMemory call: (200M / 4020M). This is a WinAPI call which tells Windows OS to remove as many pages as possible from the physical RAM. Of course, this memory is still used by the process, it remains in virtual memory and swap, waiting for the player to get into a place when it becomes needed. 5. At the "Press Enter to Start" screen: (300M / 4020M). 6. Just started the game: (480M / 4080M). As you see, measuring physical working set is simply the wrong way to measure memory consumption, because most of allocated stuff sits in swap when you start mission. If you simply stand still, then pages get slowly loaded back into physical memory (AI-s move around anyway, some other things happen). If you start running around the map, the engine has to touch more and more memory: it loads new geometry which was completely unnecessary before. So it is normal that physical working set grows, and grows much faster than when you stand still. So the question is: is this really a memory leak or abnormal memory consumption for the map of this size? If physical working set during playing does not exceed peak working set during loading, then probably the memory consumption just reflects the size of the map. I think that EAS eating 800 MB should be fixable (0004755, not for 2.06). It seems that textures do not eat physical memory, so we can forget about them. The main problem is about SpawnMapEntities eating 1900 MB, which needs more investigation. |
|
|
@Grayman, its just a regular light source - light_torchflame_moving. And I dont think this is the only reason for the high ram usage. @stgatilov, abnormal memory consumption for the map of this size sounds like the better explanation. SpawnMapEntities eating 1.9GB is as you say not good and what causes the 32bit version to crash I would have thought. And what is EAS? And moving versus standing still, ok that makes sense. As of today I have created 275 models that used to be brushwork (roof, windows, other geometry). You say thats a lot of models, but compared to the amount of stock/core models in the map the above number is small. One thing that might be causing abnormal ram use in relation to .LWO models is 30 of the 275 have tris counts from 10-80k |
|
|
Ok, after some failed attempts to measure memory more precisely, I have realized that we have forgot about the powers given to us by our ancestors =) Bikerdude, you basically need the following console command: printMemInfo Just type it in console after loading, then go to maps directory of your fms and read the text files there. It also prints the high-level stats into console. Here is the sample report I get in x64 (without debugger): Used image memory: 1,076,869,716 bytes Used model memory: 305,826,356 bytes Used sound memory: 261,989,167 bytes Used asset memory: 1,644,685,239 bytes These numbers are exactly the same in 32-bit mode. The models with 'shadowhide' in their path take about 211 MB of memory. But probably your models also include the ones without 'shadowhide', so the total should be a but larger. As you see, the pure size of assets is rather close to 2 GB limit of 32-bit mode. Add bad implementation of EAS to this, and you are somewhere near the border. I suggest using 64-bit game for your map, this is TDM's future anyway. Or trying to use less assets: either assets of smaller size or just less different models/textures in single map. |
 |
I remember running out of memory on older missions, but I am not sure if it's not an Intel driver bug. What is taking so much asset memory? Does this happen on existing missions? If not, then it's not to be fixed in 2.06. |
|
|
Here are the same stats for 1st mission of "No Honor Among Thieves" (takes 1 GB of virtual memory): Used image memory: 559,433,870 bytes Used model memory: 103,126,952 bytes Used sound memory: 175,558,812 bytes Used asset memory: 838,119,634 bytes As you see, it is less by 800 MB. So yes: the high usage comes from too many different textures used in the level, plus more different models and sounds. And it is explained by the sheer size of the map, not by any problem in the engine. The high memory usage by assets cannot be fixed because it is not a bug (and I think it is unlikely to be improved except for by mapper's efforts). One thing which will be improved in future is EAS (0004755). Aside from that, I recommend switching exclusively to 64-bit build if you create maps of this size. P.S. BTW, I have noted the printMemInfo command in two wiki pages: http://wiki.thedarkmod.com/index.php?title=Performance:_Essential_Must-Knows#Texture_Variety http://wiki.thedarkmod.com/index.php?title=Creating_Large_Areas#Possible_Errors |
|
|
@stgatilov, thand thanks for new info. So last question then, how has me converting a load of brushwork to models (33k down to 14k) using less ram..? is this that EAS you have refered too.? |
|
|
Bikerdude, I'm afraid I do not get your question. |
|
|
@stgatilov - Before I converted a lot of my brush work to models the map would crash on dmap in 2.06. The map has 33,000 brushes, so converting a bunch of the brush based func statics reduce that number down to 14k. my question was why would such a high number of brushes caused 2.06 x64/x86 to crash? |
|
|
As described in related issue 0004661, 64-bit dmap usually crashes not because of memory depletion. It crashes because data structures become inconsistent in some algorithm, most likely due to rounding errors and all such nasty stuff. I think the reason why removing brushes helped you with dmap is the following. Brushes are heavily compiled by dmap, they are used while creating BSP tree, portals, subspaces, areas. Also they take part in some additional mesh processing and shadow volumes precomputation. If you are really interested, you can try to read: http://fabiensanglard.net/doom3/dmap.php As for func_statics, they are NOT used in most of these compilcated algorithms. So the less brushes you have, the less chance that some algorithm in dmap would break. I think it is critical to use brushes to define the high-scale structure of the map. If you disable everything except brushes, they must be enough to define all the areas (rooms, passages, portals between then). All the fine details can easily be put into models/func_statics. But keep in mind that I have ZERO mapping experience and what I say here can easily turn out to be wrong =) |
|
|
In the Doom 3 game, level geometry is 99% models. Brushes are used to "seal and portalize". In theory you could make a map without brushes but you'd have to set "void sealing models" to "inline" which essentially converts them into brushes on dmap. |
- Anonymous
