View Issue Details

IDProjectCategoryView StatusLast Update
0004753The Dark ModDef / Setuppublic27.03.2018 16:57
Reporteruser81Assigned Tostgatilov  
PriorityhighSeveritycrashReproducibilityalways
Status resolvedResolutionnot fixable 
PlatformPC, Windows, x32OSWin 7/8OS VersionSp2/8.1
Product VersionTDM 2.06 
Target VersionTDM 2.06 
Summary0004753: High RAM useage over time: TDM is crashing to desktop
DescriptionOpen up shadowhide's large city map and leave open for aabout 30mins, you will see the memory start at 300-600mb rising to 2.2gb where TDM crashed to desktop.

- dumpfile - will link this shortly.
Additional InformationProblem signature:
  Problem Event Name: APPCRASH
  Application Name: TheDarkMod.exe
  Application Version: 1.0.0.1
  Application Timestamp: 5aa93125
  Fault Module Name: MSVCR120.dll
  Fault Module Version: 12.0.21005.1
  Fault Module Timestamp: 524f7ce6
  Exception Code: 40000015
  Exception Offset: 000a7676
  OS Version: 6.1.7601.2.1.0.256.48
  Locale ID: 2057
  Additional Information 1: 4435
  Additional Information 2: 44354d4bfb261a113d6758b7340c580c
  Additional Information 3: c924
  Additional Information 4: c924eb3c0494902e4839576d979b3b42
TagsNo tags attached.

Relationships

related to 0004755 resolvedstgatilov Optimize EAS memory and time consumption. 
related to 0004756 assignedstgatilov Optimize memory handling in LWO loading 
related to 0004662 closedstgatilov TDM x64 using a lot of ram during DMAP 

Activities

grayman

grayman

18.03.2018 14:17

administrator   ~0010071

Link to complete pk4?
grayman

grayman

18.03.2018 14:49

administrator   ~0010072

Trying to get as close to apples/apples as possible.

1 - Difficulty level?

2 - 32- or 64-bit TDM?

3 - Just start the mission and stand at the starting point, doing nothing?

user81

19.03.2018 12:41

  ~0010074

Last edited: 19.03.2018 13:10

View 3 revisions

Ok so fired it up and just let it run (whithout moving) and the memory skyrocketed fairly quickly, started at 400mb and crashed at 2.1GB. But the fault module was different to last time.

1 - Difficulty level - none was a dmap/map

2 - 32- or 64-bit TDM? - 32bit

3 - Just start the mission and stand at the starting point, doing nothing? - yes

Problem signature:
  Problem Event Name: APPCRASH
  Application Name: TheDarkMod.exe
  Application Version: 1.0.0.1
  Application Timestamp: 5aa93125
  Fault Module Name: nvoglv32.DLL
  Fault Module Version: 23.21.13.9101
  Fault Module Timestamp: 5a9068c5
  Exception Code: c0000005
  Exception Offset: 00af55f3
  OS Version: 6.1.7601.2.1.0.256.48
  Locale ID: 2057
  Additional Information 1: 0a9e
  Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
  Additional Information 3: 0a9e
  Additional Information 4: 0a9e372d3b4ad19135b953a78882e789

grayman

grayman

19.03.2018 12:52

administrator   ~0010076

Got answers to my questions?

user81

19.03.2018 13:08

  ~0010078

Last edited: 19.03.2018 16:07

View 3 revisions

# Full PK4 - https://mega.nz/#!PY0WTbKA!-ZW9qMtvePjh8Xj0FmjnALTKwBu2eFZTbyVla8KdT-s

# Dumpfile for above crash - https://mega.nz/#!bN8DXLaa!Un7zvmBhbCELlBAAj4YiheOKedSMQGL1k5-x3bsoCYI

Going to uninstall the nvidia driver, run driver sweaper and try both the x86 and x64 version and see if and when I get a CTD.

edit my post above.

grayman

grayman

19.03.2018 13:16

administrator   ~0010079

Also, does it crash when you run on 2.05?
grayman

grayman

19.03.2018 13:24

administrator   ~0010080

Last edited: 19.03.2018 13:24

View 2 revisions

Two things about the pk4:

1 - It's missing the startingmap.txt file, which causes it to crash at the start.

2 - The maps/assassins.script file should be named maps/city.script ???

grayman

grayman

19.03.2018 13:36

administrator   ~0010081

Also, could use a copy of your Darkmod.cfg file to replicate your cvar settings.
grayman

grayman

19.03.2018 13:46

administrator   ~0010082

Watching memory use now.

Appears to be rising around 4k/sec (or so).
grayman

grayman

19.03.2018 13:54

administrator   ~0010083

location separator 59 doesn't touch a portal.
grayman

grayman

19.03.2018 15:24

administrator   ~0010091

Initial run ...

Set up the debugger to catch the crash.

Started the mission.

Memory started around 370mB.

30 minutes later it had climbed to 440mB.

At the 60 minute mark, it was still sitting at 440mB. No crash.

Ended the test.

This was with my cfg file. Need to rerun the test with yours.

user81

19.03.2018 16:00

 

autoexec.cfg (46 bytes)

user81

19.03.2018 16:01

 

Darkmod.cfg (12,009 bytes)

user81

19.03.2018 16:02

  ~0010092

Last edited: 19.03.2018 16:06

View 3 revisions

# Attached autoexec and darkmod config files.
# Added the missing startmap and readme files, and rename assassins.script to city.script
# Fixed the VP intersecting seperator 59.
# Did a test of 2.06x64 for 30 mins went upto 480MB
# Doing a test of 2.06x86 atm, will report back.

nbohr1more

nbohr1more

19.03.2018 16:19

developer   ~0010096

I didn't see anything anomalous in your settings but you're another one who inexplicably has seta tdm_lg_split "0" (should be 1).

Also, if I recall correctly, 2.06 allows you to have Soft Shadows without requiring Index Buffers which kills perf on Nvidia so I think you can set:

seta r_useIndexBuffers "0" which I believe is the default.

user81

19.03.2018 17:00

  ~0010100

Last edited: 19.03.2018 17:03

View 2 revisions

I don't remember setting those so will change both as per above.

Back to testing with 2.06x86, its been open for an hour and the memory footprint is sitting at 2.235GB. I will leave it running but expect to crash at some point soon though.

My system specs btw -

Core i7 4790k
nVidia 1070 - 391.01
Sb Xfi - latest driver
Win 7 x64 sp1
16GB of DDR3-1333

user81

19.03.2018 17:49

  ~0010104

Last edited: 19.03.2018 18:00

View 3 revisions

To summarise I uninstalled the 391.01 nvidia driver, ran DDU (Display Driver Uninstaller), rebooted reinstall the driver (clean install)

Ok so after nearly 2hrs it(2.06-x86) had risen to 2.36GB, I then tried to move around and it jumped to 2.458GB. The game is very laggy at this point. And then moved again for a little while and it then crashed with a malloc error.

"Malloc failure for 1048576"

Im gonna loadup BCD and see if that causes the same memory leak/usage.

grayman

grayman

19.03.2018 19:18

administrator   ~0010105

Corrected the cvars NB mentioned.

Using BD's cfg files.

Started at 360mB.

404mB after 30 minutes.

411mB after 60 minutes.

No sign of a mem leak.

user81

19.03.2018 19:43

  ~0010106

Last edited: 19.03.2018 19:49

View 3 revisions

Hmmm, wtf is causing the leak on my side then I wonder. Have you tried walking around the map?

I ran BCD for 2 hrs (stood at the start location) and it never went above 400MB

Am gonna try and run the mission from a .PK4 and not a custom game shortcut as atm thats the only thing I can think of being differ between Gman and my system. I will also run it on the laptop as that has a differ GPU in-case its an nvidia thing.

user81

19.03.2018 22:24

  ~0010107

Last edited: 20.03.2018 08:21

View 2 revisions

Ok so used a .PK4 instead of the unpacked mission and normal TDM shortcut -

# Win8.1/AMD-FX4800/GTX770/4GB, after 30mins ram was at 670MB, will do a longer test tomoz.
# Win7sp1/Corei5-5200u/HD5500/8GB, after 60 mins the ram was at 640MB
# Win7sp1/Corei7-4790k/GTX1070/16GB, after after 20mins ram was at 460MB, will do a longer test tomoz.

Why would an archived(.pk4) version of the mission not use anywhere near the amount of ram compared to the uncompressed version of the same mission!!??

nbohr1more

nbohr1more

20.03.2018 03:17

developer   ~0010110

Archiving does squeeze large text files like the map file or the mtr files considerably but I have another suspicion about this.

When the mission folder is open you might have long paths like

darkmod\fms\my_mission\dds\textures\darkmod\stone\cobblestone\my_custom_stone

in which the new Video encoder algorithm might be searching for pk4's to decompress. Depending on how that works, it might get into some recursion loop when the second "darkmod" folder entry is listed in the path...

Now, since we have a workaround of packing the mission prior to testing it... can this be a 2.07 issue or is this critical for 2.06?

user81

20.03.2018 08:29

  ~0010113

And in answer to Gmans 2.05 question, TDM would run for far longer(hours) before I would see the memory usage hit 2GB.

But I will do a side-by-side test of 2.05/2.06 x86 and chart the ram useage.
stgatilov

stgatilov

20.03.2018 16:47

administrator   ~0010126

Last edited: 20.03.2018 17:31

View 2 revisions

Did not manage to reproduce it on a first try.
Things for me to try later: 1) use 32-bit version 2) use attached cfg-file 3) use 2.06 beta build instead of SVN.

UPDATE: Tried to run 32-bit build of the most recent 2.06 beta --- still no leak (process explorer usage grows very slowly, and growth speed reduces over time --- still about 400 MB after 25 minutes).

user81

20.03.2018 18:37

  ~0010127

Last edited: 20.03.2018 18:38

View 2 revisions

Did a test with the .PK4 above under 2.05 -

Ater 34mins ram was at 375MB, but as soon as I started moving around (walking through the map, 10 seconds) the ram jumped to 452MB. Then after several minutes or running around the ram use increase calmed down after 580mb mark. I will do the same test with 2.06x86 after dinner.

@Grayman, stgatilov - have you guys tried running around for a few minutes to see what your ram useage jumps to ..?

grayman

grayman

20.03.2018 20:05

administrator   ~0010128

I ran around for about a half hour.

Mem usage went up to 1.6gB, but committed memory rose in turn to around 3gB.

I assume that if memory used ever caught up to mem committed, it would crash.

user81

20.03.2018 21:57

  ~0010129

@Commited verses used, yes I think that's the case.

On the testing front I let 2.06 x86 run for 3 hrs whith the player stood at the start spot and the ram rose to 2.01GB. So I imagine if I ran around the ram useage would hit the limiiter far sooner and crash. So that's gonna be my nest test and x86 and x64 version of the current build of 2.06 which i will do tomoz.
stgatilov

stgatilov

21.03.2018 02:38

administrator   ~0010132

Last edited: 21.03.2018 02:39

View 2 revisions

Maybe I am not patient enough, but I cannot get anything looking like a leak. In my case the reserved virtual memory grows from 2 GB to 2.3 GB, which isn't critical.

Also, the term "memory leak" is usually used when allocated memory never get deallocated. I have tried to run around, then quit mission, then start it again (without closing TDM), then run around some more. After the second running time, the memory usage was not greater than after the first time. This probably indicates that the additional memory was deallocated when I quitted mission. Then it is not a "memory leak". It may even be normal memory consumption of the mission. It is still not clear why it is so large then.

The ugly workaround is to run 64-bit version. Then it won't crash at least.

user81

21.03.2018 10:42

  ~0010134

Last edited: 21.03.2018 10:44

View 3 revisions

I am fairly sure I got significantly increased ram usage on the x64 version also, and will be testing that today.

Have changed the title to reflect stgatilov's comment.

Can I ask what OS & hardware your all running?

stgatilov

stgatilov

21.03.2018 14:19

administrator   ~0010140

Last edited: 21.03.2018 15:12

View 2 revisions

Loading the map in debugger takes a very long time due to too many small memory allocations while loading LWO models. I have added a personal TODO to optimize memory handling in lwo loading (I think it is very inefficient there) after 2.06. But I think it should not cause leaks (all this memory must be gone by the time you start playing).

Anyway, bikerdude, are you sure you really need 100 MB of custom architectural LWO models? Did we ever had a mission with so many custom LWO stuff?
It is not that I discourage anyone, I'm just trying to guess what can eat so much memory compared to other missions.

Do you have any custom LOD models in this map by the way?

UPDATE: Ok, another reason for slow loading is that it has a lot of AAS/EAS. This is magnified by the fact that EAS is not quite efficient and uses STL, so it gets very slow when you run game from debugger. Another thing to look into later.

P.S. Windows 10 x64, Ryzen 1600, 16 GB Ram.

user81

21.03.2018 15:56

  ~0010141

Last edited: 21.03.2018 16:39

View 5 revisions

I had to export a bunch of brushwork to .LWO models to cut down on the amount of brushes in the map. I started with 32k which just cause TDM to crash on loading.

No, because the map is all custom built. But I might have to look at LOD if some of the worst perf spots.

So ran 2.06 x64 for 2hrs while stood at the start location and the ram hit 1.2GB, and within 1-2mins it jumped upto 1.4gb. I ran around for another 10-15 mins and it climbed to 2.5GB. Then stood still in the magic ship for a another 15mins and the ram useage just kept going up at roughly 20-1000kb per second but as soon as I went out into the garden it stopped climbing at such a high rate - sitting at 4.42GB

So I went back inside the shop and stood at various spot to see where the ram useage was highest, and it was only when I looked down at the little furnace (witch has a moving torchflame) and that the ram skyrocketed again (@20-1000kb per second to start with after a minute of standing in the spot the ram useage increased at 500KB-1MB per second)

Why the hell would a moving light cause such a massive memory usage? I am going to see if the same location does the same in 2.05

user81

21.03.2018 17:11

  ~0010142

Doing a little test to see if its my larg map or 2.06 eating ram, I have exported the above little shop with the player looking at the moving light from the little furnace. Started at 90MB, within 10mins it rose to 150MB, gonna leave it running while I have some dinner.
grayman

grayman

21.03.2018 18:22

administrator   ~0010143

What is the light entity that seems to be causing the problem?
stgatilov

stgatilov

22.03.2018 04:52

administrator   ~0010144

Last edited: 24.03.2018 06:53

View 3 revisions

I decided to look at memory usage in general.

I'll write memory consumption in format (Current working set / Reserved virtual memory) after each major loading step.
Note that this is Release x64 launched from debugger. Debugger increases memory consumption, as does the 64-bit mode.

0. Before SpawnMapEntities: (309M / 876M).
1. SpawnMapEntities/MapPopulate finished: (1272M / 1922M). This is the first big uprise in memory consumption. Models is one of the factors here (0004756), but I'm not sure how big it is.
2. SetupEAS finished: (2131M / 2750M). This takes a lot of time and highly increases memory consumption.
3. idImageManager::EndLevelLoad finished: (2039M / 4020M). It loads all the textures into OpenGL driver. I suppose it does not increase working set because OpenGL driver explicitly avoids caching the texture data on CPU side (using non-caching memory allows to interact faster with GPU, as far as I remember).
4. After Sys_SetPhysicalWorkMemory call: (200M / 4020M). This is a WinAPI call which tells Windows OS to remove as many pages as possible from the physical RAM. Of course, this memory is still used by the process, it remains in virtual memory and swap, waiting for the player to get into a place when it becomes needed.
5. At the "Press Enter to Start" screen: (300M / 4020M).
6. Just started the game: (480M / 4080M).

As you see, measuring physical working set is simply the wrong way to measure memory consumption, because most of allocated stuff sits in swap when you start mission. If you simply stand still, then pages get slowly loaded back into physical memory (AI-s move around anyway, some other things happen). If you start running around the map, the engine has to touch more and more memory: it loads new geometry which was completely unnecessary before. So it is normal that physical working set grows, and grows much faster than when you stand still.

So the question is: is this really a memory leak or abnormal memory consumption for the map of this size?
If physical working set during playing does not exceed peak working set during loading, then probably the memory consumption just reflects the size of the map.

I think that EAS eating 800 MB should be fixable (0004755, not for 2.06). It seems that textures do not eat physical memory, so we can forget about them. The main problem is about SpawnMapEntities eating 1900 MB, which needs more investigation.

user81

22.03.2018 08:50

  ~0010145

@Grayman, its just a regular light source - light_torchflame_moving. And I dont think this is the only reason for the high ram usage.

@stgatilov, abnormal memory consumption for the map of this size sounds like the better explanation. SpawnMapEntities eating 1.9GB is as you say not good and what causes the 32bit version to crash I would have thought. And what is EAS?

And moving versus standing still, ok that makes sense.

As of today I have created 275 models that used to be brushwork (roof, windows, other geometry). You say thats a lot of models, but compared to the amount of stock/core models in the map the above number is small. One thing that might be causing abnormal ram use in relation to .LWO models is 30 of the 275 have tris counts from 10-80k
stgatilov

stgatilov

24.03.2018 07:29

administrator   ~0010147

Ok, after some failed attempts to measure memory more precisely, I have realized that we have forgot about the powers given to us by our ancestors =)

Bikerdude, you basically need the following console command:
  printMemInfo
Just type it in console after loading, then go to maps directory of your fms and read the text files there. It also prints the high-level stats into console.

Here is the sample report I get in x64 (without debugger):
 Used image memory: 1,076,869,716 bytes
 Used model memory: 305,826,356 bytes
 Used sound memory: 261,989,167 bytes
 Used asset memory: 1,644,685,239 bytes
These numbers are exactly the same in 32-bit mode.
The models with 'shadowhide' in their path take about 211 MB of memory. But probably your models also include the ones without 'shadowhide', so the total should be a but larger.

As you see, the pure size of assets is rather close to 2 GB limit of 32-bit mode. Add bad implementation of EAS to this, and you are somewhere near the border. I suggest using 64-bit game for your map, this is TDM's future anyway. Or trying to use less assets: either assets of smaller size or just less different models/textures in single map.
stgatilov

stgatilov

24.03.2018 08:48

administrator  

printMemInfo.png (1,051,314 bytes)
duzenko

duzenko

24.03.2018 08:51

developer   ~0010151

Last edited: 24.03.2018 08:51

View 2 revisions

I remember running out of memory on older missions, but I am not sure if it's not an Intel driver bug.
What is taking so much asset memory?
Does this happen on existing missions? If not, then it's not to be fixed in 2.06.

stgatilov

stgatilov

24.03.2018 12:34

administrator   ~0010152

Here are the same stats for 1st mission of "No Honor Among Thieves" (takes 1 GB of virtual memory):
 Used image memory: 559,433,870 bytes
 Used model memory: 103,126,952 bytes
 Used sound memory: 175,558,812 bytes
 Used asset memory: 838,119,634 bytes
As you see, it is less by 800 MB.

So yes: the high usage comes from too many different textures used in the level, plus more different models and sounds. And it is explained by the sheer size of the map, not by any problem in the engine. The high memory usage by assets cannot be fixed because it is not a bug (and I think it is unlikely to be improved except for by mapper's efforts).

One thing which will be improved in future is EAS (0004755).

Aside from that, I recommend switching exclusively to 64-bit build if you create maps of this size.


P.S. BTW, I have noted the printMemInfo command in two wiki pages:
  http://wiki.thedarkmod.com/index.php?title=Performance:_Essential_Must-Knows#Texture_Variety
  http://wiki.thedarkmod.com/index.php?title=Creating_Large_Areas#Possible_Errors

user81

24.03.2018 15:12

  ~0010153

@stgatilov, thand thanks for new info.

So last question then, how has me converting a load of brushwork to models (33k down to 14k) using less ram..? is this that EAS you have refered too.?
stgatilov

stgatilov

24.03.2018 16:46

administrator   ~0010154

Bikerdude, I'm afraid I do not get your question.

user81

27.03.2018 15:06

  ~0010163

@stgatilov -

Before I converted a lot of my brush work to models the map would crash on dmap in 2.06. The map has 33,000 brushes, so converting a bunch of the brush based func statics reduce that number down to 14k.

my question was why would such a high number of brushes caused 2.06 x64/x86 to crash?
stgatilov

stgatilov

27.03.2018 16:05

administrator   ~0010169

Last edited: 27.03.2018 16:05

View 2 revisions

As described in related issue 0004661, 64-bit dmap usually crashes not because of memory depletion. It crashes because data structures become inconsistent in some algorithm, most likely due to rounding errors and all such nasty stuff.

I think the reason why removing brushes helped you with dmap is the following.
Brushes are heavily compiled by dmap, they are used while creating BSP tree, portals, subspaces, areas. Also they take part in some additional mesh processing and shadow volumes precomputation. If you are really interested, you can try to read: http://fabiensanglard.net/doom3/dmap.php
As for func_statics, they are NOT used in most of these compilcated algorithms. So the less brushes you have, the less chance that some algorithm in dmap would break.

I think it is critical to use brushes to define the high-scale structure of the map. If you disable everything except brushes, they must be enough to define all the areas (rooms, passages, portals between then). All the fine details can easily be put into models/func_statics.
But keep in mind that I have ZERO mapping experience and what I say here can easily turn out to be wrong =)

nbohr1more

nbohr1more

27.03.2018 16:47

developer   ~0010170

Last edited: 27.03.2018 16:57

View 3 revisions

In the Doom 3 game, level geometry is 99% models.
Brushes are used to "seal and portalize".

In theory you could make a map without brushes but you'd have to set "void sealing models" to "inline" which essentially converts them into brushes on dmap.

Issue History

Date Modified Username Field Change
18.03.2018 12:58 user81 New Issue
18.03.2018 14:17 grayman Note Added: 0010071
18.03.2018 14:49 grayman Note Added: 0010072
19.03.2018 12:41 user81 Note Added: 0010074
19.03.2018 12:42 user81 Note Edited: 0010074 View Revisions
19.03.2018 12:52 grayman Note Added: 0010076
19.03.2018 13:08 user81 Note Added: 0010078
19.03.2018 13:10 user81 Note Edited: 0010074 View Revisions
19.03.2018 13:10 user81 Note Edited: 0010078 View Revisions
19.03.2018 13:16 grayman Note Added: 0010079
19.03.2018 13:24 grayman Note Added: 0010080
19.03.2018 13:24 grayman Note Edited: 0010080 View Revisions
19.03.2018 13:36 grayman Note Added: 0010081
19.03.2018 13:46 grayman Note Added: 0010082
19.03.2018 13:54 grayman Note Added: 0010083
19.03.2018 15:24 grayman Note Added: 0010091
19.03.2018 16:00 user81 File Added: autoexec.cfg
19.03.2018 16:01 user81 File Added: Darkmod.cfg
19.03.2018 16:02 user81 Note Added: 0010092
19.03.2018 16:02 user81 Description Updated View Revisions
19.03.2018 16:03 user81 Note Edited: 0010092 View Revisions
19.03.2018 16:06 user81 Note Edited: 0010092 View Revisions
19.03.2018 16:07 user81 Note Edited: 0010078 View Revisions
19.03.2018 16:19 nbohr1more Note Added: 0010096
19.03.2018 17:00 user81 Note Added: 0010100
19.03.2018 17:03 user81 Note Edited: 0010100 View Revisions
19.03.2018 17:49 user81 Note Added: 0010104
19.03.2018 17:53 user81 Note Edited: 0010104 View Revisions
19.03.2018 18:00 user81 Note Edited: 0010104 View Revisions
19.03.2018 19:18 grayman Note Added: 0010105
19.03.2018 19:43 user81 Note Added: 0010106
19.03.2018 19:44 user81 Note Edited: 0010106 View Revisions
19.03.2018 19:49 user81 Note Edited: 0010106 View Revisions
19.03.2018 22:24 user81 Note Added: 0010107
20.03.2018 03:17 nbohr1more Note Added: 0010110
20.03.2018 08:21 user81 Note Edited: 0010107 View Revisions
20.03.2018 08:29 user81 Note Added: 0010113
20.03.2018 16:47 stgatilov Note Added: 0010126
20.03.2018 17:31 stgatilov Note Edited: 0010126 View Revisions
20.03.2018 18:37 user81 Note Added: 0010127
20.03.2018 18:38 user81 Note Edited: 0010127 View Revisions
20.03.2018 20:05 grayman Note Added: 0010128
20.03.2018 21:57 user81 Note Added: 0010129
21.03.2018 02:38 stgatilov Note Added: 0010132
21.03.2018 02:39 stgatilov Note Edited: 0010132 View Revisions
21.03.2018 10:42 user81 Note Added: 0010134
21.03.2018 10:42 user81 Summary Memory leak: TDM is crashing to desktop after 30mins => High RAM useage over time: TDM is crashing to desktop
21.03.2018 10:43 user81 Note Edited: 0010134 View Revisions
21.03.2018 10:44 user81 Note Edited: 0010134 View Revisions
21.03.2018 14:19 stgatilov Note Added: 0010140
21.03.2018 15:12 stgatilov Note Edited: 0010140 View Revisions
21.03.2018 15:56 user81 Note Added: 0010141
21.03.2018 16:17 user81 Note Edited: 0010141 View Revisions
21.03.2018 16:19 user81 Note Edited: 0010141 View Revisions
21.03.2018 16:38 user81 Note Edited: 0010141 View Revisions
21.03.2018 16:39 user81 Note Edited: 0010141 View Revisions
21.03.2018 17:11 user81 Note Added: 0010142
21.03.2018 18:22 grayman Note Added: 0010143
22.03.2018 04:52 stgatilov Note Added: 0010144
22.03.2018 05:11 stgatilov Relationship added related to 0004755
22.03.2018 05:20 stgatilov Relationship added related to 0004756
22.03.2018 05:23 stgatilov Note Edited: 0010144 View Revisions
22.03.2018 08:50 user81 Note Added: 0010145
24.03.2018 06:53 stgatilov Note Edited: 0010144 View Revisions
24.03.2018 07:29 stgatilov Note Added: 0010147
24.03.2018 08:48 stgatilov File Added: printMemInfo.png
24.03.2018 08:51 duzenko Note Added: 0010151
24.03.2018 08:51 duzenko Note Edited: 0010151 View Revisions
24.03.2018 12:34 stgatilov Note Added: 0010152
24.03.2018 12:36 stgatilov Assigned To => stgatilov
24.03.2018 12:36 stgatilov Status new => assigned
24.03.2018 12:37 stgatilov Status assigned => resolved
24.03.2018 12:37 stgatilov Resolution open => not fixable
24.03.2018 12:38 stgatilov Platform PC, Windows, x64 => PC, Windows, x32
24.03.2018 15:12 user81 Note Added: 0010153
24.03.2018 16:46 stgatilov Note Added: 0010154
27.03.2018 15:06 user81 Note Added: 0010163
27.03.2018 15:20 user81 Relationship added related to 0004662
27.03.2018 16:05 stgatilov Note Added: 0010169
27.03.2018 16:05 stgatilov Note Edited: 0010169 View Revisions
27.03.2018 16:47 nbohr1more Note Added: 0010170
27.03.2018 16:47 nbohr1more Note Edited: 0010170 View Revisions
27.03.2018 16:57 nbohr1more Note Edited: 0010170 View Revisions