View Issue Details

IDProjectCategoryView StatusLast Update
0005439The Dark ModCodingpublic18.07.2021 05:07
Reporterstgatilov Assigned Tostgatilov  
PrioritylowSeveritynormalReproducibilityN/A
Status resolvedResolutionfixed 
Product VersionTDM 2.08 
Target VersionTDM 2.10Fixed in VersionTDM 2.10 
Summary0005439: Alleviate entity limit
DescriptionMappers repeatedly hit the 8K limit on number of entitities.
Since many entities don't consume much runtime resources, removing the limit is worth investigation.
Additional InformationAlternatively, we can work on a better func_static-s inline, which would merge their renderEntities and kill them.
There prototype is discussed here:
  https://forums.thedarkmod.com/index.php?/topic/16837-when-your-getting-close-the-entity-limit-inlining/&do=findComment&comment=453909
TagsNo tags attached.

Relationships

related to 0004520 resolvedstgatilov Make savegames of x86 and x64 builds interchangeable 

Activities

stgatilov

stgatilov

28.02.2021 17:43

administrator   ~0013746

Here is the list of commits.

These two are independent preliminary fixes:
  r9156. idStrPool::FreeString now takes O(1) time instead of O(N) time.
  r9157. idBounds::FromPointRotation/FromBoundsRotation: fixed wrong results. Especially when for rotation angle > 180 degrees.
The first is needed to accelerate map freeing for many entities, and the second one is just an accidentally found clipping bug.

The main change is returning "flex list" of results from several idClip methods:
  r9158. idClip methods now store output into idFlexList with 128 elements on stack, instead of storing it into huge stack array with NUM_GENTITIES elements.
These methods are used all over the game code, so the set of changes is substantial.

Replaced ordinary array with "flex list" in more places:
  r9159. Do not create stack array with NUM_GENTITIES elements for stim/response processing.
  r9160. Do not create stack array with NUM_GENTITIES elements in idAI::Event_GetRandomTarget.
  r9161. idGameLocal::EntitiesWithinRadius now returns idClip_EntityList instead of fixed-size array.
  r9162. idPush now uses FlexList instead of fixed-size arrays.
  r9163. idPathCorner::RandomPath now uses FlexList instead of fixed-size arrays.

One more optimization commit (avoid iterating over NULLs in entities list):
  r9164. CFrobDoor::FindDoubleDoor and idSecurityCamera::PostSpawn now iterate over spawned entities only. Instead of iterating over all NUM_GENTITIES elements of entities array.
This one removes huge memory consumption for multiplayer (Duzenko has already removed this code for no apparent reason):
  r9165. Removed some multiplayer-related code which used large arrays of NUM_GENTITIES size. Also removed this constant from g_showCollisionAlongView code.

Bump dmap limit so that internal entitylimit2_gen with 60K entities can be dmapped:
  r9166. Increased MAX_INTER_AREA_PORTALS in dmap from 1024 to 10240. Also produce error if this number is exceeded.

And now we come to actual commits about increasing the limit.
  r9167. EntityPtr now stores two 32-bit integers separately, instead of packing two integers into one 32-bit number.
  r9169. Raised max number of entities from 8192 to 65536.
stgatilov

stgatilov

28.02.2021 17:46

administrator   ~0013747

There is still pretty rough place in idProgram::ScriptObjectMemory_Init.

In 64-bit build, we have an "embedded heap" for script objects, which is basically a hand-written allocator.
It allocates one large chunk of memory, which is surely enough to handle allocation requests in all cases.
Now this chunk is about 100 MB.
Of course, only a tiny amount of this memory is actually used, so I guess allocating virtual memory does not produce any problems here.
stgatilov

stgatilov

17.07.2021 17:21

administrator   ~0014167

Increasing size of idEntityPtr broken passing entities in event system.
Discussion here:
  https://forums.thedarkmod.com/index.php?/topic/16837-when-youre-getting-close-to-the-entity-limit-inlining/&do=findComment&comment=462932

I have found it because "Snowed Inn" crashes soon after startup on 2.10 dev builds, but does not crash in 2.09.
Looking at the crash, I found that idBrittleFracture::Event_Touch event on func_fracture_3 gets entity argument which is NULL.
When I tracked who posted the event, I found that the entity argument was not NULL.
So entity arguments for events usually turn into NULL pointers.

This is caused by the fact that entity arguments are stored in entity data as idEntityPtr (see idEvent::Alloc).
That is not surprising, since entity can die while event waits to be executed.
Previously sizeof(idEntityPtr) was 4, now it is 8.
However, the code still allocates only 4 bytes for it, as seen in idEventDef::Construct.
So idEntityPtr gets partially overwritten, hence the validity check fails on receiver size, and it turns into NULL.
stgatilov

stgatilov

18.07.2021 05:07

administrator   ~0014168

Fixed in svn rev 9471.

Issue History

Date Modified Username Field Change
07.12.2020 12:21 stgatilov New Issue
07.12.2020 12:21 stgatilov Status new => assigned
07.12.2020 12:21 stgatilov Assigned To => stgatilov
28.02.2021 17:43 stgatilov Note Added: 0013746
28.02.2021 17:46 stgatilov Note Added: 0013747
28.02.2021 17:46 stgatilov Status assigned => resolved
28.02.2021 17:46 stgatilov Resolution open => fixed
28.02.2021 17:46 stgatilov Fixed in Version => TDM 2.10
17.07.2021 17:21 stgatilov Note Added: 0014167
17.07.2021 17:22 stgatilov Relationship added related to 0004520
18.07.2021 05:07 stgatilov Note Added: 0014168