View Issue Details

IDProjectCategoryView StatusLast Update
0004660The Dark ModGraphicspublic18.05.2019 08:13
Reporterstgatilov Assigned Tostgatilov  
Status suspendedResolutionopen 
Product VersionTDM 2.06 
Target VersionTDM 2.08Fixed in Version 
Summary0004660: Soft shadows: sometimes light leaks near edges (in complete darkness)
DescriptionSometimes surfaces become brighter near edges because of soft shadows. I'm speaking about the case when all surfaces are dark around the edge, and everything should be in shadow.

NOTE: this is not about the famous halo effect.
Steps To ReproduceSee screenshot from "Patently Dangerous".
It is in the outside area near the beginning of the mission.
TagsNo tags attached.


related to 0004851 resolvedcabalistic Soft shadows with multisampling produces sparklies near wall edges 




11.11.2017 16:48


PatentlyDangerous_SoftShadows_EdgeLeak.jpg (411,795 bytes)


11.11.2017 16:53

developer   ~0009575

I don't understand where the algorithm manages to find unshadowed pixels near the edges. It seems to me that all the pixels surrounding the edge are in shadow, and I don't see any bright pixels without soft shadows.

I have found a workaround, which fixes the problem completely in this case. I'm pretty sure it should work well in other cases. It is: increase polygon offset for shadow volumes rendering.

The default settings are:
  r_shadowPolygonOffset -1
  r_shadowPolygonFactor 0
The issue does not happen with either this:
  r_shadowPolygonOffset -10
  r_shadowPolygonFactor 0
or with this:
  r_shadowPolygonOffset -1
  r_shadowPolygonFactor 1


11.11.2017 16:55

developer   ~0009576

Duzenko, could you guess where unshadowed pixels come from?
Is there any easy debug-way to see the contents of stencil buffer on screen?


11.11.2017 16:55

developer (4,816,783 bytes)


11.11.2017 16:56


Quicksave_0.jpg (135,018 bytes)
Quicksave_0.jpg (135,018 bytes)


12.11.2017 19:42

developer   ~0009585

You can visualize the stencil texture via interaction.fs provided you have a non-zero in r_softshadowsquality.
Base value sampled from u_stencilTexture is 128. 129 is one shadow-deep, etc.


13.11.2017 11:43

developer   ~0009597

Last edited: 13.11.2017 11:43

View 2 revisions

Added debug stencil visualizer in the end of StencilSoftShadow in interaction.fs.
Ran out of time with it but it looks like a driver bug.
If sampling from base texcoord then it returns a correct value.
If sampling from base texcoord + const offset then it returns noise for some pixels.



13.11.2017 15:29

developer   ~0009598

What if stencil buffer really contains white pixels there, but we do not see them, because those pixels are not rendered with this face?
I also tried showing stencil buffer contents, and they are black for unshifted case and contains white for shifted case. I suspect that stencil buffer there is really wrong, but we do not see it with such a way of debug visualization.


13.11.2017 15:39


stencil.png (370,560 bytes)


13.11.2017 15:40

developer   ~0009599

Added stencil.png, where you can see white lines somewhere.
Those lines are blurred and generate the artifacts.
If you increase polygon offset for shadow volumes (as suggested above), those lines disappear.

P.S. To be honest, I wonder how I got this picture. On previous attempt, I did not saw the lines there...


13.11.2017 16:14

developer   ~0009601

Ok, I see those white lines in stencil buffer only when r_useFBO is disabled. I also have 2xAA in effect in this case. When I enable r_useFBO, these lines disappear. But the issue appears regardless of r_useFBO.


13.11.2017 16:24

developer   ~0009602


I wonder if GL_EXT_polygon_offset_clamp could be used to enhance r_shadowPolygonOffset ?


13.11.2017 16:49

developer   ~0009603

Is there any problem in using "r_shadowPolygonFactor 1" without any clamping?


13.11.2017 17:05

developer   ~0009604

Last edited: 13.11.2017 17:11

View 2 revisions

I don't think so.
Shadow Polygon Offset 1 probably good enough in 99.9999% of cases.

I just remembered seeing GL_EXT_polygon_offset_clamp awhile back and thought I would ping Duzenko to see if it looks useful.

We manually adjusted regular surface offset factors in 2.03 so I don't see why we can't change shadow polygon offsets in 2.06.

Does the change cause any adverse effects when soft shadows aren't enabled? (I doubt it).



13.11.2017 19:22

developer   ~0009605

Last edited: 13.11.2017 19:22

View 2 revisions

So you suspect stencil might not have the default value for pixels not on the interaction faces? The only way to test that is to draw a full screen quad. I will try that when I have the time.
Thanks Nbohr1more for bringing depth test in the conversation. When I think about it the noise did look a lot like depth fighting.



13.11.2017 20:21

developer   ~0009606

I had to fix the no-primary-fbo soft-shadow operation first on Intel so did not have the time for fullscreen quads tonight but in the process I noticed this thing.
I start with primary fbo off and SS on. Glitch is there.
I turn the primary fbo on -> glitch goes away.
I turn off the P-FBO off -> glitch does not come back.
How does it go for you?
Also, please test that I did not break the non-Intel path while fixing on Intel.


14.11.2017 02:00

developer   ~0009609

The latest build still renders just fine on Nvidia.

That said, I will relay that the first note on "Closemouthed Shadows" has some sorta rendering issue with a self-shadow that doesn't exist when you enable f_useFBO.


20.11.2017 05:33

developer   ~0009642

VanishedOne wrote me about two places where he saw light leaking with soft shadows only, and in both places setting "r_shadowPolygonFactor 1" fixed the issue.

Maybe we simply change r_shadowPolygonFactor to 1 by default and see how the beta continues?


20.11.2017 05:52

developer   ~0009643

I noticed hairline polygon cracks in the shadows for the first guard in "The Thieves" when setting r_shadowPolygonFactor 1 without soft shadows but I haven't noticed any widespread problems with the setting.

I guess, ideally, this would hard code to 0 if soft shadows are off?

At the very least, it should be set to cvar_archive so that you can save your preferred setting from session to session.


20.11.2017 16:14

developer   ~0009644

All these cvars which must behave differently depending on other cvars is too complicated, isn't it?
If setting cvar to 1 is not an option, then it would be better to remove the cvar, and compute it on-demand depending on soft shadows cvars. But this reduces game tweakability...
Alternatively, we can add value -1, which means something like "auto" or "default", and make if default. And in this case effectively set 0 or 1 depending on soft shadows presence.


20.11.2017 16:50

developer   ~0009645

I like your last idea "-1 == auto"

I'm not terribly inclined to object to a default of 1 either as long as the cvar can be archived and we inform users about the caveats of the new default with hard shadows.


21.11.2017 17:14

developer   ~0009663

After some consideration, this idea is weird. Value -1 is quite valid for polygon offset factor: it is just the same as value 1, but with offsetting to the opposite side.
One the other hand, we can set default value of r_shadowPolygonFactor to be "default", and then check for it everywhere. But it is rather awkward to set string value to a float variable =)


21.11.2017 17:47

developer   ~0009664

Ok, I have checked the Thieves mission and I understand what you mean.

It seems that those artifacts are caused by using offsetfactor. This is because offset is different for adjacent faces of the mesh (unless they are parallel), so a crack appears between them.
For instance, it is described here:

To be honest, I do not feel that depth fighing the real issue here. Maybe we should simply leave things as they are? at least until we understand where this light comes from...


21.11.2017 18:18

developer   ~0009665

Last edited: 21.11.2017 18:19

View 3 revisions

Easy workaround for 2.06

Set the cvars to CVAR_ARCHIVE in RenderSystem_init.cpp

idCVar r_shadowPolygonOffset( "r_shadowPolygonOffset", "-1", CVAR_RENDERER | CVAR_FLOAT | CVAR_ARCHIVE, "bias value added to depth test for stencil shadow drawing" );
idCVar r_shadowPolygonFactor( "r_shadowPolygonFactor", "0", CVAR_RENDERER | CVAR_FLOAT | CVAR_ARCHIVE, "scale value for stencil shadow drawing" );

Let user know they can cure some shadow leakage via r_shadowPolygonFactor 1.

Move bug to 2.07 for further investigation.



22.11.2017 11:46

developer   ~0009673

Last edited: 22.11.2017 11:50

View 2 revisions

I tried disabled culling for stencil visualization but failed.
What works for me is r_showshadows 1 and then step inside the building corner shadow. (also r_singlelight 87 helps here)
Screenshot attached
What I can see here is a shadow surface flickering on top of the wall, classic depth fighting case.
One way to fix it is to merge both corner walls in one surface so it casts a single shadow.
I'm not sure what the flickering surface is - is it SV front cap or ???



23.11.2017 10:14

developer   ~0009680

Current version appears to be able to render the back faces for stencil debugging.
I set r_lightallbackfaces as exe run parameters in VS debugging options or alternatively you can quick save/load after toggling it on the fly.
Attached screenshot with stencil visualization on both front and back faces.


23.11.2017 10:15


Untitled.jpg (169,030 bytes)
Untitled.jpg (169,030 bytes)


23.11.2017 17:19

developer   ~0009685

Last edited: 23.11.2017 17:22

View 4 revisions

If you disable r_useFBO, then you will see the same issue without any flickering. Stencil buffer contains zero inside the wall. At least that is what I see when I use your debug code with X shift and without FBO.

It seems this effect shows every time you have a wall, and some static box object at zero distance from it (i.e. sharing a plane). In such case there is a whole vertical line which makes a reflex angle between the wall and the object. If you place a light in front of the object, so the the line is in shadow, then you get this light leaking.

I have tried to position a candle near several static objects close to walls, and I can reproduce it in all cases.
EDIT: Ok, not in all cases. But in many cases the issue shows up.



23.11.2017 17:25


PatentlyDangerous_LeakAgain.jpg (485,274 bytes)


23.11.2017 17:28

developer   ~0009686

Added a new image (LeakAgain) near the fountain.
In this case the issue happens only with r_useFBO 0.
I suppose that there is something wrong with rendering stencil in such cases, but it leads to different results in FBO and non-FBO cases. Like if some buffer is not cleared between frames or something like that.


24.11.2017 06:05

developer   ~0009687

The problem disappears if I dmap the map with shadowOptLevel set to SO_NONE instead of SO_MERGE_SURFACES. Strangely enough, this is true even if R_CreateShadowVolume is called with SO_STATIC, so the key difference is not in R_CreateShadowVolume.


26.04.2018 03:04

developer   ~0010402

Halo reduction shaders 15171, 15172, 15173 have significantly reduced this problem but it still persists.

r_shadowPolygonFactor 1

is still the best option


10.05.2018 01:58

developer   ~0010456

Last edited: 10.05.2018 03:10

View 5 revisions

I think this is a pretty easy workaround:


// If softshadows is enabled and r_shadowPolygonFactor is 0 change it to 1
// otherwise carry on using the cvars as originally intended

if ( r_softShadowsQuality.GetBool() && !r_shadowPolygonFactor.GetBool() ) {
        qglPolygonOffset ( 1.0f, -r_shadowPolygonOffset.GetFloat() );
        qglEnable (GL_POLYGON_OFFSET_FILL);
    } else {
 if ( r_shadowPolygonFactor.GetFloat() || r_shadowPolygonOffset.GetFloat() )
        qglPolygonOffset( r_shadowPolygonFactor.GetFloat(), -r_shadowPolygonOffset.GetFloat() );
        qglEnable( GL_POLYGON_OFFSET_FILL );

Edit: seems to work as expected

(edited version attached)



10.05.2018 03:10


4660_draw_stencil.cpp (8,529 bytes)


10.05.2018 16:23

developer   ~0010459

We still know that "r_shadowPolygonFactor 1" adds issues to stencil shadows. So this workaround is not perfect.
Besides, no one knows why the problem exists, and how this workaround hides it.


10.05.2018 16:31

developer   ~0010460

In the above, we aren't really changing the cvar though.

We are changing it's affect on the renderer so the cvar remains at 0
but the renderer acts as if it's 1.

It's basically a blacklist for a value-combo we know to be broken.
"Never let Polygon Factor = 0 if soft shadows is on".

I think this would reduce a lot of confusion for end users in 2.06
and we could do a deeper dive in 2.07?


11.12.2018 03:06

developer   ~0010865

In Patently Dangerous 2.07\SVN this problem barely exists now.
It is also cured in Shadow Map mode.

I suggest we move this to 2.08 or remove the release version
 target and suspend it.


11.12.2018 04:10

developer   ~0010867

I don't mind.

The problem seems to happen rarely, and we did not manage to fully understand why it happens despite much time. The anti-halo patch diminishes the problem, and shadow maps are not affected at all.


20.01.2019 14:12

developer   ~0011445

Is this the same issue?

Issue History

Date Modified Username Field Change
11.11.2017 16:48 stgatilov New Issue
11.11.2017 16:48 stgatilov Status new => assigned
11.11.2017 16:48 stgatilov Assigned To => stgatilov
11.11.2017 16:48 stgatilov File Added: PatentlyDangerous_SoftShadows_EdgeLeak.jpg
11.11.2017 16:53 stgatilov Note Added: 0009575
11.11.2017 16:55 stgatilov Note Added: 0009576
11.11.2017 16:55 stgatilov File Added:
11.11.2017 16:56 stgatilov File Added: Quicksave_0.jpg
12.11.2017 19:42 duzenko Note Added: 0009585
13.11.2017 11:43 duzenko Note Added: 0009597
13.11.2017 11:43 duzenko Note Edited: 0009597 View Revisions
13.11.2017 15:29 stgatilov Note Added: 0009598
13.11.2017 15:39 stgatilov File Added: stencil.png
13.11.2017 15:40 stgatilov Note Added: 0009599
13.11.2017 16:14 stgatilov Note Added: 0009601
13.11.2017 16:24 nbohr1more Note Added: 0009602
13.11.2017 16:49 stgatilov Note Added: 0009603
13.11.2017 17:05 nbohr1more Note Added: 0009604
13.11.2017 17:11 nbohr1more Note Edited: 0009604 View Revisions
13.11.2017 19:22 duzenko Note Added: 0009605
13.11.2017 19:22 duzenko Note Edited: 0009605 View Revisions
13.11.2017 20:21 duzenko Note Added: 0009606
14.11.2017 02:00 nbohr1more Note Added: 0009609
20.11.2017 05:33 stgatilov Note Added: 0009642
20.11.2017 05:52 nbohr1more Note Added: 0009643
20.11.2017 16:14 stgatilov Note Added: 0009644
20.11.2017 16:50 nbohr1more Note Added: 0009645
21.11.2017 17:14 stgatilov Note Added: 0009663
21.11.2017 17:47 stgatilov Note Added: 0009664
21.11.2017 18:18 nbohr1more Note Added: 0009665
21.11.2017 18:19 nbohr1more Note Edited: 0009665 View Revisions
21.11.2017 18:19 nbohr1more Note Edited: 0009665 View Revisions
22.11.2017 04:28 stgatilov Target Version => TDM 2.07
22.11.2017 11:46 duzenko Note Added: 0009673
22.11.2017 11:46 duzenko File Added: Untitled.jpg
22.11.2017 11:50 duzenko Note Edited: 0009673 View Revisions
23.11.2017 10:14 duzenko Note Added: 0009680
23.11.2017 10:15 duzenko File Deleted: Untitled.jpg
23.11.2017 10:15 duzenko File Added: Untitled.jpg
23.11.2017 17:19 stgatilov Note Added: 0009685
23.11.2017 17:20 stgatilov Note Edited: 0009685 View Revisions
23.11.2017 17:20 stgatilov Note Edited: 0009685 View Revisions
23.11.2017 17:22 stgatilov Note Edited: 0009685 View Revisions
23.11.2017 17:25 stgatilov File Added: PatentlyDangerous_LeakAgain.jpg
23.11.2017 17:28 stgatilov Note Added: 0009686
24.11.2017 06:05 stgatilov Note Added: 0009687
26.04.2018 03:04 nbohr1more Note Added: 0010402
10.05.2018 01:58 nbohr1more Note Added: 0010456
10.05.2018 01:59 nbohr1more Status assigned => feedback
10.05.2018 02:08 nbohr1more Note Edited: 0010456 View Revisions
10.05.2018 02:36 nbohr1more Note Edited: 0010456 View Revisions
10.05.2018 02:36 nbohr1more Note Edited: 0010456 View Revisions
10.05.2018 03:10 nbohr1more File Added: 4660_draw_stencil.cpp
10.05.2018 03:10 nbohr1more Note Edited: 0010456 View Revisions
10.05.2018 16:23 stgatilov Note Added: 0010459
10.05.2018 16:23 stgatilov Status feedback => assigned
10.05.2018 16:31 nbohr1more Note Added: 0010460
25.06.2018 06:11 stgatilov Relationship added related to 0004851
11.12.2018 03:06 nbohr1more Note Added: 0010865
11.12.2018 03:06 nbohr1more Status assigned => feedback
11.12.2018 04:10 stgatilov Note Added: 0010867
11.12.2018 04:10 stgatilov Status feedback => assigned
11.12.2018 04:21 nbohr1more Target Version TDM 2.07 => TDM 2.08
20.01.2019 14:12 STiFU Note Added: 0011445
18.05.2019 08:13 stgatilov Status assigned => suspended