floating point precision or rounding error?

share shaders here
Post Reply
  • Author
  • Message
Offline
*sensei*
Posts: 372
Joined: 28 Jul 2013, 23:26

floating point precision or rounding error?

Question related to floating point precision...

I have some code to separate shadows, mids and highlights based on intensity... now normally one would say this will work 100%...

grey=saturate(dot(color.xyz, 0.33333333))
highlights=grey*grey
shadows=(1-grey)*(1-grey)
midtones=1-highlights-shadows

then I do some multiplicative tinting on shadow and midtones layers and recombine them as

result=midtones+shadows+highlights

now this should work, shouldn't it? Guess not.

In the higher intensities one gets some pixels pure black as if they have been skipped. The missing pixels are so selective that it has to be 1 certain intensity value being missed.

Now if I change the math to

highlights=grey
shadows=1-grey

and keep everything else the same, no such issue exists.

I can also do

highlights=grey*grey
shadows=1-highlights

and also fine.

Im pretty puzzled why doing the first method creates those black dots, besides using 1 more layer there's nothing different. Every math I throw against that formula gives the correct output, yet in game I get some black pixels.
Is this some precision/rounding error? if so, does anyone know a fix that doesn't cause other bugs, or should I consider recoding the whole thing with using the results of shadows, mids, highlights as interpolation factors on a lerp?
I rather keep the code as it is now as the results are pretty ;)

Here's a shot where you see clearly the black pixels
Image

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: floating point precision or rounding error?

Looks like it. Floating point acts the same as cpu output (except amd cards, they give not exact results), so this is not videocard specific bug. Apply "saturate" to every 1- or -1 ops (or max/min).
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
*sensei*
Posts: 372
Joined: 28 Jul 2013, 23:26

Re: floating point precision or rounding error?

Tried that, no joy...
Anyway, recoded it now and black-dot-free, so far.

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: floating point precision or rounding error?

Strange. midtones=1-highlights-shadows code produce artifact, because (1-grey)*(1-grey) is always positive and small, so midtones=saturate(midtones) must work. Or may be later you have code which do not work when value is above 1 or do something very hard to negative values (like frac).
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
*sensei*
Posts: 372
Joined: 28 Jul 2013, 23:26

Re: floating point precision or rounding error?

don't know. This part of code runs very early in enbeffect... recoded the applygamecolorcorrections part and it was part of it, so perhaps there was a piece of code running after it that caused problems. I now made it like this to add midtones and shadow tint to HDR;

(renamed variables and simplified)

tint=_c4; //xyz=rgb, w=str
maxtint=max(max(tint.x, tint.y), tint.z);
if(maxtint==0) { //avoid dividing by 0... will always set tint to white when there's no tint (default engine value is white but devs on some weathers set tint to 0, 0, 0)
maxtint=1;
tint.xyz=1;
}
maxtint=1/maxtint;
tint.xyz*=maxtint;
hdr_intensity=dot(color.xyz, 0.33333333);
ldr_intensity.x=saturate(intensity);
ldr_intensity.y=1-ldr_intensity.x;
temp1.xyz=tint.xyz*ldr_intensity.x*ldr_intensity.y;
temp2.xyz=temp1.xyz+(hdr_intensity-ldr_intensity.x*ldr_intensity.y);
temp3.xyz=temp2.xyz*(1-shadow_str*ldr_intensity.y)+(tint.xyz*shadow_str*ldr_intensity.y);
temp2.xyz=lerp(temp3.xyz, temp1.xyz, ldr_intensity.x);
color.xyz=lerp(color.xyz, temp2.xyz, tint.w);

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: floating point precision or rounding error?

If this may happen later in code, try to force 1.001 and -0.001, if it will be buggy, you need to modify that code, may be it gives division by zero or something else like that.
Replace this

Code: Select all

maxtint=1/maxtint;
with this

Code: Select all

maxtint=1/max(maxtint, 0.00000001);
to increase performance without "if" statement.

Last argument of "lerp"s saturated would be lovely to prevent bugs with negative values, because you use subtraction earlier.
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
*sensei*
Posts: 372
Joined: 28 Jul 2013, 23:26

Re: floating point precision or rounding error?

Yes, originally had the code like

maxtint=1/(maxtint + epsilon);

and epsilon was declared earlier as 1e-10. But went over all the Skyrim image spaces, and really its a mess, so picked an if statement just to be sure. I'll try this max() approach, see if that goes well. All I hope is that no silly dev put somewhere tint at some intensity and the rgb at something like 0, 0, 0.000001 ;) but I don't trust skyrim devs anymore after trying to control with ENB what they did on Skyrim image spaces. What I noticed more than often is that they used adaptation values on images spaces to correct for wrong setup of light intensities in certain area just because easier to change 1 value than change 200. It's the most annoying thing ever. Best example is RiftenRaggedFlagon ... ceiling light in daytime is just insanely bright, unless you activate Skyrim adaptation, and more area's in Riften sewers suffer the same.

Good idea about putting the lerp's in a saturate to avoid things going bezerk later if something goes <0, thanks.

Offline
Posts: 1
Joined: 29 Jan 2015, 22:10

Re: floating point precision or rounding error?

Hey Prod, sorry to be a stalker, but I was freaking out that your files were no longer on the nexus and while I have them for myself, I hate to think that you will not be offering them in the future. You had the most amazing ENB settings and noone else had anything close. You are very gifted and I hope you well in the future either way!

If you leave you will become infamous like the guy that made Skyrim Realistic Overhaul(forgot his name lol) and people will have to go to nasty torrent sites to get your masterpiece ROFL!

Anyway, sorry to be a stalker(and sorry I hijacked this thread)

Offline
*sensei*
Posts: 372
Joined: 28 Jul 2013, 23:26

Re: floating point precision or rounding error?

I really don't care, some people believe I work for them and if I say no take very childish actions. I just code for fun (believe it or not) and I'm just an amateur. If it's not fun and people cant appreciate it being a hobby of mine then I simply don't care and remove. Maybe one day I reupload some place.

Offline
User avatar
*sensei*
Posts: 446
Joined: 17 Apr 2014, 22:12
Location: Schweden

Re: floating point precision or rounding error?

prod80
Some people seem to think modders profit from their work. Like it's a job.
We who have some sort of insight know this isn't true.
You have helped the community in ways people can't grasp. Your work has found its way into others work, including mine.
The whole community is in your depbt.
Thank you! :)

Btw, don't listen to them. And don't answer them.
_________________
| i5 3350p @3.1 | 16 GB RAM | GTX1060 | Skyrim on SSD |
My Flickr
My Soundcloud
CGI ENB
Post Reply