[FO4] Replicating retro shader?

share shaders here
  • Author
  • Message
Offline
User avatar
*sensei*
Posts: 316
Joined: 12 Aug 2013, 18:55
Location: Scotland

[FO4] Replicating retro shader?

Ages ago, a user on Reddit made this image by reducing it's palette to 16 colours, and then downscaling it:

Image

Now that looks kickass, almost like an old-timey DOS game pre-render.

How would I go about creating this effect? I assume sort of LUT in enbeffect.fx, then downscaling in enbeffectpostpass.fx. But how to do that, I have no idea.
_________________
Intel i7 6700k | AMD Vega 56 8GB | 2x16GB DDR4 @ 3000mhz | Windows 7 64bit | Creative Soundblaster X-Fi Titanium Fatal1ty Pro | Asus z170 Pro Gaming

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: [FO4] Replicating retro shader?

There is dos game shader for Deus Ex HR, you can use it for basis
http://enbdev.com/effect_en.htm
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
User avatar
*sensei*
Posts: 316
Joined: 12 Aug 2013, 18:55
Location: Scotland

Re: [FO4] Replicating retro shader?

I took a look at the postpass .fx file, and it's a bit too complex for me; I'm using KingEric's barebones one for the SMAA effect (viewtopic.php?f=7&t=4721), so I'm not sure how to add it without breaking everything.

Is it possible to use ENBEffect.fx for this shader?
_________________
Intel i7 6700k | AMD Vega 56 8GB | 2x16GB DDR4 @ 3000mhz | Windows 7 64bit | Creative Soundblaster X-Fi Titanium Fatal1ty Pro | Asus z170 Pro Gaming

Offline
User avatar
*blah-blah-blah maniac*
Posts: 17427
Joined: 27 Dec 2011, 08:53
Location: Rather not to say

Re: [FO4] Replicating retro shader?

Everything is possible, but i'm busy with Fallout 4 modding now to make anythings like this inside non mine code (i don't like to touch code of other people).
_________________
i9-9900k, 64Gb RAM, RTX 3060 12Gb, Win7

Offline
User avatar
*blah-blah-blah maniac*
Posts: 530
Joined: 30 Jan 2012, 13:18

Re: [FO4] Replicating retro shader?

I can do that for you, if you give me a little time, not at home right now. You're using the standard SMAA postpass.fx from kingeric with no other changes?

Offline
User avatar
*sensei*
Posts: 316
Joined: 12 Aug 2013, 18:55
Location: Scotland

Re: [FO4] Replicating retro shader?

Aye, I am. I would like to learn how to do it myself, but I'd need an example to show how it's done. No idea what I was doing when I tried to do it.
_________________
Intel i7 6700k | AMD Vega 56 8GB | 2x16GB DDR4 @ 3000mhz | Windows 7 64bit | Creative Soundblaster X-Fi Titanium Fatal1ty Pro | Asus z170 Pro Gaming

Offline
User avatar
*blah-blah-blah maniac*
Posts: 530
Joined: 30 Jan 2012, 13:18

Re: [FO4] Replicating retro shader?

There's more to the trick than just downscaling and degrading color range. The image looks as if the author uses a custom palette and also dithered the image with bayer I guess to reduce the banding. I can set you something up rather quickly, but as I don't like retro effects at all, you have to tell me if it does what you expect because to me, it kinda looks all same.

EDIT: Ok, so I found the respective reddit thread. Seems like the guy did this:
Stuck the image in Aseprite, then indexed the pallete to about 16 colors. After that I just lowered the size of the image and voila, 'pixel art', even if it looks a bit processed.
So it's not easy at all to replicate this. Maybe if you'd go the LUT way and do the same but then you'd need to do exactly what he did. I just semi-ported Boris' retro fx.

https://pastebin.com/LSGieS2g

use with the rest of kingeric's smaa files. I cannot test atm how it looks ingame as I don't own F4 and my SSE crashes with ENB when the game has finished loading - which makes absolutely NO sense but oh well..

Offline
User avatar
*sensei*
Posts: 316
Joined: 12 Aug 2013, 18:55
Location: Scotland

Re: [FO4] Replicating retro shader?

Unfortunately, it doesn't run: Image

It thinks that there are SMAA variables being used elsewhere, and there is no Sampler0 variable.
_________________
Intel i7 6700k | AMD Vega 56 8GB | 2x16GB DDR4 @ 3000mhz | Windows 7 64bit | Creative Soundblaster X-Fi Titanium Fatal1ty Pro | Asus z170 Pro Gaming

Offline
User avatar
*blah-blah-blah maniac*
Posts: 530
Joined: 30 Jan 2012, 13:18

Re: [FO4] Replicating retro shader?

WTF I swear I fixed that. There's also another bug I definitely fixed as it definitely ran. Seems like I didn't copy the file properly and had an old version in the clipboard... Oh well. You can get the sampler0 from the standard file, just put it anywhere above the pixel shader.

And in the pixel shader itself, replace origcolor with color.

And replace the line that declares texcoord with

float2 texcoord = floor(IN.txcoord0.xy * screenresRescaled) / screenresRescaled;

No clue why I sent you the wrong one as I had it definitely running.. The warnings you see there are always there but ENB suppresses them, as long as there is no error that aborts compilation, so I assume the original SMAA effect is buggy but not buggy enough to break it.

Offline
User avatar
*sensei*
Posts: 316
Joined: 12 Aug 2013, 18:55
Location: Scotland

Re: [FO4] Replicating retro shader?

Oh man, it looks so good. Now I need to figure out how to make LUTs, to get the colours right.

All I could find was this super expensive software that generates it from two source images. Is there a more, eh, inexpensive way of making them?
_________________
Intel i7 6700k | AMD Vega 56 8GB | 2x16GB DDR4 @ 3000mhz | Windows 7 64bit | Creative Soundblaster X-Fi Titanium Fatal1ty Pro | Asus z170 Pro Gaming
Post Reply