WebGPURenderer - Better packing with MRT

[FarazzShaikh]

Description

I have a usecase where I use a bunch of PostProcessing effects which require a lot of scene passes. I use setMRT to generate these passes.

const scenePass = TSL.pass(scene, camera);
scenePass.setMRT(
      TSL.mrt({
        output: TSL.output,
        normal: TSL.directionToColor(TSL.normalView),
        diffuse: TSL.diffuseColor,
        metalRough: TSL.vec2(TSL.metalness, TSL.roughness),
      })
);

However, this is not enough. I need to add an additional velocity pass. Doing so results in the total color attachment bytes exceeding its default maximum value of 32 bytes.

ERROR: Total color attachment bytes per sample (40) exceeds maximum (32) with formats...

I am already using UnsignedByteType for all of my passes.

outputTexture.type = THREE.UnsignedByteType;
...

Check out this fiddle with the error illustrated:

https://jsfiddle.net/farazshaikh/ztxdy6s4/14/

Solution

The solution here is to pack the passes more efficiently. However, packing the roughness/metalness into the alpha channels of the diffuseColor or normal results in unwanted alpha blending of transparent: true objects.

This is demonstrated in the following minimal Fiddle:

https://jsfiddle.net/farazshaikh/37c5uefh/2/

I am opening this issue to request an API for more efficient packing stratigies using MRT as suggested by Mugan87 here: https://discourse.threejs.org/t/webgpurenderer-postprocessing-cant-swap-beteween-results-from-2-differnet-mrt-passes/88047/5

Alternatives

The alternate solution is to set requiredLimits.maxColorAttachmentBytesPerSample in the WebGPURenderer constructor to higher than 32 bytes. But this is not scalable to all hardware.

Another alternative is to use multiple MRT passes, however, this seems to be broken as illustrated by this Fiddle: https://jsfiddle.net/farazshaikh/9dy7buqh/ . In this fiddle, I am unable to swap the MRT outputs attached to outputNode if the newly attached texture is from a different MRT pass.

I am unsure if this is a bug in ThreeJS or if I am misusing MRT.

Additional context

No response

[FarazzShaikh]

Another alternative is to use multiple MRT passes, however, this seems to be broken

Branched this issue out into #32253 as suuggested by Mugen87 on forum. (https://discourse.threejs.org/t/webgpurenderer-postprocessing-cant-swap-beteween-results-from-2-differnet-mrt-passes/88047/8)

[mrdoob]

I managed to do this packing a few weeks ago:

three.js/examples/webgpu_postprocessing_ssgi.html

Lines 76 to 84 in f303279

	const scenePass = pass( scene, camera );
	scenePass.setMRT( mrt( {
	output: output,
	// Pack metalness into diffuseColor alpha, and attenuate RGB by (1-metalness) for PBR
	diffuseColor: vec4( diffuseColor.rgb.mul( oneMinus( materialMetalness ) ), materialMetalness ),
	// Pack roughness into normal alpha channel to save MRT bandwidth
	normal: vec4( directionToColor( normalView ).rgb, materialRoughness ),
	velocity: velocity
	} ) );

[FarazzShaikh]

That great! But setting both boxes to transparent: true makes them dissapear from the diffuse and normal passes, even though I explictly pin their alpha to 1

I think this is beacuse the materialMetalness and materialRoughness is acting like their alpha when rendering the pass.

Notice that in the normal pass, only the tall box dissapears because only its roughness is explictly set to 0 whereas the small box's roughness is 1 by default.

And vice-versa for the diffuse pass and metalness.

Here is this fiddle: https://jsfiddle.net/farazshaikh/dLqvezwb/76/

Furthurmore, If i use the TSL proeprties metalness rather than materialMetalness we see that in the diffuse pass, the tall box shows its metalnessNode step pattern. The tall box is invisible where the metalness is 0 and visible where its 1

Fiddle: https://jsfiddle.net/farazshaikh/pt86cv91/5/

Is this expected? It seems like it is not and will produce incorrect results in SSGI, SSR or any other pass that uses the alpha-packed MRT passes.