help shadows optix7 about lighthouse2

[PerspectivesLab]

ok i figured out ! i was applying a scale a root node level in the gltf file that was misleading the shadow calculations, the solution was to apply scale directly at vertex level, and then exporting to gltf

[PerspectivesLab]

it seems while applying a scale to the model ( ex 0.5,0.5,0.5 ), that the source matrix calculation is causing incorrect results for shadows/shade/ambient:
in rendercore.cpp

			if (instance->transform)
			{
				T = mat4::Identity();
				memcpy( &T, instance->transform , 12* sizeof( float ) );
				invT = T.Inverted();  

			}
			else
			{
				T = mat4::Identity();
				invT = mat4::Identity();
			} 

if i use only:

				T = mat4::Identity();
				invT = mat4::Identity();

the shadows/ambient are working
could you comfirm you have the same behaviour with the gltf model provided ? with rendercore optix 7.0.
how can i make model transform scale work ? thx

[PerspectivesLab]

ok i created a pull request for that, let me know if it goes through, thx

[jbikker]

Sorry for the late reply, I missed your comment.
I think the issue is a bit deeper: when using a transform on instances the shading code must take this into account for normals by applying the same transform to these vectors. Or more accurately, to obtain corect transformed normals the inverse of the transpose of the transform must be applied. This is the same as the transform itself for orthonormal matrices, but scale matrices are not orthonormal...
I will fix this in the new year. In the meantime: the fix you proposed will not work if the model is rotated, but for now it should indeed solve your problem.

[jbikker]

I have committed a couple of fixes that should resolve your problem. I will close the issue now; if it turns out your problem persists please reopen it.