Support for shadows. about gearvrf

[Maginador]

We are planning on shadows support implementation.
The main idea is to start with hard shadows based on something like shadow mapping technique.

Our initial definitions for parameters to expose for the user :

Light Parameters (GVRLight)

UseShadows (on/off) - bool useShadow
if the light is going to use shadow or not, in the case of more than one light, error/warning.

Shadow Type (hard/soft) - enum shadowType FUTURE USE
Defines the type of shadow that will be used, it interferes in what passes sequence to use.

Softness (0-1) - float softness FUTURE USE
How soft is the shadow going to be. The amount of feather applied to the borders.

Strength (0-1) - float strength
Opacity of the shadow, 1 for full shadow and 0 for no shadow.

Resolution (low/medium/high) - enum shadowResolution
Quality of the mapping (render resolution):
low = worse quality and higher performance.
medium = not bad quality and not bad performance.
high = higher quality and worse performance.

Bias(0-1) - float shadowBias
bias to prevent artifact (the higher the bias, higher shadow offset (contact shadows may disappear if bias is too high).

Other light important Parameters

Range(1 - x) - float lightRange
If using directional light its range is infinity, otherwise range will define how far the light interfere in shadow projection.

Angle/Direction (vec3(0 - 360)) - Vector3D lightDirection
Direction from where the light comes.

Object Parameters (GVRRenderData)

Cast Shadow (on/off) - bool castShadows
if this object is going to cast shadows towards other objects.

Receive Shadow (on/off) - bool receiveShadows
if this object is going to receive shadows cast by other objects.

[fredduarte]

@thomasflynn

Here is a Work In Progress implementation of Shadow Mapping on the GVRf:
https://github.com/fredduarte/GearVRf/tree/shadowmap (3 commits)

Again, this is NOT a final implementation, but a display of the shadowing capability on the framework.

There is an attached sample project with a couple of 3D models rendering shadows,
but, on the old directory structure (Samples dir along with the Framework).

This way, to use the shadow map, all is needed is the code that follows:

...
gvrDirectionalLight = new GVRDirectionalLight(gvrContext);

scene.setDirectionalLight(gvrDirectionalLight);

gvrDirectionalLight.setLightPosition(0, 7, 10);

gvrDirectionalLight.setLightDirection(0, 0, 0);
...

There's still work to be done:

• Add multiple sources of light;
• Add Directional Light as a scene object;
• Improve final rendering visual by addressing the shadow 'acne' and other issues;

Feedback from this sample code is welcome. :-)

[Vittalbabu]

I have a proof of concept (initial checkin) of lighting in https://github.com/Vittalbabu/GearVRf/commits/light More checkins will follow after reviewing feedback.

It supports a single light with properties taken from the Material. Plan is to support individual light classes, multiple lights, lights attached to Scene instead of Material, a more robust shader, etc. Some testing has been done but main goal is to invite discussion on lighting, shadow maps and other light shaders. I will be happy to provide more detail and clarifications.

Thanks,

• Vittal

Lighting – covers multiple types of lights, multiple lights.

Proposal –

1. Allow creation of global lights (multiple types of lights and multiple instances). Light information will be removed from Material class and added to Scene
2. Keep the current implementation where RenderData holds Material for a Mesh. Each object will decide which shader to use – some shaders could have light and some not. Default Shader for a Material is Light Shader (instead of texture shader).
3. Light shader has two modes.
   • Use a texture + light properties to calculate the final color.
   • Use material properties + light properties to calculate final color.
   • The shaders can be changed by app.
4. Shadow maps are supported as follows.
   • Shadows are supported via an app settable option.
   • Initial pass to create the shadow map as texture(s).
   • Use these in the Light shader to implement shadows.
5. Provide more than one Light shader to implement different lighting algorithms.

Some background :-

How do we want to implement lighting?
• Should it be global (applied to all relevant shaders)? Or should we have the flexibility to allow individual Scene Objects to decide whether to implement lighting? Prefer global lighting model but open to suggestions.
o Global lighting imposes constraints on Material. RenderData instances with meshes should have a Material with some minimal properties to support lighting (normal, surface properties) . Otherwise lighting calculations will vary across different objects – some objects will have lighting calculated in the shader and some will not.
o How to integrate lighting in shaders? There will be a light shader that implements a reference lighting algorithm.
• Should lighting be one pass or two pass?
o One pass will be more efficient
o Two pass lighting is simpler to implement. First pass writes out a texture map and a normal map. Second pass uses this plus light data (positions, properties) to calculate lighting. We could integrate this in the post process step. All objects will be equally affected by lighting. This works with all shaders (if normal are not output we choose a default normal).

• Light algorithm.
o Can be done in vertex shader or fragment shader. Vertex shader lighting will not be accurate in some cases ((eg) specular lighting in middle of the face is usually not accurate) but is faster. Current implementation is in fragment shader.
o
• Light types – Have classes for different light types.
• Lights are attached to Scene for global illumination. To handle dynamic lights (car headlights), the light instances can hold a reference to a SceneObject. The light instance will get position and orientation information from the SceneObject.

How do shadow maps and other lighting calculations fit?

• To support shadow maps, we need textures that hold the shadow map. We can incorporate that in the lighting shader.
• Other photo-realistic lighting calculations. We can allow the app to set custom light shaders.

[Vittalbabu]

@fredduarte The approach looks good. Let's discuss if my proposal above fits your design and illumination models in general.

[fredduarte]

Hi, @Vittalbabu.

This approach implemented on the pull-request ( [https://github.com//pull/381] ) i've made a couple of days ago
was mostly implemented by a former work partner of ours, Rodrigo ([email protected]).
He helped us a lot getting to this solution, so, i've asked him to give some
feedback on your comments above. I translated and cited them above [hope my english is
OK to read and i didn't say anything stupid :-) ]. I've added a little input as well.

I have a basic understanding of the shadow mapping technique and lighting in general,
so, from now on i'll try my best to answer any questions you have, and see if we can
improve this solution even further.

Below are the answers (sorry for the long post):

I have a proof of concept (initial checkin) of lighting in https://github.com/Vittalbabu/GearVRf/commits/lightMore checkins will follow after reviewing feedback.
It supports a single light with properties taken from the Material. Plan is to support individual light classes, multiple lights, lights attached to Scene instead of Material, a more robust shader, etc. Some testing has been done but main goal is to invite discussion on lighting, shadow maps and other light shaders. I will be happy to provide more detail and clarifications.

• Important note: lighting over the object itself is commonly done with light points, making it possible to apply it in a single shader, which helps a lot on the performance.
  On the shadowmap case, extra rendering steps are necessary to obtain the shadowmap from the lights (on the implemented shadowmap case, there is only one light).

Lighting – covers multiple types of lights, multiple lights.

• Several light points can be implemented to calculate the lighting, but, i believe that to calculate several shadowmaps for each light point would be impracticable.
  (So, i believe the ideal would be to have 1 directional light that causes shadow and several other light points that would create the lighting effects, but,
  wouldn't generate shadows on others)

Proposal –

1 Allow creation of global lights (multiple types of lights and multiple instances). Light information will be removed from Material class and added to Scene

a) To organize in multiple types of light sounds interesting.
b) About removing the material, i believe it should work in conjunction with the Material class, because the final color of the object is the combination of the light
and the interaction with the object itself, in this case, the material type.

1. Keep the current implementation where RenderData holds Material for a Mesh. Each object will > decide which shader to use – some shaders could have light and some not. Default Shader for a Material is Light Shader (instead of texture shader).

a) So, as far as i was able to analyze the framework structure, the TextureShader already has an implementation related to lighting, but,
so far, i see no problem in separating it on a LightShader.

1. Light shader has two modes. • Use a texture + light properties to calculate the final color. • Use > material properties + light properties to calculate final color. • The shaders can be changed by app.

a) As i said above, i think we should use the second option (light + material), but, if we were to provide one or the other for performance reasons i believe it won't be an issue.

1. Shadow maps are supported as follows. • Shadows are supported via an app settable option. • > Initial pass to create the shadow map as texture(s). • Use these in the Light shader to implement shadows.

a) So, here we implemented the class ShadowShader, which is capable of taking advantage of other shaders to calculate the color that will be applied
in conjunction with the shadowmap to render the shadows. (Note: all objects can generate shadows, mas, only the static objects can receive shadows)

b) On our version, all this work is already done. It is only necessary to the the DirectionalLight on the Scene. (Less control for the dev, but, with more easiness of use).

Some background :­

How do we want to implement lighting?

a) Excelent question, as said on the beginning, a basic lighting (DirectionalLight) was implemented for the shadowmapping.
In fact, we can have a good range of lighting possibilities here.

• Should it be global (applied to all relevant shaders)? Or should we have the flexibility to allow individual Scene Objects to decide whether to implement lighting? Prefer global lighting model but open to suggestions.

a) About the framework's current approach to be flexible and not global, i believe that it would be better as global..., but, it's just an opinion.

o Global lighting imposes constraints on Material. RenderData instances with meshes should have > a Material with some minimal properties to support lighting (normal, surface properties).Otherwise > lighting calculations will vary across different objects – some objects will have lighting calculated in > the shader and some will not.

a) So, as said before, the light and the material should be calculated in a combined way, and yes the light has priority, but, it shouldn't ignore the material of the object.
Ex.: A green light illuminates a blue cube, the result ends up being black. That is, following the primary RGB colors (that of course depending on the material).

b) As for normals, surface properties, yes that is essential for the fast calculations.

c) To shut off light over some objects is good for optimization.

o How to integrate lighting in shaders? There will be a light shader that implements a reference lighting algorithm.

a) So, on the ShadowShader we implemented a very basic one. But, since i'm taking advantage of the other shaders,
as long as the objects position buffer doesn't change, other shaders will be used automatically increasing the visual quality according to the selected shaders.

• Should lighting be one pass or two pass?

a) Shadowmap necessarily has to render on more than one camera.

b) We are using 2 (actually 3) : (light), (color from other shaders), (light + color == shadow).

o One pass will be more efficient

a) we believe it is not possible to utilize shadowmap this way.

o Two pass lighting is simpler to implement. First pass writes out a texture map and a normal map.

Second pass uses this plus light data (positions, properties) to calculate lighting. We could integrate this in the post process step. All objects will be equally affected by lighting. This works with > all shaders (if normal are not output we choose a default normal).

a) Disregarding the shadowmap, and if we consider only the light, we believe that your talking about leaving the light information pre-calculated on a texture in a First step,
and on the second step apply it to other shaders. This approach has many similarities with the method of capturing the final color of all the other shaders that i'm using,
and combining it to generate the shadows (almost like a "post-process" as you said on the text). If implemented creatively, i think we could achieve good results with this approach.

• Light algorithm.

o Can be done in vertex shader or fragment shader. Vertex shader lighting will not be accurate in > some cases ((eg) specular lighting in middle of the face is usually not accurate) but is faster. Current implementation is in fragment shader.

a) We are combining both (vertex and fragment), using the main part on the fragment shader. As said earlier it improves precision.

• Light types – Have classes for different light types.

a) We implemented a class called DirectionalLight. On it, there are options for rendering the shadowmap with orthogonal or perspective camera.
At the moment, this class is separated from the standard class Light, because, this class Light, considers only light points, in a simple way.

• Lights are attached to Scene for global illumination. To handle dynamic lights (car headlights), the light instances can hold a reference to a SceneObject. The light instance will get position and orientation information from the SceneObject.

a) We also have to do this as well on our code. :)

How do shadow maps and other lighting calculations fit?

a) So, we are taking advantage of the color calculations from the other shaders, therefore, if the color/image is with the right lighting, the shadowmap will use this automatically.
It is the same principle of rendering in 2 steps, mas, in our case, we render it in 3 ( light, color and [light + color == shadow]).

• To support shadow maps, we need textures that hold the shadow map. We can incorporate that in the lighting shader.

a) In this case, we already have the shadowmap and in have it a way that can integrate automatically with all the other shaders.

• Other photo-realistic lighting calculations. We can allow the app to set custom light shaders

a) About that, it is a case for studying, but, regarding custom light shaders, i believe the framework already has a custom shader class.

@fredduarte The approach looks good. Let's discuss if my proposal above fits your design and illumination models in general.

a) On our approach there are definitely points for improvement regarding the integration with the framework.

b) In general, it works as follows:
1 - We on Render::renderCamera() if the scene has a DirectionalLight and in case it does, it enters a special mode
where it renders the scene 3 times.
1.1 The first rendering is from the light perspective;
1.2 The second rendering is from the camera perspective in a way to process all shaders (including animation and other possible future lights);
1.3 The third rendering is the union, joining the 2 steps from above to generate the shadows (note: it is performed over the 3D Models and not the 2D plane (with 3D we have more precision))

BR,

[fredduarte]

Hi, @thomasflynn
i know the pull-request for the shadow mapping was merged already.
But, is Vittalbabu still working on the lighting branch previously discussed here?
[https://github.com/Vittalbabu/GearVRf/commits/light]

[thomasflynn]

@Vittalbabu could you respond please. Perhaps @fredduarte can help you finish your commit while you're tracking down that performance problem.

[Vittalbabu]

@fredduarte

I had some issues with the lighting equation and moved to using the lighting equations implemented in texture shader. The framework in [https://github.com/Vittalbabu/GearVRf/commits/light] can support multiple lights but it is incomplete in the sense the light parameters passed from application to Java to jni will need to change based on proper implementation of multiple lights.

If you are still unable to access my branch, could you implement multiple lights in the following manner --

1. Implement the vertex+fragment shaders, uniforms, etc. for multiple lights. You can modify the texture_shader class and hardcode a certain number of lights.
2. If you can access my branch, you can make the changes directly to light_shader.
3. Once the shader is working, we will update the light parameters sent from app to Java to jni and remove the hardcoded lights. App will properly set the lights.

One goal is to avoid re-compiling shaders if the number of lights change. We can set a limit on the number of lights (10?) and always allocate uniforms for this many lights.

If you need any further clarifications, I can provide them.

Thanks,

• Vittal

[fredduarte]

@danke-sra Please check #433
I've made some changes to support the previously created Light class.
Changed the shadow sample as well.

[danke-sra]

@fredduarte Thanks for the update. I will take a look ASAP.

[thomasflynn]

support for multiple lights and shadow maps has been merged.