[enhancement, needs investigation] Add customizable visual effects about neothesia

[Rasmez]

Interesting... sounds like a plan!

I still need to make (basically from garbage) a MIDI to USB cable to be able to use my keyboard as input, so... 1 problem at the time. Thank you so much for the clarification and orientation, greatly appreciated, good luck and keep the good work mate.

[PolyMeilex]

Hi, first, thanks for taking the time to write this.

First of all, I'm not familiar with the Rust programming language to say if this is out of the scope of the project in terms of difficulty or something

Language does not matter too much in this case, but it's worth noting out that Neothesia does not use any game engine like SeeMusic (Unity) or Keysight (Unreal), so adding effects is not just a matter of slapping unity particle system on top of a keyboard. The system has to be designed from the ground up, ideally with customizability in mind. It's a lot more work but it opens up a lot of possibilities.

SeeMusic and Keysight have some visual effects in them, but they are not fully customizable

Oh boy, even without customizability those are beautiful, it's hard to compeate, they are simply gorgeous.

Add a way to add custom GLSL shaders

Yeah, that's cool, but it's a little bit problematic because Neothesia supports Vulkan, DX12, and Metal graphics backends, and GLSL is supported only by one of those, so GLSL has to be precompiled to Spirv binary format. It's done in the build, not in runtime. So either the user would have to provide precompiled GLSL, or WGSL would be used instead, as it is a language designed from the ground up with this use case in mind.

Another question is, is shader enough? Let's say that the user wants to use a different particle movement system than the default one, how would one specify that? Do we create a customizable computation shader that does all the positioning stuff? And I have no idea if all effects are possible to write just in shaders, the lack of proper persistent state will be problematic. Using frame buffers for some effects will also be impossible as it requires CPU programming.

Tradeoffs

To be honest I don't see any serious ones, as long as those effects are optional.
The only tradeoff is that it would increase code complexity quite significantly. I wouldn't be too surprised to see that majority of the code in the codebase would become dedicated only to this feature exclusively. It's like building a midi visualizer framework of some sort.

It's a cool idea but I'm not sure if I can pull this off alone, it's quite a complex design problem. Some simple predefined effects will be added sooner or later, but making it customizable is whole another level.

Note to future self:
Glava does allow custom shaders to be used (but without particles), maybe it's worth reading through it.

[Rasmez]

Quite unfortunate indeed... well... we can't have anything we want ¯_(ツ)_/¯

I guess that I'll need to add Rust to my list of languages, learn the thing and "port" the project to Godot, to have the functionality that I want... ಠ_ʖಠ

I'll watch closely the project, when it mature a little bit, I'll use it... maybe... I need to add a way to plug my old Casio CTK-480 keyboard to a computer first tho... Ah... the perks of poverty in a socialist hell... darn it (ṧ︵ṧ)

[PolyMeilex]

If you have any questions about how this (or any other synthesia like) project works, just let me know. I will try to give a helpful answers.

You can also read through:
Old synthesia codebase (c++, Old OpenGL) (yes it used to be open source)

When it comes to Neothesia, If you are just interested in file read and timing calculations you can just look at lib_midi directory. It's totally Neothesia independent so it's a grate starting point, because after all the calculations are done you can easily handle displaying of notes in Godot yourself.

My discord tag is: poly#1945
My discord server sgeZuVA

Fell free