babylonjs / babylonnative Goto Github PK

• RTT textures are flipped verticaly.
  This is more complex to fix than the loaded textures.

I tried to download .js script instead of loading resource assets using https on the GitHub raw repo. I get an CURLE_UNSUPPORTED_SCHEME error code from curl_url_set. Apparently it has something to do with proxies (?)

BROWSER CAPABILITIES

• Complete implementation of XMLHttpRequest
• Investigate Web workers
• Investigate WebAssembly

ARCHITECTURE

• Propose and implement proper InputManager

NATIVE ENGINE

• Finalize Babylon Native contract
• Correctly support device loss
• Skyboxes
• Fix stub for URL.createObjectURL in engine
• Fix stub for URL.revokeObjectURL in engine
• Fix stub for Blob in engine
• Investigate BGFX multi-threading
• drawElementsType and drawArrayType should be more robust
• Refactor NativeEngine.ts to share more with other engines
• Implement setViewport
• Fix TODO's in NativeEngine.ts
• Complete implementation of NativeEngine.createTexture
• Implement instancing support
• Implement XHR progress support
• Implement GLB support
• Ensure proper support for Unicode file paths
• Revise Tools.LoadScript to not depend on document
• Handle backbuffer, depth, and stencil in ViewClearState
• Ensure BindFrameBuffer is setting all the necessary state
• Investigate limiting frame buffer SetUpView to only happen after reset (may not have meaningful perf impact)
• Refactor OpenXR support to minimize/eliminate per-frame allocations

TEST INFRASTRUCTURE

• Continuous integration
• Pass all asset generator tests
• Pass all Babylon.js tests
• Create test automation

DOCUMENTATION

• Create React Native sample project
• Create Vue sample project
• Create InputManager sample project
• Automate documentation (if API is large enough?)
• Document Babylon Native contract
• Investigate XAML example

CROSS PLATFORM

• Support Android
• Support iOS
• Support macOS
• Support XAML by implementing ISwapChainPanel constructor
• Ensure proper support for used texture types across platforms

JAVASCRIPT INFRASTRUCTURE

• Enable V8 debugging
• Correctly propagate and handle JavaScript errors
• Investigate JSI support

Ordered by complexity

• the project compiles for x86 and x86_64 but has never been tested on the simulator
• V8 .so are poorly handled in the APK for the Native Activity project. It needs to be fixed before it can run
• V8 must be installed manually for local builds. This should be done automatically in CMake
• Screen ratio is not updated with screen rotation
• Add support for SSL
• Shader compliance and parsing between BJS and bgfx is not great for OpenGL. Expect issues when dealing with more complex shaders
• Front facing is inverted (you see the inside of the cube). A bgfx repo update will be done shortly and should be done on Master @CedricGuillemet
• Asset loading is using a callback mecanism that supports APK assets loading. It will be replaced by a plugin extension (#101)
• getHostDocument is missing in native engine definition ( https://github.com/BabylonJS/BabylonNative/pull/30/files#diff-61eebfcda2bd45c2180b579c8f4fcd42R51 ) => PR : BabylonJS/Babylon.js#7286

Call stack:

>	TestApp.exe!bx::debugBreak() Line 38	C++
 	TestApp.exe!bgfx::CallbackStub::fatal(const char * _filePath, unsigned short _line, bgfx::Fatal::Enum _code, const char * _str) Line 70	C++
 	TestApp.exe!bgfx::fatal(const char * _filePath, unsigned short _line, bgfx::Fatal::Enum _code, const char * _format, ...) Line 438	C++
 	TestApp.exe!bgfx::setViewFrameBuffer(unsigned short _id, bgfx::FrameBufferHandle _handle) Line 4714	C++
 	TestApp.exe!babylon::NativeEngine::SetViewPort(const Napi::CallbackInfo & info) Line 1349	C++
 	TestApp.exe!Napi::ObjectWrap<babylon::NativeEngine>::InstanceVoidMethodCallbackWrapper::__l2::<lambda>() Line 3409	C++
 	TestApp.exe!Napi::details::WrapCallback<void * <lambda>(void)>(Napi::ObjectWrap<babylon::NativeEngine>::InstanceVoidMethodCallbackWrapper::__l2::void * <lambda>(void) callback) Line 61	C++
 	TestApp.exe!Napi::ObjectWrap<babylon::NativeEngine>::InstanceVoidMethodCallbackWrapper(napi_env__ * env, napi_callback_info__ * info) Line 3411	C++
 	TestApp.exe!`anonymous namespace'::ExternalCallback::Callback(void * callee, bool isConstructCall, void * * arguments, unsigned short argumentCount, void * callbackState) Line 172	C++

Looks like the ViewID mechanism is no longer functioning as it used to.

check NativeEngine::Impl::SetState

Remove the win32 code in console.cpp and allow for the client to register output functions/lambda. Test apps will be responsible for the output (outputDebugString, android_log_output,...)

https://github.com/BabylonJS/Babylon.js/blob/120cdb8cec65f678af92e37b319b87eb54fd079a/src/Engines/nativeEngine.ts#L265-L275

Object URL and blobs are currently not supported in the Native Engine; they are "hacked out," as shown at the link. Implement in order to achieve feature parity, enabling load progress.

Spheres rendering differently than expected.

https://github.com/BabylonJS/BabylonNative/blob/master/TestApp/Scripts/experience.js#L12-L26

Pursuant to a conversation and changes in PR #30

https://github.com/BabylonJS/BabylonNative/blob/master/TestApp/Source/UWP/App.cpp#L240L243

DPI changes should likely be handled inside the platform-specific Runtime extensions (RuntimeWin32, RuntimeUWP, etc.) and only get communicated down to lower layers as resolution changes.

To access local files in the APK (scripts/*.js), Android asks to use an asset manager. You can't open the file directly and basically, it's not even a file in the filesystem but a bunch of bytes in a zip. For now, I've added a function, per platform that does the job if curl doesn't succeed at loading the resource.

Add support for extremely basic input (comparable to OpenXR's Khronos simple_controller) for XR scenarios, including full routing up to Babylon. This will make it easier for Babylon.js to test and iterate on WebXR support using knowable and controllable inputs from the underlying implementation.

We're currently forcing BGFX to be single-threaded, which isn't its favorite thing to do, only because it aligned more directly with how we prototyped the integration. Evaluate what it would take/mean to enable multithreading in BgfxEngine.cpp.

When generating project with CMake, you can only create one testapp target.

• win32 or UWP on Windows
• ios or macos on Mac
  I was wondering if we should not create a testapp_win32 and testapp_UWP target on Windows, in the same generated project.
  and both targets on mac as well.

We currently have a mixture of patterns for checking platforms throughout our CMakeLists.txt files, mixing variables like WINDOWS_STORE and APPLE in with our usage of BABYLON_NATIVE_PLATFORM. Let's discuss (here in this issue?) what the preferred pattern should be, then propagate pattern everywhere that makes sense and, if there are exceptions where the pattern should be broken, clearly designate these exceptions with comments.

Just a reminder here
microsoft/arcana.cpp#4

• Ensure that ViewClearState::Update is actually setting all the appropriate flags, or open issues/documentation to remedy that later:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.h

  Lines 86 to 92 in 1b40d7b

  	void Update() const
  	{
  	bgfx::setViewClear(m_viewId, m_flags, Color(), m_depth, m_stencil);
  	// discard any previous set state
  	bgfx::discard();
  	bgfx::touch(m_viewId);
  	}

• Consider optimizing FrameBufferManager::Bind by only calling SetUpView when absolutely necessary (which may be as infrequently as on context loss):

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.h

  Lines 175 to 185 in 1b40d7b

  	void Bind(FrameBufferData* data)
  	{
  	m_boundFrameBuffer = data;
  	// TODO: Consider doing this only on bgfx::reset(); the effects of this call don't survive reset, but as
  	// long as there's no reset this doesn't technically need to be called every time the frame buffer is bound.
  	m_boundFrameBuffer->SetUpView(GetNewViewId());
  	// bgfx::setTexture()? Why?
  	// TODO: View order?
  	}

• bgfx currently reorders FrameBuffer submissions according to its own default order. Enforce the provided order so that behavior exactly matches the Web behavior:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.h

  Line 184 in 1b40d7b

  // TODO: View order?

  @CedricGuillemet
• Investigate

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Line 1279 in 8873731

  throw std::exception{/* Does this case even make any sense? */};

• Do we need these enums anymore? If not, remove them: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngineImpl.h#L295-L297 (dead link, enums already removed)
• Remove NativeEngine::Impl::GetEngine once the real mechanism for creating/destroying engines is implemented:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.h

  Line 326 in 1b40d7b

  Napi::Value GetEngine(const Napi::CallbackInfo& info); // TODO: Hack, temporary method. Remove as part of the change to get rid of NapiBridge.

  (@syntheticmagus)
• Consider options for removing extricated bgfx internals from NativeEngine.cpp:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Lines 21 to 23 in 1b40d7b

  	#define BGFX_UNIFORM_FRAGMENTBIT UINT8_C(0x10) // Copy-pasta from bgfx_p.h
  	#define BGFX_UNIFORM_SAMPLERBIT UINT8_C(0x20) // Copy-pasta from bgfx_p.h
  	#define BGFX_RESET_FLAGS (BGFX_RESET_VSYNC | BGFX_RESET_MSAA_X4 | BGFX_RESET_MAXANISOTROPY)

• Discover implications of “num” value on non-D3D11 pipelines and implement/document accordingly: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L115
• Revisit empty bgfx::VertexDecl hack in NativeEngine::Impl::CreateVertexBuffer:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Lines 539 to 543 in 1b40d7b

  	// HACK: Create an empty valid vertex decl which will never be used. Consider fixing in bgfx.
  	bgfx::VertexLayout vertexLayout;
  	vertexLayout.begin();
  	vertexLayout.m_stride = 1;
  	vertexLayout.end();

• Experiment with ways to remove the infamous dFdy hack: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L375-L377 (@bghgary)
• Set the Z offset in SetState:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Lines 786 to 787 in 1b40d7b

  	// TODO: zOffset
  	const auto zOffset = info[1].As<Napi::Number>().FloatValue();

  [captured in #1053]
• Stubs: SetZOffset, GetZOffset, SetDepthTest, GetDepthWrite, SetDepthWrite, SetColorWrite. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L573-L611 @CedricGuillemet
• SetIntArray stubs. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L634-L660 @CedricGuillemet
• SetFloatArray stubs. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L679-L698 @CedricGuillemet
• SetMatrix stubs. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L711-L723 @CedricGuillemet
• SetBool stub. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L781-L786 @CedricGuillemet
• Mipmaps: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L805 @CedricGuillemet
• Texture sampling, wrap, and anisotropic stubs. Implement or delete: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L902-L926 @CedricGuillemet
• Handle viewport in DrawIndexed: https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeEngine.cpp#L1009 @CedricGuillemet
• Draw stub. Implement or delete:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Lines 1344 to 1353 in 1b40d7b

  	void NativeEngine::Draw(const Napi::CallbackInfo& info)
  	{
  	const auto fillMode = info[0].As<Napi::Number>().Int32Value();
  	const auto elementStart = info[1].As<Napi::Number>().Int32Value();
  	const auto elementCount = info[2].As<Napi::Number>().Int32Value();
  	// STUB: Stub.
  	// bgfx::submit(), right? Which means we have to preserve here the state of
  	// which program is being worked on.
  	}

  @CedricGuillemet
• Confirm that GetRenderWidth and GetRenderHeight are returning the correct values:

  BabylonNative/Plugins/NativeEngine/Source/NativeEngine.cpp

  Lines 1375 to 1383 in 1b40d7b

  	Napi::Value NativeEngine::GetRenderWidth(const Napi::CallbackInfo& info)
  	{
  	return Napi::Value::From(info.Env(), bgfx::getStats()->width);
  	}
  	Napi::Value NativeEngine::GetRenderHeight(const Napi::CallbackInfo& info)
  	{
  	return Napi::Value::From(info.Env(), bgfx::getStats()->height);
  	}

  @CedricGuillemet
• Fix stub for URL.createObjectURL in engine [captured in #1240]
• Fix stub for URL.revokeObjectURL in engine [captured in #1240]
• Fix stub for Blob in engine [captured in #1241]

Repro steps:

1. Launch default TestApp experience
2. Wait for experience to finish loading, then press "R" to recreate the Babylon Native Runtime.

Result:
Every time the Runtime is recreated, memory usage climbs.

Expected Behavior:
All Runtime resources should be discarded successfully when the Runtime is recreated, so there should be no leftovers to occupy memory after pressing "R".

crossplatform audio engine
https://github.com/jarikomppa/soloud
Do you know any other one?

Centralising all the CURL issues here. This might be used as requirements for a CURL replacement.

• Support non-blocking curl
  Our usage of libcurl is currently the blocking synchronous version. We should use the non-blocking asynchronous version.
• UWP with broad file system access in libcurl fails
  For UWP, we have <rescap:Capability Name="broadFileSystemAccess" /> set in the manifest and have enabled in the settings to allow the app.
• Curl doesn't handle binary to text conversions. We need to deal with the BOM and text-encoding, etc. See here.

libcurl eventually calls the C open API (which eventually calls CreateFileW), but this returns -1.

We currently have a fallback mechanism to use Windows::Storage::StorageFile::GetFileFromPathAsync instead, but it would be nice to see if we can get libcurl to work with broadFileSystemAccess.
Curl and local file access
To access local files in the APK (scripts/*.js), Android asks to use an asset manager. You can't open the file directly and basically, it's not even a file in the filesystem but a bunch of bytes in a zip. For now, I've added a function, per platform that does the job if curl doesn't succeed at loading the resource.
Issue moved to plugin #119

• Remove workaround for cURL integration in UWP
  

  BabylonNative/Library/CMakeLists.txt

  Lines 48 to 53 in ab9fd87

  	# TODO: Certain parts of cURL's functionality are gated behind WINAPI checks
  	# that cause the functionality to become unavailable in UWP. Find a better way
  	# to ensure that functionality is enabled, then remove the following workaround.
  	if(WINDOWS_STORE)
  	target_compile_definitions(libcurl PRIVATE "WINAPI_PARTITION_DESKTOP=1")
  	endif()

• CMake libcurl taking too much time
  CMake 3min40 to generate solution per target on Azure with at least 2min30 taken only by libcurl
  There must be some flags to change to speed up the solution generation time.
• Security issue on Android with OpenSSL. I had to disable host and peer verification because of curl unable to check that. This is far beyond my knowledge and should be addressed by someone competent.

curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 0);
curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 0);

Our usage of libcurl is currently the blocking synchronous version. We should use the non-blocking asynchronous version.

resubmit PR with changes for 1 build system on Android

500Mb allocated while loading Sponza. It looks like a lambda capture. Any idea @bghgary ?

Override BGFX's default view order to ensure rendering happens in the order expected by the JavaScript engine.

Looks like something became incompatible as a result of the Promise work done for the OpenXR integration.

Replace winrt::Windows::Web::Http::HttpStatusCode in XMLHttpRequest.h with something crossplatofmr (custom enum?)

In many places where we create JavaScript types using NAPI, we store off a copy of the constructor as a private static Napi::FunctionReference within the C++ type. This will probably not work if/once we have multiple simultaneous runtimes as the constructors for the second runtime will replace those of the first. Moreover, this is not probably not necessary, since the constructors should be owned by the JavaScript global context (at least in the majority of cases) and probably shouldn't be kept by native at all.

CMake 3min40 to generate solution per target on Azure with at least 2min30 taken only by libcurl
There must be some flags to change to speed up the solution generation time.

Work will require the XrPlugin to have access to the task chain, so should be undertaken after the work to switch NativeEngine to us runtime global state stored with NAPI so that XrPlugin can take advantage of the same mechanism.

https://github.com/BabylonJS/Babylon.js/blob/78e72971ce16c387c7c3cf02dd318f5b6aa0c939/src/Cameras/XR/webXRSessionManager.ts#L10-L12

https://github.com/BabylonJS/Babylon.js/blob/78e72971ce16c387c7c3cf02dd318f5b6aa0c939/src/Cameras/XR/webXRSessionManager.ts#L167-L169

https://github.com/BabylonJS/BabylonNative/blob/master/Library/Source/NativeXr.cpp#L971-L978

The order of destruction is not quite working as intended in RuntimeImpl::BaseThreadProcedure right now. As it's currently written, the JavaScript environment (env) is correctly tied to the lifetime of the thread on which it was created, which is strictly less than the lifespan of the RuntimeImpl itself. However, the dispatcher and task chain which represent the thread are actually members of the RuntimeImpl; therefore, they outlive the thread and are destroyed after the the thread has terminated; in other words, they strictly outlive env. The problem with this is that the capturing lambdas we use inside the task system end up holding onto unsafe references (i.e., Napi::FunctionReference) to objects that are owned by env. These objects are of course destroyed when the env dies, and the dispatcher and task chain's attempts to destroy them again therefore cause the program to crash on close.

I can currently think of two ways to address this. Option 1 is to manually clear the task chain and dispatcher in the shutdown phase of the thread, thereby causing them to be empty before the env is destroyed (this will require changes to Arcana). Option 2 is to make the dispatcher and task chain not be members of the RuntimeImpl, but instead make them beholden to the thread in the same way env is, perhaps by encapsulating all this thread state in an object which can be used to guarantee destruction order. Option 2 is probably the more correct of these alternatives; it'll just require some minor rearchitecting of the RuntimeImpl to achieve the desired result without losing the ability to enqueue work before the thread has actually started.

In a number of places, NativeEngine.cpp is using bgfx::copy to create locally owned copies of data from Babylon.js (this line, for example). This is probably redundant, as Babylon.js is already keeping copies to deal with device loss and other problems, so it should be possible to have the NativeEngine operate on refs alone so long as it's given access to the permanently-stored versions of data on the JavaScript side, not temporary copies. Fixing will require changes on both managed and native sides.

As a starting point, we want to try to separate out the Console callbacks in the runtime constructor and the NAPI object wrap of the Console into its own "module" and the TestApp is responsible for adding it if needed.

Future runtime objects will all be done using this kind of pattern.

V8 is not part of the repo. We will use Nugget or another package mgmt. to download it during build.
V8 .so must be placed in TestApp\Source\Android\app\src\main\jniLibs${ABI}
It's used to link against the Android app and copied by the gradle build system into the apk for dynamic JNI linking. .so for arm64, arm32, x86, x86-64 must be present when releasing a public apk.

Headers include is set in CMakeLists.txt external build of the Android TestApp.

Take care of commented deletion
https://github.com/BabylonJS/BabylonNative/pull/18/files#diff-0ac7323dc9a9ce79677fe03a04509a80L53
As GetArrayBufferAllocator method doesn't seem to be present

The purple color visible in this screenshot is the default color we provide to BGFX, but Babylon.js should be overriding this color with the Playground's blue. This is indicative that the clear state may not be being set properly for the default view.

Here are the issues so far I've found while trying to compile for Android:

• Exception with string and missing noexcept keyword in Arcana. Will do a PR for that
• libcurl Win32 headers conflict. I had to hack it with a 'set(CMAKE_USE_WINSSL 0)' in the cmakelists. There must be a better fix/config
• bgfx.cmake just a couple lines for Android. There might be more than that to do. Will do a PR
• V8. Unable to compile for Android arm/arm64. I tried various tags, various toolchains, manually tried to create the .a. there is always something failing. Compiles and link with the nugget package!
• Clang templates shitstorm. I commented the lines to get something compiled. No fix yet. I will try to make a repro on Windows with clang so anyone can try to find a solution.

Avoid switching to xcode 11 if possible and find native functions. like NSBundle

• Find a way to rename uniform buffers in the compiler like what shaderc does. Instead of string search and replace

Screenshot:

Happens about 1/5 times I run the app

Some problems I encountered:

• shader uniforms inside a struct named 'Frame' when linking shader, the size of that struct differs between VS and FS resulting in a non linking shader. I did a string replace for now to remove it as bgfx doesn't support uniform struct
• bgfx only supports mat4x4 and vec4 uniform I get a driver error when trying to set a vec4 on a vec3 uniform
• for now, I tweak the shader string instead of passing in glslang. It might not be useful to pass the shader with glslang on Android. I see multiple benefits: smaller binary, faster shader loading. Might evaluate that in the future.

Done for WIN32
missing for

• UWP
• Android
• iOS
• MacOS

Just a reminder here
microsoft/arcana.cpp#4

JSC needs more work:

• handle dispose & destructor callback
• find a way to link externals and c++ object on the JSC side, without another map
• private object on class instead of std::map
• test with gltf loading
• reference counting
• define_class is the Hotpoint for all the method and properties definition.

Right now, the only known missing feature for this is the ISwapChainPanel constructor path for RuntimeUWP. TODO and implementation starting point here:

See BabylonJS/Babylon.js@baedfc7 for the stop gap solution.

Ideally, this should be done in the native layer so that we don't keep crossing the JS to native boundary.

For UWP, we have <rescap:Capability Name="broadFileSystemAccess" /> set in the manifest and have enabled in the settings to allow the app.

libcurl eventually calls the C open API (which eventually calls CreateFileW), but this returns -1.

We currently have a fallback mechanism to use Windows::Storage::StorageFile::GetFileFromPathAsync instead, but it would be nice to see if we can get libcurl to work with broadFileSystemAccess.