mkkellogg / gaussiansplats3d Goto Github PK
View Code? Open in Web Editor NEWThree.js-based implementation of 3D Gaussian splatting
License: MIT License
Three.js-based implementation of 3D Gaussian splatting
License: MIT License
npm run build
[email protected] build
npm run build-library && npm run build-demo
[email protected] build-library
npx rollup -c && mkdir -p ./build/demo/lib && cp ./build/gaussian-splat-3d* ./build/demo/lib/
./src/index.js → ./build/gaussian-splat-3d.umd.cjs...
(!) Broken sourcemap
https://rollupjs.org/troubleshooting/#warning-sourcemap-is-likely-to-be-incorrect
Plugins that transform code (such as "base64") should generate accompanying sourcemaps.
created ./build/gaussian-splat-3d.umd.cjs in 131ms
./src/index.js → ./build/gaussian-splat-3d.module.js...
(!) Broken sourcemap
https://rollupjs.org/troubleshooting/#warning-sourcemap-is-likely-to-be-incorrect
Plugins that transform code (such as "base64") should generate accompanying sourcemaps.
created ./build/gaussian-splat-3d.module.js in 70ms
命令语法不正确。
When I want to test the demos on my local clone, after having run npm run build
there is no folder ./build/demo/assets/data/stump/[*.splat]
. Therefore running npm run demo
will eventually not open any demo samples.
Not sure if I am doing something wrong, but if not, would there be any chance to include the demo splats in the repo?
I setup an example here:
https://stackblitz.com/edit/github-ugddhz?file=index.html,src%2Fapp.js
The file is loading, the CORS are working, too. But nevertheless I do not see my file. It is a ply file from poly.cam that I converted with your converter to make it a splat file. But other files are there, too (also in the "local" public folder), and they won't work, either.
Am I doing something wrong?
And is there another place to ask question or can I ask here?
PS: I should add that the converted splat file is displayed correctly on the website demo page, where it is possible to upload a file...
Thanks for the great development.
When I create an environment with docker container in wsl on windows, if the port number mapped in docker-compose.yml and the port number described in server.js are the same, the server cannot start and does not output any errors. I imagine this is related to something in this library. Do you have any idea what it could be?
For example, in the following example, node server.js
will not start, and i must either change the mapping to 8080:8081 or devise a way to change the port specification in server.js to 8082.
version: '3' services: app: container_name: app build: context: . dockerfile: Dockerfile restart: always tty: true volumes: - type: bind source: . target: /app ports: - 8080:8080
If a .ply
file is converted to .ksplat
, can the .ksplat
file be converted back to .ply
without the original .ply
file?
Nicely done ...
Unfortunately, at the moment with the View Your Own Scene on the demo page, it's very unintuitive as to how to set the Camera orientation parameters to get an uploaded splat view to change with the controls as expected - I tried half a dozen times, and always failed. Would it be possible to make this clearer, or easier to use?
Is there any other way to run without setting CORS,Thank you
Super nice work, it renders super smooth on my Macbook pro :)
I "optimized "the ply to splat with your function but it still doens't render on my android phone (not super fast but not slow Pixel 7)
Any ideas of optimization? I tested the 3 examples too, it doesn"t work.
Thanks!
Hey great work here so far @mkkellogg!
I'd love to help out where we can on a WebXR viewer for Splats.
We have our own internal renderer right now running on WebXR but it's closed source. Ideally we could support a open source renderer in a popular framework like Three.js to make them more accessible.
We have optimized splats to make them really small while retaining most of the quality. We have the splat here in the repo running at 90FPS on Quest 3 in our WebXR renderer. Hope this can be useful for testing out in the headset since most of the data out there is too large to be useful.
https://github.com/gmix-tech/small_splats
Let me know how I can help out.
Feel free to ping me at [email protected]
When using an external scene, the SplatMesh object does not react in any way to changes in the camera's Up vector. All examples in the description use the negative direction of the camera Y axis, but in ThreeJS the positive Y direction is the default. Can you tell me where to change your code in order to either implement a dependence on the Up vector of the camera or at least change the direction of the SplatMesh rendering?
Hi. Can you tell me what sorter.wasm file is for and how to compile it? I'm trying to build this package in my Angular project. When trying to create a build, the following errors appear:
./src/app/services/engine/GaussianSplatsViewer/worker/SortWorker.js:96:38-48 - Error: export 'default' (imported as 'SorterWasm') was not found in './sorter.wasm' (possible exports: __wasm_call_ctors, emscripten_tls_init, sortIndexes)
./src/app/services/engine/GaussianSplatsViewer/worker/sorter.wasm - Error: Module not found: Error: Can't resolve 'env' in 'E:\Projects\artwork-threed-visualizer\VoyagerFrontend\src\app\services\engine\GaussianSplatsViewer\worker'
I see the compile_wasm.sh script, I ran it using PowerShell but no new file was generated. I also don’t see anything in the package.json file that relates to wasm. Can you tell me what these files are needed for and how to configure the build correctly?
Or is sorter.wasm just a binary file that needs to be imported into a JS file as a string?
EDIT:
It doesn't matter anymore. I used fetch to download this file. And to fix the error with self.crossOriginIsolated, I set some headers in the angular.json file. Maybe this will help someone in the future.
Viewer.js
fetch('assets/libs/sorter.wasm').then((response) => response.arrayBuffer())
.then((sortWorkerBuffer) => {
const base64String = this.arrayBufferToBase64(sortWorkerBuffer);
...
});
angular.json
"serve": {
"builder": "@angular-builders/custom-webpack:dev-server",
"options": {
"browserTarget": "artwork-threed-visualizer:build",
"headers": {
"Cross-Origin-Opener-Policy": "same-origin",
"Cross-Origin-Embedder-Policy": "credentialless"
}
},
Setting camera UP vector on file loading works great for orienting my model. But I would love to use the orientation feature instead so the model can exist in default world space. However, it does not appear to work how I'd expect. I would assume it would yield the same visual effect as changing the camera up i.e. the camera orbits the camera on the correct plane.
Can you confirm the expected behavior/functionality and usage?
const up = [0.1294521689414978, -0.4815174639225006, -0.8668235540390015]
const viewer = new Viewer({
rootElement: containerRef.current,
//cameraUp: up,
initialCameraPosition: [-1, 4, 6],
initialCameraLookAt: [0, 0, 0],
});
const orientation = new THREE.Quaternion();
orientation.setFromUnitVectors(
new THREE.Vector3(0, 1, 0),
new THREE.Vector3(up[0], up[1], up[2]).normalize()
);
viewer.init();
viewer.loadFile(file, {
splatAlphaRemovalThreshold: 5,
halfPrecisionCovariancesOnGPU: true,
position: [0, 0, 0],
orientation: orientation.toArray(),
})
.then(() => {
console.log('File loaded');
viewer.start();
});
I dug through the code to look for potential bugs, but came up empty handed. I lose track of the splatMesh.quaternion but my guess is because its a member of a threejs parent class.
Thanks!
Dear author,
I'm a new user for java. Could you please tell me where to put the code you provide to convert .ply file to .splat file? Thank you.
Best regards,
hhc
I tried to rotate 180° along the Z axis in the model direction.
I tried to rotate using addSplatScene with params.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<title>3D Gaussian Splat Demo - Garden</title>
<script type="text/javascript" src="js/util.js"></script>
<script type="importmap">
{
"imports": {
"three": "./lib/three.module.js",
"@mkkellogg/gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
}
}
</script>
<style>
body {
background-color: #000000;
height: 100vh;
margin: 0px;
}
</style>
</head>
<body>
<script type="module">
import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
import * as THREE from 'three';
const quaternion = new THREE.Quaternion();
const euler = new THREE.Euler( 0.0, 0.0, Math.PI);
quaternion.setFromEuler(euler);
const viewer = new GaussianSplats3D.Viewer({
"dynamicScene": false,
});
let path = 'assets/data/garden/garden';
path += isMobile() ? '.ksplat' : '_high.ksplat';
viewer.addSplatScene(path,{
"rotation": quaternion.toArray()
})
.then(() => {
viewer.start();
});
</script>
</body>
</html>
I got the result as shown in the following image.
Next, I tried to rotate using dynamicScene.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta http-equiv="x-ua-compatible" content="ie=edge">
<title>3D Gaussian Splat Demo - Garden</title>
<script type="text/javascript" src="js/util.js"></script>
<script type="importmap">
{
"imports": {
"three": "./lib/three.module.js",
"@mkkellogg/gaussian-splats-3d": "./lib/gaussian-splats-3d.module.js"
}
}
</script>
<style>
body {
background-color: #000000;
height: 100vh;
margin: 0px;
}
</style>
</head>
<body>
<script type="module">
import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d';
import * as THREE from 'three';
const quaternion = new THREE.Quaternion();
const euler = new THREE.Euler( 0.0, 0.0, Math.PI);
quaternion.setFromEuler(euler);
const viewer = new GaussianSplats3D.Viewer({
"dynamicScene": true,
});
let path = 'assets/data/garden/garden';
path += isMobile() ? '.ksplat' : '_high.ksplat';
viewer.addSplatScene(path)
.then(() => {
const splatScene = viewer.getSplatScene(0);
splatScene.quaternion.copy(quaternion);
viewer.start();
});
</script>
</body>
</html>
I got the result as shown in the following image.
I wanted to reproduce the result of dynamicScene with addSplatScene.
How can I reproduce this using addSplatScene? Also, is this behavior the correct behavior for the inconsistency?
In case somebody uses Vite: Vite seems to require some special CORS configuration to be able to read SharedBufferArray. A solution would be this:
chaosprint/vite-plugin-cross-origin-isolation#3
or this vite plugin:
https://github.com/chaosprint/vite-plugin-cross-origin-isolation
Your work is a good one. What I would like to ask is Deploying this project to an accessible web page is accessible on the computer side, but not on the mobile (iPhone is ok) side, and it shows the problem of loading all the time.
Do you know where you need to change your source code?
best regards,
npm run build
[email protected] build
npm run build-library && npm run build-demo
[email protected] build-library
npx rollup -c && mkdir -p ./build/demo/lib && cp ./build/gaussian-splat-3d* ./build/demo/lib/
./src/index.js → ./build/gaussian-splat-3d.umd.cjs...
created ./build/gaussian-splat-3d.umd.cjs in 124ms
./src/index.js → ./build/gaussian-splat-3d.module.js...
created ./build/gaussian-splat-3d.module.js in 64ms
命令语法不正确。
would be great to PR some of those classes to mrdoob/three.js?
I have tried the online demo and also set up a website for viewing and configured the proper CORS settings in Nginx, as well as implemented HTTPS. The site is functioning fast and well on Windows browsers and iOS. However, it is not working on the default browser (webview) in Android. I checked the 'crossOriginIsolated' variable using dev tools and found that it is always false, regardless of the CORS settings I apply. Interestingly, the site works well on the Firefox mobile browser on the same device.
Upon consulting MDN, I discovered that while the 'crossOriginIsolated' variable is available in webview, the term 'SharedArrayBuffer' is never supported in webview. Additionally, MDN indicated that Chrome on Android supports SharedArrayBuffer, but it still failed to run on Chrome for Android.
Do you have any ideas on how to resolve this issue and get the website running on Android browsers? A fix would be great help and enable a broader range of users. Thanks.
Hi is it possible to get hot-reload working with it?
i am using bun run build
and bun run demo
but a hot reload will be awesome to have
Hello! I'm the author of gsplat.js, in which I'm using the splat format as provided in antimatter15/splat
I have opened an issue on splat compression, and I think it would be great if we can have a universal representation, with a consistent header to support different compression methods.
I can replicate your compressed format, but maybe we can open a common format repo with test files, so we can stay on the same page?
What do you think?
Thanks for your great work !
I have a program
I use the nginx proxy pass port, and add the following configuraions:
add_header 'Cross-Origin-Embedder-Policy' 'require-corp';
add_header 'Cross-Origin-Opener-Policy' 'same-origin';
But appear:
DOMException: Failed to execute 'postMessage' on 'DedicatedWorkerGlobalScope': SharedArrayBuffer transfer requires self.crossOriginIsolated .
How can i resolve this program~
Thanks
Hi,
I was wondering why the file size of .splat files are so high (100+mb) compared to Kevin's .splat files(25mb).
Skimming through the codebase, this implementation also discards the spherical harmonics coefficient. Is there any specific reason for the larger file size ?
Thank you ! :)
There is a 2GB size limitation for JavaScript strings. Will there be any support for handling large files in JavaScript, such as file partitioning or other techniques?
Thanks!
After:
npm run demo
I get:
Server running at 127.0.0.1:8080
HTTP(500)) Request for ./build/demo/ -> Server error.
HTTP(404) Request for ./build/demo/favicon.ico -> File not found.
When I navigate to local host URL it says this:
Sorry, check with the site admin for error: EISDIR ..
When I used SERVEZ running locally I get the index page to load but the links to the three example pages don't load splats.
I have demo data installed here:
GaussianSplats3D-main\build\demo\assets\data
\garden
\stump
\truck
All three directories have associated splat files.
Any ideas what I might be doing wrong?
very useful to map this to unity, unreal type controls with joystick.
I have added camera viewpoint movement in the scene, although it may not be suitable for every model. Perhaps if I can obtain the camera poses from the trained model, it would be possible to automatically handle the camera motion trajectory. What do you think?
Hello. I was looking to work on Gaussian Splat with Three.js and found this GitHub repository.
However, I immediately ran into a major obstacle.
Even after installing and uploading to the server, and testing with the codes provided in the sample, I can't see the ksplat on my server. The splat files we worked on also don't upload.
In the sample page console, there are a few more Views.js files that appear, but
Views.js doesn't show up in our console.
What could be the problem?
` import * as THREE from 'three';
import * as GaussianSplats3D from 'gaussian-splats-3d';
const viewer = new GaussianSplats3D.Viewer({
'cameraUp': [0, -1, -0.54],
'initialCameraPosition': [-3.15634, -0.16946, -0.51552],
'initialCameraLookAt': [1.52976, 2.27776, 1.65898]
});
viewer.init();
let path = './shoose.splat';
viewer.loadFile(path, {
'halfPrecisionCovariancesOnGPU': true
})
.then(() => {
viewer.start();
});`
There is an issue that has been bothering me, which is how to select coordinate points in the Splat scene through mouse clicks in order to conveniently add regular meshes to the scene. However, when I tried to implement it using the raycaster.intersectObjects() method, it didn’t work.
Hi @mkkellogg !
I'm experiecing some strange behaviour and i was wondering if you could help me/had an idea of what could be happening.
I have a ksplat that i'm trying to playback but depending on the tilt of the camera a lot of the splats disappear.
This doesn't happen when using the SIBR viewer (or antimatter15 implementation for what i can tell, but i might be wrong, as i just used the online viewer and is not too user friendly).
For what is worth i think it's to do with the sorting: i removed the toHalfFloat
for the covariance and i could see all the points without the splats, and they were still disappering. Then again, i'm not sure why just sorting the splats would make them disappear, so maybe it's something else.
I attacched a test i've done:
converted_file.zip
If you try with:
'cameraUp': [0, 0, -1],
'initialCameraPosition': [9993.60576, -277.27384, -435.28904],
'initialCameraLookAt': [9293.52797, 251.08879, 28.23262],
and slightly tilt up/down you should see them disappearing
Note on the file: when i was playing around with the bucketSize
and having huge files back, i did something to cull out the model given a point and a radius.
What i've attacched is just a culled version fo the entire model, i was trying to check if there was something "in front" that would occlude the rest, i can assure you this happens on the full model as well, this is just simplier to attach 😄
Hi @mkkellogg !
I was wondering what was your opinion on adding another parameter that can be passed to the viewer and that would tide together MaximumDistanceToSort/MaximumDistanceToRender/SplatBufferBucketBlockSize and the Camera.far?
Is there any drawback?
Did i miss some other variable?
When using the mouse wheel to zoom in on the scene, some objects disappear partially. I have tried setting the camera's far clipping plane to 2000, but it doesn't seem to have any effect.
Hey,
I'd like to hear your stance on this topic.
Personally I think it's a great developer tool and a great aid when exposing the API to the end user.
I'd happily convert the existing code if you'd like me to.
In a context of SPA web apps we need a method of the Viewer to release hogged up resources.
The issue is: If the viewer is integrated into a SPA framework (Blazor in my case), upon navigating to a "page" with no viewer and then back again to the viewer, there will be a significant performance and stuttering issues up to a point of unusability. Resources from each load/visit of the viewer are still hogging clients memory. This is because resources aren't freed since a navigation in a SPA means just an in-place html replace and NOT a full page reload. I checked memory usage upon each viewer initialization, and it keeps adding up but not freeing the memory since SPA frameworks dont know how to do that with .js resources. SPAs only handle their component disposal which has nothing to do with resources that are used up by .js libraries. This will eventually result in memory leak.
Usually .js libraries offer some kind of a destroy or dispose method. Is there a way for the viewer to do the same? If not I would grately appreciate some sort of an implementation in the future release.
Some splat imported from other compressors have different compression levels.
namely splats imported from playcanvas compressor have 190 as compression level and splats from polycam website have 97 as level and in SplatBuffer.js at line 58 it throws an error as it's not able to load these files.
is there any way to fix this behaviour?
Hello, I am not very familiar with Three.js but have worked with Webgl and was wondering where in the code I might be able to offset the origin to draw the splat in a different location. I presumed it should be in the SplatBuffer where I have tried modifying the position functions to shift the splat. But all I have achieved is some of the guassians being shifted and not all. So I was wondering if you would know off the top of your head where this would take place. I also tried adding a translation matrix buildPreComputedBuffers() and that didn't seem to work. If you could point me in a direction where this would be achievable in your code that would me much appreciated
Hi, im wondering if theres a way in the code to set the internal render resolution, so maintain the same size viewer but for example set the model itself to be at 0.5* the resolution?
Latest changes possibly caused a problem - will happily process a file from local system; but only shows a black screen for any of the demo items.
Any thoughts on how lighting sim could be done in the future strictly inside three js?
Thx for your great work !
I have run successfully locally,
How could i do to run it online?
thx for your reply!
This library is very useful and perfectly meets my needs. I would like to implement a loading animation similar to LumaAI based on this. It seems that they may also control the 'splatAlphaRemovalThreshold' parameter to achieve this, as seen here: https://lumalabs.ai/capture/443a844e-525f-41e1-9750-1dff62782466. I roughly examined the code and it seems that the 'splatAlphaRemovalThreshold' parameter is already passed to the 'loadFileToSplatBuffer' function before rendering, and afterwards, it seems that the parameter cannot be modified to adjust the rendering behavior. Therefore, I would like to know if I have missed anything or if there is any way to achieve this, thanks!
I've just written my own renderer:
https://github.com/fynv/web_gaussian_splats
, which is heavily based on the great work here. Thanks very mcuh!
One thing I noticed is that if I don't remove the splats with alpha == 0,
there can be some obvious performance defficiency (fragment shader bound).
The indication is that there are a signicant number of splats with alpha ==0.
The culling is currently done when building the "splat-tree" structure, where "splatAlphaRemovalThreshold" is defaulted to 1.
It seems to me that for alpha == 0, it doesn't even make sense to store these splats in the .splat data in the first place.
Can we just drop these splats during data conversion?
My current workflow is:
Super Splash
.splat
with GaussianSplats3D.splat
The issue I encounter is that the orientation of a splat is obviously treated very differently among software. I am aware of the Y/Z-axis, but in this case the splat is not 90 degrees off (which I would expect, when y and z axis were treated differently), but the splat is completely turned around.
The screenshot shows what I have in Super Splash
and the video here:
https://dl.rowild.at/GaussianSplats3D_upside_down.mp4
shows what GaussianSplats3D
is doing with the (converted from .ply
) .splat
.
I am aware that the distance of the camera has to be adjusted in most cases. (Of course I would love to know if that is the case, because the .splat
is scaled differently or is there really just a difference ein placing the camera.)
But, as mentioned above: the splat is turned upside-down in GaussianSplats3D
– and that does not really seem to be the case, because the z-axis is defined differently.
Not sure, if my thinking is totally wrong and splats work completely different from meshes, or if there is something else off...
I know my way around three.js so-lala and would like to think that, along with the documentation, I can re-define the OrbitControls and the camera so that my splat is handled properly. But it seems that currently these adjustments have to be done on a splat-to-splat basis.
The compression effect of .ksplat
model on .ply
is great! I wonder if it is possible to display .ksplat
models in Cesium?
For example, displaying a .ksplat
model of a building on Cesium using a Cesium plugin, and ensuring compatibility with Cesium's browsing features like zooming, panning, and rotating...
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.