lzy
/
threepipe
Tarkkaile 1
Äänestä 0
Fork 0
Koodi Ongelmat 0 Pull-pyynnöt 0 Julkaisut 0 Wiki Activity
Selaa lähdekoodia

Add gaussian splatting plugin

master
Palash Bansal 2 vuotta sitten
vanhempi
bee1d696ec
commit
0bcc426d7d
No account linked to committer's email address
39 muutettua tiedostoa jossa 1526 lisäystä ja 1 poistoa
Jaettu näkymä Näytä diff tilastot
  1. 42
    0
      README.md
  2. 1
    0
      examples/index.html
  3. 54
    0
      examples/splat-load/index.html
  4. 2
    1
      examples/tweakpane-editor/index.html
  5. 2
    0
      examples/tweakpane-editor/script.ts
  6. 64
    0
      plugins/gaussian-splatting/package-lock.json
  7. 60
    0
      plugins/gaussian-splatting/package.json
  8. 50
    0
      plugins/gaussian-splatting/src/gaussian-splats-3d/GaussianSplatsPlugin.ts
  9. 40
    0
      plugins/gaussian-splatting/src/global.d.ts
  10. 5
    0
      plugins/gaussian-splatting/src/index.ts
  11. 3
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/ThreeGaussianSplatPlugin.css
  12. 74
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/ThreeGaussianSplatPlugin.ts
  13. 30
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/BufferPool.ts
  14. 3
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/ISort.d.ts
  15. 53
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/SortWorkerManager.ts
  16. 4
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/build-in-docker.sh
  17. 7
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/build.sh
  18. 107
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.cpp
  19. 16
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.js
  20. BIN
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.wasm
  21. 9
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/watch.sh
  22. 59
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/worker.ts
  23. 182
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/geometry/GaussianSplatGeometry.ts
  24. 22
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/index.ts
  25. 51
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/loaders/SplatLoader.ts
  26. 105
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialExtension.ts
  27. 25
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialPhysical.ts
  28. 73
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialRaw.ts
  29. 26
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialUnlit.ts
  30. 9
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/materials/util.ts
  31. 96
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/mesh/GaussianSplatMesh.ts
  32. 18
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.main.frag.glsl
  33. 15
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.main.vert.glsl
  34. 10
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.pars.frag.glsl
  35. 58
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.pars.vert.glsl
  36. 11
    0
      plugins/gaussian-splatting/src/three-gaussian-splat/shaders/index.ts
  37. 41
    0
      plugins/gaussian-splatting/tsconfig.json
  38. 10
    0
      plugins/gaussian-splatting/typedoc.json
  39. 89
    0
      plugins/gaussian-splatting/vite.config.js

+ 42
- 0
README.md Näytä tiedosto

@@ -127,6 +127,7 @@ To make changes and run the example, click on the CodePen button on the top righ
- [@threepipe/plugins-extra-importers](#threepipeplugins-extra-importers) - Plugin for loading more file types supported by loaders in three.js
- [@threepipe/plugin-blend-importer](#threepipeplugin-blend-importer) - Blender to add support for loading .blend file
- [@threepipe/plugin-geometry-generator](#threepipeplugin-geometry-generator) - Generate parametric geometry types that can be re-generated from UI/API.
- [@threepipe/plugin-gaussian-splatting](#threepipeplugin-gaussian-splatting) - Gaussian Splatting plugin for loading and rendering splat files

## Getting Started

@@ -3310,3 +3311,44 @@ generator.generators.custom = new CustomGenerator('custom') // Extend from AGeom
generator.uiConfig.uiRefresh?.()
```

## @threepipe/plugin-gaussian-splatting

Exports [GaussianSplattingPlugin](https://threepipe.org/plugins/gaussian-splatting/docs/classes/GaussianSplattingPlugin.html) which adds support for loading .blend files.

It uses [`three-gaussian-splat`](./plugins/gaussian-splatting/src/three-gaussian-splat), a rewrite of [@zappar/three-guassian-splat](https://github.com/zappar-xr/three-gaussian-splat) (and [gsplat.js](https://github.com/huggingface/gsplat.js) and [antimatter15/splat](https://github.com/antimatter15/splat)) for loading splat files and rendering gaussian splats.

[Example](https://threepipe.org/examples/#splat-load/) —
[Source Code](./plugins/gaussian-splatting/src/index.ts) —
[API Reference](https://threepipe.org/plugins/gaussian-splatting/docs)

NPM: `npm install @threepipe/plugin-gaussian-splatting`

Note: This is still a WIP.

Currently working:
* Importing .splat files (just array buffer of gaussian splat attributes)
* ThreeGaussianSplatPlugin (Same as GaussianSplattingPlugin), add importer and update events to the viewer
* GaussianSplatMaterialExtension for adding gaussian splat functionality to any material like Unlit, Physical
* GaussianSplatMesh a subclass of Mesh2 for holding the gaussian splat geometry and a material with gaussian splat extension. also handles basic raycast in the splat geometry. (assuming simple points)
* GaussianSplatGeometry holds the geometry data and and the sort worker. Computes correct bounding box and sphere.
* SplatLoader for loading splat files and creating the geometry and material.
* GaussianSplatMaterialUnlit, GaussianSplatMaterialRaw
* GaussianSplatMaterialPhysical, working but normals are hardcoded to 0,1,0

TBD:
* Exporting/embedding splat files into glb
* Rendering to depth/gbuffer
* Estimate normals/read from file
* Lighting in GaussianSplatMaterialPhysical

```typescript
import {ThreeViewer} from 'threepipe'
import {GaussianSplattingPlugin} from '@threepipe/plugin-gaussian-splatting'

const viewer = new ThreeViewer({...})
viewer.addPluginSync(GaussianSplattingPlugin)

// Now load any .splat file.
const model = await viewer.load<GaussianSplatMesh>('path/to/file.splat')

```

+ 1
- 0
examples/index.html Näytä tiedosto

@@ -260,6 +260,7 @@
<li><a href="./ktx2-load/">KTX2 Load </a></li>
<li><a href="./ktx-load/">KTX Load </a></li>
<li><a href="./blend-load/">BLEND Load </a></li>
<li><a href="./splat-load/">SPLAT Load<br/>(Gaussian Splatting) </a></li>
<li><a href="./extra-importer-plugins/">Extra(3ds, 3mf, collada, amf, bvh, vox, gcode, mdd, pcd, tilt, wrl, ldraw, vtk, xyz) Load </a></li>
</ul>
<h2 class="category">Export</h2>

+ 54
- 0
examples/splat-load/index.html Näytä tiedosto

@@ -0,0 +1,54 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Blend Load</title>
<meta name="viewport" content="width=device-width, initial-scale=1">
<!-- Import maps polyfill -->
<!-- Remove this when import maps will be widely supported -->
<script async src="https://unpkg.com/es-module-shims@1.6.3/dist/es-module-shims.js"></script>

<script type="importmap">
{
"imports": {
"threepipe": "./../../dist/index.mjs",
"@threepipe/plugin-gaussian-splatting": "./../../plugins/gaussian-splatting/dist/index.mjs"
}
}

</script>
<style id="example-style">
html, body, #canvas-container, #mcanvas {
width: 100%;
height: 100%;
margin: 0;
overflow: hidden;
}
</style>
<script type="module" src="../examples-utils/simple-code-preview.mjs"></script>
<script id="example-script" type="module">
import {_testFinish, ThreeViewer} from 'threepipe'
import {GaussianSplattingPlugin} from '@threepipe/plugin-gaussian-splatting'

const viewer = new ThreeViewer({canvas: document.getElementById('mcanvas')})
viewer.addPluginsSync([GaussianSplattingPlugin])

async function init() {

await viewer.setEnvironmentMap('https://threejs.org/examples/textures/equirectangular/venice_sunset_1k.hdr')
const result = await viewer.load('https://asset-samples.threepipe.org/splat/bonsai.splat', {
autoCenter: true,
autoScale: true,
})
console.log(result)
}

init().then(_testFinish)
</script>
</head>
<body>
<div id="canvas-container">
<canvas id="mcanvas"></canvas>
</div>

</body>

+ 2
- 1
examples/tweakpane-editor/index.html Näytä tiedosto

@@ -21,7 +21,8 @@
"@threepipe/plugin-tweakpane-editor": "./../../plugins/tweakpane-editor/dist/index.mjs",
"@threepipe/plugin-extra-importers": "./../../plugins/extra-importers/dist/index.mjs",
"@threepipe/plugin-blend-importer": "./../../plugins/blend-importer/dist/index.mjs",
"@threepipe/plugin-geometry-generator": "./../../plugins/geometry-generator/dist/index.mjs"
"@threepipe/plugin-geometry-generator": "./../../plugins/geometry-generator/dist/index.mjs",
"@threepipe/plugin-gaussian-splatting": "./../../plugins/gaussian-splatting/dist/index.mjs"
}
}


+ 2
- 0
examples/tweakpane-editor/script.ts Näytä tiedosto

@@ -42,6 +42,7 @@ import {HierarchyUiPlugin, TweakpaneEditorPlugin} from '@threepipe/plugin-tweakp
import {BlendLoadPlugin} from '@threepipe/plugin-blend-importer'
import {extraImportPlugins} from '@threepipe/plugin-extra-importers'
import {GeometryGeneratorPlugin} from '@threepipe/plugin-geometry-generator'
import {GaussianSplattingPlugin} from '@threepipe/plugin-gaussian-splatting'

async function init() {

@@ -95,6 +96,7 @@ async function init() {
GeometryGeneratorPlugin,
Object3DWidgetsPlugin,
Object3DGeneratorPlugin,
GaussianSplattingPlugin,
...extraImportPlugins,
])


+ 64
- 0
plugins/gaussian-splatting/package-lock.json Näytä tiedosto

@@ -0,0 +1,64 @@
{
"name": "@threepipe/plugin-gaussian-splatting",
"version": "0.1.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@threepipe/plugin-gaussian-splatting",
"version": "0.1.0",
"license": "Apache-2.0",
"dependencies": {
"threepipe": "file:./../../src/"
},
"devDependencies": {
"@types/emscripten": "^1.39.10",
"comlink": "^4.4.1"
}
},
"../../../three-gaussian-splat": {
"name": "@zappar/three-gaussian-splat",
"version": "0.0.1",
"extraneous": true,
"license": "MIT",
"dependencies": {
"@types/emscripten": "^1.39.10",
"comlink": "4.4.1"
},
"devDependencies": {
"@types/node": "^14.11.2",
"@types/three": "0.150.1",
"gts": "^3.1.1",
"lil-gui": "^0.18.2",
"parcel": "^2.10.3",
"path-browserify": "^1.0.1",
"process": "^0.11.10",
"three": "^0.150.1",
"typescript": "^4.9.4"
},
"engines": {
"node": ">=18.0.0"
},
"peerDependencies": {
"three": "*"
}
},
"../../src": {},
"node_modules/@types/emscripten": {
"version": "1.39.10",
"resolved": "https://registry.npmjs.org/@types/emscripten/-/emscripten-1.39.10.tgz",
"integrity": "sha512-TB/6hBkYQJxsZHSqyeuO1Jt0AB/bW6G7rHt9g7lML7SOF6lbgcHvw/Lr+69iqN0qxgXLhWKScAon73JNnptuDw==",
"dev": true
},
"node_modules/comlink": {
"version": "4.4.1",
"resolved": "https://registry.npmjs.org/comlink/-/comlink-4.4.1.tgz",
"integrity": "sha512-+1dlx0aY5Jo1vHy/tSsIGpSkN4tS9rZSW8FIhG0JH/crs9wwweswIo/POr451r7bZww3hFbPAKnTpimzL/mm4Q==",
"dev": true
},
"node_modules/threepipe": {
"resolved": "../../src",
"link": true
}
}
}

+ 60
- 0
plugins/gaussian-splatting/package.json Näytä tiedosto

@@ -0,0 +1,60 @@
{
"name": "@threepipe/plugin-gaussian-splatting",
"description": "Gaussian Splatting for Threepipe",
"version": "0.1.0",
"devDependencies": {
"comlink": "^4.4.1",
"@types/emscripten": "^1.39.10"
},
"dependencies": {
"threepipe": "file:./../../src/"
},
"clean-package": {
"remove": [
"clean-package",
"scripts",
"devDependencies",
"//",
"markdown-to-html"
],
"replace": {
"dependencies": {
"threepipe": "^0.0.23"
}
}
},
"type": "module",
"main": "dist/index.js",
"module": "dist/index.mjs",
"types": "dist/index.d.ts",
"files": [
"dist",
"src"
],
"scripts": {
"new:pack": "npm run prepare && clean-package && npm pack && clean-package restore",
"new:publish": "npm run prepare && clean-package && npm publish --access public && clean-package restore",
"prepare": "npm run build",
"build": "rimraf dist && vite build",
"dev": "NODE_ENV=development vite build --watch",
"docs": "rimraf docs && npx typedoc"
},
"author": "repalash <palash@shaders.app>",
"license": "Apache-2.0",
"keywords": [
"three",
"three.js",
"threepipe",
"gaussian-splatting",
"ml",
"ai"
],
"bugs": {
"url": "https://github.com/repalash/threepipe/issues"
},
"homepage": "https://github.com/repalash/threepipe#readme",
"repository": {
"type": "git",
"url": "git://github.com/repalash/threepipe.git"
}
}

+ 50
- 0
plugins/gaussian-splatting/src/gaussian-splats-3d/GaussianSplatsPlugin.ts Näytä tiedosto

@@ -0,0 +1,50 @@
// import {AViewerPluginSync, createStyles, IViewerEvent, ThreeViewer} from 'threepipe'
// import styles from './SamplePlugin.css'
// import * as GaussianSplats3D from '@mkkellogg/gaussian-splats-3d'
//
// // import * as GaussianSplats3D from 'gle-gs3d'
//
// export class GaussianSplatsPlugin extends AViewerPluginSync<string> {
// public static readonly PluginType: string = 'GaussianSplatsPlugin'
// enabled = true
// dependencies = []
// toJSON: any = null
//
// constructor() {
// super()
// }
//
// splats: any
// private _ready = false
// onAdded(viewer: ThreeViewer) {
// super.onAdded(viewer)
// createStyles(styles)
// this.splats = new GaussianSplats3D.Viewer({
// 'selfDrivenMode': false,
// 'renderer': viewer.renderManager.webglRenderer,
// 'camera': viewer.scene.mainCamera,
// 'useBuiltInControls': false,
// // 'ignoreDevicePixelRatio': false,
// // 'gpuAcceleratedSort': true,
// // 'halfPrecisionCovariancesOnGPU': true,
// 'sharedMemoryForWorkers': false,
// // 'integerBasedSort': false,
// // 'dynamicScene': false,
// // 'webXRMode': GaussianSplats3D.WebXRMode.None,
// })
// this.splats.init()
// // this.splats.loadFile('https://generic-cors-proxy.repalash.workers.dev/https://zappar-xr.github.io/three-gaussian-splat-example/bonsai.5148b146.splat').then(()=>{
// this.splats.addSplatScene('https://generic-cors-proxy.repalash.workers.dev/https://projects.markkellogg.org/threejs/assets/data/garden/garden_high.ksplat').then(()=>{
// this._ready = true
// })
// }
//
// protected _viewerListeners = {
// postFrame: (_: IViewerEvent) => {
// if (!this._ready) return
// console.log('postframe')
// this.splats.update()
// this.splats.render()
// },
// }
// }

+ 40
- 0
plugins/gaussian-splatting/src/global.d.ts Näytä tiedosto

@@ -0,0 +1,40 @@
declare module '*.txt' {
const content: string
export default content
}
declare module '*.glsl' {
const content: string
export default content
}
declare module '*.vert' {
const content: string
export default content
}
declare module '*.frag' {
const content: string
export default content
}
declare module '*.module.scss' {
const content: any
export default content
export const stylesheet: string
}
declare module '*.module.css' {
const content: any
export default content
export const stylesheet: string
}
declare module '*.css' {
const content: string
export default content
}
declare module '*.css?inline' { // for vite
const content: string
export default content
}

// export {}

// hack for typedoc
// eslint-disable-next-line @typescript-eslint/naming-convention
// declare type OffscreenCanvas = HTMLCanvasElement

+ 5
- 0
plugins/gaussian-splatting/src/index.ts Näytä tiedosto

@@ -0,0 +1,5 @@
import {ThreeGaussianSplatPlugin} from './three-gaussian-splat'

export class GaussianSplattingPlugin extends ThreeGaussianSplatPlugin {}

export * as threeGaussianSplat from './three-gaussian-splat'

+ 3
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/ThreeGaussianSplatPlugin.css Näytä tiedosto

@@ -0,0 +1,3 @@
html{

}

+ 74
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/ThreeGaussianSplatPlugin.ts Näytä tiedosto

@@ -0,0 +1,74 @@
import {AViewerPluginSync, createStyles, IGeometryEvent, ILoader, Importer, ThreeViewer} from 'threepipe'
import styles from './ThreeGaussianSplatPlugin.css?inline'
import {GaussianSplatMesh} from './index'
import {AnyOptions} from 'ts-browser-helpers'
import {SplatLoader} from './loaders/SplatLoader'
import {SortWorkerManager} from './cpp-sorter/SortWorkerManager'
import {GaussianSplatGeometry} from './geometry/GaussianSplatGeometry'

export class ThreeGaussianSplatPlugin extends AViewerPluginSync<string> {
public static readonly PluginType: string = 'ThreeGaussianSplatPlugin'
enabled = true
dependencies = []
toJSON: any = null

constructor() {
super()
}

splats: GaussianSplatMesh[] = []
private _ready = false
onAdded(viewer: ThreeViewer) {
super.onAdded(viewer)
createStyles(styles)
viewer.assetManager.importer.addImporter(this._importer)
viewer.scene.addEventListener('mainCameraUpdate', this._activeCameraUpdate)
viewer.scene.addEventListener('geometryUpdate', this._geometryUpdate)
this._ready = true
}

onRemove(viewer: ThreeViewer) {
viewer.assetManager.importer.removeImporter(this._importer)
}

private _activeCameraUpdate = () => {
if (!this._ready || this.isDisabled()) return
this.splats.forEach(async splat=>splat.update(this._viewer!.scene.mainCamera, this._viewer!.renderManager.webglRenderer))
}

private _geometryUpdate = (event: IGeometryEvent) => {
if (!this._ready || this.isDisabled() || !(event.geometry as GaussianSplatGeometry)?.isGaussianSplatGeometry) return
event.geometry!.appliedMeshes.forEach(async(splat: GaussianSplatMesh)=>splat.update ? splat.update(this._viewer!.scene.mainCamera, this._viewer!.renderManager.webglRenderer) : undefined)
}

private _sortWorkerManager = new SortWorkerManager() // todo: dispose?
protected _importer = new Importer(class extends SplatLoader implements ILoader {
onDispose: (mesh: GaussianSplatMesh)=>void = ()=>{return}
onCreate: (mesh: GaussianSplatMesh)=>void = ()=>{return}
transform(res: GaussianSplatMesh, _: AnyOptions): any {
res.addEventListener('dispose', ()=>this.onDispose(res))
this.onCreate(res)
return res
}
}, ['splat'], [], true, (l)=>{
if (!l) return l
l.sortWorkerManager = this._sortWorkerManager
l.onDispose = (mesh: GaussianSplatMesh)=>{ // todo: dispose should only remove from GPU?
this.splats = this.splats.filter(splat=>splat !== mesh)
}
l.onCreate = (mesh: GaussianSplatMesh)=>{
this.splats.push(mesh)
}
l.onGeometryLoad = (_)=>{
// console.log('geometry loaded')
// console.log(geometry.boundingBox)
this._viewer?.setDirty()
}
return l
})

// protected _viewerListeners = {
// preRender: (_: IViewerEvent) => {
// },
// }
}

+ 30
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/BufferPool.ts Näytä tiedosto

@@ -0,0 +1,30 @@
export class BufferPool {
private pool: ArrayBuffer[];
private bufferSize: number;

constructor(bufferSize: number, initialPoolSize: number = 0) {
this.bufferSize = bufferSize;
this.pool = [];
this.initPool(initialPoolSize);
}

private initPool(initialPoolSize: number): void {
for (let i = 0; i < initialPoolSize; i++) {
this.pool.push(new ArrayBuffer(this.bufferSize));
}
}

public getBuffer(): ArrayBuffer {
if (this.pool.length > 0) {
return this.pool.pop()!;
} else {
return new ArrayBuffer(this.bufferSize);
}
}

public returnBuffer(buffer: ArrayBuffer): void {
if (buffer.byteLength === this.bufferSize) {
this.pool.push(buffer);
}
}
}

+ 3
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/ISort.d.ts Näytä tiedosto

@@ -0,0 +1,3 @@
export interface MainModule {
runSort(_0: number, _1: number, _2: number, _3: number): void;
}

+ 53
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/SortWorkerManager.ts Näytä tiedosto

@@ -0,0 +1,53 @@
import {Remote, transfer, wrap} from 'comlink'
import type {WasmSorter} from './worker'

export const SPLAT_ROW_LENGTH = 3 * 4 + 3 * 4 + 4 + 4

function trimBuffer(_buffer: Uint8Array, _maxSplats: number, _vertexCount: number): {buffer: Uint8Array; vertexCount: number} {
const actualVertexCount = Math.min(_vertexCount, _maxSplats)
const actualBufferSize = SPLAT_ROW_LENGTH * actualVertexCount
const buffer = _buffer.slice(0, actualBufferSize)
return {buffer, vertexCount: actualVertexCount}
}

export class SortWorkerManager {
private _workers: Remote<WasmSorter>[] = []
private _maxWorkers = 8

private _workerCtor: new (vertexCount: number, globalBuffer: Uint8Array) => Remote<WasmSorter>

onError = (e: any) => {
console.error(e)
console.error(
'ERROR: Line ', e.lineno, ' in ', e.filename, ': ', e.message
)
}

constructor() {
const worker = new Worker(new URL('../cpp-sorter/worker', import.meta.url), {type: 'module'})
worker.addEventListener('error', this.onError, false)
this._workerCtor = wrap(worker) as any
}

async createWorker(data: Uint8Array, maxSplats = 1000000) {
if (this._workers.length < this._maxWorkers) {
const vertexCount = Math.floor(data.length / SPLAT_ROW_LENGTH)
const bufferInfo = trimBuffer(data, maxSplats, vertexCount)
const globalBuffer = transfer(bufferInfo.buffer, [bufferInfo.buffer.buffer])
const worker = await new this._workerCtor(vertexCount, globalBuffer)
await worker.load()
this._workers.push(worker)
return worker
}
console.error('Max workers reached')
throw new Error('Max workers reached')
}

async disposeWorker(worker: Remote<WasmSorter>) {
const index = this._workers.indexOf(worker)
if (index !== -1) {
this._workers.splice(index, 1)
await worker.dispose()
}
}
}

+ 4
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/build-in-docker.sh Näytä tiedosto

@@ -0,0 +1,4 @@
#!/bin/bash
set -e
DOCKER_IMAGE="emscripten/emsdk:3.1.50"
docker run --rm -v "$(pwd)":/src -w /src $DOCKER_IMAGE bash build.sh

+ 7
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/build.sh Näytä tiedosto

@@ -0,0 +1,7 @@
#!/bin/bash
set -e

rm -rf sort.wasm sort.js sort.d.ts
rm -rf pthread_sort.wasm pthread_sort.js pthread_sort.worker.js
em++ -pthread -O3 -s EXPORT_ES6=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s ENVIRONMENT=web,worker -flto --embind-emit-tsd=ISort.d.ts sort.cpp -o pthread_sort.js -s WASM=1 -lembind -s EXPORTED_FUNCTIONS='["_malloc", "_free"]' -s AGGRESSIVE_VARIABLE_ELIMINATION=1 -s ELIMINATE_DUPLICATE_FUNCTIONS=1
em++ -O3 -s EXPORT_ES6=1 -s ALLOW_MEMORY_GROWTH=1 -s MODULARIZE=1 -s ENVIRONMENT=web,worker -flto --embind-emit-tsd=ISort.d.ts sort.cpp -o sort.js -s WASM=1 -lembind -s EXPORTED_FUNCTIONS='["_malloc", "_free"]' -s AGGRESSIVE_VARIABLE_ELIMINATION=1 -s ELIMINATE_DUPLICATE_FUNCTIONS=1

+ 107
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.cpp Näytä tiedosto

@@ -0,0 +1,107 @@
#include <emscripten/bind.h>
#include <algorithm>
#include <cstdint>
#include <cmath>
#include <vector>
#include <numeric>
#include <stdio.h>
#include <cstring>


union DepthIndex {
struct {
float depth;
uint32_t index;
};

bool operator < (const DepthIndex& other) const {
return depth < other.depth;
}
};

uint32_t floatToUInt(float f) {
uint32_t u;
std::memcpy(&u, &f, sizeof(float));
uint32_t mask = -int32_t(u >> 31) | 0x80000000;
return u ^ mask;
}

void radixSort(std::vector<DepthIndex>& depthIndices) {
const int n = depthIndices.size();
std::vector<DepthIndex> temp(n);

for (int shift = 0; shift < 32; shift += 8) {
size_t count[256] = {};
for (auto& di : depthIndices) {
count[(floatToUInt(di.depth) >> shift) & 0xFF]++;
}
size_t pos[256];
pos[0] = 0;
for (int i = 1; i < 256; i++) {
pos[i] = pos[i - 1] + count[i - 1];
}
for (auto& di : depthIndices) {
int index = (floatToUInt(di.depth) >> shift) & 0xFF;
temp[pos[index]++] = di;
}
depthIndices.swap(temp);
}
}


using namespace emscripten;


void runSort(int viewProjPtr, int bufferPtr, int vertexCount, int combinedPtr) {
auto *viewProj = reinterpret_cast<float *>(viewProjPtr);
uint8_t *buffer = reinterpret_cast<uint8_t *>(bufferPtr);
auto *combined = reinterpret_cast<float *>(combinedPtr);

// Calculate lengths based on vertexCount
int quatLength = 4 * vertexCount;
int scaleLength = 3 * vertexCount;
int centerLength = 3 * vertexCount;
int colorLength = 4 * vertexCount;

// Calculate offsets for each array within the combined array
int quatOffset = 0;
int scaleOffset = quatOffset + quatLength;
int centerOffset = scaleOffset + scaleLength;
int colorOffset = centerOffset + centerLength;

std::vector<DepthIndex> depthIndices(vertexCount);
for (int i = 0; i < vertexCount; ++i) {
auto *f_buffer = reinterpret_cast<float *>(buffer + 32 * i);
depthIndices[i].depth = 10000 - (viewProj[2] * f_buffer[0] + viewProj[6] * f_buffer[1] + viewProj[10] * f_buffer[2]);
depthIndices[i].index = i;
}

radixSort(depthIndices);

for (int i = 0; i < vertexCount; ++i) {
uint32_t index = depthIndices[i].index;
auto *f_buffer = reinterpret_cast<float *>(buffer + 32 * index);

combined[quatOffset + 4 * i + 0] = (buffer[32 * index + 28 + 0] - 128) / 128.0f;
combined[quatOffset + 4 * i + 1] = (buffer[32 * index + 28 + 1] - 128) / 128.0f;
combined[quatOffset + 4 * i + 2] = (buffer[32 * index + 28 + 2] - 128) / 128.0f;
combined[quatOffset + 4 * i + 3] = (buffer[32 * index + 28 + 3] - 128) / 128.0f;

combined[centerOffset + 3 * i + 0] = f_buffer[0];
combined[centerOffset + 3 * i + 1] = f_buffer[1];
combined[centerOffset + 3 * i + 2] = f_buffer[2];

combined[colorOffset + 4 * i + 0] = buffer[32 * index + 24 + 0] / 255.0f;
combined[colorOffset + 4 * i + 1] = buffer[32 * index + 24 + 1] / 255.0f;
combined[colorOffset + 4 * i + 2] = buffer[32 * index + 24 + 2] / 255.0f;
combined[colorOffset + 4 * i + 3] = buffer[32 * index + 24 + 3] / 255.0f;

combined[scaleOffset + 3 * i + 0] = f_buffer[3];
combined[scaleOffset + 3 * i + 1] = f_buffer[4];
combined[scaleOffset + 3 * i + 2] = f_buffer[5];
}
}

EMSCRIPTEN_BINDINGS(my_module) {
function("runSort", &runSort);
}

+ 16
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.js
File diff suppressed because it is too large
Näytä tiedosto


BIN
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/sort.wasm Näytä tiedosto


+ 9
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/watch.sh Näytä tiedosto

@@ -0,0 +1,9 @@
#!/bin/bash
set -e
./build.sh
fswatch -o sort.cpp | while read line
do
echo "File changed, rebuilding..."
./build.sh
echo "Done."
done

+ 59
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/cpp-sorter/worker.ts Näytä tiedosto

@@ -0,0 +1,59 @@
import {expose, transfer} from 'comlink'
import type {MainModule} from './ISort'
// @ts-expect-error no types
import workerPromise from './sort'
// import sharedArrayBufferWorkerPromise from './sort'
import {BufferPool} from './BufferPool'

export class WasmSorter {
public module: EmscriptenModule & MainModule
private _viewProjPtr: number
private _globalBufferPtr: number
private _combinedPtr: number
private _bufferPool: BufferPool

constructor(private _vertexCount: number, private _globalBuffer: Uint8Array) {
const combinedLength = this._calculateCombinedLength()
// we're double buffering, so 2 is the magic number here
this._bufferPool = new BufferPool(combinedLength, 2)
}

public async load() {
// const sharedABSupported = typeof SharedArrayBuffer !== 'undefined'
// this.module = await (sharedABSupported ? sharedArrayBufferWorkerPromise() : workerPromise())
this.module = await workerPromise()
this._viewProjPtr = this.module._malloc(16 * Float32Array.BYTES_PER_ELEMENT)
this._globalBufferPtr = this.module._malloc(this._vertexCount * 32)
this._combinedPtr = this.module._malloc(this._calculateCombinedLength())
this.module.HEAPU8.set(this._globalBuffer, this._globalBufferPtr)
}

public runSort(viewProj: Float32Array): ArrayBuffer {
this.module.HEAPF32.set(viewProj, this._viewProjPtr / Float32Array.BYTES_PER_ELEMENT)
this.module.runSort(this._viewProjPtr, this._globalBufferPtr, this._vertexCount, this._combinedPtr)

const byteStart = this._combinedPtr
const byteLength = this._calculateCombinedLength()
const bufferToTransfer = this._bufferPool.getBuffer()
new Uint8Array(bufferToTransfer).set(new Uint8Array(this.module.HEAPU8.buffer, byteStart, byteLength))
return transfer(bufferToTransfer, [bufferToTransfer])
}

public returnBuffer(buffer: ArrayBuffer): void {
this._bufferPool.returnBuffer(buffer)
}

private _calculateCombinedLength(): number {
return 4 * 4 * this._vertexCount + 3 * 4 * this._vertexCount + 3 * 4 * this._vertexCount + 4 * 4 * this._vertexCount * Float32Array.BYTES_PER_ELEMENT
}

public dispose(): void {
if (this.module) {
this.module._free(this._viewProjPtr)
this.module._free(this._globalBufferPtr)
this.module._free(this._combinedPtr)
}
}
}

expose(WasmSorter)

+ 182
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/geometry/GaussianSplatGeometry.ts Näytä tiedosto

@@ -0,0 +1,182 @@
import type {WasmSorter} from '../cpp-sorter/worker'
import {Remote, transfer} from 'comlink'
import {
Box3,
BufferAttribute,
Camera,
IGeometry,
iGeometryCommons,
InstancedBufferAttribute,
InstancedBufferGeometry,
type IObject3D,
Matrix4,
PerspectiveCamera,
Sphere,
Vector3,
} from 'threepipe'

export class GaussianSplatGeometry extends InstancedBufferGeometry implements IGeometry {
constructor(
private _worker: Remote<WasmSorter>,
private _vertexCount: number,
private _maxSplats: number,
private _onLoad?: (geometry: GaussianSplatGeometry) => void,
) {
super()
iGeometryCommons.upgradeGeometry.call(this)
this.initAttributes()
}

readonly isGaussianSplatGeometry = true

assetType: 'geometry' // dont set the value here since its checked in upgradeGeometry
setDirty = iGeometryCommons.setDirty
refreshUi = iGeometryCommons.refreshUi
appliedMeshes = new Set<IObject3D>()

private _viewProj: number[] = []
private _sortRunning = false

private _initialized = false

private _centersBuffer: Float32Array = new Float32Array(0)

public async update(camera: PerspectiveCamera | Camera, meshMatrixWorld: Matrix4) {
if (this._sortRunning || !this._initialized || !this._worker) {
return
}

camera.updateMatrixWorld(true)

this._viewProj = new Matrix4().multiply(camera.projectionMatrix).multiply(camera.matrixWorldInverse).multiply(meshMatrixWorld).elements

this._sortRunning = true
const viewProj = new Float32Array(this._viewProj)
const result = await this._worker.runSort(viewProj)

const {quat, scale, center, color} = this._extractViews(result)

if (this._centersBuffer.length !== center.length) this._centersBuffer = new Float32Array(center)
else this._centersBuffer.set(center)

;(this.attributes.color as InstancedBufferAttribute).array = color;
(this.attributes.quat as InstancedBufferAttribute).array = quat;
(this.attributes.scale as InstancedBufferAttribute).array = scale;
(this.attributes.center as InstancedBufferAttribute).array = this._centersBuffer
// (this.attributes.center as InstancedBufferAttribute).array = center

const pms = Promise.all([
new Promise<void>(resolve => (this.attributes.color as InstancedBufferAttribute).onUpload(resolve)),
new Promise<void>(resolve => (this.attributes.quat as InstancedBufferAttribute).onUpload(resolve)),
new Promise<void>(resolve => (this.attributes.scale as InstancedBufferAttribute).onUpload(resolve)),
// new Promise<void>(resolve => (this.attributes.center as InstancedBufferAttribute).onUpload(resolve)),
])

this.attributes.color.needsUpdate = true
this.attributes.quat.needsUpdate = true
this.attributes.scale.needsUpdate = true
this.attributes.center.needsUpdate = true

this.setDirty()

await pms

await this._worker.returnBuffer(transfer(result, [result]))

this._sortRunning = false
}

async initAttributes() {
const viewProj = new Float32Array(this._viewProj)
const result = await this._worker.runSort(viewProj)
const {quat, scale, center, color} = this._extractViews(result)

this.setAttribute('color', new InstancedBufferAttribute(color, 4, true))
this.setAttribute('quat', new InstancedBufferAttribute(quat, 4, true))
this.setAttribute('scale', new InstancedBufferAttribute(scale, 3, true))
this.setAttribute('center', new InstancedBufferAttribute(center, 3, true))
this.setAttribute('position', new BufferAttribute(new Float32Array([1, -1, 0, 1, 1, 0, -1, -1, 0, -1, 1, 0]), 3, true))

this.attributes.position.needsUpdate = true
this.setIndex(new BufferAttribute(new Uint16Array([0, 1, 2, 2, 3, 0]), 1, true))
this.instanceCount = Math.min(quat.length / 4, this._maxSplats)
this.computeBoundingBox()
this.computeBoundingSphere()

this._initialized = true
this.setDirty()
this._onLoad && this._onLoad(this)
}

private _extractViews(receivedBuffer: ArrayBuffer): {quat: Float32Array; scale: Float32Array; center: Float32Array; color: Float32Array} {
const combined = new Float32Array(receivedBuffer)

const quatLength = 4 * this._vertexCount
const scaleLength = 3 * this._vertexCount
const centerLength = 3 * this._vertexCount
const colorLength = 4 * this._vertexCount

const quatOffset = 0
const scaleOffset = quatOffset + quatLength
const centerOffset = scaleOffset + scaleLength
const colorOffset = centerOffset + centerLength

const quat = combined.subarray(quatOffset, quatOffset + quatLength)
const scale = combined.subarray(scaleOffset, scaleOffset + scaleLength)
const center = combined.subarray(centerOffset, centerOffset + centerLength)
const color = combined.subarray(colorOffset, colorOffset + colorLength)

return {quat, scale, center, color}
}

computeBoundingBox() {
if (!this.getAttribute('center')) return super.computeBoundingBox()

const box = this.boundingBox ?? (this.boundingBox = new Box3())
box.setFromBufferAttribute(this.getAttribute('center') as InstancedBufferAttribute)

if (isNaN(this.boundingBox!.min.x) || isNaN(this.boundingBox!.min.y) || isNaN(this.boundingBox!.min.z)) {
console.error('GaussianSplatGeometry.computeBoundingBox(): Computed min/max have NaN values. The "position" attribute is likely to have NaN values.', this)
}
}

computeBoundingSphere() {

if (this.boundingSphere === null) this.boundingSphere = new Sphere()

const position = this.attributes.center

if (position && (position as any).isGLBufferAttribute) {
console.error('THREE.BufferGeometry.computeBoundingSphere(): GLBufferAttribute requires a manual bounding sphere. Alternatively set "mesh.frustumCulled" to "false".', this)
this.boundingSphere.set(new Vector3(), Infinity)
return
}

if (position) {
// first, find the center of the bounding sphere
const center = this.boundingSphere.center
if (!this.boundingBox) this.computeBoundingBox()
this.boundingBox!.getCenter(center)

// second, try to find a boundingSphere with a radius smaller than the
// boundingSphere of the boundingBox: sqrt(3) smaller in the best case

let maxRadiusSq = 0
const vector = new Vector3()

for (let i = 0, il = position.count; i < il; i++) {
vector.fromBufferAttribute(position, i)
maxRadiusSq = Math.max(maxRadiusSq, center.distanceToSquared(vector))
}

this.boundingSphere.radius = Math.sqrt(maxRadiusSq)

if (isNaN(this.boundingSphere.radius)) {
console.error('THREE.BufferGeometry.computeBoundingSphere(): Computed radius is NaN. The "position" attribute is likely to have NaN values.', this)
}

}

}

}

+ 22
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/index.ts Näytä tiedosto

@@ -0,0 +1,22 @@
/**
* ThreeGaussianSplatPlugin: Threejs utilities and threepipe plugin and material extension for Gaussian Splatting
*
* @license
* A port of -
* https://github.com/zappar-xr/three-gaussian-splat
* MIT License (c) 2023 Zappar Limited
* Which is based on -
* gsplat.js, MIT License (c) 2023 Dylan Ebert
* antimatter15/splat, MIT License (c) 2023 Kevin Kwok
*/

export {GaussianSplatMaterialExtension} from './materials/GaussianSplatMaterialExtension'
export {GaussianSplatMaterialPhysical} from './materials/GaussianSplatMaterialPhysical'
export {GaussianSplatMaterialRaw} from './materials/GaussianSplatMaterialRaw'
export {GaussianSplatMaterialUnlit} from './materials/GaussianSplatMaterialUnlit'
export {computeFocalLengths} from './materials/util'
export {GaussianSplatGeometry} from './geometry/GaussianSplatGeometry'
export {GaussianSplatMesh} from './mesh/GaussianSplatMesh'
export {ThreeGaussianSplatPlugin} from './ThreeGaussianSplatPlugin'
export {SplatLoader} from './loaders/SplatLoader'
export {gaussianSplatShaders} from './shaders'

+ 51
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/loaders/SplatLoader.ts Näytä tiedosto

@@ -0,0 +1,51 @@
import {FileLoader, ILoader, IMaterial, Loader, LoadingManager} from 'threepipe'
import {SortWorkerManager, SPLAT_ROW_LENGTH} from '../cpp-sorter/SortWorkerManager'
import {GaussianSplatGeometry} from '../geometry/GaussianSplatGeometry'
import {GaussianSplatMesh} from '../mesh/GaussianSplatMesh'
import {GaussianSplatMaterialUnlit} from '../materials/GaussianSplatMaterialUnlit'

export class SplatLoader extends Loader implements ILoader {
sortWorkerManager: SortWorkerManager

materialConstructor = (_: GaussianSplatGeometry): IMaterial|undefined => new GaussianSplatMaterialUnlit()

constructor(manager?: LoadingManager) {
super(manager)
}

onGeometryLoad = (_: GaussianSplatGeometry)=>{
return
}

public load(url: string, onLoad?: (data: GaussianSplatMesh) => void, onProgress?: (event: ProgressEvent) => void, onError?: (event: ErrorEvent) => void): void {
// const path = ( scope.path === '' ) ? LoaderUtils.extractUrlBase( url ) : scope.path;
const loader = new FileLoader(this.manager)
loader.setPath(this.path)
loader.setResponseType('arraybuffer')
loader.setRequestHeader(this.requestHeader)
loader.setWithCredentials(this.withCredentials)
loader.load(url, async(buffer) => {
try {
const data = new Uint8Array(buffer as ArrayBuffer)
const maxSplats = 1000000
const vertexCount = Math.floor(data.length / SPLAT_ROW_LENGTH)
const worker = await this.sortWorkerManager.createWorker(data, maxSplats)
const geometry = new GaussianSplatGeometry(worker, vertexCount, maxSplats, this.onGeometryLoad)
const mesh = new GaussianSplatMesh(geometry, this.materialConstructor(geometry) as any)
// const mesh = new GaussianSplatMesh(geometry, new UnlitMaterial() as any)
mesh.rotation.x = Math.PI
if (!url.startsWith('blob:') && !url.startsWith('data:'))
mesh.name = url.split('/').pop()!.split('?')[0]
onLoad && onLoad(mesh)
} catch (e) {
if (onError) onError(e)
else console.error(e)
this.manager.itemError(url)
}
}, onProgress, onError)
}

public async loadAsync(url: string, onProgress?: (event: ProgressEvent) => void): Promise<GaussianSplatMesh> {
return new Promise((resolve, reject) => this.load(url, resolve, onProgress, reject))
}
}

+ 105
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialExtension.ts Näytä tiedosto

@@ -0,0 +1,105 @@
import {
Camera,
IMaterial,
MaterialExtension,
PerspectiveCamera,
Shader,
shaderReplaceString,
Vector2,
WebGLRenderer,
} from 'threepipe'
import {computeFocalLengths} from './util'
import {gaussianSplatShaders} from '../shaders'

export class GaussianSplatMaterialExtension implements MaterialExtension {
readonly isGaussianSplatMaterialExtension = true
extraUniforms = {
viewport: {value: new Vector2()},
focal: {value: new Vector2()},
minAlpha: {value: 0.02},
}
parsFragmentSnippet = gaussianSplatShaders.pars_frag
parsVertexSnippet = gaussianSplatShaders.pars_vert
shaderExtender = (shader: Shader, material: IMaterial) => {
shader.vertexShader = shaderReplaceString(shader.vertexShader,
'#include <begin_vertex>',
'vec3 transformed = vec3( center );')
if (shader.vertexShader.includes('#include <beginnormal_vertex>')) {
// todo: get correct normal by rendering to depth and then sampling the normal
shader.vertexShader = shaderReplaceString(shader.vertexShader,
'#include <beginnormal_vertex>',
'\nobjectNormal = vec3(1,0,0);\n', {append: true})
}
shader.vertexShader = shaderReplaceString(shader.vertexShader,
'#include <project_vertex>',
gaussianSplatShaders.main_vert, {append: true})

if (!material.isGBufferMaterial && !material.userData.isGBufferMaterial) {
shader.fragmentShader = shaderReplaceString(shader.fragmentShader,
'#include <map_fragment>',
shaderReplaceString(
gaussianSplatShaders.main_frag,
/gl_FragColor\s*=/, // because of minification
'diffuseColor*='
), {prepend: true})
}
// eslint-disable-next-line no-constant-condition
if (0)
shader.fragmentShader = shaderReplaceString(shader.fragmentShader,
'#include <output_fragment>',
'\ngl_FragColor=diffuseColor;',
// '\ngl_FragColor=vec4(reflectedLight.directDiffuse, 1.);',
// '\ngl_FragColor=vec4(geometryNormal, 1.);',
{append: true})
return shader
}
isCompatible = (material: IMaterial) =>
!!material.isPhysicalMaterial ||
!!material.isUnlitMaterial ||
!!material.isGBufferMaterial ||
!!material.userData.isGBufferMaterial

setDirty?: () => void

private _currentCamera?: PerspectiveCamera | Camera
private _renderer?: WebGLRenderer

public set minAlpha(value: number) {
this.extraUniforms.minAlpha.value = value
this.setDirty && this.setDirty()
}


constructor() {
window.addEventListener('resize', this._refresh)
}

dispose() {
// todo: add again on added to mesh?
window.removeEventListener('resize', this._refresh)
}

update(camera: PerspectiveCamera | Camera, renderer: WebGLRenderer): void {
if (this._currentCamera === camera && this._renderer === renderer) return
this._renderer = renderer
this._currentCamera = camera
this._refresh()
}

private _refresh = (): void => {
if (!this._currentCamera) return
const size = new Vector2()
this._renderer?.getSize(size)
const dpr = this._renderer?.getPixelRatio() || 1
let fov = 75
let aspect = size.x / size.y

if (this._currentCamera instanceof PerspectiveCamera) {
fov = this._currentCamera.fov
aspect = this._currentCamera.aspect
}

this.extraUniforms.focal.value = computeFocalLengths(size.x, size.y, fov, aspect, dpr)
this.extraUniforms.viewport.value = new Vector2(size.x * dpr, size.y * dpr)
}
}

+ 25
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialPhysical.ts Näytä tiedosto

@@ -0,0 +1,25 @@
import {PhysicalMaterial} from 'threepipe'
import {GaussianSplatMaterialExtension} from './GaussianSplatMaterialExtension'

export class GaussianSplatMaterialPhysical extends PhysicalMaterial {
readonly isGaussianSplatMaterialPhysical = true

gsplatExtension = new GaussianSplatMaterialExtension()

constructor() {
super({
depthTest: true,
depthWrite: false,
transparent: true,
vertexColors: false,
})
// this.userData.renderToGBuffer = true
// this.userData.renderToDepth = true
this.registerMaterialExtensions([this.gsplatExtension])
}

dispose() {
this.gsplatExtension.dispose()
return super.dispose()
}
}

+ 73
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialRaw.ts Näytä tiedosto

@@ -0,0 +1,73 @@
import {Camera, PerspectiveCamera, ShaderMaterial2, Vector2, WebGLRenderer} from 'threepipe'
import {computeFocalLengths} from './util'
import {gaussianSplatShaders} from '../shaders'

export class GaussianSplatMaterialRaw extends ShaderMaterial2 {
private _currentCamera?: PerspectiveCamera | Camera
private _renderer?: WebGLRenderer

readonly isGaussianSplatMaterialRaw = true

public set minAlpha(value: number) {
this.uniforms.minAlpha.value = value
this.needsUpdate = true
}

constructor() {
super({
uniforms: {
viewport: {value: new Vector2()},
focal: {value: new Vector2()},
minAlpha: {value: 0.02},
},
fragmentShader: `${gaussianSplatShaders.pars_frag}
void main () {
${gaussianSplatShaders.main_frag}
}`,
vertexShader: `${gaussianSplatShaders.pars_vert}
void main () {
gl_Position = projectionMatrix * modelViewMatrix * vec4(center, 1);
${gaussianSplatShaders.main_vert}
}`,
depthTest: true,
depthWrite: false,
transparent: true,
}, true)

window.addEventListener('resize', this._refresh)
}

private _refresh = (): void => {
if (!this._currentCamera) return

const size = new Vector2()
this._renderer?.getSize(size)
const width = size.x
const height = size.y

const dpr = this._renderer?.getPixelRatio() || 1

let fov = 75
let aspect = width / height

if (this._currentCamera instanceof PerspectiveCamera) {
fov = this._currentCamera.fov
aspect = this._currentCamera.aspect
}

this.uniforms.focal.value = computeFocalLengths(width, height, fov, aspect, dpr)
this.uniforms.viewport.value = new Vector2(width * dpr, height * dpr)
}

dispose() {
// todo: add again on added to mesh?
window.removeEventListener('resize', this._refresh)
return super.dispose()
}

update(camera: PerspectiveCamera | Camera, renderer: WebGLRenderer): void {
this._renderer = renderer
this._currentCamera = camera
this._refresh()
}
}

+ 26
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/materials/GaussianSplatMaterialUnlit.ts Näytä tiedosto

@@ -0,0 +1,26 @@
import {UnlitMaterial} from 'threepipe'

import {GaussianSplatMaterialExtension} from './GaussianSplatMaterialExtension'

export class GaussianSplatMaterialUnlit extends UnlitMaterial {
readonly isGaussianSplatMaterialUnlit = true
gsplatExtension = new GaussianSplatMaterialExtension()

constructor() {
super({
depthTest: true,
depthWrite: false,
transparent: true,
vertexColors: false,
})
// this.userData.renderToGBuffer = true
// this.userData.renderToDepth = true
this.registerMaterialExtensions([this.gsplatExtension])
}

dispose() {
this.gsplatExtension.dispose()
return super.dispose()
}
}


+ 9
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/materials/util.ts Näytä tiedosto

@@ -0,0 +1,9 @@
import {MathUtils, Vector2} from 'threepipe'

export const computeFocalLengths = (width: number, height: number, fov: number, aspect: number, dpr: number) => {
const fovRad = MathUtils.degToRad(fov)
const fovXRad = 2 * Math.atan(Math.tan(fovRad / 2) * aspect)
const fy = dpr * height / (2 * Math.tan(fovRad / 2))
const fx = dpr * width / (2 * Math.tan(fovXRad / 2))
return new Vector2(fx, fy)
}

+ 96
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/mesh/GaussianSplatMesh.ts Näytä tiedosto

@@ -0,0 +1,96 @@
import {GaussianSplatGeometry} from '../geometry/GaussianSplatGeometry'
import {Camera, IMaterial, Mesh2, PerspectiveCamera, WebGLRenderer} from 'threepipe'
import type {GaussianSplatMaterialRaw} from '../materials/GaussianSplatMaterialRaw'
import {GaussianSplatMaterialUnlit} from '../materials/GaussianSplatMaterialUnlit'
import {Matrix4, Ray, Raycaster, Sphere, Vector3} from 'three'
import {GaussianSplatMaterialExtension} from '../materials/GaussianSplatMaterialExtension'

export class GaussianSplatMesh extends Mesh2<GaussianSplatGeometry, IMaterial> {
readonly isGaussianSplatMesh = true

constructor(geometry: GaussianSplatGeometry, material: IMaterial) {
super(geometry, material)
this.frustumCulled = false
}

public async update(camera: PerspectiveCamera | Camera, renderer: WebGLRenderer) {
if ((this.material as any as GaussianSplatMaterialRaw)?.isGaussianSplatMaterialRaw) {
(this.material as any as GaussianSplatMaterialRaw).update(camera, renderer)
} else if (this.material) {
const ext =
(this.material as GaussianSplatMaterialUnlit).gsplatExtension ??
this.material.materialExtensions.find(e=>
(e as GaussianSplatMaterialExtension).isGaussianSplatMaterialExtension) as GaussianSplatMaterialExtension
ext && ext.update(camera, renderer)
}
this.updateMatrixWorld(true)
return this.geometry.update(camera, this.matrixWorld)
}

raycast(raycaster: Raycaster, intersects: any[]) {

const geometry = this.geometry
const matrixWorld = this.matrixWorld
// const threshold = raycaster.params.Points?.threshold ?? .01
const threshold = .02

// Checking boundingSphere distance to ray

if (geometry.boundingSphere === null) geometry.computeBoundingSphere()

const sphere = new Sphere()
sphere.copy(geometry.boundingSphere!)
sphere.applyMatrix4(matrixWorld)
sphere.radius += threshold

if (!raycaster.ray.intersectsSphere(sphere)) return

//

const inverseMatrix = new Matrix4()
const ray = new Ray()

inverseMatrix.copy(matrixWorld).invert()
ray.copy(raycaster.ray).applyMatrix4(inverseMatrix)

const localThreshold = threshold / ((this.scale.x + this.scale.y + this.scale.z) / 3)
const localThresholdSq = localThreshold * localThreshold

const attributes = geometry.attributes
const positionAttribute = attributes.center

const position = new Vector3()
for (let i = 0, l = positionAttribute.count; i < l; i++) {

position.fromBufferAttribute(positionAttribute, i)

const rayPointDistanceSq = ray.distanceSqToPoint(position)

if (rayPointDistanceSq < localThresholdSq) {

const intersectPoint = new Vector3()

ray.closestPointToPoint(position, intersectPoint)
intersectPoint.applyMatrix4(matrixWorld)

const distance = raycaster.ray.origin.distanceTo(intersectPoint)

if (distance < raycaster.near || distance > raycaster.far) return

intersects.push({

distance: distance,
distanceToRay: Math.sqrt(rayPointDistanceSq),
point: intersectPoint,
face: null,
object: this,

})


}
}

}

}

+ 18
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.main.frag.glsl Näytä tiedosto

@@ -0,0 +1,18 @@
vec2 d = (vCenter - 2.0 * (gl_FragCoord.xy/viewport - vec2(0.5, 0.5))) * viewport * 0.5;

float power = -0.5 * (vConic.x * d.x * d.x + vConic.z * d.y * d.y) + vConic.y * d.x * d.y;

if (power > 0.0) discard;
float alpha = min(0.99, vColor.a * exp(power));
if(alpha < minAlpha) discard;

gl_FragColor = vec4(vColor.rgb, alpha);
//vec2 d = (vCenter - 2.0 * (gl_FragCoord.xy/viewport - vec2(0.5, 0.5))) * viewport * 0.5;
//
//float power = -0.5 * (vConic.x * d.x * d.x + vConic.z * d.y * d.y) + vConic.y * d.x * d.y;
//
//if (power > 0.0) discard;
//float alpha = min(0.99, vColor.a * exp(power));
//if(alpha < minAlpha) discard;
//
//diffuseColor *= vec4(vColor.rgb, alpha);

+ 15
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.main.vert.glsl Näytä tiedosto

@@ -0,0 +1,15 @@
vec3 cov2d = compute_cov2d(center, scale, quat);
float det = cov2d.x * cov2d.z - cov2d.y * cov2d.y;
vec3 conic = vec3(cov2d.z, cov2d.y, cov2d.x) / det;
float mid = 0.5 * (cov2d.x + cov2d.z);
float lambda1 = mid + sqrt(max(0.1, mid * mid - det));
float lambda2 = mid - sqrt(max(0.1, mid * mid - det));
vec2 v1 = 7.0 * sqrt(lambda1) * normalize(vec2(cov2d.y, lambda1 - cov2d.x));
vec2 v2 = 7.0 * sqrt(lambda2) * normalize(vec2(-(lambda1 - cov2d.x),cov2d.y));

vColor = color;
vConic = conic;
vCenter = vec2(gl_Position) / gl_Position.w;

vPosition = vec2(vCenter + position.x * (position.y < 0.0 ? v1 : v2) / viewport);
gl_Position = vec4(vPosition, gl_Position.z / gl_Position.w, 1);

+ 10
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.pars.frag.glsl Näytä tiedosto

@@ -0,0 +1,10 @@
// https://github.com/vincent-lecrubier-skydio/react-three-fiber-gaussian-splat
precision mediump float;

varying vec4 vColor;
varying vec3 vConic;
varying vec2 vCenter;

uniform vec2 viewport;
uniform vec2 focal;
uniform float minAlpha;

+ 58
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/shaders/gsplat.pars.vert.glsl Näytä tiedosto

@@ -0,0 +1,58 @@
// https://github.com/vincent-lecrubier-skydio/react-three-fiber-gaussian-splat
precision mediump float;

#ifndef SHADER_NAME // isRawShaderMaterial
attribute vec3 position;
uniform mat4 modelViewMatrix;
uniform mat4 projectionMatrix;
mat3 transpose(mat3 m) { return mat3(m[0][0], m[1][0], m[2][0], m[0][1], m[1][1], m[2][1], m[0][2], m[1][2], m[2][2]); }
#endif

attribute vec4 color;
attribute vec4 quat;
attribute vec3 scale;
attribute vec3 center;

uniform vec2 focal;
uniform vec2 viewport;
//uniform vec3 sphereCenter;
//uniform vec3 planeNormal;
//uniform float planeDistance;

varying vec4 vColor;
varying vec3 vConic;
varying vec2 vCenter;
varying vec2 vPosition;
//varying float vDistance;
//varying float vPlaneSide;

mat3 compute_cov3d(vec3 scale, vec4 rot) {
mat3 S = mat3(
scale.x, 0.0, 0.0,
0.0, scale.y, 0.0,
0.0, 0.0, scale.z
);
mat3 R = mat3(
1.0 - 2.0 * (rot.z * rot.z + rot.w * rot.w), 2.0 * (rot.y * rot.z - rot.x * rot.w), 2.0 * (rot.y * rot.w + rot.x * rot.z),
2.0 * (rot.y * rot.z + rot.x * rot.w), 1.0 - 2.0 * (rot.y * rot.y + rot.w * rot.w), 2.0 * (rot.z * rot.w - rot.x * rot.y),
2.0 * (rot.y * rot.w - rot.x * rot.z), 2.0 * (rot.z * rot.w + rot.x * rot.y), 1.0 - 2.0 * (rot.y * rot.y + rot.z * rot.z)
);
mat3 M = S * R;
return transpose(M) * M;
}

vec3 compute_cov2d(vec3 center, vec3 scale, vec4 rot){
mat3 Vrk = compute_cov3d(scale, rot);
vec4 t = modelViewMatrix * vec4(center, 1.0);
vec2 lims = 1.3 * 0.5 * viewport / focal;
t.xy = min(lims, max(-lims, t.xy / t.z)) * t.z;
mat3 J = mat3(
focal.x / t.z, 0., -(focal.x * t.x) / (t.z * t.z),
0., focal.y / t.z, -(focal.y * t.y) / (t.z * t.z),
0., 0., 0.
);
mat3 W = transpose(mat3(modelViewMatrix));
mat3 T = W * J;
mat3 cov = transpose(T) * transpose(Vrk) * T;
return vec3(cov[0][0] + 0.3, cov[0][1], cov[1][1] + 0.3);
}

+ 11
- 0
plugins/gaussian-splatting/src/three-gaussian-splat/shaders/index.ts Näytä tiedosto

@@ -0,0 +1,11 @@
import fragmentParsShaderSource from '../shaders/gsplat.pars.frag.glsl'
import vertexParsShaderSource from '../shaders/gsplat.pars.vert.glsl'
import vertexShaderSource from '../shaders/gsplat.main.vert.glsl'
import fragmentShaderSource from '../shaders/gsplat.main.frag.glsl'

export const gaussianSplatShaders = {
main_frag: '\n' + fragmentShaderSource + '\n',
main_vert: '\n' + vertexShaderSource + '\n',
pars_frag: fragmentParsShaderSource,
pars_vert: vertexParsShaderSource,
}

+ 41
- 0
plugins/gaussian-splatting/tsconfig.json Näytä tiedosto

@@ -0,0 +1,41 @@
{
"compilerOptions": {
"baseUrl": "./src",
"rootDir": "./src",
"allowJs": false,
"checkJs": false,
"skipLibCheck": true,
"allowSyntheticDefaultImports": true,
"experimentalDecorators": true,
"isolatedModules": true,
"module": "es2020",
"noImplicitAny": true,
"declaration": true,
"declarationMap": true,
"declarationDir": "dist",
"outDir": "dist",
"noImplicitThis": true,
"noUnusedLocals": true,
"noUnusedParameters": true,
"removeComments": false,
"preserveConstEnums": true,
"moduleResolution": "node",
"emitDecoratorMetadata": false,
"sourceMap": true,
"target": "ES2021",
"strictNullChecks": true,
"lib": [
"es2020",
"esnext",
"dom"
]
},
"include": [
"src/**/*"
],
"exclude": [
"node_modules",
"**/*.spec.ts",
"dist"
]
}

+ 10
- 0
plugins/gaussian-splatting/typedoc.json Näytä tiedosto

@@ -0,0 +1,10 @@
{
"extends": [
"../../typedoc.json"
],
"entryPoints": [
"src/index.ts"
],
"name": "Threepipe Gaussian Splatting Plugin",
"readme": "none"
}

+ 89
- 0
plugins/gaussian-splatting/vite.config.js Näytä tiedosto

@@ -0,0 +1,89 @@
import {defineConfig} from 'vite'
import json from '@rollup/plugin-json';
import dts from 'vite-plugin-dts'
import packageJson from './package.json';
import license from 'rollup-plugin-license';
import replace from '@rollup/plugin-replace';
import glsl from 'rollup-plugin-glsl';
import path from 'node:path';

const isProd = process.env.NODE_ENV === 'production'
const { name, version, author } = packageJson
const {main, module, browser} = packageJson

const globals = {
'three': 'threepipe', // just incase someone uses three
'threepipe': 'threepipe',
}

export default defineConfig({
optimizeDeps: {
exclude: ['uiconfig.js', 'ts-browser-helpers'],
},
base: '',
// define: {
// 'process.env': process.env
// },
build: {
sourcemap: true,
minify: false,
cssMinify: isProd,
cssCodeSplit: false,
watch: !isProd ? {
buildDelay: 1000,
} : null,
lib: {
entry: 'src/index.ts',
formats: isProd ? ['es', 'umd'] : ['es'],
name: name,
fileName: (format) => (format === 'umd' ? main : module).replace('dist/', ''),
},
outDir: 'dist',
emptyOutDir: isProd,
commonjsOptions: {
exclude: [/uiconfig.js/, /ts-browser-helpers/],
},
rollupOptions: {
output: {
// inlineDynamicImports: false,
globals,
},
external: Object.keys(globals),

},
},
plugins: [
isProd ? dts({tsconfigPath: './tsconfig.json'}) : null,
replace({
'from \'three\'': 'from \'threepipe\'',
delimiters: ['', ''],
}),
replace({
'process.env.NODE_ENV': JSON.stringify(isProd ? 'production' : 'development'),
preventAssignment: true,
}),
glsl({ // todo: minify glsl.
include: 'src/**/*.glsl',
}),
json(),
// postcss({
// modules: false,
// autoModules: true, // todo; issues with typescript import css, because inject is false
// inject: false,
// minimize: isProduction,
// // Or with custom options for `postcss-modules`
// }),
license({
banner: `
@license
${name} v${version}
Copyright 2022<%= moment().format('YYYY') > 2022 ? '-' + moment().format('YYYY') : null %> ${author}
${packageJson.license} License
`,
thirdParty: {
output: path.join(__dirname, 'dist', 'dependencies.txt'),
includePrivate: true, // Default is false.
},
}),
],
})

Write Preview
Loading…
Peruuta
Tallenna