Mixed Reality Capture module for WebXR and Three.js

Overview

Reality Mixer JS

This is a small Mixed Reality Capture module for WebXR + Three.js.

You can use this module to allow users to record your WebXR experiences in Mixed Reality.

Unlike the original Reality Mixer or LIV, the Mixed Reality composition is happening on the browser itself. However, users will need another external application to record their browser (e.g. OBS).

Keep in mind that this is still just a prototype and that I'm not a frontend developer. Feel free to open PRs and contribute to the project.

TO-DOs

  • Create a calibration page to allow users to calibrate the position and orientation of their camera, and to configure their chroma key (green screen).

How to test the example

  • Clone this repository.

  • Run npm ci to download the dependencies.

  • Run http-server to start the HTTP server (that can be downloaded by running npm install -g http-server).

  • WebXR and navigator.mediaDevices require HTTPS. You could use a tool like localtunnel for testing. You can run npm install -g localtunnel to download it and then you can run lt --port 8080 --subdomin 127.0.0.1 in a separate terminal.

  • Open your browser and navigate to https://{your HTTPS domain}/examples/webxr_vr_ballshooter.html

Your browser will ask for permission to access your camera, and it'll ask for permission to use your VR headset once you click on the WebXR button.

You'll need to edit the code (the calibration) to set the correct position, orientation, and fov of your camera, and the correct parameters for your green screen.

API

import * as THREE from 'three';
import * as MRC from 'reality-mixer';

let mixedRealityCapture;
let scene, renderer, camera;

// ...

const cameraCalibration = new MRC.CameraCalibration(
   1920, // width of the video
   1080, // height of the video
   38, // vertical field of view in degrees
   [0, 1.5, 0], // vector with the position of the camera in scene coordinates
   [0, 0, 0, 1] // quaternion with the orientation of the camera
);

const chromaKey = new MRC.ChromaKey(
   [0, 1, 0], // chroma key color (red, green, blue values from 0 to 1)
   0.25, // similarity (0 to 1)
   0 // smoothness (0 to 1)
);

const calibration = new MRC.Calibration(
   cameraCalibration,
   chromaKey,
   4, // Delay (in number of frames) between the real camera and the virtual camera
);

// ... Initialize your Three.js scene and renderer here ...

// Hide your renderer when you want to display the Mixed Reality output
renderer.domElement.style.display = "none";

// Create a new Mixed Reality Capture session
mixedRealityCapture = new MRC.MixedRealityCapture( calibration );

// Add the Mixed Reality Output to the document
document.body.appendChild( mixedRealityCapture.domElement );

// ...

// You should call this whenever the window resizes
mixedRealityCapture.onWindowResize();

// ...

// Render the Mixed Reality composition after rendering your scene

renderer.render( scene, camera );

mixedRealityCapture.render( renderer.xr, scene );

Alternatively, you can instantiate the calibration with a JSON provided by the user:

// ...

const json = `
{ 
    "schemaVersion": 1, 
    "camera": { 
        "width": 1280, 
        "height": 720, 
        "fov": 38, 
        "position": [0, 1.5, 0], 
        "orientation": [0, 0, 0, 1] 
    }, 
    "chromaKey": {
        "color": [0, 1, 0],
        "similarity": 0.25,
        "smoothness": 0
    },
    "delay": 4
}
`;

const calibrationData = JSON.parse( json );

const calibration = MRC.Calibration.fromData( calibrationData );

// ...

mixedRealityCapture = new MRC.MixedRealityCapture( calibration );

// ...
You might also like...

📷 Detects your face and adds filters from your webcam. You can capture and download images.

📷 Detects your face and adds filters from your webcam. You can capture and download images.

Snapchat Filters on WebCam 📷 Detects your face and adds filters from your webcam. You can capture and download images. 👉 Visit site 📌 Screenshots ?

Apr 27, 2022

Based on Google Chrome recorder, implement UI interface capture and notify the result to the target mailbox

Based on Google Chrome recorder, implement UI interface capture and notify the result to the target mailbox

chrome-recoder-crawler README-CN Modify the .js file exported by Google Chrome recorder. By default, the innerText property of the node operated in th

Oct 18, 2022

MagicCap is a image/GIF capture suite for Mac and Linux

MagicCap is a image/GIF capture suite for Mac and Linux

MagicCap is a image/GIF capture suite for Mac and Linux. You can get a precompiled copy from the releases page of this GitHub page.

Sep 15, 2022

This project is a web application that allows the user to select a location on a map, display the selected region, and capture an image of that region. The captured image is then applied as a texture to a 3D cuboid using BabylonJS.

Map-Babylon Client repo commit details https://github.com/hashmat-noorani/mapbox-babylon-client Server repo commit details https://github.com/hashmat

Mar 21, 2023

A quick capture plugin for Obsidian, all data from your daily notes.

A quick capture plugin for Obsidian, all data from your daily notes.

Obsidian Memos 中文文档 A new way for you to quick capture an idea in Obsidian. Which is highly based on the awesome open source project: memos and awesom

Jan 3, 2023

A social media platform aimed to capture the essence of all popular, existing social media platforms

A social media platform aimed to capture the essence of all popular, existing social media platforms

Social Fuel Reimagining Social Media, step by step 📌 About A social media platform aimed to capture the essence of all popular, existing social media

Feb 12, 2022

An application to capture screenshots automatically of your screen!

 An application to capture screenshots automatically of your screen!

TASCA : The Automatic Screenshot Capturing Application An application to capture screenshots automatically of your screen! Report Bug · Request Featur

Jan 27, 2022

A WebApp to make Obsidian Clippers or Bookmarklets - to capture webpages or data from the web

A WebApp to make Obsidian Clippers or Bookmarklets - to capture webpages or data from the web

A WebApp to make Obsidian Clippers or Bookmarklets - to capture webpages or data from the web

Dec 4, 2022

Create beautiful quotes that capture your attention.

Create beautiful quotes that capture your attention.

Features Easy text highlights via normal rich text formatting Yellow = Bold, Blue = Italic Export to png, jpeg, svg, or copy png to clipboard Supports

Dec 15, 2022
Owner
Fabio
Fabio
Hemsida för personer i Sverige som kan och vill erbjuda boende till människor på flykt

Getting Started with Create React App This project was bootstrapped with Create React App. Available Scripts In the project directory, you can run: np

null 4 May 3, 2022
Kurs-repo för kursen Webbserver och Databaser

Webbserver och databaser This repository is meant for CME students to access exercises and codealongs that happen throughout the course. I hope you wi

null 14 Jan 3, 2023
Idle Game Based on A Dark Room with bits of NGU Idle and others mixed in.

An-Empty-World Main Story: You wake up in a strange place, surrounded by debris. Unaware of how you got there, it becomes immediately aware that you'r

Andre Schoolman 2 Mar 26, 2022
Mixed Messages is a simple Node.js application, that will print a randomized fake fact to the terminal each time it is ran.

Mixed Messages - Fake Fact Generator Mixed Messages is a simple Node.js application, That will print a randomized fake fact to the terminal each time

Parietic 2 Jan 10, 2022
A framework for NodeJS deveopers bringing dreams to reality!

ELIXOR A framework for NodeJS enthusiasts! What does it do? Short Answer Elixor is a framework for NodeJS developers that lets you build cross-platfor

SKYLIX 9 Jun 25, 2022
GPU-accelerated Augmented Reality for the web.

MARTINS.js WebAR engine Create amazing Augmented Reality experiences with MARTINS.js, a GPU-accelerated Augmented Reality engine for the web. Get star

Alexandre Martins 19 Dec 16, 2022
Create parallel reality of your Substrate network.

Chopsticks Create parallel reality of your Substrate network. Install Make sure you have setup Rust environment (>= 1.64). Clone repository with submo

Acala Foundation 37 Dec 15, 2022
Text Engraving & Extrusion demo based on Three.js is implemented with Typescript and webpack5. Used THREE-CSGMesh as the core tech to achieve engraving and extrusion results

Text Engraving & Extrusion Text Engraving & Extrusion demo is implemented using Three.js, with Typescript and webpack5. Used THREE-CSGMesh as the core

Jiahong Li 3 Oct 12, 2022
three.js examples. if you are first in learning three.js , this will give you much help.

three-projected-material Three.js Material which lets you do Texture Projection on a 3d Model. Installation After having installed three.js, install i

null 22 Nov 2, 2022
An npm package for demonstration purposes using TypeScript to build for both the ECMAScript Module format (i.e. ESM or ES Module) and CommonJS Module format. It can be used in Node.js and browser applications.

An npm package for demonstration purposes using TypeScript to build for both the ECMAScript Module format (i.e. ESM or ES Module) and CommonJS Module format. It can be used in Node.js and browser applications.

Snyk Labs 57 Dec 28, 2022