webxr-ar-vr-expert · diff
git:20260907.57a4b90 to git:20260912.c3d58ff
103 added, 64 removed. Audit A to A.
---
name: webxr-ar-vr-expert
- description: "Expert guide for WebXR (Web-based Virtual and Augmented Reality) development using Babylon.js and Three.js. Covers device compatibility, immersive sessions, controllers, and hit-testing."
- author: "Roedy Rustam"
+ description: "Expert guide for WebXR spatial computing, Web-based Virtual Reality (VR), Mixed Reality (MR), and Augmented Reality (AR) using Babylon.js and Three.js. Covers Apple Vision Pro, Meta Quest 3, hand tracking, and hit-testing / Panduan ahli WebXR spatial computing (VR/AR/MR)."
+ author: "vibes-plug-swarm"
---
- # WebXR AR/VR Expert
+ # WebXR Spatial Computing Expert (VR, AR & MR)
[English](#english) | [Bahasa Indonesia](#bahasa-indonesia)
---
<a name="english"></a>
## English
### Orchestration & Integration
- Connects and orchestrates with relevant domain skills like `brainstorming`, `zero-to-prod-orchestrator`, and `project-context-mapper` to ensure cohesive execution.
+ Connects and orchestrates with relevant domain skills:
+ - `web-3d-graphics-expert` — WebGPU/WebGL 3D scene rendering, lighting, and model optimization.
+ - `web-game-engine-expert` — Spatial physics, grabbing interactions, and fixed-timestep motion.
+ - `apple-ecosystem-expert` — Apple Vision Pro visionOS Safari WebXR features and spatial guidelines.
+ - `glsl-shader-expert` — Passthrough blending and spatial holographic shaders.
+ - `performance-web-vitals` — Maintaining rock-solid 72/90/120 FPS to prevent simulator sickness.
### Description
- Production-grade guidance for building immersive **Augmented Reality (AR)** and **Virtual Reality (VR)** experiences in the browser using the **WebXR Device API**. Focuses on implementation via **Babylon.js** (which has industry-leading WebXR support) and **Three.js** / React Three Fiber.
+ Production-grade guidance for architecting immersive **Virtual Reality (VR)**, **Mixed Reality (MR)**, and **Augmented Reality (AR)** in the browser via the **WebXR Device API**. Covers modern head-mounted displays (**Meta Quest 3/3S**, **Apple Vision Pro**) and mobile AR, focusing on Babylon.js and Three.js with `@react-three/xr`.
### Trigger Conditions
Activate this skill when the user is:
- - Building AR/VR experiences for Oculus/Meta Quest, Apple Vision Pro, or mobile AR devices.
- - Setting up an `immersive-vr` or `immersive-ar` WebXR session.
- - Implementing controller tracking, hand tracking, or teleportation in 3D space.
- - Using AR Hit-Testing to place digital objects on physical surfaces (floors/tables).
+ - Developing spatial computing web applications or immersive web experiences.
+ - Setting up WebXR sessions (`immersive-vr`, `immersive-ar`).
+ - Implementing Hand Tracking (joint poses, pinch gestures) or spatial controller interactions.
+ - Adding Mixed Reality passthrough or depth sensing.
+ - Implementing AR Hit-Testing to snap 3D models onto real-world planes (floors, walls, tabletops).
+ - Optimizing XR framerates (fixed foveated rendering) to eliminate motion sickness.
- ### Core Concepts
+ ---
- #### 1. Babylon.js WebXR Setup
- Babylon.js provides the most robust default WebXR experience.
+ ### Key Capabilities & Headset Targets (2026)
+
+ | Target Platform | Session Mode | Key Features |
+ | :--- | :--- | :--- |
+ | **Meta Quest 3 / 3S** | `immersive-vr`, `immersive-ar` | Full-color stereo passthrough, high-precision hand tracking, fixed foveation levels 0-3. |
+ | **Apple Vision Pro (visionOS)** | `immersive-vr`, `immersive-ar` | Eye-gaze + pinch interaction, 90/120Hz display refresh, strict privacy hand tracking. |
+ | **Mobile AR (Android / iOS WebKit)** | `immersive-ar` | Surface hit-testing, ambient light estimation, instant plane detection. |
+
+ ---
+
+ ### 1. Babylon.js WebXR Setup (Gold Standard)
+
+ Babylon.js provides the most comprehensive out-of-the-box WebXR implementation with integrated teleportation, controller profiles, and physics:
+
```typescript
- import { Scene, FreeCamera, Vector3, Engine } from '@babylonjs/core';
+ import { Scene, WebXRExperienceHelper, WebXRFeatureName } from '@babylonjs/core';
- async function setupXR(scene: Scene) {
- // Initialize default WebXR experience
- // This automatically adds the "Enter VR/AR" button, teleportation, and controller models
+ export async function initBabylonXR(scene: Scene) {
const xr = await scene.createDefaultXRExperienceAsync({
uiOptions: {
- sessionMode: 'immersive-vr', // or 'immersive-ar'
+ sessionMode: 'immersive-ar', // Fallback to 'immersive-vr' if AR is unsupported
+ referenceSpaceType: 'local-floor'
},
- optionalFeatures: true
- });
-
- // Example: Listen for controller input
- xr.input.onControllerAddedObservable.add((controller) => {
- controller.onMotionControllerInitObservable.add((motionController) => {
- // Handle button presses
- });
+ optionalFeatures: ['hit-test', 'hand-tracking', 'anchors', 'plane-detection']
});
+
+ // Enable Fixed Foveated Rendering to reclaim 20-30% GPU performance on Quest
+ const foveation = xr.baseExperience.featuresManager.enableFeature(
+ WebXRFeatureName.FOVEATION,
+ 'latest',
+ { foveationLevel: 2 }
+ );
+
+ // Listen for Hand Tracking
+ const handTracking = xr.baseExperience.featuresManager.enableFeature(
+ WebXRFeatureName.HAND_TRACKING,
+ 'latest',
+ { xrInput: xr.input }
+ );
+
+ return xr;
}
```
- #### 2. Three.js / React Three Fiber Setup
- In R3F, use `@react-three/xr`.
+ ---
+
+ ### 2. React Three Fiber Setup (`@react-three/xr`)
+
```tsx
import { Canvas } from '@react-three/fiber';
- import { XR, createXRStore } from '@react-three/xr';
+ import { XR, createXRStore, Hands, Controllers } from '@react-three/xr';
- const store = createXRStore();
+ const store = createXRStore({
+ foveation: 1,
+ handTracking: true
+ });
- export default function App() {
+ export function SpatialApp() {
return (
<>
- <button onClick={() => store.enterVR()}>Enter VR</button>
+ <button className="xr-btn" onClick={() => store.enterAR()}>
+ Enter Mixed Reality
+ </button>
<Canvas>
<XR store={store}>
- <mesh>
- <boxGeometry />
- <meshBasicMaterial color="red" />
+ <ambientLight intensity={0.8} />
+ <directionalLight position={[5, 10, 5]} />
+ <Controllers />
+ <Hands />
+ {/* Spatial 3D Objects */}
+ <mesh position={[0, 1.2, -1]}>
+ <boxGeometry args={[0.3, 0.3, 0.3]} />
+ <meshStandardMaterial color="royalblue" />
</mesh>
</XR>
</Canvas>
</>
);
}
```
- #### 3. AR Hit-Testing
- In AR mode, "Hit-Testing" allows you to detect real-world surfaces.
- - Ensure `immersive-ar` is used.
- - Request the `hit-test` feature when initializing the session.
- - Use the hit-test results (a matrix/position) to place a reticle or object.
-
- ### Best Practices
- - **Performance is Critical:** VR requires a consistent 72 to 90 FPS (or 120 FPS on Apple Vision Pro). Dropped frames cause motion sickness.
- - **Polycount:** Keep polygon counts and draw calls low.
- - **UI in XR:** Standard HTML UI will not work in immersive mode. You must use 3D UI (e.g., Babylon's GUI 3D system or Three.js spatial HTML layers if supported).
-
---
- ### Integration with Other Skills (MANDATORY)
- This skill works best when combined with:
- - `web-3d-graphics-expert` — For building the 3D scene that XR will visualize.
- - `apple-ecosystem-expert` — For optimizing WebXR compatibility with iOS (Safari) and visionOS.
- - `web-game-engine-expert` — For adding physics to VR interactions (e.g., throwing objects).
-
- ### Referenced By Orchestrators (MANDATORY)
- - `brainstorming` — Add to "3D & Graphics" or "Spatial Computing".
- - `zero-to-prod-orchestrator` — Phase 5 (Frontend / XR Implementation).
+ ### Best Practices for Spatial Performance
+ 1. **Rock-Solid Framerates:** VR requires stable 72, 90, or 120 FPS. Any frame drop causes instant vestibular nausea.
+ 2. **Fixed Foveated Rendering (FFR):** Reduces fragment shader resolution in the user's peripheral vision, freeing substantial GPU overhead.
+ 3. **No 2D DOM Overlays in VR:** Standard HTML elements cannot render inside an immersive session. Render spatial 3D UI panels using world-space 3D meshes or canvas textures.
---
<a name="bahasa-indonesia"></a>
## Bahasa Indonesia
### Integrasi Orkestrasi
- Terhubung dan mengorkestrasi skill domain yang relevan seperti `brainstorming`, `zero-to-prod-orchestrator`, dan `project-context-mapper` untuk memastikan eksekusi yang kohesif.
+ Terhubung dan mengorkestrasi skill domain yang relevan:
+ - `web-3d-graphics-expert` — Rendering scene 3D WebGPU/WebGL, pencahayaan, dan optimasi model.
+ - `web-game-engine-expert` — Fisika spasial, interaksi menggenggam objek, dan pergerakan fixed-timestep.
+ - `apple-ecosystem-expert` — Fitur WebXR Safari di Apple Vision Pro visionOS dan pedoman spasial.
+ - `glsl-shader-expert` — Shader hologram dan blending passthrough AR.
+ - `performance-web-vitals` — Menjaga kestabilan 72/90/120 FPS demi mencegah *motion sickness*.
### Deskripsi
- Panduan ahli untuk pengembangan *Virtual Reality* (VR) dan *Augmented Reality* (AR) di web menggunakan **WebXR API**. Difokuskan pada implementasi menggunakan **Babylon.js** dan **Three.js** untuk ekosistem *Spatial Computing*.
+ Panduan produksi untuk membangun aplikasi spasial imersif **Virtual Reality (VR)**, **Mixed Reality (MR)**, dan **Augmented Reality (AR)** di browser melalui **WebXR Device API**. Mencakup headset modern (**Meta Quest 3/3S**, **Apple Vision Pro**) serta perangkat mobile AR.
### Kondisi Pemicu
- - Mengembangkan aplikasi VR/AR untuk Meta Quest, Apple Vision Pro, atau Mobile AR.
- - Mengimplementasikan pelacakan pengontrol (controller), pelacakan tangan (hand tracking), atau teleportasi.
- - Menggunakan AR Hit-Testing untuk meletakkan objek 3D di permukaan dunia nyata.
+ Aktifkan skill ini ketika pengguna:
+ - Mengembangkan aplikasi web *Spatial Computing* (VR/AR/MR).
+ - Menginisialisasi sesi WebXR (`immersive-vr`, `immersive-ar`).
+ - Mengimplementasikan *Hand Tracking* (pelacakan sendi tangan, gerakan cubit/pinch) atau kontroler spasial.
+ - Mengaktifkan *Passthrough* warna Mixed Reality.
+ - Menggunakan AR *Hit-Testing* untuk menempatkan objek 3D di lantai atau meja fisik.
+ - Mengoptimalkan performa VR dengan *Fixed Foveated Rendering* (FFR).
- ### Panduan Singkat
- - **Performa Mutlak:** Jaga FPS tetap stabil di angka 72-90 FPS. *Frame drop* pada VR menyebabkan rasa mual (motion sickness) pada pengguna. Batasi jumlah poligon dan *draw calls*.
- - **Babylon.js sebagai Prioritas:** Untuk proyek XR berskala besar, prioritaskan Babylon.js karena dukungan bawaannya untuk interaksi VR, fisika, dan UI 3D jauh lebih matang dibanding library lain.
- - **UI Spasial:** Ingat bahwa elemen HTML biasa tidak muncul di dalam sesi `immersive-vr` atau `immersive-ar`. Anda harus membuat UI berupa objek 3D (seperti panel yang melayang di udara).
- - **Hit-Testing:** Untuk AR, gunakan fitur *Hit-Test* yang disediakan WebXR API untuk mengenali lantai atau meja sebagai tempat menaruh objek.
+ ---
+
+ ### Panduan Inti Pengembangan Spasial
+
+ 1. **Stabilitas Frame Rate Mutlak**:
+ - Targetkan 72–90 FPS tanpa toleransi *stutter*. Pada kacamata VR, keterlambatan frame (*latency hitch*) langsung memicu rasa mual (*motion sickness*).
+ 2. **Fixed Foveated Rendering (FFR)**:
+ - Aktifkan fitur foveation pada Meta Quest untuk menurunkan resolusi tepi lensa (periferi mata), menghemat 20-30% daya GPU.
+ 3. **Antarmuka 3D Spasial (Bukan HTML DOM)**:
+ - Elemen HTML standar tidak dapat ditampilkan di dalam sesi imersif. Buat UI berupa panel 3D melayang (*world-space meshes*).
+ 4. **Hit-Testing untuk AR**:
+ - Manfaatkan *Hit-Test API* bawaan WebXR untuk mendeteksi bidang nyata permukaan meja atau lantai secara instan sebelum meletakkan objek 3D.