Augmented Object Intelligence

Can a vision-language model's implicit 3D spatial understanding, combined with depth-accurate AR anchoring via ARCore, deliver educationally meaningful knowledge overlays on arbitrary physical objects — without pre-training on those objects?

Three-layer architecture with clean separation of concerns:

• ARCore — spatial computation, raw depth sensing, ambient occlusion, 3D anchor placement, motion tracking
• Gemini Vision — semantic understanding, component hierarchy generation, UV position estimation for each component on the visible object surface
• Compose overlay — label rendering at projected anchor positions, drill-down interaction

The key interface: Gemini returns normalized UV coordinates [0,1] for each component's estimated position on the visible surface. ARCore converts those 2D coordinates to 3D world positions via depth-aware hit testing. Labels anchor to real geometry and occlude correctly behind real objects.

• No pre-training on specific objects — works on anything Gemini can identify
• Depth-accurate occlusion — virtual labels correctly hide behind real geometry
• Hierarchical drill-down — tap a component to go deeper into its structure
• AI-generated knowledge — not lookup tables, fully generated per object
• Related to XR-Objects (Google UIST 2024) but extends it with depth occlusion, hierarchical structure, and AI-generated content for arbitrary objects

Raw depth over smoothed depth — deliberate performance decision for mid and low-end device support. Smoothed depth has higher computational overhead; raw depth is noisier but the performance budget freed is worth it for real-time AR.

UV coordinate approach — Gemini has implicit 3D spatial understanding from video training data. It can estimate where on a visible object surface each component is located with sufficient accuracy for AR label placement. ARCore provides the metric precision that Gemini cannot.

ARCore depth visualization with custom shaders and ambient occlusion working on device. Gemini integration returning structured JSON with component hierarchy and UV coordinates working. UV → 3D anchor → Compose label pipeline in progress.

Technologies: Kotlin, ARCore Depth API, Gemini Vision API, Jetpack Compose, Android, OpenGL ES, GLSL shaders