visionOS 2 adds the ability to track 3D objects that app developers prespecify.
As well as improving hand tracking and enhancing scene understanding, visionOS 2 introduces 3D object tracking, an entirely new tracking capability.
These tracking improvements weren’t announced in the main WWDC24 keynote, but Apple engineers detailed them in developer sessions available online.
visionOS 2 Also Improves Apple Vision Pro’s Hand Tracking
visionOS 2 brings major improvements to hand tracking on Apple Vision Pro and expands the headset’s understanding of the environment around you.
Since launch Vision Pro has been able to track specified 2D images, such as markers and game boards. With visionOS 2 it can track 3D objects too.
The developer must provide tracked 3D objects as USDZ 3D models to the Create ML tool on their Mac before app compilation. Create ML will train a neural network for each object, which will be baked into the app.
That means users can’t provide their own arbitrary 3D objects, and tracking new objects would require an app update. Still, this could be a very useful feature for business and education use cases, or tracking common objects.
The system will provide a bounding box with scale and orientation, which developers can use to anchor virtual objects to tracked objects or even to augment them with 3D effects.
We’ll be very curious to see what visionOS developers end up building with this new capability, which opens up the ability to truly augment specific objects in your space, not just the geometry of the environment.
The first visionOS 2 beta is available to install already, and the stable release is set for fall later this year.
This article was originally published on uploadvr.com