As underlined last time, ￼there are three layers in an ARKit application: Rendering, Tracking and Scene Understanding. Last time we analyzed in great detail how Rendering is done in Metal using a custom view. ARKit uses Visual Inertial Odometry for accurate Tracking of the world around it and to combine camera sensor data with CoreMotion data. No additional calibration is necessary for image stability while we are in motion. In this article we look at Scene Understanding - ways of describing scene attributes by using plane detection, hit-testing and light estimation. ARKit can analyze the scene presented by the camera view and find horizontal planes such as floors. First, we need to enable the plane detection feature (which is off by default) by simply adding one more line before running the session configuration:
Note that only horizontal plane detection is possible with the current API version.
The ARSessionObserver protocol’s methods are used for handling session errors, tracking changes and interruptions:
However, there are other delegate methods that belong to the ARSessionDelegate protocol (which extends ARSessionObserver) that let us work with anchors. Put a print() call inside the first one:
Let’s move to the Renderer.swift file now. First, create a few class properties we need to work with. These variables will help us create and display a debug plane on the screen:
Next, in setupPipeline() we create the buffer:
We need to create new vertex and fragment functions for our plane, as well as new render pipeline and depth stencil states. Right before the line where the command queue is created, add these lines:
Next, in setupAssets() we need to create a new Model I/O plane mesh and then create the Metal mesh from it. At the end of the function add these lines:
Next, in updateBufferStates() we need to update the address of the buffer where the plane resides. Add the following lines:
Next, in updateAnchors() we need to update the transform matrices and the anchors count. Add the following lines before the loop:
Then, inside the loop replace the last three lines with the following lines:
We had to rotate the plane 90 degrees by the Z axis so we can make it horizontal. Notice that we used a custom method named rotationMatrix() so let’s define it. We have seen this matrix in the early articles when we first introduced 3D transforms:
Next, in drawAnchorGeometry() we need to make sure we have at least one anchor before drawing it. Replace the first line with this one:
Next, let’s finally create the drawDebugGeometry() function that draws our plane. It’s very similar to the anchor drawing function:
There is one more thing left to do in Renderer and that is - call this function in update() right above the line where we end the encoding:
Finally, let’s go to the Shaders.metal file. We need a new struct with just the vertex position passed via a vertex descriptor:
In the vertex shader we update the vertex position using the model-view matrix:
And last, in the fragment shader we give the plane a bold color to make it noticeable in the view:
If you run the app, you should be able to see a rectangle added when the app detects a plane, like this:
What we could do next is update/remove planes as we detect more or as we move away from the previously detected one. The other delegate methods can help us achieve just that. Then, we could look at collisions and physics. Just a thought for the future.
I want to thank Caroline for being the designated (plane) detective for this article! The source code is posted on Github as usual.