Model I/O was introduced in 2015 for iOS 9 and OS X 10.11 and it is a framework that helps us create more realistic and interactive graphics. We can use it to import/export 3D assets, to describe lighting, materials and environments, to bake lights, to subdivide and voxelize meshes, and for physical based rendering. Model I/O easily integrates our assets with our code in various 3D APIs:
In order to import an asset we simply do:
var url = URL(string: "/Users/YourUsername/Desktop/imported.obj")
let asset = MDLAsset(url: url!)
To export an asset we do:
url = URL(string: "/Users/YourUsername/Desktop/exported.obj")
try! asset.export(to: url!)
Model I/O will save both the .obj file and an additional .mtl file that contains information about the object materials, such as in this example:
# Apple ModelI/O MTL File: exported.mtl
newmtl material_1
Kd 0.8 0.8 0.8
Ka 0 0 0
Ks 0 0 0
ao 0 0 0
subsurface 0 0 0
metallic 0 0 0
specularTint 0 0 0
roughness 0.9 0 0
anisotropicRotation 0 0 0
sheen 0.05 0 0
sheenTint 0 0 0
clearCoat 0 0 0
clearCoatGloss 0 0 0
Integrating Model I/O with Metal takes four steps:
Step 1: set up the render pipeline state
First we create a vertex descriptor so we can pass input to the vertex function. The vertex descriptor is needed to describe the vertex attribute inputs to a render state pipeline. We need 3 x 4 bytes for the vertex position, 4 x 1 byte for color, 2 x 2 bytes for the texture coordinates and 4 x 1 byte for ambient occlusion. At the end we tell the descriptor how large (24) our stride is in total:
let vertexDescriptor = MTLVertexDescriptor()
vertexDescriptor.attributes[0].offset = 0
vertexDescriptor.attributes[0].format = MTLVertexFormat.float3 // position
vertexDescriptor.attributes[1].offset = 12
vertexDescriptor.attributes[1].format = MTLVertexFormat.uChar4 // color
vertexDescriptor.attributes[2].offset = 16
vertexDescriptor.attributes[2].format = MTLVertexFormat.half2 // texture
vertexDescriptor.attributes[3].offset = 20
vertexDescriptor.attributes[3].format = MTLVertexFormat.float // occlusion
vertexDescriptor.layouts[0].stride = 24
let renderPipelineDescriptor = MTLRenderPipelineDescriptor()
renderPipelineDescriptor.vertexDescriptor = vertexDescriptor
let rps = device.newRenderPipelineStateWithDescriptor(renderPipelineDescriptor)
Step 2: set up the asset initialization
We need to also create a Model I/O vertex descriptor to describe the layout of the vertex attributes in a mesh. We use a model named Farmhouse.obj that also has a texture Farmhouse.png (both already added to the sample project for you):
let desc = MTKModelIOVertexDescriptorFromMetal(vertexDescriptor)
var attribute = desc.attributes[0] as! MDLVertexAttribute
attribute.name = MDLVertexAttributePosition
attribute = desc.attributes[1] as! MDLVertexAttribute
attribute.name = MDLVertexAttributeColor
attribute = desc.attributes[2] as! MDLVertexAttribute
attribute.name = MDLVertexAttributeTextureCoordinate
attribute = desc.attributes[3] as! MDLVertexAttribute
attribute.name = MDLVertexAttributeOcclusionValue
let mtkBufferAllocator = MTKMeshBufferAllocator(device: device!)
let url = Bundle.main.url(forResource: "Farmhouse", withExtension: "obj")
let asset = MDLAsset(url: url!, vertexDescriptor: desc, bufferAllocator: mtkBufferAllocator)
Next, we load the texture for our asset:
let loader = MTKTextureLoader(device: device)
let file = Bundle.main.path(forResource: "Farmhouse", ofType: "png")
let data = try Data(contentsOf: URL(fileURLWithPath: file))
let texture = try loader.newTexture(with: data, options: nil)
Step 3: set up MetalKit mesh and submesh objects
We are now creating the meshes and submeshes needed in the final, fourth step. We also compute the Ambient Occlusion, which is a measure of geometry obstruction, and it tells us how much of the ambient light actually reaches any given pixel or point of our object, and and how much of this light is blocked by surrounding meshes. Model I/O provides a UV mapper that creates a 2D texture and wraps it around the object’s 3D mesh. For every pixel in the texture we can compute the ambient occlusion value, which is just one extra float added for each vertex:
let mesh = asset.object(at: 0) as? MDLMesh
mesh.generateAmbientOcclusionVertexColors(withQuality: 1, attenuationFactor: 0.98, objectsToConsider: [mesh], vertexAttributeNamed: MDLVertexAttributeOcclusionValue)
let meshes = try MTKMesh.newMeshes(from: asset, device: device!, sourceMeshes: nil)
Step 4: set up Metal rendering and drawing of meshes
Finally, we configure the command encoder with the mesh data that it needs to draw:
let mesh = (meshes?.first)!
let vertexBuffer = mesh.vertexBuffers[0]
commandEncoder.setVertexBuffer(vertexBuffer.buffer, offset: vertexBuffer.offset, at: 0)
let submesh = mesh.submeshes.first!
commandEncoder.drawIndexedPrimitives(submesh.primitiveType, indexCount: submesh.indexCount, indexType: submesh.indexType, indexBuffer: submesh.indexBuffer.buffer, indexBufferOffset: submesh.indexBuffer.offset)
Next, we will work on our shader functions. First we set up our structs for the vertices and uniforms:
struct VertexIn {
float4 position [[attribute(0)]];
float4 color [[attribute(1)]];
float2 texCoords [[attribute(2)]];
float occlusion [[attribute(3)]];
};
struct VertexOut {
float4 position [[position]];
float4 color;
float2 texCoords;
float occlusion;
};
struct Uniforms {
float4x4 modelViewProjectionMatrix;
};
Notice that we are matching the information we set up in the vertex descriptor, with the VertexInstruct. For the vertex function, we use a [[stage_in]] attribute because we are passing per-vertex inputs as an argument to this function:
vertex VertexOut vertex_func(const VertexIn vertices [[stage_in]],
constant Uniforms &uniforms [[buffer(1)]],
uint vertexId [[vertex_id]])
{
float4x4 mvpMatrix = uniforms.modelViewProjectionMatrix;
float4 position = vertices.position;
VertexOut out;
out.position = mvpMatrix * position;
out.color = float4(1);
out.texCoords = vertices.texCoords;
out.occlusion = vertices.occlusion;
return out;
}
The fragment function reads the per-fragment inputs passed from the vertex function and also processes the texture we passed via the command encoder:
fragment half4 fragment_func(VertexOut fragments [[stage_in]],
texture2d<float> textures [[texture(0)]])
{
float4 baseColor = fragments.color;
return half4(baseColor);
}
If you run the playground, you will see this output image:
That’s a pretty dull white model. Let’s apply the ambient occlusion to it by replacing the last line in the fragment function with these lines:
float4 occlusion = fragments.occlusion;
return half4(baseColor * occlusion);
If you run the playground again, you will see this output image:
The ambient occlusion also seems a bit raw and that is because our model is quite flat, without any curves or surface irregularities which would give way more credit to the realism that the ambient occlusion brings. Next, let’s apply the texture. Replace the last line in the fragment function with these lines:
constexpr sampler samplers;
float4 texture = textures.sample(samplers, fragments.texCoords);
return half4(baseColor * texture);
If you run the playground again, you will see this output image:
The texture looks really great on this model, but it would look even more realistic if we brought the ambient occlusion back. Replace the last line in the fragment function with this line:
return half4(baseColor * occlusion * texture);
If you run the playground again, you will see this output image:
Not bad for a few lines of code, right? Model I/O is such a great framework for 3D graphics and game programmers. There are a couple of articles on the web about using Model I/O with SceneKit, however, I thought using it with Metal is even more interesting! The source code is posted on Github as usual.
Until next time!
Everything about the Metal API 