ARKit: What it is, How it is Done, and Why You Care

This post is based on a presentation I have been giving, with the goal of enabling companies to make more informed choices about tools to use in the creation of Augmented Reality (AR) experiences. ARKit, Apple’s recent addition to its iOS frameworks, is a good step in the adoption of AR for widespread use. As I expressed in a previous post, entitled “Why Should Your Company Choose an Augmented Reality or Virtual Reality Solution?“, the decision to use AR or VR depends on the requirements you define to meet your company’s goals. Similarly, if you decide on an AR solution, the decision to use ARKit depends on whether it is the correct tool for the AR experience you intend to implement. So let’s consider what ARKit is, how it is done, and why you care!

What is ARKit

Maybe, before that, we should make sure we are all on the same page, and ask …

What is AR?

  • AR creates the illusion that virtual objects are placed into the physical world, based on what your camera sees
  • As opposed to VR, which creates the illusion that the user is placed a virtual world, which might be a simulated world or a 360 video, with or without user interaction

The user of an AR experience will continue to see the real world. The AR application will augment that world with objects that must appear to remain where they are placed in that real world.

So then, what is ARKit?

ARKit is a high level API for developing AR on mobile devices

  • Well, if they are made by Apple
  • And they have a recent enough processor: iOS devices with A9 chips or later, including iPhone 6S or later, iPad Pro, iPad 2017 or later
  • And if they are running the iOS 11 operating system, or later

Indeed, those really are not very restrictive requirements, as Apple users do tend to stay fairly up to date with models and operating systems. Those who do not are also not likely to be the users that are chomping at the bit to try AR. That said, if part of your goal is to introduce the AR experience to those who might not even be looking for it, then this is something to be considered.

ARKit Supports 6 Degrees of Freedom

In AR implementations that provide only 3 degrees of freedom, virtual objects continue to appear properly placed when the orientation of your device rotates around its x, y, and z axes, also known as pitch, yaw and roll. With an additional 3 degrees of freedom, placement will also remain accurate when your device changes position in the real world, moving forward or back, up or down, and left or right. This allows the user to get closer to, and even move around the virtual objects. Six degrees of freedom is not unique to ARKit, but it is an important improvement to an AR experience and is one of the things ARKit handles very well.

How Is It Done?

 When I present this topic live, I typically give a real-time demo at this point, building a very basic, and then a slightly more involved AR app using ARKit. In the context of this post, I will stick to

  • How is it done by ARKit

and post future articles about

  • How is it done by your team!

Please follow me to see future posts, including video demos

ARKit accomplishes 6 Degrees of Freedom using Tracking, Scene Understanding, and Rendering, which work as follows.

Tracking

ARKit’s tracking includes

  • World tracking

Real time tracking of the device’s relative position in the real world. ARKit uses data based on measurements made by the camera, from pixels to points in the real world. The system notes changes in distance from frame to frame, and uses that to estimate movement of the device

  • Visual Inertial Odometry (VIO)

ARkit also gathers data using Visual Inertial Odometry.

Huh?

Let me explain further, with the help of some definitions I found on the web

vis·u·al

adjective: visual

  1. relating to seeing or sight.
  2. “visual perception”

 

in·er·tial

adjective: inertial

  1. relating to or arising from inertia.

(of navigation or guidance) depending on internal instruments that measure a craft’s acceleration and compare the calculated position with stored data.

That is clearly a nautical or aeronautic reference. The internal instruments ARKit uses for VIO are the accelerometer and gyroscope in our phone or iPad, which together are called the Inertial Measurement Unit or (IMU)

So finally, I looked up Odometry.

Sweet … but then I did find this pertinent description:

Odometry is the use of data from motion sensors to estimate change in position over time

That description goes on to explain its use in robotics, to estimate the position of moving robots, relative to a starting location. That same concept is incorporated into the Visual Inertial Odometry that powers the tracking in ARKit

So we have World Tracking and VIO, and in fact your pose is calculated both ways, and combined using a Kalman Filter, which determines which is the better estimate of your “real” position and provides that to the ARKit SDK. In using ARKit, you don’t need to know the complexities of a Kalman Filter. ARKit will do a good job at making the best choice it can, even if

  • the World Tracking is unable to get constant updates, perhaps because the camera is temporarily blocked
  • or the VIO is unable to get recent enough updates, perhaps because the system is otherwise taxed or there was an interruption

Scene Understanding

Another key ARKit feature is Scene understanding, which includes

  • Plane detection

ARKit’s ability to detect horizontal planes and create anchor points based on those planes to which you can anchor objects, or to which objects can react using physics

  • Hit Testing

Allows you to get an interaction with the real world, so that, for instance, the user can tap the screen and you can detect which virtual object the user was intending to “tap”. That can even include tapping on a detected plane. Then some behavior can be programmed, based on that tap

  • Light estimation

Correctly lights your virtual geometry, so it better matches the current lighting of the scene. For instance, if you have a virtual object that was created in a bright light that is appropriate for a sunlit room, its appearance will be adjusted, when rendered in the same room at night, under an overhead lamp.

 

Rendering

 Rendering in ARKit is done using some long existing iOS rendering frameworks, with some added classes to support ARKit.

  • SceneKit

A popular Scene and Node based interface for building worlds. It is the most natural rendering framework for the type of development I expect to see in most ARKit apps

  • SpriteKit

Used for 2D sprite based rendering. AR apps that add virtual content to otherwise flat worlds, rendered only at the screen level, might be better served using this framework

  • Metal

Apple’s GPU based graphics rendering. Apps that need to maximize the graphics and the computing potential of the devices might be best served using Metal

Using Tracking, Scene Understanding and Rendering, ARKit allows you to create AR apps that work great on iOS devices running iOS11 on most iPhones in use today.

Why Do You Care?

 

  • Apple cares

I don’t mean that in a Kool-Aid drinking, Apple fanboy/girl way. Apple is a pervasive force and thought leader. The priority they are giving ARKit is helping grow recognition and adoption in the space. Apple’s app ecosystem is mature and effective, and there are many Apple device users, who do tend to keep up to date with the latest hardware and software

  • Google cares

It didn’t take long for Google to reveal their own software-based AR solution: ARCore, which I will cover in a future article. Google’s quick follow-up demonstrates that they also see the value in a solution that doesn’t require special hardware configurations, and that they recognize the importance Apple is giving it. Competition is good, and of course Google’s solution also creates possibilities for Android implementations

  • Unity cares

There is already a Unity plugin for ARKit, with which I have had some early success. I am eager to experiment further, and to share demos with you

and if those titans of the industry aren’t enough:

  • Preferred App cares

For those who aren’t paying attention: That’s us! The fact that we care doesn’t necessarily mean it’s the right choice, but it is one of the choices, and the current spotlight holder!

Please follow us to read more articles about other choices, and to read our views on how they compare and contrast

So … Do You Care?

I kept this post at a level that would be general enough for company decision makers and content creators to understand the key points that should be considered. I provided some starting points from which you can further your research, to support your analysis on a level that’s appropriate for your interest and involvement. Preferred App would be happy to offer guidance and assist with the assessment of how ARKit, or other AR or VR solutions can fit into your business landscape.

Howard Olah-Reiken is president of Preferred App Consulting, where we advise and mentor companies looking to incorporate Augmented Reality, Virtual Reality, Digital eyewear and other emerging technology into their enterprise. We build prototypes and lead engineering teams in solution development. Howard, a member of the VR/AR Association Enterprise Committee, works actively to build the AR/VR community.