What’s ARKit?

ARKit is a new framework in Apple’s iOS11 that allows programmers to deliver a real-time on-screen augmented reality (AR) view of the world for iPhone and iPad. This AR view is delivered to the user by using the device’s camera and sensors via a system known as Visual Inertial Odometry or “VIO.” VIO maps the space around a user using the camera, gyroscopes and accelerometers in such a way that the resulting measurements create a virtual playfield upon which almost any kind of content can be displayed and interacted with.

ARKit can be used to create very simple apps that help you measure a space or an object by pointing and tapping on the screen to mark off edges of the room or the object being measured (the results are accurate to within millimeters - which is very impressive); it can also present a surprisingly complex interactive experience from first person shooters, to racing simulations, virtual portals you can literally walk through, virtual tabletop games, scientific simulations, and much, much more.

As amazing as the capabilities are out of the box, there is one critical element of AR that Apple doesn’t address with their framework… how, exactly, do you get data and content into your AR app? Except for the most simplistic of AR simulations, apps with depth, that deliver great experiences will need a system to provide a continuous flow of information to power their AR activities. Simply put, augmented reality needs a data layer.

So, What’s a Data Layer?

AR apps are driven by data. For simple apps like a virtual board game this isn’t too much of an issue as you can probably install a collection of static data into your app at build time. But, what if your application needs to be out in the real world? What if your app needs to respond to the changing environment, load real-time data, or load new or updated scenes in response to the storyline of your app? Or, an even harder question for all mobile apps: what if your app needs to keep working interactively offline if your user’s internet connection is unavailable? If any of these are true, your AR app needs a reliable data layer to keep the information flowing.

Fortunately, the Realm Platform is a cross platform system for delivering exactly this service. Using RMP your application can operate in a mode where your users get application data synchronized transparently and without any complicated programmatic machinery revolving around the serialization and deserialization of object data – this is the AR “data layer.”

Automatic Data Sync, Reactive Programming

Before we get into the details of how RMP will make the care and feeding of AR apps easier, let’s look at a schematic of what we’re talking about from 3 perspectives:

• A simple self-contained app:

As we can see clearly the self contained AR app is certainly very simple; all of its content is embedded - it’s also a dead-end from the point of expandability. Every time an update to the content is needed a new version of the app must be produced/released

• A traditional full stack SQL or SQLite based app:

The SQL based app has more flexibility- you can query a back end to gather rows from a DB, but at the end of the day the machinery involved in moving data around – to say nothing of data model management – on multiple platforms – is quite expensive. The infrastructure required to make this possible is usually very extensive, and, unfortunately very brittle. The smallest changes in the stack, the models or data formats can cascade through the application causing issues for both users and developers.

• An RMP-based app:

Finally, the RMP based app enables dynamic updating of not only the main app data, but allows for easy customization of data delivered down to the specific user. We should point out that this works for both iOS and Android because the design of the Realm programming model is that developers interact with native platform objects, not XML, JSON, or SQL. This means that not only is software development easier, but updates can be bidirectional if required merely by updating the objects. The Realm Object Server takes care of propagating updates across the system, managing distribution, data format conversion (for example from iOS to Android and vice versa) as well as conflict management and resolution and even data schema changes as you evolve your application’s data models.

Powering up AR with Services integration

What if, for example, if your AR application needs to make use of a real-time weather service to be able to provide a deep/compelling experience? Realm’s Enterprise Edition employs a system called global event handlers that can be employed to integrate the Realm Object Server with any external service. You could deploy an event handler to observe the user’s location, and then connect to a weather API to integrate the data directly into your AR experience and have those updates appear in your user’s copy of the app in real time.

The API integration isn’t just one way either - using Realm’s Data Connectors, changes to the Realm Database on a user’s device can effect changes to and accept changes from legacy databases like Oracle, DB2 and MongoDB as well.

How Simple Can We Make It?

Of course, every AR app will be different, but let’s look at an interesting but non-trivial example. Say we we wanted to make a simple AR app on iOS that just showed Yelp reviews in mid-air in front of every restaurant the user might pass by …how much code are we talking about…? Let’s start with the data model:

import RealmSwift

class ARPlace : Object {
    @objc dynamic var latitude = 0.0
    @objc dynamic var longitude = 0.0
    @objc dynamic var elevation = 0.0
    @objc dynamic var bearing = 0.0
    @objc dynamic var activationRadius = 10.0 // radius in meters around which the app should notice & do something with/about this place 
    @objc dynamic var deviceLookAngle = 0.0 // direction the camera needs to be facing to see this object
    @objc dynamic var longitude = “”
    @objc dynamic var placeName = “”	// e.g.,Joe’s Diner
    @objc dynamic var placeDescription = “” // e.g., A down-home meat-n-potatoes place
    @objc dynamic var reviewURL= “”	// URL to load into our magic floating AR browser
}

Here we define a simple structure that contains location data and simple metadata about what can be found at a given location, and that’s it! It captures everything we need to know and provides a link to get more info that we will later display in our augmented reality view.

On the Realm Object Server side you’d have a small script that watches for connected users (i.e., users running your app), and looks for changes in their location, and then copies ARPlace records into a Realm on their device that represents restaurants within 100 meters of their current location. The whole script is just a few dozen lines, but can be summarized in pseudocode as follows:

repeat (forever) {
watchforChangesInUserRealms {
    if aUser.userLocationChange > 100meters {
      nearByReataurants = askYelpForRestaurantsNear(usersLatLon)
      populateUsersRealmWith(nearByReataurants)//*see note
    }
}

The mere act of creating new rows in the user’s realm will cause those data to be updated on the user’s device with no other action required on the programmer’s part. These objects are, as previously noted native objects so there is no parsing of JSON or XML to contend with. Of course (as per our “note” above), if the user’s Realm already contains some of the records you just retrieved you can update them or not depending on your business requirements.

But, you get the picture: Getting data of any kind from your back-end system out to your users is the same as writing it to the Realm. You can do this writing of data preemptively too - if your app has access (for example) to the user’s calendar your app could “know” they will be near a given location at a certain date/time and could pre-populate their ARPlaces data so your app is all set and ready to provide a great experience. This is also perfect for “offline first” applications where you want to ensure your user has access to important data no matter how sketchy their mobile data connection might be.

How does this look on the client side? Super simple:

You open the Realm, make a query to get some records and register a notification token so Realm will let you know when there are changes to the result set for the records you asked for.

It can all be done in fewer than 30 lines:

import RealmSwift

let currentLocation = <current location via CoreLocation>
let myRealmConfig = Realm.config(<username/password or other authtoken>)
let myRealm = try! Realm(myRealmConfig)
let (nortLatOffset, southLatOffset, eastLonOffset, westLonOffset = getBoxAroundLocation(currentLocation, radius: 100)
Let searchPredicate = NSPredicate("lat < %f AND lat > %f AND lon < %f AND lon  %f”, nortLatOffset, southLatOffset, eastLonOffset, westLonOffset)
var nearbyPlaces = myRealm.objects(ARPlace.self).filter(searchPredicate)

This code at the top of your view controller sets up all you need to get continuous updates to data as your user moves around. The last bit we need is a way to be notified as the result set changes so we can update our view. Fortunately Realm makes this easy too:

let notificationToken = self.nearbyPlaces?.addNotificationBlock { [weak self] (changes: RealmCollectionChange) in
     switch changes {
     case .initial:// first time through - display our data
		break
     case .update(_, let deletions, let insertions, let modifications):
// you get a list of each … modify your view as needed
  	   break
     case .error(let error):
         break
            }
 } // of notification handler

The only thing we’ve left out of this example is a small function (getBoxAroundLocation) to calculate the 100 meter boundary around the user’s location (which is external to the use of Realm, but we’ll show you how we did that in our upcoming webinar). Everything else is in support of your app displaying and/or reacting to changes to the data in your Realm. All of the data you would need in order to display these restaurant reviews in your AR scene automatically appears in the sync’d Realm.

Wait Just a Second.. Couldn’t I Do That With SQLite?

Well, sure you could… but you’d be doing it all by-hand: pushing around JSON or XML blobs that would have to be serialized, deserialized, converted back into native Swift objects. You’d also be pulling this data rather than it just showing up at the right time whenever your user needed it. This would cause all sorts of unnecessary network traffic to say nothing of extra latency at every single step, degrading the performance of your app. “Offline first” operation would be off the table as there would be no underlying object and data management framework work with.

And, finally, if you upgrade your app and change the data model, well… you’d still have to support all your old code and SQL queries and ensuring it all still works on all possible configurations of your end-user’s devices as well as older versions of your app… it would be complicated.

With the Realm Platform, all the data marshalling, copying, bi-directional data synchronization, offline-first support, conflict resolution, model updating, etc happens transparently: You open your Realm, register a listener and are notified when changes happen. Done.

Plot Twist: Realm <3 ARCore Too!

Did we mention this is cross-platform? Apple’s ARKit may be getting the lion’s share of the press coverage since Apple’s World-Wide Developer’s Conference, but Google hasn’t been asleep at the wheel: In early September Google announced their own Augmented Reality framework called ARCore. ARCore is very similar to ARKit; it provides all the visual and spatial/environment management info and programming hooks needed to build similar AR experiences but adds in Google’s “Project Tango” sensor systems into the mix. (Project Tango is a hardware beacon system – similar to Apple’s iBeacons – which can be used to enhance the AR experience by providing even finer degrees of user position tracking.)

The great news is that since Realm is platform agnostic the Realm Platform can help you deliver a top-shelf unified AR experience by delivering a cross platform data layer: One Realm that can seamlessly deliver data to both iOS and Android Augmented Reality applications simultaneously.

Wanna See It in Action? Tune in to the Realm AR Webinar

This article gives just a small taste of what’s possible when you mix Augmented Reality and Realm - stay tuned for an upcoming Realm webinar where we’ll go into more depth of how you can power your AR experiences. We’ll cover design principles, event handling with Realm Object Server, integration of 3rd party APIs, observing user locations in order to proactively load data and more!

Next Up: Getting Started with the Realm Mobile Platform - Guided Demo