ARCore Primer | Android System Design

ARCore is Google's platform for building augmented reality experiences on Android. It uses the device's camera and IMU to understand the physical world, allowing virtual 3D content to appear anchored in the real environment.

Three Foundational Capabilities

Motion tracking: tracks the device's position and orientation relative to the world using Visual Inertial Odometry (VIO)
Environmental understanding: detects flat surfaces (floor, table, walls) and estimates their boundaries
Light estimation: measures ambient light intensity and direction so virtual objects can be lit realistically

Sceneform vs ARSceneView (Alternatives)

Sceneform was deprecated. In 2024, the recommended approach is:

ARCore SDK directly for custom rendering (OpenGL / Vulkan)
io.github.sceneview:arsceneview (community maintained Sceneform successor) for simpler 3D object placement

Basic ARCore Session

// implementation("com.google.ar:core:1.43.0")

class ArActivity : AppCompatActivity() {
    private var arSession: Session? = null
    private lateinit var surfaceView: GLSurfaceView

    override fun onResume() {
        super.onResume()

        // Check ARCore availability
        when (ArCoreApk.getInstance().checkAvailability(this)) {
            ArCoreApk.Availability.SUPPORTED_INSTALLED -> createSession()
            ArCoreApk.Availability.SUPPORTED_NOT_INSTALLED,
            ArCoreApk.Availability.SUPPORTED_APK_TOO_OLD -> {
                // Prompt user to install/update ARCore via Play Store
                ArCoreApk.getInstance().requestInstall(this, true)
            }
            else -> {
                Toast.makeText(this, "ARCore not supported", Toast.LENGTH_LONG).show()
                finish()
            }
        }
    }

    private fun createSession() {
        try {
            val session = Session(this)
            val config = Config(session).apply {
                planeFindingMode = Config.PlaneFindingMode.HORIZONTAL_AND_VERTICAL
                lightEstimationMode = Config.LightEstimationMode.ENVIRONMENTAL_HDR
                updateMode = Config.UpdateMode.LATEST_CAMERA_IMAGE
            }
            session.configure(config)
            arSession = session
            session.resume()
        } catch (e: UnavailableException) {
            handleArException(e)
        }
    }

    override fun onPause() {
        super.onPause()
        arSession?.pause()
    }
}

Per-Frame Processing

The AR update loop runs every frame in the GL thread:

// Called from GLSurfaceView.Renderer.onDrawFrame()
fun onDrawFrame() {
    val frame = arSession?.update() ?: return
    val camera = frame.camera

    if (camera.trackingState != TrackingState.TRACKING) return  // not localized yet

    // Process newly detected planes
    for (plane in arSession!!.getAllTrackables(Plane::class.java)) {
        if (plane.trackingState == TrackingState.TRACKING) {
            drawPlaneIndicator(plane)
        }
    }

    // Handle user tap — hit test against detected planes
    pendingTap?.let { tap ->
        val hitResults = frame.hitTest(tap)
        hitResults.firstOrNull { it.trackable is Plane }?.let { hit ->
            val anchor = hit.createAnchor()
            placeObject(anchor)
        }
        pendingTap = null
    }
}

Anchors and Trackables

// Anchor: a fixed point in the physical world
// Creating an anchor from a hit test result:
val anchor = hitResult.createAnchor()

// Placing content at an anchor:
val pose = anchor.pose  // Pose = position + rotation in world space
val modelMatrix = FloatArray(16)
pose.toMatrix(modelMatrix, 0)
// Use modelMatrix to position your 3D object in OpenGL

// Persistent anchors (Cloud Anchors) — shared across devices
val hostListener = object : Session.HostCloudAnchorCallback {
    override fun onHostComplete(cloudAnchorId: String, state: CloudAnchorState) {
        if (state == CloudAnchorState.SUCCESS) {
            saveToServer(cloudAnchorId)  // share this ID with other devices
        }
    }
}
arSession.hostCloudAnchorWithTtl(anchor, 1, hostListener)  // TTL in days

Depth API

On supported devices, ARCore provides a depth map — a 16-bit depth image at the resolution of the color camera:

val config = Config(session).apply {
    depthMode = if (session.isDepthModeSupported(Config.DepthMode.AUTOMATIC)) {
        Config.DepthMode.AUTOMATIC
    } else {
        Config.DepthMode.DISABLED
    }
}

// Per frame: get depth image
val depthImage = frame.acquireDepthImage16Bits()
// depthImage is a 16-bit depth map in millimeters
// Use for occlusion rendering: virtual objects hidden behind real-world surfaces
depthImage.close()  // always close!

Key Takeaways

Concept	Rule
Session lifecycle	`resume()` in `onResume`, `pause()` in `onPause`
Tracking state	Check `camera.trackingState == TRACKING` before rendering
Hit testing	Tap → `frame.hitTest()` → filter for `Plane` → `createAnchor()`
Anchors	Anchor positions virtual objects to real-world coordinates
Depth API	Use for occlusion; check `isDepthModeSupported` before enabling
ARCore install	Check and prompt for ARCore installation before creating a session