Yury Korolev yury

Introduction

ARKit face tracking is shrouded in a ton of myth and mystique.

I reverse engineered the internal face tracking API, and this is how it actually works.

Disclaimer

This is based on reverse engineering the API of Apple's face detection framework and testing it with my own code. Some of it is educated guesswork based on names and strings. I did not reverse engineer the details of how ARKit/FaceKit works, but given what I've seen, I'm quite confident in the overall big picture.

Using CAHostingLayer in SwiftUI

Background

SwiftUI provides more than just UIHostingController / _UIHostingView (iOS) and NSHostingController / NSHostingView (macOS) for embedding SwiftUI views into UIKit/AppKit.

Starting from iOS 18.0 / macOS 15.0, SwiftUI also offers CAHostingLayer<Content: View> — a CALayer subclass that can host SwiftUI content directly at the layer level. This API is currently marked as SPI (@_spi(ForUIKitOnly) / @_spi(ForAppKitOnly)) and is used internally by UIKit, AppKit, and WebKit.

Accessing CAHostingLayer without Apple Internal SDK

local streaming test with HLS low latency tools

Install tools

https://download.developer.apple.com/Developer_Tools/HLS_Low_Latency_Beta_Tools/LowLatency_HLS_Prerelease_Tools73.dmg

Start web server

go run /usr/local/share/llhlstools/ll-hls-origin-example.go -dir /Library/WebServer/Documents -http :8000

Setting up ssh public key authentication on macOS using a YubiKey 4

I largely followed Florin's blog post, but have a few notes to add regarding issues I encountered:

Basic setup notes

I used a YubiKey 4, while the blog describes using a YubiKey NEO. I'm sure a YubiKey 5 would also work. I'm also running macOS 10.13.6.
I installed GPGTools as recommended. However, as I'll note later, it seems that gpg-agent only automatically starts when gpg is used; for ssh, you'll need to ensure it's running.
Before generating your keys, decide what key size you want to use. If you run the list command inside gpg --edit-card, look for the Key attributes line to see what is currently selected. On my YubiKey 4, it defaulted to 2048 bits for all keys:

Key attributes ...: rsa2048 rsa2048 rsa2048

	//
	// ViewController.swift
	// UIGlassContainerEffect-Demo
	//
	// Created by Seb Vidal on 04/11/2025.
	//

	import UIKit
	import MapKit

	import Foundation
	import MLCompute
	import PlaygroundSupport

	let iPage = PlaygroundPage.current
	iPage.needsIndefiniteExecution = true

	/*
	* Apple says MLCMatMulLayer does a ``batch matrix multiplication''
	* but didn't make clear its meaning.

	//In your view, declare a UIViewRepresentable as a UIButton type as seen below.
	//This is important as you cannot do it as a UIBarButtonItem (at least as I've found in Xcode 12 beta 2).


	struct MenuButtonView: UIViewRepresentable {

	typealias UIViewType = UIButton

	let saveAction = UIAction(title: "") { action in }
	let saveMenu = UIMenu(title: "", children: [

	/// Fixed swipe gesture NavigationLink
	struct NavigationLink<Destination: View, Label:View>: View {
	var destination: Destination
	var label: () -> Label

	public init(destination: Destination, @ViewBuilder label: @escaping () -> Label) {
	self.destination = destination
	self.label = label
	}

	# in /etc/systemd/system
	[Unit]
	Description=Bluetooth Agent
	[Service]
	ExecStart=/usr/bin/bt-agent -c NoInputNoOutput
	Type=simple
	[Install]
	WantedBy=multi-user.target