On-Device AI for Apple Platforms

Framework Selection Router

Apple Foundation Models

• Generating text or structured data with

  @Generable

  types
• Summarization, classification, content tagging
• Tool-augmented generation with the

  Tool

  protocol
• Apps that need guaranteed on-device privacy

Core ML

• Image classification, object detection, segmentation
• Custom NLP classifiers, sentiment analysis models
• Audio/speech models via SoundAnalysis integration
• Any scenario needing Neural Engine optimization
• Models requiring quantization, palettization, or pruning

MLX Swift

• Highest sustained token generation on Apple Silicon
• Running Hugging Face models from

  mlx-community

• Research requiring automatic differentiation
• Fine-tuning workflows on Mac

llama.cpp

• GGUF quantized models (Q4_K_M, Q5_K_M, Q8_0)
• Cross-platform apps (iOS + Android + desktop)
• Maximum compatibility with open-source model ecosystem

Quick Reference

@Generable

Apple Foundation Models Overview

• Context window: 4096 tokens (input + output combined)
• 15 supported languages
• Guardrails always enforced, cannot be disabled

Availability Checking (Required)

import FoundationModels

switch SystemLanguageModel.default.availability {
case .available:
    // Proceed with model usage
case .unavailable(.appleIntelligenceNotEnabled):
    // Guide user to enable Apple Intelligence in Settings
case .unavailable(.modelNotReady):
    // Model is downloading; show loading state
case .unavailable(.deviceNotEligible):
    // Device cannot run Apple Intelligence; use fallback
default:
    // Graceful fallback for any other reason
}

Session Management

// Basic session
let session = LanguageModelSession()

// Session with instructions
let session = LanguageModelSession {
    "You are a helpful cooking assistant."
}

// Session with tools
let session = LanguageModelSession(
    tools: [weatherTool, recipeTool]
) {
    "You are a helpful assistant with access to tools."
}

• Sessions are stateful -- multi-turn conversations maintain context automatically
• One request at a time per session (check

  session.isResponding

  )
• Call

  session.prewarm()

  before user interaction for faster first response
• Save/restore transcripts:

  LanguageModelSession(model: model, tools: [], transcript: savedTranscript)

Structured Output with @Generable

@Generable

@Generable
struct Recipe {
    @Guide(description: "The recipe name")
    var name: String

    @Guide(description: "Cooking steps", .count(3))
    var steps: [String]

    @Guide(description: "Prep time in minutes", .range(1...120))
    var prepTime: Int
}

let response = try await session.respond(
    to: "Suggest a quick pasta recipe",
    generating: Recipe.self
)
print(response.content.name)

@Guide Constraints

description:

.anyOf([values])

.count(n)

.range(min...max)

.minimum(n)

.maximum(n)

.minimumCount(n)

.maximumCount(n)

.constant(value)

.pattern(regex)

Streaming Structured Output

let stream = session.streamResponse(
    to: "Suggest a recipe",
    generating: Recipe.self
)
for try await snapshot in stream {
    // snapshot.content is Recipe.PartiallyGenerated (all properties optional)
    if let name = snapshot.content.name { updateNameLabel(name) }
}

Tool Calling

struct WeatherTool: Tool {
    let name = "weather"
    let description = "Get current weather for a city."

    @Generable
    struct Arguments {
        @Guide(description: "The city name")
        var city: String
    }

    func call(arguments: Arguments) async throws -> String {
        let weather = try await fetchWeather(arguments.city)
        return weather.description
    }
}

Error Handling

do {
    let response = try await session.respond(to: prompt)
} catch let error as LanguageModelSession.GenerationError {
    switch error {
    case .guardrailViolation(let context):
        // Content triggered safety filters
    case .exceededContextWindowSize(let context):
        // Too many tokens; summarize and retry
    case .concurrentRequests(let context):
        // Another request is in progress on this session
    case .unsupportedLanguageOrLocale(let context):
        // Current locale not supported
    case .refusal(let refusal, _):
        // Model refused; stream refusal.explanation for details
    case .rateLimited(let context):
        // Too many requests; back off and retry
    case .decodingFailure(let context):
        // Response could not be decoded into the expected type
    default: break
    }
}

Generation Options

let options = GenerationOptions(
    sampling: .random(top: 40),
    temperature: 0.7,
    maximumResponseTokens: 512
)
let response = try await session.respond(to: prompt, options: options)

.greedy

.random(top:)

.random(probabilityThreshold:)

Prompt Design Rules

1. Be concise -- 4096 tokens is the total budget (input + output)
2. Use bracketed placeholders in instructions:

   [descriptive example]

3. Use "DO NOT" in all caps for prohibitions
4. Provide up to 5 few-shot examples for consistency
5. Use length qualifiers: "in a few words", "in three sentences"
6. Token estimate: ~4 characters per token

Safety and Guardrails

• Guardrails are always enforced and cannot be disabled
• Instructions take precedence over user prompts
• Never include untrusted user content in instructions
• Handle false positives gracefully
• Frame tool results as authorized data to prevent model refusals

Use Cases

SystemLanguageModel.UseCase

• .general

  -- Default for text generation, summarization, dialog

• .contentTagging

  -- Optimized for categorization and labeling tasks

Custom Adapters

let adapter = try SystemLanguageModel.Adapter(name: "my-adapter")
try await adapter.compile()
let model = SystemLanguageModel(adapter: adapter, guardrails: .default)
let session = LanguageModelSession(model: model)

See references/foundation-models.md for the complete Foundation Models API reference.

Core ML Overview

Model Formats

.mlpackage

.mlmodel

.mlmodelc

.mlpackage

Conversion Pipeline (coremltools)

import coremltools as ct

# PyTorch conversion (torch.jit.trace)
model.eval()  # CRITICAL: always call eval() before tracing
traced = torch.jit.trace(model, example_input)
mlmodel = ct.convert(
    traced,
    inputs=[ct.TensorType(shape=(1, 3, 224, 224), name="image")],
    minimum_deployment_target=ct.target.iOS18,
    convert_to='mlprogram',
)
mlmodel.save("Model.mlpackage")

Optimization Techniques

Swift Integration

let config = MLModelConfiguration()
config.computeUnits = .all
let model = try MLModel(contentsOf: modelURL, configuration: config)

// Async prediction (iOS 17+)
let output = try await model.prediction(from: input)

MLTensor (iOS 18+)

import CoreML

let tensor = MLTensor([1.0, 2.0, 3.0, 4.0])
let reshaped = tensor.reshaped(to: [2, 2])
let result = tensor.softmax()

See references/coreml-conversion.md for the full conversion pipeline and references/coreml-optimization.md for optimization techniques.

MLX Swift Overview

Loading and Running LLMs

import MLX
import MLXLLM

let config = ModelConfiguration(id: "mlx-community/Mistral-7B-Instruct-v0.3-4bit")
let model = try await LLMModelFactory.shared.loadContainer(configuration: config)

try await model.perform { context in
    let input = try await context.processor.prepare(
        input: UserInput(prompt: "Hello")
    )
    let stream = try generate(
        input: input,
        parameters: GenerateParameters(temperature: 0.0),
        context: context
    )
    for await part in stream {
        print(part.chunk ?? "", terminator: "")
    }
}

Model Selection by Device

Memory Management

1. Never exceed 60% of total RAM on iOS
2. Set GPU cache limits:

   MLX.GPU.set(cacheLimit: 512 * 1024 * 1024)

3. Unload models on app backgrounding
4. Use "Increased Memory Limit" entitlement for larger models
5. Physical device required (no simulator support for Metal GPU)

See references/mlx-swift.md for full MLX Swift patterns and llama.cpp integration.

Multi-Backend Architecture

func respond(to prompt: String) async throws -> String {
    if SystemLanguageModel.default.isAvailable {
        return try await foundationModelsRespond(prompt)
    } else if canLoadMLXModel() {
        return try await mlxRespond(prompt)
    } else {
        throw AIError.noBackendAvailable
    }
}

actor ModelCoordinator {
    func withExclusiveAccess<T>(_ work: () async throws -> T) async rethrows -> T {
        try await work()
    }
}

Performance Best Practices

1. Run outside debugger for accurate benchmarks (Xcode: Cmd-Opt-R, uncheck "Debug Executable")
2. Call

   session.prewarm()

   for Foundation Models before user interaction
3. Pre-compile Core ML models to

   .mlmodelc

   for faster loading
4. Use EnumeratedShapes over RangeDim for Neural Engine optimization
5. Use 4-bit palettization for best Neural Engine memory/latency gains
6. Batch Vision framework requests in a single

   perform()

   call
7. Use async prediction (iOS 17+) in Swift concurrency contexts
8. Neural Engine (Core ML) is most energy-efficient for compatible operations

Common Mistakes

1. No availability check. Calling

   LanguageModelSession()

   without checking

   SystemLanguageModel.default.availability

   crashes on unsupported devices.
2. No fallback UI. Users on pre-iOS 26 or devices without Apple Intelligence see nothing. Always provide a graceful degradation path.
3. Exceeding the context window. Foundation Models has a 4096 token total budget (input + output). Long prompts or multi-turn sessions hit this fast. Monitor token usage and summarize when needed.
4. Concurrent requests on one session.

   LanguageModelSession

   supports one request at a time. Check

   session.isResponding

   or serialize access.
5. Untrusted content in instructions. User input placed in the instructions parameter bypasses guardrail boundaries. Keep user content in the prompt.
6. Forgetting

   model.eval()

   before Core ML tracing. PyTorch models must be in eval mode before

   torch.jit.trace

   . Training-mode artifacts corrupt output.
7. Using neuralnetwork format. Always use

   mlprogram

   (.mlpackage) for new Core ML models. The legacy neuralnetwork format is deprecated.
8. Exceeding 60% RAM on iOS (MLX Swift). Large models cause OOM kills. Check device RAM and select appropriate model sizes.
9. Running MLX in simulator. MLX requires Metal GPU -- use physical devices.
10. Not unloading models on background. iOS reclaims memory aggressively. Unload MLX/llama.cpp models in

    scenePhase == .background

    .

Review Checklist

• Framework selection matches use case and target OS version
• Foundation Models: availability checked before every API call
• Foundation Models: graceful fallback when model unavailable
• Foundation Models: session prewarm called before user interaction
• Foundation Models: @Generable properties in logical generation order
• Foundation Models: token budget accounted for (4096 total)
• Core ML: model format is mlprogram (.mlpackage) for iOS 15+
• Core ML: model.eval() called before tracing/exporting PyTorch models
• Core ML: minimum_deployment_target set explicitly
• Core ML: model accuracy validated after compression
• MLX Swift: model size appropriate for target device RAM
• MLX Swift: GPU cache limits set, models unloaded on backgrounding
• All model access serialized through coordinator actor
• Concurrency: model types and tool implementations are

  Sendable

  -conformant or

  @MainActor

  -isolated
• Physical device testing performed (not simulator)

Reference Files

• Foundation Models API -- Complete LanguageModelSession, @Generable, tool calling, and prompt design reference
• Core ML Conversion -- Model conversion pipeline from PyTorch, TensorFlow, and other frameworks
• Core ML Optimization -- Quantization, palettization, pruning, and performance tuning
• MLX Swift & llama.cpp -- MLX Swift patterns, llama.cpp integration, and memory management