Failed to create context on ios

[pkarthic]

I have tested it in iphone 13 128gb
ios: 18.3.1

flutter: [AI] Engine load failed: LlamaException: Failed to load model from /var/mobile/Containers/Data/Application/62C93405-097E-48E4-AC60-ACCA50C506DF/Documents/qwen2-0_5b-instruct-q4_0.gguf (Exception: Exception: Failed to create context)
flutter: [AI] Stack trace: #0 LlamaEngine.loadModel (package:llamadart/src/core/engine/engine.dart:178:7)

#1 Future.timeout. (dart:async/future_impl.dart:1061:7)

#2 AiInferenceNotifier.loadModel (package:flutter_water_level/data/providers/ai_inference_provider.dart:23:7)

#3 _DashboardShellState.initState. (package:flutter_water_level/presentation/views/dashboard_shell.dart:41:21)

I have reduced the context size to 256 and set threads limits to 2, but still failed creating context.

[leehack]

Thanks for the details. The stack trace shows the GGUF file is getting past the model-load step and failing during context creation (llama_init_from_model).

One thing to check: reducing contextSize and threads does not disable the iOS Metal/offload path. Could you try forcing CPU for the first pass?

await engine.setNativeLogLevel(LlamaLogLevel.debug);

await engine.loadModel(
  modelPath,
  modelParams: const ModelParams(
    preferredBackend: GpuBackend.cpu,
    gpuLayers: 0,
    contextSize: 256,
    numberOfThreads: 2,
    numberOfThreadsBatch: 2,
    batchSize: 64,
    microBatchSize: 1,
  ),
);

If that loads successfully, the issue is likely Metal/offload initialization or memory pressure on the iPhone 13 rather than the model file itself. After that, try Metal again with a small fixed offload count, for example preferredBackend: GpuBackend.metal and gpuLayers: 4, then 8/16.

Could you also paste the native log lines around the failure with setNativeLogLevel(LlamaLogLevel.debug) enabled? The current Dart exception collapses the native reason into Failed to create context, so the native log should tell us whether Metal allocation, command queue/library creation, or context memory reservation is failing.

[pkarthic]

@leehack Update: Working on iPhone 13 (A15) with preferredBackend: GpuBackend.cpu

The suggested params worked. The model loads and runs successfully on iOS with:

await engine.setNativeLogLevel(LlamaLogLevel.debug);
await engine.loadModel(
  modelPath,
  modelParams: const ModelParams(
    preferredBackend: GpuBackend.cpu,
    gpuLayers: 0,
    contextSize: 256,
    numberOfThreads: 2,
    numberOfThreadsBatch: 2,
    batchSize: 64,
    microBatchSize: 1,
  ),
);

Root cause from the native logs:

Metal shader compilation fails entirely on A15 (iPhone 13, iOS):

ggml_metal_library_init: error: Error Domain=MTLLibraryErrorDomain Code=3 "Compiler encountered an internal error"
ggml_metal_device_init: error: failed to create library

So trying Metal with gpuLayers: 4/8/16 would not help — the Metal library itself does not compile, regardless of offload count.

Additional finding:
libggml-cpu.dylib is also missing from the app bundle:
load_backend: failed to load libggml-cpu.dylib: dlopen(libggml-cpu.dylib, 0x0006): tried: ... (no such file)
However, the static CPU fallback inside llama.cpp activates when preferredBackend: GpuBackend.cpu is set, and all 24 layers run on CPU successfully.

Device info:
Device: iPhone 13
GPU: Apple A15 (MTLGPUFamilyApple8, MTLGPUFamilyMetal3)
Model: qwen2.5-0.5b-instruct-q4_k_m.gguf
The fix on your end would be to resolve the Metal shader compilation error in the bundled iOS binary, and optionally bundle libggml-cpu.dylib so the CPU backend loads via dylib rather than falling back to the static path.

[leehack]

I opened two draft PRs to make this testable before a native release:

• Native fix: Disable iOS Metal residency sets by default llamadart-native#15
• Test helper in llamadart: Prefer local native bundles when explicitly enabled #170

The native PR disables Metal residency sets on iOS/tvOS/visionOS while keeping Metal enabled. I also triggered a non-publishing b9159 build from that PR branch so it matches the current llamadart native pin. The ios-device-arm64 job passed, and the artifact is available here:

• Run: https://github.com/leehack/llamadart-native/actions/runs/26518762182
• Artifact: native_apple_ios-device-arm64 (libllamadart.dylib)

To test it locally:

1. Check out llamadart from the helper PR branch and use it as a local path dependency in your app:

dependency_overrides:
  llamadart:
    path: ../llamadart

2. Download and unzip the native_apple_ios-device-arm64 artifact from the workflow run above.
3. Copy libllamadart.dylib into the local llamadart checkout at:

third_party/bin/ios/libllamadart.dylib

4. Run your Flutter app with the local native bundle override enabled:

export LLAMADART_ALLOW_LEGACY_LOCAL_BUNDLES=true
flutter clean
flutter pub get
flutter run

Then retry the Metal path, not the CPU workaround. If this works, we can merge the native fix, publish a new native bundle, and then bump llamadart to consume it normally.

[pkarthic]

@leehack confirmed working on iPhone 13 (A15, iOS 18).

With the fix/prefer-local-native-bundles branch + b9159 ios-device-arm64 artifact:

ggml_metal_library_init: using embedded metal library
ggml_metal_device_init: GPU name: MTL0 (Apple A15 GPU)
load_tensors: offloaded 16/25 layers to GPU

No more MTLLibraryErrorDomain Code=3. All Metal compute pipelines compiled and loaded successfully. Model runs on preferredBackend: GpuBackend.metal with gpuLayers: 16.

Please let us know when this is merged and published so we can remove the local path override and use the released version.

[leehack]

Thanks for confirming the PR artifact fixed this on iPhone 13 / A15.

This is now merged and published:

• Native fix: Disable iOS Metal residency sets by default llamadart-native#15
• Fixed native bundle release: leehack/llamadart-native@b9371, rebuilt from merge commit 140b2c5 in https://github.com/leehack/llamadart-native/actions/runs/26567122506
• Dart package: llamadart 0.6.17 is published and pins the fixed b9371 native bundle.

You should be able to remove the local path dependency override and stop using LLAMADART_ALLOW_LEGACY_LOCAL_BUNDLES. Update normally and rerun:

flutter pub upgrade llamadart
flutter clean
flutter pub get
flutter run

If you tested main earlier and already cached the first b9371 bundle before the native assets were refreshed, delete .dart_tool/llamadart/native_bundles once so the app redownloads the fixed iOS bundle.

Closing this as fixed. Please comment again if 0.6.17 still fails on the A15 Metal path.

[pkarthic]

@leehack 0.6.17 from pub.dev still fails on the same device (iPhone 13 / A15):

MTLCompiler: Compilation failed with XPC_ERROR_CONNECTION_INTERRUPTED on 3 try
ggml_metal_library_init: error: Error Domain=MTLLibraryErrorDomain Code=3 "Compiler encountered an internal error"
ggml_metal_device_init: error: failed to create library

Note the new XPC_ERROR_CONNECTION_INTERRUPTED line the system Metal compiler process is being killed mid-compilation. This did not appear with the PR artifact (b9159 from Actions run 26518762182), which compiled the Metal library successfully in ~12 seconds.

The b9159 pre-merge artifact worked. the b9371 published bundle does not. It looks like the two builds may differ in how the Metal library is embedded (pre-compiled vs JIT at runtime). We are staying on GpuBackend.cpu in the meantime.

[leehack]

@pkarthic Hmmm. Let me check again. Thanks for the report.

[leehack]

I checked the release path and this is not the same failure as the first local test artifact. The confirmed-good artifact was built from the native fix branch with llama_cpp_tag=b9159 (run 26518762182). The published llamadart 0.6.17 package uses llamadart-native@b9371; that native release run was dispatched with llama_cpp_tag=latest, which resolved to b9371 (run 26567122506). So the wrapper fix is present, but the underlying llama.cpp/Metal source changed between the artifact you tested and the published bundle.

I also downloaded the published llamadart-native-ios-arm64-b9371.tar.gz and compared it with the working PR artifact. The b9371 dylib still has the embedded __ggml_metallib section, so this does not look like a missing-upload or missing-embed problem. llama.cpp's embedded Metal path still compiles that embedded source at runtime via newLibraryWithSource; b9371 added Metal source changes after b9159, and the A15 runtime compiler is dying while compiling that newer source (XPC_ERROR_CONNECTION_INTERRUPTED / MTLLibraryErrorDomain Code=3). There are no Metal-source changes between b9371 and the current upstream latest I checked (b9404), so simply rebuilding latest is unlikely to change this.

For now, please stay on the CPU workaround or the previously tested local artifact. I am going to treat the release fix as either: (1) publish a corrected native/package path based on the known-good b9159 + iOS no-residency wrapper combination, or (2) produce a new b9371 test artifact with a targeted Metal mitigation before republishing. I will not ask you to keep retesting 0.6.17 as-is because the published iOS bundle is materially different from the artifact that passed on your iPhone 13.

[pkarthic]

@leehack Thanks for the detailed breakdown. We'll stay on GpuBackend.cpu in the meantime and wait for the corrected release. Happy to test again whenever you have a new artifact or publish ready.

[leehack]

@pkarthic Just out of curiosity, do you actually notice a performance difference between Metal and the CPU on that device? Since I haven't seen any issue tickets related to this specific problem in the llama.cpp upstream, I suspect it might take some time to address.

Instead, I'm considering adding a version override feature so users can easily bypass the native binary. This would allow you to pin it to version b9159, which is currently working for your device.

[pkarthic]

@leehack Yes, clear difference on iPhone 13 (A15):

• Metal (b9159): model loaded and was ready to use almost instantly after the shader compilation step
• CPU: model took 4–5 seconds to finish loading before it was ready to use

So the engine initialization is significantly faster on Metal.