Run Gemma and VLMs on mobile with llama.cpp

Written by Georgios Soloupis, AI and Android GDE.

Gemma’s logo.

Running the Gemma 3 model and several Vision-Language Models (VLMs) entirely offline on mobile devices is now also feasible using llama.cpp, a highly optimized, quantized inference framework for LLMs. Thanks to its efficient use of CPU resources, llama.cpp allows these models (originally trained for larger systems) to be compressed and executed in real-time on mobile hardware without requiring cloud connectivity. By converting models like Gemma 3 to GGUF format and applying quantization (e.g., Q4_K_M or Q5_0), users can deploy them on-device. This offline capability empowers users with greater privacy, speed, and autonomy, making powerful multimodal AI accessible even in low-connectivity or edge environments.

Running large language models (LLMs) entirely offline is not a brand-new development. Several implementations have existed for months, primarily focusing on text-only LLMs rather than vision-language models (VLMs). Libraries such as MediaPipe, llama.cpp, and Apple’s MLX have made notable progress in enabling on-device inference. More recently, with the introduction of MediaPipe’s GenAI API, models like Gemma 3 can now run on mobile devices at usable speeds. In parallel, llama.cpp has expanded its support beyond pure LLMs, offering efficient edge deployment of VLMs via the optimized .gguf format, making it possible to run both text-based and multimodal models entirely offline.

This blog post focuses on deploying a llama.cpp binding on Android using llama.rn, a React Native integration. Below are the key details about the app:

Features:

• Local LLM Inference: Runs GGUF-formatted models directly on an Android device’s CPU.
• Text & Vision Models: Demonstrates initialization and inference for both text-only (Gemma) and multimodal vision (Gemma, SmolVLM2, etc) models.
• Standard Completion: Fetches the entire model response at once.
• Streaming Completion: Receives the response token-by-token for a real-time, chatbot-like experience.
• Multimodal Prompts: Shows how to structure prompts that include both text and images for vision-language tasks.
• Modern Android Stack: Built with Kotlin, Coroutines, and a declarative UI using Jetpack Compose.
• Performance Metrics: Displays the generation speed in tokens per second (T/s).

Below are the most significant details of the mobile application:

1. Native Library Loading: The MyApplication class initializes SoLoader, a utility from Facebook that loads the native *.so C++ libraries compiled from llama.cpp. This is a crucial first step.

class MyApplication : Application() {
    override fun onCreate() {
        super.onCreate()
        // Initialize SoLoader. This is the crucial step.
        // It tells the app how to find and load the native libraries.
        SoLoader.init(this, /* native exopackage */ false)
    }
}

2. RNLlama Bridge: In MainActivity, an instance of RNLlama is created by providing it with a ReactApplicationContext. This acts as the bridge between our Kotlin code and the native C++ functions.

override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        // The RNLlama class needs a ReactApplicationContext. We can create one from our app context.
        val reactContext = ReactApplicationContext(applicationContext)
        rnLlm = RNLlama(reactContext)
        ...
}

3. Model Initialization:

• The initializeLlmContext() and initializeVisionLlmContext() methods prepare a set of parameters (as a WritableMap) specifying the model file path, context size (n_ctx), and other configurations.
• For vision models, it’s critical to set ctx_shift: false and load a corresponding multimodal projector (mmproj) file.

4. Completion Flow:

• A user action triggers a completion (e.g., startVisionCompletion()).
• The prompt is formatted into the chat structure the model expects using rnLlm.getFormattedChat().
• For vision tasks, an image is encoded to a Base64 data URI and passed in the media_paths parameter.
• The rnLlm.completionStream() method is called, which executes the inference on a background thread.
• A StreamCallback receives generated tokens in real-time, which are appended to a state variable, causing the Jetpack Compose UI to update automatically.

Image that is used during VLM inference.

This is the result of running SmolVLM2 model with the app, using the above image.

Mobile’s screenshot with the result after inference.

The current implementation supports CPU-only inference, but performance is impressive, especially with the lightweight SmolVLM2 model. If your device has ample RAM, you can also run the more advanced Gemma 3 4B model, which includes vision capabilities. Gemma files can be downloaded here.
More models can be obtained at this page.

Full android project can be build from this repository. The binding is based on the llama.rn project here.

Conclusion
In this blog post we explored how to run Gemma 3 and vision-language models (VLMs) entirely offline on Android using llama.cpp and its React Native binding, llama.rn. We highlighted the technical setup for deploying GGUF-formatted models on mobile, enabling both text-only and multimodal inference with impressive performance. The app is built with a modern Android stack (Kotlin, Jetpack Compose) and supports real-time streaming, vision-text prompts, and lightweight models like SmolVLM2. With proper RAM, even larger models like Gemma 3 4B can run locally, showcasing the growing potential for private, on-device AI.