Hey, my name is Cormac Brick. I work on Google AI Edge, which is a way of bringing models to the edge. This is something we use internally for our own products, and it's also something we make available as open-source products. As part of this, we also work really closely with the Gemma team because they publish a lot of models that are targeted to edge devices as well.

A quick bit of background on me: I've been focusing on edge AI for the last ten years or so at this point. I started off in 2016 running a GoogLeNet on a USB key hardware accelerator plugged into a Raspberry Pi at NeurIPS 2016. It's been a lot of fun over the years. I then joined Intel, where I led the architecture for the NPU that goes into all of their laptops these days. Three years ago, I moved to Google to work as a tech lead on Edge AI here, and that's been great fun—a crazy few years, as you can imagine.

I'm happy to share both where we're up to and also give a kind of overview of where mobile AI is up to today. What is the state of the art? What are the different patterns we see for model deployment?

The two things I want to focus on in this talk are tiny LLMs and agent skills. Tiny LLMs are very, very small models, while agent skills are now possible on-device, though larger models are needed to make those work fluently. Both are exciting new directions that have only recently been made possible.

As recently as last week, when the DeepMind team launched Gemma 2, we supported them with an Android and iOS app that has opened up new possibilities for what we can do on mobile. We'll deep-dive into that, as well as looking at what we can do with tiny models. Those are the two halves of the presentation.

Okay, so that's the intro. First, I'll give an overview of AI on the edge, looking at small language models and tiny language models. Then, we'll take a look at Gemma models, just because that's pretty topical, and the types of performance we see for Gemma models on various types of edge devices, since that's what our team does.

Next, we'll look at agent skills, which we built on top of the latest generation of Gemma models that can run on Android, iOS, and many other platforms. In the second half, we're going to look at tiny models—how you can fine-tune and deploy a tiny model to edge devices. Finally, I have an example of a real app that our team built using tiny LLMs based on Gemma technology.

Firstly, since you are already in this room, you're probably interested in edge AI, so this goes without saying. But there are a lot of benefits to running on the edge. There are latency and UX improvements for really sensitive in-the-loop tasks, like live voice translation, for example.

That's something our team shipped on Pixel last year, where you can do live voice translation. It is very challenging to do that with the required latency using a cloud service, so being able to do that offline and on the edge is key to meeting latency and user experience requirements.

Privacy is definitely a major factor as well. People use AI within messaging apps, and they prefer their messages to stay on their phone, encrypted and fully private. That is a good example of where we're seeing LLMs getting deployed to assist in those types of use cases where privacy is important.

Offline use is another obvious benefit, and then there are cost savings. We see this being increasingly relevant for laptop users, where there is a trend towards experimenting with small language models to handle some of the token processing they might otherwise do with a desktop agentic workflow. Some folks are interested in exploring those savings as well.

Our team enables people to build apps. We do MediaPipe, LiteRT-LM (which is an LLM runtime that works on mobile and edge), and LiteRT (which is just a standard inference framework, previously known as TensorFlow Lite).

We've used that over the years to ship in Google Photos, YouTube Shorts—if you've used any of the funny effects in YouTube Shorts, those are things our team built in collaboration with the YouTube team using MediaPipe and various deep learning tracking models under the hood. That is built on the same MediaPipe and LiteRT technology. All Android phones today use this as part of system services, or third-party apps build on a version of the stack that ships with Android.

But our stack also runs far beyond Android. Even though it is available as a system service in Android, you can take the same `.tflite` file and ship it to iOS, macOS, Linux, Windows, web, or IoT devices.

There is one caveat: that same file deploys on CPU and GPU, but for the NPU, we need to do some special compilation, resulting in a specialized NPU file. But certainly for the Gemma models we published last week, it is true that one file can work on CPU and GPU across a wide set of devices.

Some of the things we're seeing for LLMs on-device are privacy-centric features, voice agents, and local agents, where tool calling is a very popular workflow. Within our stack, LiteRT-LM is the component we'll look at a bit later that helps us run tiny LLMs on-device, cross-platform, and fast because we support all the different hardware accelerators.

This is an important concept: we see two trends happening today. One is system-level GenAI. For very large models, the way they tend to show up on mobile phones isn't that when you launch a single app, it downloads a four-billion-parameter model just to help you find a good restaurant in whatever app you're looking in.

Instead, the trend is to build larger models into the OS. We call this system-level GenAI. These models tend to be in the two- to five-billion-parameter range, depending on the OS. This is a choice we see being made by both the Android team, who we work closely with, and the Apple Intelligence team.

We see similar choices being made by both mobile OS vendors where there's a central model built-in. On Android, that's called AI Core, and there are things like summarization APIs and an increasing set of APIs available for developers, including a Prompt API. That's available on premium Android devices, and premium Apple devices have their own Apple Intelligence. As a trend, this is highly relevant. If you want to leverage a built-in model, this is a great way to go.

The other trend we see is in-app GenAI, which is where tiny LLMs, or TLMs as we're calling them in this presentation, are more relevant. While system GenAI is generally customized via prompting or skills—and uses a foundation model pre-loaded on the device—in-app GenAI is generally custom-built for specific tasks and loaded with the app or the webpage.

We also deploy some of these on the web. Generally, they are targeted at wider reach, so they work not just on premium devices but on all devices, because reach is important for many of the application developer teams we work with.

Surprisingly, if you fine-tune for a single task, we've seen really strong performance on simple tasks like summarization, transcription, or voice-to-action type things. We can get really reliable performance from models in the 100- to 500-million-parameter range, depending on the complexity of the task.

A good example here is we launched function Gemma in December, which was a 270-million-parameter model dedicated to function calling. We were able to show that for voice-to-function calling across ten different functions relevant to Android developers, our internal evaluation reached 85% to 90% reliability using that very small model. This was widely deployable to iOS and Android devices, and it's actually something you can play with in a sample app we'll look at later.

That's one use of a tiny LLM. We're seeing more interest from application developer teams now in fine-tuning models to deploy as in-app GenAI. The difference is that on the left, customization is via prompting or skills, whereas on the right, we encourage people to do some degree of fine-tuning to make a tiny LLM work in practice.

Certainly, below 500 million parameters, that's true. Maybe for 500 million parameters and above, we can do more general-purpose tasks, but for really tiny models—especially less than 500 million parameters—our experience is that you need to fine-tune to get production-level reliability.

Now I'm going to talk about Gemma 2. The models launched last week fall into that system GenAI category. We can do lots of powerful things with Gemma 2, as you'll see.

First, let's talk about sizes. There are two small sizes: Gemma 2 2B and Gemma 2 4B (effective 2B and effective 4B). Gemma 2 2B is called that because it only needs about two billion parameters to be present in RAM to run, which is a limiting factor on device. Gemma 2 4B has been optimized to run with four billion parameters on-device.

There are more parameters used by the model, but those other parameters are used for per-layer embeddings (PLE). DeepMind has talks on this later in the week where you can get more details, but the TL;DR is that the other parameters are used for per-layer embeddings. In our runtime, we don't actually need to load all of those parameters into RAM. We memory-map them and only need to load one line of the per-layer embedding table at a time during the autoregressive loop. We actually only need to load a few hundred or thousands of bytes of that in order to do the next-token inference. As you go through inference, you don't ever end up requiring to load the whole PLE table into memory, and the OS does a reasonably good job of evicting memory used by older PLE tokens. That's why we have this idea of "effective" parameters.

The smaller models run on a variety of platforms. The 2B and 4B models are on the AI Core roadmap, meaning these models are available now for experimentation, and at the appropriate point in time, the Android team will integrate them into AI Core so they'll be available more broadly on a wider set of devices. There is another talk this week from Olli from the AI Core team, who will probably share more details about exactly what that roadmap looks like.

Omar from the Google DeepMind team is also going to do a deeper dive on the Gemma world. The bottom two models on the slide are also relevant for the edge, though they are not the focus of my talk today because they are optimized for laptops. Those sizes have been optimized to run well on consumer-grade laptops, albeit ones that have perhaps 32 gigabytes of RAM. I'm going to focus less on those other two models, even though they are very powerful and useful.

We can see that both the 2B and 4B models have excellent performance on a wide range of tasks, particularly on knowledge and reasoning. But one of the big step-ups from my perspective as a user of the models, compared to the last generation, is that they have built-in function calling and built-in thinking.

That combination of thinking plus function calling is what unlocks our ability to do skills on-device. As you'll see in a second, we can describe a skill, give it to the model, and the model can pick it up and use it. This allows us to bring a pattern that has proven very popular in the last few months over to mobile, allowing for new types of mobile experiences.

Also, the 2B and 4B models are multimodal, so they support audio, image, and text, while the larger models support just image and text.

Another change from a deployment point of view is that this is the first time the Gemma models are released under a standard Apache 2.0 license, which means they are more usable by more people. Again, you'll hear more about this in Omar's talk, but I thought I'd mention it here.

That's Gemma in general. Now, let's do a deep dive into Gemma on our runtime and platform. As shown in the picture earlier, we have a single LiteRT-LM file, which is a LiteRT file containing the tokenizer and the other components we need in one package to run a model. That single model runs across all of these classes of devices: mobile, desktop, and embedded. We're pretty excited to do more in the embedded space, particularly with image input, where there is a lot of scope for new IoT use cases.

This is a bit of an eye chart, but let's dig into performance. I would say this is a snapshot as of today; we are continuing to work on this both within our team and with partners like Intel, Raspberry Pi, and Qualcomm to optimize these numbers.

For the two-billion-parameter model, we can get really compelling performance: a high-end Android phone can do thousands of tokens per second on a GPU, and we see similar numbers on a MacBook.

The bottom two rows show that on a Raspberry Pi, we can get about 13 tokens per second, which is sufficient to do simple image analysis use cases with reasonable latency. The bottom row is us running on a hardware accelerator. This is a Qualcomm IoT robotics development platform, and there we see pretty compelling performance as well because we've used the NPU, which gives a lot better performance.

We have corresponding performance on the 4B model, which works across a wide set of devices with proportionally less prefill and decode performance given the size and the number of parameters we need to fetch. But broadly, they are available on lots of platforms.

Now, what can we do with those models? Lots of things. One of the things I wanted to talk about, because it's net-new rather than just showing image analysis, audio transcription, or translation, is agent skills on-device.

We have an app available on both iOS and Android called Google AI Gallery. This allows you to do basic AI chat, ask questions about an image, do transcription or translation starting from audio, or run audio-to-function-calling use cases. The second option from the left is agent skills, and I'm going to do a deeper dive on that today.

Hopefully, this video will play. Oh, that is very annoying, the sound is not working. I will fix this before we post the slides.

Morning. Let's log a new mood journal entry.

Since this is playing but the sound is not working, I'll explain.

I got eight hours of sleep and I'm looking forward to hanging out with Amy today.

This is a journal skill app or a mood tracker where it logs your mood and sleep.

Analyze the trend in my mood over the last seven days.

What's happening here with the mood tracker app is that it logs your moods or observations to a diary, and then the LLM is able to go back in and summarize the content.

What's interesting here is that this isn't a custom fine-tune. It is just us giving a particular skill with a little bit of JavaScript to the model, and then through a free-text interface, the model is able to pick up and use that skill.

It is also able to call multiple skills. There are two skills actually happening here: one is the mood tracker skill, and the other is the map skill. Now it is using a query Wikipedia skill to look for the latest information from the Oscars. Instead of having a model that is pinned in time, it becomes really easy to extend it with skills such as map lookup, an interactive journal that you can add or subtract entries from via voice, or adding more modern and relevant knowledge. We're showing Wikipedia in this case.

This is another fun skill that someone on the team developed: the mood music skill, which calls a web service to compose music based on a single image and plays it. It is able to compose some lo-fi music to go with a person's breakfast. It's a new paradigm in how we are able to easily extend these models, and to do so in a pretty low-code way, as you'll see when we dig into how this works under the hood.

To save time, I won't play all of these videos because you've seen many of them. One pattern we find interesting is augmenting the knowledge base. We can produce rich, interactive content, like flashcards for visualizations, where instead of summarizing something in three bullet points, you can use a JavaScript skill to show those bullet points as a card instead. If the model thinks a card display will be more helpful for the user, it will use that.

Let's skip forward to how we built them. What's actually happening is that the way we've built the skills makes them efficient; the instructions can be loaded on-demand. This uses a principle of progressive disclosure or conditional depth.

Instead of an MCP (Model Context Protocol) workflow where you need to describe everything about all the functions you have, the way we've structured these skills is that there is a one-line description. The agent sees these one-line descriptions, and if a skill sounds interesting, it asks to load the full skill. We teach it a skill to load a skill. It then goes in, loads the skill, and finds out all the details about how to use it and what function calls it can use.

This pattern is particularly important for token efficiency and reliability on edge models. If we had to load all the details for all the skills into the edge model, that would be a lot of context for it to reason over, which would ultimately hurt performance. The lighter-weight models are great to run on-device, but in terms of reasoning over very long context windows, a condensed context window will up your batting average in terms of the quality metrics you're looking at to ship an app.

The second part is tool access, which helps us integrate new tools dynamically. We think of them in terms of input tools (how to get more information, which could be Wikipedia or a weather service) and output tools (presenting new outputs to the user, like showing something on maps or via cards). You can have skills to extend both the input and output patterns that a model can perform. It also helps us bring in domain-specific knowledge bases, so that skill calling Wikipedia could easily be calling an internal customer CRM or fetching data from a local RAG system.

Within our structure, we have a `skill.md` file and optionally scripts or assets. The `skill.md` contains the metadata we always process.

The example here shows extracting text and tables from a PDF. We trigger on PDF extraction, and the instructions are only loaded when the model thinks it requires that skill. This pattern is particularly important. Within scripts, we also support writing custom JavaScript that can be rendered within the app, which is a very easy way to extend within both iOS and Android systems.

To dig a bit deeper: we have our system prompt that we ship with the app, and then there is a set of skill descriptions added to the system prompt. When the user asks for something, the model decides whether to trigger the skill by reading its metadata. It then calls the `load_skill` app under the hood.

The tool response from that function call is the content of the `skill.md` file, which is placed into the context window so the model now knows about those functions. It then calls `run_javascript`, which runs the JavaScript we picked up from the skill file, and we return a response.

One other thing to mention as part of this workflow: when we were targeting reliability, we implemented constrained decoding within the runtime, which is tuned to only apply to the output when we are calling a tool.

The constrained decoding can restrict the output to the specific tool you are supposed to be calling. In this system, instead of having generic JSON constraints, we know there is a finite set of tools the model is supposed to be able to use, so we can employ stronger constrained decoding. This helps us have a more reliable system overall.

We find this helps a lot for the two-billion-parameter model. As models get more capable, the margin you get from this strict constrained decoding is less essential compared to running a very large model, like a ten-billion-parameter or larger model. But for really small models on-device, this is a very helpful tool that provides stronger guardrails, allowing us to increase quality so we have something useful in production.

Within the app, we support toggling the skills you want to use. Even for the skill descriptions, you can decide how many you want to have live at any given point in time. This video shows loading a virtual piano skill, which allows you to tap the keys and play sounds. That is just more JavaScript.

You can also load custom skills. You can write a skill yourself and load it from a URL, which is a fun way to prototype a skill with Gemma for an app idea you have. The skills can also use an API key, so if you need to use a web service, you can prompt the user to input a secret key.

On our GitHub page, we have a discussion section where users are posting skills they have written themselves. We can take the community skills we like and feature them in the app, showcasing useful creations to the wider community. This video shows a third-party animal intro skill, which is a kids' feature added to the app.

The key point here is that there is a really low barrier to extending the model in a way that is relevant to downstream app users. This is very easy, and we're going to go a little deeper now to see just how easy it is.

Going one step deeper into the skill architecture: we have our own orchestrator containing a skill registry, and we call the `load_skill` function to load the skills. Within the skills, there are JavaScript skills and native intents.

For the Android system, we are able to call Android system intents or native intents directly in your skill—for example, if you wanted to turn Wi-Fi on or off. Any intents exposed to Android that are available in JavaScript can be used.

Then there is the roleplay `skill.md`, which contains the persona and scenario data, followed by the specific skill and resources. As I mentioned before, this can be JavaScript that runs entirely locally for a fully offline experience, or, like in the music composer example, it can call a web API requiring an API key.

The predefined tools we have under the hood include `load_skill`, `run_javascript`, and `run_intent`. The orchestrator is able to make the overall system work using just these three tools.

Let's test our app.

This is our Gemini product manager showing the restaurant roulette skill.

It's good to find a restaurant. Please reply to me in English.

This is a restaurant roulette skill. It was so easy to develop skills using tools like Gemini CLI or anti-gravity. We actually had the team develop about 80 of these skills, so we had lots of options to show, and it was a lot of fun for folks to build them.

Here we can look at the structure itself of Restaurant Roulette. Under the hood, it requires a secret key to be set to true, searches for ten restaurants, and then returns locations and cuisines. This is the `index.js` file, where we render a simple web view to show the spinning wheel.

You can obviously code all of this yourself. We have the full source code for the examples, and we have both the source code and a skill spec on GitHub to help you get started with detailed instructions if you want to try writing your own skill.

The pattern we actually used most was using skills to write skills. Using tools like Gemini CLI, Cloud Code, or anti-gravity was our favorite pattern, where we would say, "I want to write a skill for AI Edge Gallery." This example works in Gemini CLI, where we provide the documentation and some examples of skills to read, and then describe the skill we want to build, which in this case was an offline archiver.

The idea is that based on pictures you took around London, for example, it would be able to recognize something like a Churchill statue you walked past, research a bit about Churchill for you, and then on your flight back, you could go into this skill to read a bunch of information about the things you saw. That was the inspiration for this one. It fetches Wikipedia content, stores it locally, and creates an index.

With the CLI, we can also ask it to test the skill itself because we have an ADB skill in Gemini CLI. You can say, "You have access to a phone connected via ADB," and Gemini CLI uses its Android ADB skill to test that the app works and that the skill does what it says. You can put this in a prompt, ask it to iterate, and it will perform some basic validation and return the results.

If this plays, it will show an example of doing that. This worked really robustly. Out of the roughly 80 skills our team did internally, probably more than half were vibe-coded, at least initially. This reduces the barrier for folks to extend an LLM to do new and useful things for their audience.

Are all the examples here running on the smallest model, or is it on-device?

That's a great question. To repeat for the recording: are all of the examples running on the smallest model? In this case, the examples are running on the 4B model. The skills will work with the 2B model as well, and you can try that, but your mileage may vary. You might need simpler skills or fewer skills. But all the examples you've seen are running on the 4B model.

Have you tried using the Chrome prompt API with the skills to run a local model in the browser?

We haven't, no. But that is an interesting thing to try.

A quick shout-out to this link. A few things to mention about AI Edge Gallery: first, it is an open-source project that builds on top of the LiteRT-LM tooling that you saw earlier.

The app itself, as well as being fun to use, allows you to prototype app ideas, see if a feature is possible in a model on a phone, test how fast it runs, or verify if a skill is actually viable. In addition to model prototyping, you have full access to the source code. It builds on top of the same infrastructure, using open-source LiteRT and LiteRT-LM acceleration under the hood.

If you see something you like in the gallery, you can deploy the Gemma model some other way, or you can take the underlying open-source APIs, pull a model from our Hugging Face page, and run it directly.

We also have a set of discussions on the gallery repository showing community skills that have been uploaded, ranging from cat entertainment to blackjack. People have posted these skills, so if you develop one, feel free to post it there to make it discoverable. The ones that seem compelling or useful to many people can be added to our third-party preferred skills list in the app. We saw a reasonable amount of activity on this over the weekend, and it has only been live since last Thursday.

Next, I wanted to switch gears. Everything we saw with skills applies to the 2B and 4B models, which on mobile phones will mostly be destined for system GenAI in production. For IoT, desktop, or edge applications, you could load that model yourself and run these skill use cases in production on something like the Qualcomm IoT platform we saw earlier.

But to deploy models in-app today and ship them in production apps, we see more people using smaller models—what we call tiny models, which are models with less than one billion parameters. These are the kinds of models we work with teams to deploy within their apps, so I want to briefly take a look at what that workflow looks like.

LiteRT-LM is the engine that powers the gallery app. It is an open-source project with C++ and Java APIs, with Swift APIs coming soon. It also received a Python API last week, which is relevant for IoT developers who like to work in Python. It takes an LLM file and contains the required components to fully run an autoregressive loop and expose it via easy-to-use APIs.

It is cross-platform, and where relevant, the same API can be used with hardware acceleration, like the Qualcomm example. For other smaller models in the past, we've published versions that work on MediaTek and Intel silicon. Over the course of the year, as more small models become available, we will see broader NPU support.

The workflow starts with Transformers. We use a package called LiteRT-Torch, which has PyTorch-native optimizations for LiteRT and has quantization built into the workflow. That gives you a LiteRT-LM file. You can then prototype it with the gallery app or deploy it directly to your production use case using LiteRT-LM on whichever platform you want to work with.

For advanced use cases, we have the LiteRT-Torch generative API. If you want to write your own tiny model from scratch and train it, we support that workflow, as well as standard fine-tuning workflows. The generative API provides building blocks for LLMs in native PyTorch that give excellent performance when run on-device.

This gets into the weeds: this is our stack to deploy on NPUs. There are two main takeaways from this slide. First, under the hood, we invest an awful lot of work in optimization libraries: XNNPACK (which is a CPU optimization library) and GPU backends. Many of Google's first-party apps rely on both of these libraries, so we are highly motivated to ensure they have excellent performance and work on the widest possible set of devices.

These use what we call the JIT workflow, where we produce a single artifact—a LiteRT-LM file—that can work across CPU and GPU and get deployed to many types of devices. For NPUs, we need something more specialized. We need to call a vendor compiler plugin upfront, which uses an ahead-of-time (AOT) compilation workflow to produce an artifact specific to a particular NPU. Within our runtime, we then dispatch the work to the device driver of the NPU. Both of these paths are available through a consistent API, so while the JIT versus AOT distinction affects the build workflow, the actual app development workflow is very similar across NPU, CPU, and GPU.

Export and inference is simple using LiteRT-Torch. I know I've talked a lot about Gemma models because it's Gemma week for us, but it is worth calling out that we support third-party models as well. A Qwen 1.5B parameter model is shown on the right, and those Qwen models also work in the app. The app supports loading any LiteRT-LM file, running it, and getting benchmark stats.

Is that the latest Pixel that you're simulating here?

To be honest, I don't know definitively. We do a lot of testing on Pixel and on the S25 as well, so I assume it's one of those two devices.

In the latest release, we've added an optional icon underneath each chat, and if you click it, it shows you the prefill and decode stats for the model. You can find any model on Hugging Face, load it in the app, and run it to see the benchmark stats on a particular phone. On desktop, you can also use a command-line tool like `litert-lm-run` to do desktop prototyping and understand how models work there.

This is another third-party model: FastVLM from Apple. It is a really nice VLM model that is only 500 million parameters. This is running with hardware acceleration on Qualcomm, which is why it runs so fast. I think this is running on an S25 with Qualcomm silicon, and we have it running in a loop saying "describe the scene" with video input. It runs incredibly fast.

This is a good example of what is possible with a model that is feasible to deploy on-device. We haven't done 4-bit quantization in this case, but had we done that, it would only require about 250 or 260 megabytes in your app to deliver this experience.

This is an example of a general-purpose 500-million-parameter model. It has been trained on a fairly narrow set of tasks, like scene descriptions or image-to-description, so it's a narrow use case but still general-purpose. The smaller models we have, like Gemma 2 270M, typically require fine-tuning to perform a specific task. This is a great example of a highly useful general-purpose tiny model.

Other examples of general-purpose tiny models include small transcription models or narrow pairwise translation models, many of which we support and host on our Hugging Face page if you want to check them out.

This slide shows a handful of models I wanted to talk about. Function Gemma is one we published in partnership with Google DeepMind. This is a model you can further fine-tune for function calling, and there are Colab notebooks available showing how to format datasets and fine-tune it.

The next two are Mobile Actions and Tiny Garden. Mobile Actions is the example I mentioned earlier, featuring ten different mobile actions we fine-tuned ourselves, achieving around 86% reliability. Tiny Garden is a game built inside the gallery app that you can play with, which is another example of a voice-to-function-calling fine-tuned model.

Is that model pre-trained, or does it have an IT suffix?

It is instruction fine-tuned (IT). When DeepMind publishes models, they sometimes have a PT (pre-trained) suffix and sometimes an IT (instruction-tuned) suffix. PT is a model published right after pre-training, before any fine-tuning. That is helpful for expert users because you can do all of your own instruction fine-tuning, fully control the model's personality, and avoid trying to unlearn previous fine-tuning.

The way we teach the model to do function calling is via instruction fine-tuning itself. So when we publish a model for further fine-tuning for function calling, it already has a function-calling personality, and that's what we wanted to publish. In this case, the model is meant for further fine-tuning, but it is an IT model.

More typically, for larger models in the Gemma family, we provide both IT and PT checkpoints, depending on whether you have a large volume of data and want to do full fine-tuning yourself. There is a separate conversation on sovereign AI use cases that some of the DeepMind team will cover later in the week, which involves starting with our pre-trained checkpoint and adding a large corpus of sovereign or enterprise data to fully fine-tune a model to do specific tasks.

Another example is Embedding Gemma. Technically it is not an LLM, but it is a text embedding model we published with the Gemma team last September for RAG use cases. It is a very strong embedding model that only takes up 300 million parameters. It's another example of a high-utility tiny model that is useful for on-device use cases.

I've already covered this briefly: AOT compilation is our workflow for hardware acceleration on-device, which is best for distributing small models to many platforms.

LiteRT-LM is built on LiteRT and supports all of these types of models. You can also find non-LLM models on Hugging Face. This is relevant because when you're building a complex app, you typically need other models around the LLM, such as a voice activity detection (VAD) model or a denoising model. Many of these are available using the standard LiteRT runtime. LiteRT-LM is the runtime that handles the full autoregressive loop, and while there are LLM models available for it, there are also many supporting non-autoregressive models available via the LiteRT API that are typically used to deploy a full app.

For advanced usage, you can fully customize models. If you wanted to write your own Llama, Moonshine, Phi, or Qwen variant, you can do so using our codebase.

That covers the tiny models, examples, and workflows. There is a lot we can do now with very tiny models. The 500M class is available for standard features, and for smaller LLMs, we've had a lot of success fine-tuning them for apps.

Next, I'll show an example of an app we built using tiny LLMs, which is currently available on iOS, called AI Edge Eloquent.

This is a transcription model. As a speaker, you might notice I say "uh" and "um" a lot. If you got a direct transcript of this presentation, it wouldn't be great because of all the interjections. Eloquent was built for that transcription user story. It does dictation but includes an automatic polishing step that removes all the interjections and filler words. If you want to dictate a message for later use, it cleans up idioms of speech.

Another feature it has is a biasing list. We found that if we talk about LLMs and say, "Have you trained a LoRA for this?", any standard transcription service will translate "LoRA" to the name "Laura" or "Lau". Instead, you can provide a list of keywords or technical terms, and the model will bias towards those words because that's how people actually spell them.

The key features are: it runs entirely offline, it has its own biasing dictionary, and it cleans up the text in a polished step. Overall, this gives a neat offline experience with no ongoing costs, which goes back to the cost motivation we discussed at the start.

Is that last polishing step done by setting special tokens in the model, or is it done by overlaying a map or something? How is that applied?

Give me two slides, and I'll answer your question. But that's a great question: is the polishing done inside the main model or outside, and how is biasing applied? We'll answer that shortly.

Let's talk about personalization. You can add terms like "LoRA", which is an example near and dear to our hearts. If you connect it to your Google account, it can import details from Gmail, look for unusual words, and add them to the biasing list, or you can add them manually. People usually put names in here because models frequently get uncommon names wrong, as well as technical terms. Gianning and Surreal are two of the people on the team who developed the app, so those are the examples we have.

The text polishing engine involves two steps. First, microphone input goes into a speech recognition engine that delivers an unfiltered transcription. In the bottom half, we have the personalization flow where you get a set of uncommon or unique words. Both of these go into the text polishing engine, which is a dedicated mini LLM just for text polishing.

We could have built one LLM to do all of this, but one of the realities of mobile development is modularity. You might use that same transcription engine elsewhere in your app and not want to pay the weight cost of the polishing model there, or you can reuse the weights across multiple places. Modularity is a pattern we see emerging as we build these types of apps. It is also easier to debug because you can inspect what is happening in the middle.

The ASR engine and text generation components are derived from Gemma technology. These aren't officially published Gemma models, but we've followed the same workflow we advise others to use. We've taken a smaller Gemma model—something from the Gemma 270M lineage—and fine-tuned it to create a transcription engine and a text polishing engine.

To give you insight into that workflow, we use a much stronger LLM in the cloud to generate synthetic data that corresponds to the target task. Once you have a few million or tens of million examples of synthetic data, you put that into a fine-tuning workflow with the base tiny model to get a derived model.

We've used that internally to ship a stronger note-taking app, and we're using this flow for many other use cases internally supporting Google's first-party products. Smaller Gemma models are highly powerful for this type of use case, and we're seeing good mileage from this approach.

That is what the text polishing engine does. It was instruction fine-tuned to accept a system prompt like, "These are your special words. Please correct anything that sounds like these words to the correct spelling, and remove interjections, lack of clarity, or phrases like 'scratch that'."

Is that a skill?

In this case, it's not a skill because it's a tiny model; we've actually trained the behavior directly into the model. With a four-billion-parameter model, you could probably do a version of this as a skill, which would be an interesting side project.

But for tiny models, the playbook we see as most useful is generating synthetic data with a larger model, picking an off-the-shelf base like Gemma 270M, fine-tuning it, and then using quantization. This allows you to ship a compelling, narrow feature to a wide set of users, powered by an LLM that works on many devices.

Do you have a GitHub repository showing the workflow for how to fine-tune it? It's not just standard fine-tuning; there is warming up the training and other steps.

With the Gemma 270M release, we do have a Colab notebook. Both Gemma 270M and Function Gemma have Colab notebooks that show how to perform full fine-tuning.

Regarding the fine-tuning, what was your experience with the function calling example? You mentioned an 80% or 85% chance of hitting the ten functions. Was that before or after fine-tuning, and what were the numbers before?

Before fine-tuning, it was around 40-something percent, and we finished at 86% overall. Within that 86%, we had ten functions, and perhaps two of them dragged our average down. There were eight simple functions that had over 93% reliability, and then two complex ones brought the average down to 86%. There is a blog post you can read for more details on that.

We see this with smaller models all the time. Our experience is that fine-tuning provides a gain of 20 to 40 points on the evaluation, which is a significant win for tiny models in the 200- to 270-million-parameter range. Fine-tuning is essential for most tasks unless you have a model that was published specifically for a narrow task from the outset, like some off-the-shelf transcription models. But if you want to perform your own narrow task and ship to many devices, fine-tuning is currently the workflow of choice.

Will we get a codelab that can use the web?

That's a good question. We have a Colab to get as far as the LiteRT file. Our support for LiteRT-LM on the web is a work in progress. Please check our GitHub repository for the latest status.

Will we get a fine-tuning manual for Gemma 2 by any chance?

What was announced last week for Gemma 2 were the small and medium-sized models. In the past, we also published tiny models. The first models we shipped for Gemma 2 were released last week, but there will be more Gemma models in the future.

There is some fine-tuning documentation already available for the larger models. For the tiny models we've published so far—and if you are reading or listening to this talk a few months in the future, please search the web for the latest information—they are Gemma tiny models, which have fine-tuning workflows available for Function Gemma and Gemma 270M. For Gemma 2 as released last week, there are fine-tuning recipes in Vertex AI, JAX, and elsewhere. Check out the other Gemma talks later in the week for more details.

The Eloquent app is currently available on iOS if anyone wants to give it a try.

Is it available in Europe? I could not find it on the App Store, though it shows up in a web browser.

Ah, so that needs to be enabled. That's helpful feedback, and I'll pass it along.

To wrap up: system GenAI with medium-sized or small models will be appearing on a mobile device near you soon. Those same models are excellent for use in embedded systems and platforms.

Given the current memory limitations on mobile phones, which aren't likely to increase significantly soon due to cost, tiny models are where it's at for wide deployment. We hope to make these easier to use by making stronger models available through our partnership with Google DeepMind, and we want to make the fine-tuning workflows as accessible as possible.

I'm happy to share what we've been doing on both of these fronts. We have about nine minutes left if anyone has any other questions.

That's a great question regarding safety on edge models. First, the Gemma team spends an awful lot of time on this, and I would defer to them for safety questions on the models published last week. It is top of mind for them.

Additionally, for system GenAI within AI Core, what actually ships as part of the OS (rather than the raw models we published last week) typically includes an input and output safety checker implemented by the system vendor as an aftermarket addition, since they have specific requirements for their product.

For really tiny models, safety is important, but the way we deploy them is within a narrow API or task. With Eloquent, the risk profile is different because it's more of a regenerative app than a generative app; it isn't going to make things up on the fly. Tiny models generally have a narrower functional scope, which means that while you still need to address safety, it is a more constrained problem to solve.

Is there a place where I can look for how to deploy small—or maybe a little bit bigger—Gemma models on a 5090? Because I have a 5090, I want to deploy on it and try it out. Is there a way I can look it up?

So, on a 5090... I'm not sure if we have specific documentation for that, but our tool does support NVIDIA GPUs. NVIDIA is also a Gemma partner, and they have supported some of the Gemma launches in the past. So, if you look on NVIDIA's own web page, you may see some of that through their TensorRT-LLM. I know they've supported some of the Gemma launches in the past, but I can't point you to specific documentation as a fast answer. Those are the two places I would check.

Yeah.

Okay.

Yeah, that's all right. So, the examples that you show for agent skills are obviously a single skill execution—you run the skill, and that's a single skill in the system. Would you sort of change the architecture for multi-skill?

It's a question of... so, we do actually support multi-skill execution. Within the app, if you download the app, you can actually define the skills you want loaded and you can just toggle them on or off. Then you can easily say—if you're very specific in your prompt, like, "Look up this topic in Wikipedia, summarize it to three bullets, and then display as flashcards"—if you're really specific, that's going to work.

Frankly, we've only had this model a couple of weeks, so we're still putting miles on the clock and seeing what the boundaries are for how much we can do skill stacking and skill chaining. We're literally still in the mode of putting miles on the clock there. Most of the examples we were publishing were single skill, but even in the diarization app, if you recall, Alice—the person doing that—asked to summarize her mood or what time she was meeting up with Amy, and then Wikipedia came up to search for something. Within that example, she was able to show, within a single conversation, individual turns using individual skills.

But skill stacking within an individual prompt, I believe we've seen that work where you're pretty explicit, but also, frankly, we're still learning the boundaries of what's possible with these classes of models.

Sorry, a follow-up question. So, for the decision to identify which skills are important for the problem, and given that it's a small model, do you do instruction tuning from a larger model to build that intelligence, or is it naturally good at it?

The model was trained to be really good at agentic workflows and function calling. Nothing in the Gemma model was trained specifically for our skill pattern; that just came afterwards. When we got the model and started playing with it, we were like, "Wow, this works! Can we do this?" So, it was more that kind of workflow, and there's no specific training for our skill structure.

But the Google DeepMind team spent lots of time doing general-purpose thinking and function calling. They did lots of great work there to give us a general-purpose model that is really strong, but there's nothing special for our app. So, if you had a slightly different take on skills—and maybe there's a better skill architecture than the one we've shown you today—it's entirely possible you could expect to be pretty successful. There's no hidden secret sauce in what we've shown.

I'll take the next question. I think you're next.

I'm building an agent, and the first big challenge is context, particularly once you start doing the agentic workflows. Could you talk a little bit about the context window on these models?

Yeah, I would defer you to the Gemma team for official guidance. The medium-sized models have a context window of 128K, and for the smaller models, I would actually need to double-check.

32K.

32K. Yeah, that's what I was wondering, if it was 32K. Our implementation, like in Gallery, defaults to about 8K or 12K just for performance reasons, but the models do support up to 32K for the 2B and 4B models, and the other models support up to 128K.

With 32K, do you use a lot of memory for that?

I wonder if we have stats for that in the model card. For the 2B and the 4B models, the memory footprint for a larger context is not as bad as you think. The team actually optimized that metric for those models because they were targeted for edge use cases. So, the amount of KV cache that's required for each input token was something that was optimized, and the models behave pretty well on that front. I don't have a number off the top of my head for bytes per input token to give you, but it's good for its model class, is what I would say.

And a second question: do you have an iOS version?

Yeah, the AI Edge Gallery works on both iOS and Android.

In terms of being open source, is there a macOS version?

macOS... That's a good point. I will put that in the "coming soon" bucket. It's certainly one of the items on our to-do list because we only published the iOS app for the first time in January, whereas the Android app has been available since last summer.

But our intention is to have a "what you see is what you get" experience for developers. So, you can use the app, have fun, experiment with the models, and then also get the source code to see how it's built. That is certainly our intention.

Probably the last question, because we're at time.

Is there a trade-off between fine-tuning individual models for a specific task and the actual amount of memory they consume on devices if you're chaining models together?

So, to clarify, for which model is your question?

I just downloaded the 2B model.

The 2B model, yeah. If you've got multiple models, presumably... For 2B, we would recommend customization via skills or via prompting, not via fine-tuning. For the small models that are published, we would recommend customization through skills and prompting.

For tiny models, we would recommend customization through fine-tuning if you were deploying a smaller model. The other path that is available to you for the small models is LoRA fine-tuning. I know Apple supports that in their foundation model framework, and you can check with the AI Core speaker if that's on their roadmap. There's an AI Core speaker who has an AMA coming up, and you can check with him about their roadmap for this.

But certainly, if you were deploying that on an embedded system—like if somebody asked me about deploying the 2B on a robotics platform—I would be like, "Absolutely, you should fine-tune LoRAs for each of your tasks." Our runtime supports loading the model and hot-swapping LoRAs, so you don't even need to load and unload the model to load and unload LoRA adapters. It's built for that particular use case, like robotics or IoT platforms. Those LoRAs then are much smaller—it depends on the rank you choose, but they're maybe 8 to 100 megabytes in that kind of range depending on the rank you use.

Thanks a lot.

Yeah. Cool, all right, that's a wrap. I'm going to let everybody get lunch. Thank you.