Lesson 17.6 · 25 min

On-Device ML: A TensorFlow Lite Android Walkthrough

How does a model trained in Python end up recognising a hand-drawn digit inside an Android app, with no internet at all?

In short: TensorFlow Lite (now also called LiteRT) is a runtime for running trained models on phones and small devices. We train a model in Python, convert it to a compact .tflite file (usually quantised to 8-bit integers), bundle it in the app's assets, load it with an Interpreter in Kotlin, and call run() on preprocessed input. We also simulate int8 inference in numpy to see why the quantised model is 4× smaller yet gives almost the same answers.

Why TensorFlow Lite exists

In the previous lesson we saw why we might run a model on the device: low latency, privacy, offline use and no server bill. But a model trained with full TensorFlow on a laptop is not ready for a phone. The training library is large, the model file stores 32-bit floats, and the code expects a desktop CPU or a big GPU. We need a small, fast engine that only does one thing: run an already-trained model.

TensorFlow Lite (TFLite) is that engine. It has two halves: a converter, which runs on our computer and turns a trained model into a compact .tflite file (a FlatBuffer, a binary format that can be read without unpacking), and an interpreter, a small runtime library inside the app that loads that file and executes it on the phone's CPU, GPU or other accelerators. In 2024 Google renamed TFLite to LiteRT ("Lite Runtime"); the file format and the core ideas are the same, and older org.tensorflow:tensorflow-lite packages remain widely used, so we will see both names.

Think of it like a printed recipe card Training is the test kitchen with every pot and ingredient, where a chef experiments for weeks. The .tflite file is the final recipe card: only the steps needed to cook the dish, with measurements rounded to what a home cook can use. The interpreter is the home cook who follows the card quickly in a small kitchen (the phone).

Our running example: a handwritten digit recogniser. The user draws a digit 0–9 on the screen, and the app instantly says which digit it is. The model is trained on MNIST, a classic dataset of 70,000 grayscale 28 × 28 pixel images of handwritten digits.

The end-to-end workflow

Two words appear in every step. A tensor is just a multi-dimensional array of numbers: a 28 × 28 image is a 2-D tensor, a batch of one image with one colour channel is a 4-D tensor of shape [1, 28, 28, 1]. The input and output signature is the agreed shape and type of these tensors. Most bugs in on-device ML come from the app feeding a tensor that does not match what the model was trained on.

Converting and quantising the model

After training, conversion takes a few lines of Python. Setting optimizations turns on quantisation. Giving a representative dataset (a few hundred real inputs) lets the converter measure the range of every intermediate value, so it can quantise activations too, not just weights. This is called post-training quantisation because it happens after training, with no retraining.

convert.py (needs TensorFlow; shown for reference)

import tensorflow as tf
# model = a trained tf.keras model; x_train = MNIST images scaled to [0, 1]
converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]     # enable quantisation
def representative_dataset():
for img in x_train[:200]:                          # ~100-500 samples is typical
yield [img[None, ..., None].astype("float32")]  # shape [1, 28, 28, 1]
converter.representative_dataset = representative_dataset
tflite_bytes = converter.convert()
with open("digits.tflite", "wb") as f:
f.write(tflite_bytes)

Quantisation maps each real number to an 8-bit integer using a scale (how much one integer step is worth) and a zero point (which integer stands for 0.0). TFLite's int8 scheme uses symmetric quantisation for weights (zero point 0, often one scale per output channel) and asymmetric quantisation for activations (a zero point lets the range be shifted, for example to cover [0, 6] after a ReLU).

Setting up the Android project

Project setup, step by step

1. Add the runtime dependency: In the app module's build.gradle.kts add the TFLite library, for example implementation("org.tensorflow:tensorflow-lite:<version>"). Newer projects can use the LiteRT artifact instead; check the current docs for the exact coordinates and version.
2. Put the model in assets: Copy digits.tflite into app/src/main/assets/. Everything in assets is packaged inside the APK and readable at runtime.
3. Stop the build from compressing it: Add androidResources { noCompress += "tflite" } (older Gradle: aaptOptions { noCompress "tflite" }). The interpreter memory-maps the file, and a compressed asset cannot be memory-mapped.
4. Create the Interpreter once: Loading a model takes time and memory, so create the Interpreter when the screen or app starts and reuse it for every prediction, then close() it when done.
5. Run inference off the main thread: Even a fast model can take tens of milliseconds. Run it in a background coroutine so the UI never freezes.

Memory-mapping means the operating system makes the file appear as if it were in memory without copying it all into the app's heap; pages are read from storage when touched. This makes loading fast and keeps memory use low, which is why noCompress matters.

The Kotlin integration code

Here is a complete classifier class. It loads the model, converts a Bitmap into the float tensor the model expects, runs inference, and returns the best digit with its probability. It runs on a device, so there is no console output to show here; the next section simulates the same math in Python.

DigitClassifier.kt

import android.content.Context
import android.graphics.Bitmap
import android.graphics.Color
import org.tensorflow.lite.Interpreter
import java.io.FileInputStream
import java.nio.ByteBuffer
import java.nio.ByteOrder
import java.nio.MappedByteBuffer
import java.nio.channels.FileChannel
class DigitClassifier(context: Context) {
private val interpreter: Interpreter
init {
val options = Interpreter.Options().apply { setNumThreads(4) }
interpreter = Interpreter(loadModel(context, "digits.tflite"), options)
}
private fun loadModel(context: Context, name: String): MappedByteBuffer {
val fd = context.assets.openFd(name)
FileInputStream(fd.fileDescriptor).use { stream ->
return stream.channel.map(
FileChannel.MapMode.READ_ONLY, fd.startOffset, fd.declaredLength)
}
}
fun classify(drawing: Bitmap): Pair<Int, Float> {
val img = Bitmap.createScaledBitmap(drawing, 28, 28, true)
val input = ByteBuffer.allocateDirect(4 * 28 * 28).order(ByteOrder.nativeOrder())
for (y in 0 until 28) for (x in 0 until 28) {
val p = img.getPixel(x, y)
val gray = (Color.red(p) + Color.green(p) + Color.blue(p)) / 3f
input.putFloat(gray / 255f)          // same [0, 1] scaling as training
}
val output = Array(1) { FloatArray(10) } // shape [1, 10]
interpreter.run(input, output)
val probs = output[0]
val best = probs.indices.maxByOrNull { probs[it] } ?: 0
return best to probs[best]
}
fun close() = interpreter.close()
}

The most common bug: preprocessing mismatch MNIST digits are white strokes on a black background. If our drawing canvas is black ink on white paper, we must invert the pixels (1f - gray / 255f) or the model sees a completely different kind of image and answers nonsense with high confidence. The same goes for channel order, value range ([0, 1] vs [−1, 1]) and image orientation.

From an Activity or ViewModel we would call it inside a background coroutine, for example val (digit, conf) = withContext(Dispatchers.Default) { classifier.classify(bitmap) }, then update a TextView on the main thread with the result.

Simulating int8 inference in numpy

What actually happens inside the interpreter for a quantised layer? The phone multiplies 8-bit integers, adds the products into 32-bit integer accumulators (so they do not overflow), and converts back to real units once at the end. Integer math is cheaper in energy and is what many mobile accelerators are built for. Let us simulate one dense layer of a tiny digit-like classifier (64 inputs, 10 classes) and compare it with float32.

int8_inference_sim.py

# Simulate what TFLite does on a phone: run a float model and its int8 version.
import numpy as np
rng = np.random.default_rng(0)
# A tiny "trained" classifier: 64 input features -> 10 classes (like 8x8 digits)
W = rng.normal(0, 0.3, (64, 10)).astype(np.float32)
b = rng.normal(0, 0.1, 10).astype(np.float32)
X = rng.random((500, 64)).astype(np.float32)          # 500 test "images" in [0, 1]
def quantize(t, num_bits=8):
"""Symmetric per-tensor int8: real ≈ scale * q, q in [-127, 127]."""
scale = np.abs(t).max() / 127
q = np.clip(np.round(t / scale), -127, 127).astype(np.int8)
return q, scale
# Converter (once, on a laptop) quantizes W; the phone quantizes each input
Wq, w_scale = quantize(W)
Xq, x_scale = quantize(X)
bq = np.round(b / (w_scale * x_scale)).astype(np.int32)   # bias kept in int32
# Phone step: integer matmul with int32 accumulation, then rescale once
acc = Xq.astype(np.int32) @ Wq.astype(np.int32) + bq      # pure integer math
logits_int8 = acc * (w_scale * x_scale)                   # back to real units
logits_fp32 = X @ W + b
pred_fp = logits_fp32.argmax(1)
pred_q = logits_int8.argmax(1)
print("weight scale:", round(float(w_scale), 5), " example W[0,0]:",
round(float(W[0, 0]), 4), "-> q =", int(Wq[0, 0]),
"-> back to", round(float(Wq[0, 0] * w_scale), 4))
print("model size fp32:", W.nbytes + b.nbytes, "bytes; int8:", Wq.nbytes + bq.nbytes, "bytes")
print("max |logit error|:", round(float(np.abs(logits_fp32 - logits_int8).max()), 4))
print("predictions that agree:", f"{(pred_fp == pred_q).mean():.1%}")
probs = np.exp(logits_int8[0] - logits_int8[0].max()); probs /= probs.sum()
print("image 0 -> class", pred_q[0], "with prob", round(float(probs.max()), 3))

Output:

weight scale: 0.00921  example W[0,0]: 0.0377 -> q = 4 -> back to 0.0368
model size fp32: 2600 bytes; int8: 680 bytes
max |logit error|: 0.0384
predictions that agree: 99.2%
image 0 -> class 9 with prob 0.708

Pause and think: In the simulation, 4 of the 500 predictions changed after quantisation. Why do a few change while most stay the same?

Quantisation adds a small error (here at most 0.04) to each logit. When the top two classes are far apart, that error cannot change which is bigger. Only images where two classes were nearly tied flip. That is why we always re-measure accuracy on a real test set after converting.

Hardware delegates and the alternatives

By default the interpreter runs on the CPU using optimised kernels. A delegate hands all or part of the model graph to other hardware: the GPU delegate runs float (and some quantised) models on the phone GPU; vendor NPU delegates target dedicated AI chips. Android's older NNAPI path was deprecated in Android 15, and Google now points developers to newer LiteRT acceleration options, so check the current documentation before choosing. If a delegate does not support an operation, that part falls back to the CPU, which can make a "GPU" model slower than expected.

Other runtimes such as ONNX Runtime Mobile and PyTorch's ExecuTorch fill the same role for models trained in PyTorch. The workflow is the same shape: train, export to a mobile format, bundle, load, run.

Worked example, step by step

The numpy simulation used the simple, symmetric scheme with zero point 0. Activations use the other scheme from the formula: a scale and a zero point. Let us quantise one activation by hand. Suppose the representative dataset showed that a layer's outputs always lie between 0 and 6 (the example range mentioned earlier for values after a ReLU).

Quantising an activation in the range [0, 6]

1. Find the scale: int8 has 256 levels, so 255 steps cover the range: scale = (6 − 0) / 255 ≈ 0.02353.
2. Find the zero point: We want the lowest real value, 0.0, to land on the lowest integer, −128. So the integer that stands for 0.0 is zero_point = −128.
3. Quantise 1.5: q = round(1.5 / 0.02353) + (−128) = 64 − 128 = −64.
4. Decode it: real ≈ 0.02353 × (−64 − (−128)) = 0.02353 × 64 ≈ 1.506. The error is 0.006, at most half a step.
5. A value outside the range: 7.2 would need q = 306 − 128 = 178, which does not fit in int8. It is clamped to 127 and decodes to 6.0. Everything above 6 looks the same to the model. This is saturation.
6. Why not symmetric here?: A symmetric scheme covers [−6, 6], so half of its levels stand for negative values that never occur after a ReLU. Its step is 6 / 127 ≈ 0.0472, twice as coarse as 0.0235.

The same arithmetic explains two failures. If the representative samples are not typical, the recorded range is too narrow and real inputs saturate. And if a tensor contains one extreme value, the scale must stretch to reach it, so every ordinary value is rounded on a much coarser grid. Both show up as an accuracy drop after conversion with no error message, which is why we compare the converted model with the original on a test set.

Practice: try it yourself

We will write the scale-and-zero-point scheme ourselves in numpy. First we repeat the hand calculation, then we measure how much better the asymmetric scheme is on [0, 6] activations, and finally we watch one outlier weight spoil the precision of a thousand normal ones.

practice_zero_point.py

import numpy as np
rng = np.random.default_rng(4)
def asym_params(lo, hi):
"""Scale and zero point that map the real range [lo, hi] onto int8."""
scale = (hi - lo) / 255
zero_point = int(round(-128 - lo / scale))
return scale, zero_point
def quantize(x, scale, zero_point):
return np.clip(np.round(x / scale) + zero_point, -128, 127).astype(np.int8)
def dequantize(q, scale, zero_point):
return scale * (q.astype(np.float32) - zero_point)
# 1) One activation by hand, for a layer whose outputs lie in [0, 6]
scale, zp = asym_params(0.0, 6.0)
for real in (0.0, 1.5, 6.0, 7.2):
q = quantize(np.array(real), scale, zp)
print(f"real {real:3.1f} -> q = {int(q):4d} -> back to "
f"{float(dequantize(q, scale, zp)):.3f}")
print(f"scale = {scale:.5f}, zero point = {zp}")
# 2) Same activations, symmetric vs asymmetric: which wastes fewer levels?
acts = rng.uniform(0, 6, size=5000).astype(np.float32)
sym_scale = 6.0 / 127                            # symmetric covers [-6, 6]
err_sym = np.abs(dequantize(quantize(acts, sym_scale, 0), sym_scale, 0) - acts)
err_asym = np.abs(dequantize(quantize(acts, scale, zp), scale, zp) - acts)
print(f"mean error  symmetric: {err_sym.mean():.4f}   asymmetric: {err_asym.mean():.4f}")
# 3) One outlier weight stretches the scale for everyone else
w = rng.normal(0, 0.2, size=1000).astype(np.float32)
for label, weights in [("no outlier", w), ("one weight = 8.0", np.append(w, 8.0))]:
s = np.abs(weights).max() / 127
err = np.abs(dequantize(quantize(weights, s, 0), s, 0) - weights)[:1000]
print(f"{label:17s}: scale {s:.4f}, mean error {err.mean():.4f}, "
f"levels used by normal weights: {len(np.unique(quantize(w, s, 0)))}")

Output:

real 0.0 -> q = -128 -> back to 0.000
real 1.5 -> q =  -64 -> back to 1.506
real 6.0 -> q =  127 -> back to 6.000
real 7.2 -> q =  127 -> back to 6.000
scale = 0.02353, zero point = -128
mean error  symmetric: 0.0118   asymmetric: 0.0058
no outlier       : scale 0.0049, mean error 0.0013, levels used by normal weights: 187
one weight = 8.0 : scale 0.0630, mean error 0.0161, levels used by normal weights: 21

Now change it:

• Call asym_params(-2.0, 6.0) in part 1. Predict the new scale and, more interesting, the zero point: which integer now stands for 0.0?
• In part 2, draw the activations from rng.uniform(-6, 6, ...) but leave both schemes as they are. Predict which scheme now has the larger error, and why.
• In part 3, change the outlier from 8.0 to 0.8. Predict roughly how many levels the normal weights use now.

Pause and think: Our representative dataset holds only faint, thin digits, so the converter records a maximum activation of 3.0 for some layer. In real use, bold drawings produce values up to 6. What happens, and how do we fix it?

The scale and zero point are set for the range [0, 3], so every activation above 3 is clamped to the top integer and decodes to 3.0. The layer can no longer tell 'strong' from 'very strong', and accuracy drops on bold drawings, with no error raised. The fix is a representative dataset that covers the real variety of inputs, so the recorded ranges match what the app will see.

Pause and think: In the outlier run, the weight 8.0 itself is stored almost perfectly. So why does it hurt the model?

Because one scale is shared by the whole tensor. To reach 8.0 with 127 steps, each step must be about 0.063, but the ordinary weights are mostly within ±0.6, so they now fall on only 21 distinct values instead of 187. Their rounding error grows about twelve-fold, and there are a thousand of them. This is also why a separate scale per output channel helps: an outlier then only coarsens its own channel.

Common mistakes and when not to use it

• Compressed asset. Forgetting noCompress makes memory-mapping fail when loading the model.
• Shape or type mismatch. Feeding a [28, 28] float array when the model wants [1, 28, 28, 1], or floats when a fully integer model wants int8, throws an error or returns garbage. Inspect the model's input and output tensors before writing app code.
• Different preprocessing. Value range, colour channels, inversion and resizing must match training exactly.
• Creating an Interpreter per prediction. Loading is slow; create it once and reuse it (an Interpreter is not thread-safe, so use one per thread or synchronise).
• Running on the main thread. Causes jank or "app not responding" dialogs.
• Not measuring accuracy after quantisation. Usually the drop is small, but some models (especially with outlier values) lose more; compare the .tflite model against the original on a test set.

When not to use TFLite on-device Skip it when the model is far too large for phones, when a ready-made API (like ML Kit) already does the job, or when the model must change daily. In those cases cloud inference or a packaged API is less work.

Key takeaways

• TFLite/LiteRT = a converter (makes a compact .tflite file) plus an interpreter (runs it on the device).
• Workflow: train in Python, convert (often to int8), put the file in assets uncompressed, load once, run off the main thread.
• int8 quantisation stores real ≈ scale × (q − zero_point); integer math with int32 accumulation is ~4× smaller and cheap on phones.
• Most on-device bugs are input mismatches: shape, type, value range, colour, inversion.
• Always re-measure accuracy after conversion, and test on low-end devices.

Key terms

• TensorFlow Lite / LiteRT: Google's runtime for running trained models on mobile and embedded devices; renamed LiteRT in 2024.
• .tflite file: A FlatBuffer file holding the converted model graph and weights.
• Interpreter: The TFLite runtime object that loads a .tflite model and runs inference.
• Tensor: A multi-dimensional array of numbers, such as an image of shape [1, 28, 28, 1].
• Post-training quantisation: Converting a trained float model to lower-precision integers without retraining.
• Representative dataset: A small set of real inputs the converter uses to measure value ranges for quantisation.
• Delegate: A plugin that runs part or all of a TFLite model on other hardware such as a GPU or NPU.

← 17.5 Cloud vs Edge: Where Should Your Model Run? · 17.7 LLM Routing: Directing Each Query to the Best Model →