Qwen3-1.7B LiteRT-LM Model
This repository contains LiteRT-LM variants of Qwen/Qwen3-1.7B optimized for on-device text generation.
Available Artifact
Conversion Notes
Qwen3-1.7B_dynamic_wi4b32_afp32.litertlm is a dynamic INT4 variant (block-32 weights, FP32 activations). It was converted through the LiteRT Torch (litert-torch) path and quantized with AI Edge Quantizer. This artifact incorporates LiteRT-LM GPU graph optimizations, including composite ops for RoPE, fused QKV, and fused Gate/Up projections, and is configured with static prefill memory allocation.
How to Use
Command-Line Interface
- Install the prerequisites:
- Run the command in CLI:
Edge Gallery App
- Get the App: Install the app from Google Play or download the latest APK from the GitHub releases page.
- Importing the Model: Navigate to the Model manager within the app and click the "+" (plus) icon in the bottom-right corner. Two options will appear:
- Import from HF (Recommended): Select this option, and a dialog box will appear showing an example Hugging Face model URL. Enter the HF link for the desired
.litertlm model and click submit. The model will then appear in your list, and you can proceed to download it (a Hugging Face account login is required).
- From local model file: First, download the
.litertlm model directly to your Android device, OR download it to your computer and push it via ADB (e.g., adb push Qwen3-1.7B_dynamic_wi4b32_afp32.litertlm /sdcard/Download/). Then, select this option, choose the downloaded file from your storage, configure your preferred parameters, and tap "Import".
For full details on importing models and other features, see the Edge Gallery App Wiki.
To build the demo app from source, please follow the instructions from the GitHub repository.
Python
- Install the prerequisites:
- Download the model file:
- Run inference:
Performance (on device, measured)
Community measurement on a physical Samsung Galaxy S26 (SM-S942Q, Snapdragon 8 Elite Gen 5 / SM8850, Android 16): litert_lm_advanced_main from the litert-lm v0.16.0 release, GPU backend OpenCL (LITERT_CL), one fixed 205-token prompt text (202 tokens under this tokenizer), --benchmark. Two runs per backend taken back-to-back — cells show the range (the phone warms across a series; the CPU's first run also pays cold file I/O). Peak RSS is the process VmHWM. Before quoting, every file/backend pair was run with a real prompt and produced a correct text answer; the GPU rows are full delegation (e.g. Qwen3-1.7B_dynamic_wi4b32_afp32: decode 994/994 ops on LITERT_CL, nothing on the CPU fallback).
What the table says:
- On the S26 GPU the wi4b32 file is the one to run: 40.7 tok/s decode against 18.1 for the int8 file — the GPU graph optimizations described in Conversion Notes above, visible in measurement. It is also the fastest prefill and the lowest time-to-first-token on the card.
- The wi4b32 file is a GPU specialist. On CPU it decodes at 8 tok/s and prefills at 48–76 tok/s — int4 block-32 weights suit the GPU path, not XNNPACK. If you need CPU, the int8
Qwen3_1.7B.litertlm prefills 3–5× faster than wi4b32 on the same CPU.
- Peak RSS is 2.4–3.7× lower on the GPU for both files (854–1036 MB against 2.5–3.2 GB).
- GPU engine init is the one cost the table hides: 4.4–8.3 s per process against 2.2–3.5 s on CPU.
Integration
Ready to integrate this into your product? Get started in the LiteRT-LM documentation.