Sample Android App for JNI library #57

Acs176 · 2025-02-19T16:41:32Z

Description

This PR includes a sample android application under the android-app directory. It includes all the necessary .so's to run WhisperKit with any delegate. It's meant to have just very basic functionality (record, play audio, transcribe audio).

The only step needed to run the app is copying the openai_whisper-tiny/ folder into the assets folder of the android project. Currently I only support this model, can be updated in the future.

Features

Audio Transcription

There's a dropdown to select the audio file and a Transcribe button to do just that.

Recording

Press the record button once to start a microphone recording, press again to end it. The file will be stored with the name MicInput.wav and can be selected from the dropdown to play it or transcribe it.

Known issues

Model caching doesn't work correctly with QNN delegate and it takes several minutes to load the model when starting the app (if using that delegate).
Issues running in Android Studio emulator. I've experienced some strange crashes when running on the emulator which I suspect are related to the QNN .so files not loading successfully

v-prgmr · 2025-02-19T16:50:46Z

This is awesome, i am going to try it out now. Thank you so much 🚀

v-prgmr · 2025-02-19T21:24:06Z

.apk works like a charm. Just ran the tiny model on a S24 Ultra SM8650. Great work Thank you! @Acs176

Acs176 · 2025-02-20T09:38:41Z

Great news @v-prgmr! Could you share some latency data when running the transcribe on that device?
So far I've only been able to run it on an SM7450 (added the soc manually to the allowed list) and get around 1400ms transcribing the jfk.wav file.

Acs176 · 2025-02-20T14:28:57Z

I updated the PR because I noticed a bug. The lib directory to access the .so files was being hardcoded in the c++ side. The android app sets this directory on runtime when loading the libraries (it puts some hash in the path), so these .so files were not being loaded correctly into the app and you couldn't use QNN unless you had previously run the adb-push.sh script, which puts the files in the folder that TranscribeTask.cpp indicated.

I allowed the WhisperKitRunner to receive the libs path through the NativeWhisperKit from the Android app on runtime. This ensures that the libraries for QNN load correctly.

I also added a section in the screen of the app to display tflite logs from logcat. That way you get some more feedback in case you run from apk.

Try it out again @v-prgmr, you should get much faster responses if QNN was not loading correctly.

v-prgmr · 2025-02-20T14:59:58Z

@Acs176 just pulled the latest commit from your fork and rebuilt the app and ran it on SM8650.

Here are the screenshots for jfk.wav and english_test2.wav

bpkeene · 2025-02-26T17:46:55Z

Review is in progress! Will post feedback shortly

yuguolong · 2025-03-03T05:38:53Z

Great work!

bpkeene · 2025-03-06T13:37:55Z

androidApp/app/src/main/assets/MicInput.wav

Can we remove the .wav files? We have a couple representative test files in $SOURCE/test/ already, perhaps a script could move them to app assets as part of the app build process?

fixed a script to avoid duplicating test files and .so's

bpkeene · 2025-03-06T13:38:17Z

androidApp/app/src/main/assets/english_test1.wav

ditto, and so forth for other .wav files

bpkeene · 2025-03-06T13:39:07Z

androidApp/app/src/main/java/com/whispertflite/MainActivity.kt

+
+            // QNN libraries
+            "QnnSystem",    // Core system library for QNN
+            "QnnGpu",       // GPU support for QNN


Is there a way to handle the Qnn Skel & Stubs in the gradle file?

I haven't found a way to avoid loading them manually with System.loadLibrary()

bpkeene · 2025-03-06T13:40:35Z

androidApp/app/src/main/java/com/whispertflite/MainActivity.kt

+        )
+
+        fun loadAllLibraries() {
+            libraries.forEach { lib ->


I am surprised we can load QnnHtpV75Skel on SoCs that do not correspond to it (or, non-QCOM SoCs).

We should load minimal dependencies (see base list in CMakeLists) and then have soft linking / SoC check before linking to Qnn libs, what do you think?

bpkeene · 2025-03-06T13:43:06Z

androidApp/app/src/main/java/com/whispertflite/MainViewModel.kt

+        resultState = mutableStateOf("")
+        statusState = mutableStateOf("LOADING MODEL (This may take a few minutes when using QNN)")
+
+        waveFile = File(sdcardDataFolder!!.absolutePath + "/" + microphoneInputFileName)


If we remove the initial assets, perhaps we could initialize this via a safe file loader- if the file exists, load it; else, create a writeable file with .wav extension ?

if we don't remove them, would be preferable to have the existing assets included in the app build rather than duplicated in the repo

went with the solution to check if the file is already existing

bpkeene · 2025-03-06T13:51:21Z

androidApp/app/src/main/java/com/whispertflite/utils/WaveUtil.java

+import java.nio.ByteOrder;
+import java.nio.charset.StandardCharsets;
+
+public class WaveUtil {


javax.sound.sampled has .wav utilities that may be useful and is in the standard jdk, can avoid manually writing the .wav header magic-

https://stackoverflow.com/questions/3297749/java-reading-manipulating-and-writing-wav-files

this library is sadly not supported in android projects

understood, thank you for clarifying!

Can we use more standard interfaces? https://developer.android.com/reference/android/media/.

Some of these should allow us to use standard interfaces

bpkeene · 2025-03-06T13:54:01Z

androidApp/app/src/main/java/org/libsdl/app/HIDDeviceBLESteamController.java

+import java.util.LinkedList;
+import java.util.UUID;
+
+class HIDDeviceBLESteamController extends BluetoothGattCallback implements HIDDevice {


What is this for? It seems there is some chromebook specific considerations as well?

bpkeene · 2025-03-06T13:55:12Z

androidApp/app/src/main/java/org/libsdl/app/HIDDeviceManager.java

+import java.util.Iterator;
+import java.util.List;
+
+public class HIDDeviceManager {


do we need these HID / USB considerations if the app is instead installed to the device after being built on host?

is there a source or reference for this file?

bpkeene · 2025-03-06T13:58:46Z

androidApp/app/src/main/java/org/libsdl/app/HIDDeviceManager.java

+        return false;
+    }
+
+    private boolean isXboxOneController(UsbDevice usbDevice, UsbInterface usbInterface) {


seems unrelated ? ditto for a few other functions

bpkeene · 2025-03-06T13:59:44Z

androidApp/app/src/main/java/org/libsdl/app/SDLActivity.java

+    private static final int SDL_MAJOR_VERSION = 3;
+    private static final int SDL_MINOR_VERSION = 2;
+    private static final int SDL_MICRO_VERSION = 1;
+/*


can we remove the commented code?

bpkeene · 2025-03-06T14:02:53Z

androidApp/app/src/main/java/org/libsdl/app/SDLActivity.java

+package org.libsdl.app;
+
+import android.app.Activity;
+import android.app.AlertDialog;


why do we need SDL initialized or interfacing in the native layer?

There is initialization & handling of this in the internal cpp library. I would think we can use standard audio functionality at the app layer for audio capture & file read/write, lightweight calls in to whisperkit native layer from the app and the developer doesn't need to know about SDL / audio resampling - they pass the file, they receive the transcription

Is that understanding correct?

This one maybe we can solve. I've been trying hard to avoid having to embed this SDL app into our app, but if I don't do it, I am met with this error:

2025-03-11 09:50:50.851 14525-14525 m.whispertflite com.whispertflite A java_vm_ext.cc:616] JNI DETECTED ERROR IN APPLICATION: JNI FindClass called with pending exception java.lang.ClassNotFoundException: Didn't find class "org.libsdl.app.SDLActivity" on path: DexPathList[[dex file "/data/data/com.whispertflite/code_cache/.overlay/base.apk/classes5.dex", zip file "/data/app/~~1muUDF0X1sTtD6EmTn-Juw==/com.whispertflite-BC8hIf5K5XQDH5WQKpRPBA==/base.apk"],nativeLibraryDirectories=[/data/app/~~1muUDF0X1sTtD6EmTn-Juw==/com.whispertflite-BC8hIf5K5XQDH5WQKpRPBA==/lib/arm64, /data/app/~~1muUDF0X1sTtD6EmTn-Juw==/com.whispertflite-BC8hIf5K5XQDH5WQKpRPBA==/base.apk!/lib/arm64-v8a, /system/lib64, /product_h/lib64, /system_ext/lib64]]

If I try to load the SDL3 .so, I also get this error unless the app is embedded like I did. I'm unsure on what else to try in order to load all .so's successfully without having to include that app in the code.

Any input regarding this? @bpkeene

bpkeene · 2025-03-06T14:08:05Z

androidApp/app/src/main/jniLibs/arm64-v8a/libQnnDsp.so

We don't want to add these .so files to the github repo - they're actually pulled in as part of the Docker environment creation and placed in .source/ , can you reference them as part of the app build process to put in the .apk? (ditto elsewhere)

bpkeene

looked over things, can we make changes with respect to java-SDL (remove?) and not distribute the qnn libs? They are already pulled in as part of the docker environment creation, we should be able to reference them appropriately as part of the gradle build system

Acs176 · 2025-03-06T15:21:02Z

@bpkeene thanks for the very thorough review! I will get back to you and start fixing the issues asap :)

…reen

nyhryan · 2025-04-30T13:18:53Z

Hey, I am trying to run your sample app. I have cloned your branch android-app and tried to run with Galaxy S25. However the launch will fail. I can see some of .so files were failed to be loaded. (libQnnDspV66Skel.so, libQnnHtpV68Skel.so, libQnnHtpV69Skel.so...)

These warnings have libQnnDspV66Skel.so" is 32-bit instead of 64-bit messages.

And it fails with Failed to register methods of org/libsdl/app/SDLAudioManager...

Here's a log from logcat when I tried to run in AndroidStudio to Galaxy S25. https://pastebin.com/E13JqP1A

I am currently using M1 Macbook Air, macOS 15.4.1

make setup (confirmed everything is installed)
make download-models
make env
make build (inside make env docker environment)
(plus I ran make build qni,  make build qnn)

I have run these following commands. Plus ran your scripts/build_android_app.sh, copied models/openai_whisper-tiny folder into the assets folder.

bpkeene reviewed Mar 6, 2025

View reviewed changes

androidApp/app/src/main/assets/english_test1.wav Outdated

Copy link

Contributor

bpkeene Mar 6, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto, and so forth for other .wav files

bpkeene reviewed Mar 6, 2025

View reviewed changes

bpkeene requested changes Mar 6, 2025

View reviewed changes

Acs176 added 5 commits March 10, 2025 15:13

push android app for whisperkit

2e0fc05

update readme with android info

e843749

allow getting native libs dir from android app, add tflite logs on sc…

a769dbc

…reen

remove local.properties and update .gitignore

ddebbbf

remove redundant files, safely create mic input file

89dd831

Acs176 force-pushed the android-app branch from b822c64 to 89dd831 Compare March 10, 2025 14:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sample Android App for JNI library #57

Sample Android App for JNI library #57

Acs176 commented Feb 19, 2025

v-prgmr commented Feb 19, 2025

v-prgmr commented Feb 19, 2025

Acs176 commented Feb 20, 2025

Acs176 commented Feb 20, 2025

v-prgmr commented Feb 20, 2025

bpkeene commented Feb 26, 2025

yuguolong commented Mar 3, 2025

bpkeene Mar 6, 2025

Acs176 Mar 10, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

Acs176 Mar 10, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

Acs176 Mar 10, 2025

bpkeene Mar 6, 2025

Acs176 Mar 10, 2025

bpkeene Mar 15, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

bpkeene Mar 6, 2025

Acs176 Mar 11, 2025

Acs176 Mar 25, 2025

bpkeene Mar 6, 2025

bpkeene left a comment

Acs176 commented Mar 6, 2025

nyhryan commented Apr 30, 2025 •

edited

Loading

Sample Android App for JNI library #57

Are you sure you want to change the base?

Sample Android App for JNI library #57

Conversation

Acs176 commented Feb 19, 2025

Description

Features

Audio Transcription

Recording

Known issues

v-prgmr commented Feb 19, 2025

v-prgmr commented Feb 19, 2025

Acs176 commented Feb 20, 2025

Acs176 commented Feb 20, 2025

v-prgmr commented Feb 20, 2025

bpkeene commented Feb 26, 2025

yuguolong commented Mar 3, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bpkeene left a comment

Choose a reason for hiding this comment

Acs176 commented Mar 6, 2025

nyhryan commented Apr 30, 2025 • edited Loading

nyhryan commented Apr 30, 2025 •

edited

Loading