Using Tensorflow-Lite GPU delegate in Android's Native environment with C-API

user_5456541 0
Viewed 1308

Info

I'm using Tensorflow-Lite in Android's Native environment via the C-API (following these instructions) but runtime is significantly longer compared to the GPU delegate via the Java API (on ART).

The JNI AAR file (2.2) offers C-headers and a shared-library, but it seems that the shared-library doesn't contain the GPU delegate, but only a framework to configure delegates on (TfLiteDelegate object and TfLiteDelegateCreate()).

** It doesn't provide any TfLiteGpuDelegateV2Create() or tflite namespace access, for example.

Trials

  • I've tried to include a libtensorflowlite_gpu_delegate.so in the project with cmake but though it seems to build and link OK - the library isn't accessible via Native code.
  • I tried following c_api.h's example of delegate usage, but I can't seem to configure a GPU delegate.
  • Docker container doesn't include toolchain (trying to build shared library in tensorflow/tensorflow:latest-devel-gpu Tensorflow Docker image with bazel build -c opt --config android_arm64 tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so fails with cc_toolchain_suite '@local_config_cc//:toolchain' does not contain a toolchain for cpu 'arm64-v8a')

Question

How can I run an inference with the GPU delegate in Android's Native environment using the C-API?

tensorflow-lite android-ndk gpu delegates c
1 Answers

I managed to do it as follows:

1. Clone and configure tensorflow

Clone tensorflow repo from GitHub, cd into it and run ./configure. There it is important to answer Would you like to interactively configure ./WORKSPACE for Android builds? [y/N] with y and correctly specify the Android NDK and SDK directories.

2. Build libtensorflow-lite_gpu_delegate with bazel

I succeeded building the GPU delegate shared library with

bazel build -c opt --cxxopt=--std=c++11 --config android_arm64 tensorflow/lite/delegates/gpu:libtensorflowlite_gpu_delegate.so

I built against Android NDK 18.1.5063045 with minimum API level 27. Note that I only tested for the android_arm64 architecture, I cannot give guarantees for other architectures.

(The time I compiled TensorFlow the HEAD pointed at commit 0f8a27183657972c8ba2bce150e1364179ded6f9.)

3. Update CMakeLists.txt

The relevant lines are the following:

include_directories(
    /Users/<name>/tensorflow/tensorflow/lite/delegates/gpu # for Mac 
)

add_library(tensorflow-lite_gpu_delegate SHARED IMPORTED)
set_target_properties(tensorflow-lite_gpu_delegate PROPERTIES IMPORTED_LOCATION
    /private/var/tmp/_bazel_<name>/fe60511640322ef6962b77bab4b291e3/execroot/org_tensorflow/bazel-out/arm64-v8a-opt/bin/tensorflow/lite/delegates/gpu/libtensorflowlite_gpu_delegate.so) # I obtained this path pressing Cmd+Option+C on the libtensorflow-lite_gpu_delegate.so file on Mac, might be different on your OS

target_link_libraries(
    tensorflow-lite_gpu_delegate
    )

4. Use GPU delegate in code

The relevant lines are the following:

#include <delegate.h>

auto *delegate = TfLiteGpuDelegateV2Create(/*default options=*/nullptr);

// Create the model and interpreter options.
TfLiteModel *model = TfLiteModelCreate(/* create as usual */);
TfLiteInterpreterOptions* options = TfLiteInterpreterOptionsCreate();
TfLiteInterpreterOptionsAddDelegate(options, delegate);

// Create the interpreter.
TfLiteInterpreter *interpreter = TfLiteInterpreterCreate(model, options);

Note: For me, the GPU delegate did not yield a great improvement in inference speed. This might be due to my model using operations that are not supported by the GPU delegate (the set of supported ops seems to be quite small right now) and therefore have to be computed on CPU.

user_22069760
Related