Inference error with TensorFlow C++ on iOS: "Invalid argument: Session was not created with a graph before Run()!"

Question

I am trying to run my model on iOS using TensorFlow's C++ API. The model is a SavedModel saved as a .pb file. However, calls to Session::Run() result in the error:

"Invalid argument: Session was not created with a graph before Run()!"

In Python, I can successfully run inference on the model with the following code:

with tf.Session() as sess:
    tf.saved_model.loader.load(sess, ['serve'], '/path/to/model/export')
    result = sess.run(['OutputTensorA:0', 'OutputTensorB:0'], feed_dict={
        'InputTensorA:0': np.array([5000.00] * 1000).reshape(1, 1000),
        'InputTensorB:0': np.array([300.00] * 1000).reshape(1, 1000)
    })
    print(result[0])
    print(result[1])

In C++ on iOS, I try to mimick this working snippit as follows:

tensorflow::Input::Initializer input_a(5000.00, tensorflow::TensorShape({1, 1000}));
tensorflow::Input::Initializer input_b(300.00, tensorflow::TensorShape({1, 1000}));

tensorflow::Session* session_pointer = nullptr;

tensorflow::SessionOptions options;
tensorflow::Status session_status = tensorflow::NewSession(options, &session_pointer);

std::cout << session_status.ToString() << std::endl; // prints OK

std::unique_ptr session(session_pointer);

tensorflow::GraphDef model_graph;

NSString* model_path = FilePathForResourceName(@"saved_model", @"pb");
PortableReadFileToProto([model_path UTF8String], &model_graph);

tensorflow::Status session_init = session->Create(model_graph);

std::cout << session_init.ToString() << std::endl; // prints OK

std::vector outputs;
tensorflow::Status session_run = session->Run({{"InputTensorA:0", input_a.tensor}, {"InputTensorB:0", input_b.tensor}}, {"OutputTensorA:0", "OutputTensorB:0"}, {}, &outputs);

std::cout << session_run.ToString() << std::endl; // Invalid argument: Session was not created with a graph before Run()!

The methods FilePathForResourceName and PortableReadFileToProto are taken from the TensorFlow iOS sample found here.

What is the problem? I noticed that this happens regardless of how simple the model is (see my issue report on GitHub), which means the problem is not with the specifics of the model.

user_9715143 · Answer

One addition to the very comprehensive explanation above:

@jshapy8 is right in saying "You will have to find the .cc that contains REGISTER_OP("YourOperation") and add it to tf_op_files.txt" and there is a process that can simplify that a bit:

## build the print_selective_register_header tool. Run from tensorflow root
bazel build tensorflow/python/tools:print_selective_registration_header
bazel-bin/tensorflow/python/tools/print_selective_registration_header \
--graphs=/model_frozen.pb > ops_to_register.h

This creates a .h file that lists only the ops needed for your specific model.

Now when compiling your static libraries follow the Build By Hand instructions here

The instructions say to do the following:

make -f tensorflow/contrib/makefile/Makefile \
TARGET=IOS \
IOS_ARCH=ARM64

But you can pass a lot to the makefile specific to your needs and I've found the following your best bet:

make -f tensorflow/contrib/makefile/Makefile \
TARGET=IOS IOS_ARCH=ARM64,x86_64 OPTFLAGS="-O3 -DANDROID_TYPES=ANDROID_TYPES_FULL -DSELECTIVE_REGISTRATION -DSUPPORT_SELECTIVE_REGISTRATION"

In particular you are telling it here to compile for just two of the 5 architectures to speed up compiling time (full list is: i386 x86_64 armv7 armv7s arm64 and obviously takes longer) - IOS_ARCH=ARM64,x86_64 - and then you are telling it not to compile for ANDROID_TYPES_SLIM (which will give you the Float/Int casting issues referred to above) and then finally you are telling it to pull all the necessary ops kernel files and include them in the make process.

Update . Not sure why this wasn't working for me yesterday, but this is probably a cleaner and safer method:

build_all_ios.sh OPTFLAGS="-O3 -DANDROID_TYPES=ANDROID_TYPES_FULL -DSELECTIVE_REGISTRATION -DSUPPORT_SELECTIVE_REGISTRATION"

If you want to speed things up edit compile_ios_tensorflow.sh in the /Makefile directory. Look for the following line:

BUILD_TARGET="i386 x86_64 armv7 armv7s arm64"

and change it to:

BUILD_TARGET="x86_64 arm64"

Inference error with TensorFlow C++ on iOS: "Invalid argument: Session was not created with a graph before Run()!"

2 Answers