error while loading lstm model

[billamiable]

Hi, I noticed that LSTM model import is supported in new version. I would like to test loading an ONNX model by my own. I'm new to glow, is there any guidance of how to doing that? Any help is appreciated!

Update:
I tried to use model-runner to load lstm model. But I encountered the following error:
root@f9ae9f7a10c7:/home/build_Release_bundles/bin# ./model-runner -model /home/glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm_defaults/model.onnx
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1223 01:43:36.430075 3673 Error.cpp:119] exitOnError(Error) got an unexpected ErrorValue:
Error message: No node under name Y
Error return stack:
/home/glow/lib/Importer/ProtobufLoader.cpp:142
/home/glow/lib/Importer/ONNXModelLoader.cpp:2266
/home/glow/lib/Importer/ONNXModelLoader.cpp:2382
/home/glow/lib/Importer/ONNXModelLoader.cpp:2394
*** Check failure stack trace: ***
./model-runner[0x7abe5f]
./model-runner[0x7aa282]
./model-runner[0x7ac538]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7efc2ecdd390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7efc2de67428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7efc2de6902a]
/usr/lib/x86_64-linux-gnu/libglog.so.0(+0x9e49)[0x7efc2e79de49]
/usr/lib/x86_64-linux-gnu/libglog.so.0(+0xb5cd)[0x7efc2e79f5cd]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage9SendToLogEv+0x283)[0x7efc2e7a1433]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage5FlushEv+0xbb)[0x7efc2e79f15b]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google15LogMessageFatalD2Ev+0xe)[0x7efc2e7a1e1e]
./model-runner[0x2811bcc]
./model-runner[0x6fd97f]
./model-runner[0x4cf588]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7efc2de52830]
./model-runner[0x4c6099]
Aborted

Anyone knows how to deal with that?

[ponnamsairam]

we have unrolled our network in this way,
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features),unroll=True)).
As mentioned #2738 (comment)
But still we are facing the same issue "No node under name lstm_h".

so suggest us alternative solution by using lstm in glow
I am running a LSTM model which is in onnx format but i am getting error as below:

./bin/model-compiler -m /home/hemanth/lstm_new.onnx -emit-bundle=/home/Desktop
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1223 11:42:51.549001 30189 Error.cpp:119] exitOnError(Error) got an unexpected ErrorValue:
Error message: No node under name lstm_h
Error return stack:
/home/glow/lib/Importer/ProtobufLoader.cpp:110
/home/glow/include/glow/Importer/CommonOperatorLoader.h:584
/home/glow/include/glow/Importer/CommonOperatorLoader.h:1128
/home/glow/lib/Importer/ONNXModelLoader.cpp:1763
/home/glow/lib/Importer/ONNXModelLoader.cpp:1940
/home/glow/lib/Importer/ONNXModelLoader.cpp:2028
/home/glow/lib/Importer/ONNXModelLoader.cpp:2042
*** Check failure stack trace: ***
#0 0x00000000006a512a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (./bin/model-compiler+0x6a512a)
#1 0x00000000006a30bc llvm::sys::RunSignalHandlers() (./bin/model-compiler+0x6a30bc)
#2 0x00000000006a3227 SignalHandler(int) (./bin/model-compiler+0x6a3227)
#3 0x00007fa0fcc41390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007fa0fbdcb428 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35428)
#5 0x00007fa0fbdcd02a abort (/lib/x86_64-linux-gnu/libc.so.6+0x3702a)
#6 0x00007fa0fca0ae49 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0x9e49)
#7 0x00007fa0fca0c5cd (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb5cd)
#8 0x00007fa0fca0e433 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xd433)
#9 0x00007fa0fca0c15b google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb15b)
#10 0x00007fa0fca0ee1e google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xde1e)
#11 0x00000000025cd9ca glow::detail::exitOnError(char const*, unsigned long, glow::detail::GlowError) (./bin/model-compiler+0x25cd9ca)
#12 0x00000000005ff7db glow::ONNXModelLoader::ONNXModelLoader(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, llvm::ArrayRef<char const*>, llvm::ArrayRef<glow::Type const*>, glow::Function&, glow::detail::GlowError*, bool) (./bin/model-compiler+0x5ff7db)
#13 0x0000000000454540 main (./bin/model-compiler+0x454540)
#14 0x00007fa0fbdb6830 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)
#15 0x00000000004fb909 _start (./bin/model-compiler+0x4fb909)
Aborted (core dumped)

[jfix71]

CC: @mciprian13

[billamiable]

I tried to track the error and found that it occurs in getNodeValueByName function. Dive deep into the code, it shows llvm::StringMap nodeValueByName_ in class ProtobufLoader have neither Y node nor lstm_h node mentioned in @ponnamsairam 's comment.
Recall that loading LSTM is not done by creating an LSTM node, I suspect it has something to deal with the error.

[mciprian13]

For compiling an LSTM model try using the model-compiler tool and NOT the model-runner which is not intended for general purpose use. Use the following command (similar to how @ponnamsairam used it):
model-compiler -model=model.onnx -backend=CPU -emit-bundle=<bundle-dir>
This generates a library. An application code must be written afterwards to use the library code. More details you can find here:
https://github.com/pytorch/glow/blob/master/docs/AOT.md
I used ONNX models with LSTM multiple times using the model-compiler without problems. I suspect that the model itself might be corrupt (e.g. the LSTM module uses values/tensors which do not have producers).
Can you provide the ONNX model to debug it myself?

[ponnamsairam]

we have converted lstm model from keras to onnx

from numpy import array
from keras.models import Sequential
from keras.layers import LSTM
from keras.layers import Dense
import keras2onnx
from tf2onnx import tfonnx
import onnxruntime

split a univariate sequence into samples

def split_sequence(sequence, n_steps):
X, y = list(), list()
for i in range(len(sequence)):
# find the end of this pattern
end_ix = i + n_steps
# check if we are beyond the sequence
if end_ix > len(sequence)-1:
break
# gather input and output parts of the pattern
seq_x, seq_y = sequence[i:end_ix], sequence[end_ix]
X.append(seq_x)
y.append(seq_y)
return array(X), array(y)

define input sequence

raw_seq = [10, 20, 30, 40, 50, 60, 70, 80, 90]

choose a number of time steps

n_steps = 3

split into samples

X, y = split_sequence(raw_seq, n_steps)

reshape from [samples, timesteps] into [samples, timesteps, features]

n_features = 1
X = X.reshape((X.shape[0], X.shape[1], n_features))

define model

model = Sequential()
#model.add(unroll=True)
model.add(LSTM(50, activation='relu', input_shape=(n_steps, n_features),unroll=True))
model.add(Dense(1))
model.compile(optimizer='adam', loss='mse')

#sess = onnxruntime.InferenceSession(temp_model_file)

fit model

model.fit(X, y, epochs=200, verbose=0)

demonstrate prediction

x_input = array([70, 80, 90])
x_input = x_input.reshape((1, n_steps, n_features))
onnx_model = keras2onnx.convert_keras(model, model.name)
import onnx
onnx_filename = 'lstm_new.onnx'
onnx.save_model(onnx_model, onnx_filename)
#yhat = model.predict(x_input, verbose=0)
#print(yhat)

onnx file generated is not in readable format so please convert above code to onnx format and debug it

[billamiable]

I tried the model-compiler, it shows another error
Btw, the onnx model I used is inside the glow directory, the path is glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm_defaults/model.onnx

root@f9ae9f7a10c7:/home/build_Release_bundles# ./bin/model-compiler -model /home/glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm_defaults/model.onnx -emit-bundle .
Y
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1224 13:33:53.138015 449 Backend.h:94] Saving a bundle is not supported by the backend
*** Check failure stack trace: ***
./bin/model-compiler[0x72935f]
./bin/model-compiler[0x727782]
./bin/model-compiler[0x729a38]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x11390)[0x7f571b039390]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x38)[0x7f571a1c3428]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7f571a1c502a]
/usr/lib/x86_64-linux-gnu/libglog.so.0(+0x9e49)[0x7f571aaf9e49]
/usr/lib/x86_64-linux-gnu/libglog.so.0(+0xb5cd)[0x7f571aafb5cd]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage9SendToLogEv+0x283)[0x7f571aafd433]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google10LogMessage5FlushEv+0xbb)[0x7f571aafb15b]
/usr/lib/x86_64-linux-gnu/libglog.so.0(_ZN6google15LogMessageFatalD2Ev+0xe)[0x7f571aafde1e]
./bin/model-compiler[0x21a4dab]
./bin/model-compiler[0x4c8e40]
./bin/model-compiler[0x4ced6d]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0)[0x7f571a1ae830]
./bin/model-compiler[0x4c6099]
Aborted

@mciprian13 Please have a look. Thanks!

[mciprian13]

In your example you forgot the option -backend=CPU. The error Saving a bundle is not supported by the backend appears because the default backend is the Interpreter which cannot save a bundle.
I will try the steps you mentioned soon.

[billamiable]

It seems work! Thank you so much! @mciprian13

[mciprian13]

So does your original model with the LSTM module work using the model-compiler? Do I have to go through the steps you mentioned (the steps in Python)?

[ponnamsairam]

I tried the model-compiler, it shows another error

./model-compiler -model /home/glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm/lstm_new1.onnx -backend=CPU
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1226 10:52:01.982949 5554 ModelCompiler.cpp:34] Check failed: emittingBundle() Bundle output directory not provided. Use the -emit-bundle option!
*** Check failure stack trace: ***
#0 0x00000000006a512a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (./model-compiler+0x6a512a)
#1 0x00000000006a30bc llvm::sys::RunSignalHandlers() (./model-compiler+0x6a30bc)
#2 0x00000000006a3227 SignalHandler(int) (./model-compiler+0x6a3227)
#3 0x00007fe56fde3390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007fe56ef6d428 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35428)
#5 0x00007fe56ef6f02a abort (/lib/x86_64-linux-gnu/libc.so.6+0x3702a)
#6 0x00007fe56fbace49 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0x9e49)
#7 0x00007fe56fbae5cd (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb5cd)
#8 0x00007fe56fbb0433 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xd433)
#9 0x00007fe56fbae15b google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb15b)
#10 0x00007fe56fbb0e1e google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xde1e)
#11 0x0000000000454b15 main (./model-compiler+0x454b15)
#12 0x00007fe56ef58830 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)
#13 0x00000000004fb909 _start (./model-compiler+0x4fb909)
Aborted (core dumped)

[billamiable]

I tried the model-compiler, it shows another error

./model-compiler -model /home/glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm/lstm_new1.onnx -backend=CPU
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1226 10:52:01.982949 5554 ModelCompiler.cpp:34] Check failed: emittingBundle() Bundle output directory not provided. Use the -emit-bundle option!
*** Check failure stack trace: ***
#0 0x00000000006a512a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (./model-compiler+0x6a512a)
#1 0x00000000006a30bc llvm::sys::RunSignalHandlers() (./model-compiler+0x6a30bc)
#2 0x00000000006a3227 SignalHandler(int) (./model-compiler+0x6a3227)
#3 0x00007fe56fde3390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007fe56ef6d428 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35428)
#5 0x00007fe56ef6f02a abort (/lib/x86_64-linux-gnu/libc.so.6+0x3702a)
#6 0x00007fe56fbace49 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0x9e49)
#7 0x00007fe56fbae5cd (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb5cd)
#8 0x00007fe56fbb0433 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xd433)
#9 0x00007fe56fbae15b google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb15b)
#10 0x00007fe56fbb0e1e google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xde1e)
#11 0x0000000000454b15 main (./model-compiler+0x454b15)
#12 0x00007fe56ef58830 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)
#13 0x00000000004fb909 _start (./model-compiler+0x4fb909)
Aborted (core dumped)

you need to include -emit-bundle= option.

[Compiler-team-1]

I tried the model-compiler, it shows another error
./model-compiler -model /home/glow/thirdparty/onnx/onnx/backend/test/data/node/test_lstm/lstm_new1.onnx -backend=CPU
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1226 10:52:01.982949 5554 ModelCompiler.cpp:34] Check failed: emittingBundle() Bundle output directory not provided. Use the -emit-bundle option!
*** Check failure stack trace: ***
#0 0x00000000006a512a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (./model-compiler+0x6a512a)
#1 0x00000000006a30bc llvm::sys::RunSignalHandlers() (./model-compiler+0x6a30bc)
#2 0x00000000006a3227 SignalHandler(int) (./model-compiler+0x6a3227)
#3 0x00007fe56fde3390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007fe56ef6d428 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35428)
#5 0x00007fe56ef6f02a abort (/lib/x86_64-linux-gnu/libc.so.6+0x3702a)
#6 0x00007fe56fbace49 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0x9e49)
#7 0x00007fe56fbae5cd (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb5cd)
#8 0x00007fe56fbb0433 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xd433)
#9 0x00007fe56fbae15b google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb15b)
#10 0x00007fe56fbb0e1e google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xde1e)
#11 0x0000000000454b15 main (./model-compiler+0x454b15)
#12 0x00007fe56ef58830 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)
#13 0x00000000004fb909 _start (./model-compiler+0x4fb909)
Aborted (core dumped)

you need to include -emit-bundle= option.

I have included -emit-bundle option , but it is giving same error .

/glow/build/bin$ ./model-compiler -model "/home/hemanth/lstm_new1.onnx" -backend=CPU -emit-bundle=build/
WARNING: Logging before InitGoogleLogging() is written to STDERR
F1227 14:24:43.546838 6900 Error.cpp:119] exitOnError(Error) got an unexpected ErrorValue:
Error message: No node under name lstm_h
Error return stack:
/home/glow/lib/Importer/ProtobufLoader.cpp:110
/homeglow/include/glow/Importer/CommonOperatorLoader.h:584
/homeglow/include/glow/Importer/CommonOperatorLoader.h:1128
/home/glow/lib/Importer/ONNXModelLoader.cpp:1763
/home/glow/lib/Importer/ONNXModelLoader.cpp:1940
/home/glow/lib/Importer/ONNXModelLoader.cpp:2028
/home/glow/lib/Importer/ONNXModelLoader.cpp:2042
*** Check failure stack trace: ***
#0 0x00000000006a512a llvm::sys::PrintStackTrace(llvm::raw_ostream&) (./model-compiler+0x6a512a)
#1 0x00000000006a30bc llvm::sys::RunSignalHandlers() (./model-compiler+0x6a30bc)
#2 0x00000000006a3227 SignalHandler(int) (./model-compiler+0x6a3227)
#3 0x00007fb1cca2d390 __restore_rt (/lib/x86_64-linux-gnu/libpthread.so.0+0x11390)
#4 0x00007fb1cbbb7428 gsignal (/lib/x86_64-linux-gnu/libc.so.6+0x35428)
#5 0x00007fb1cbbb902a abort (/lib/x86_64-linux-gnu/libc.so.6+0x3702a)
#6 0x00007fb1cc7f6e49 (/usr/lib/x86_64-linux-gnu/libglog.so.0+0x9e49)
#7 0x00007fb1cc7f85cd (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb5cd)
#8 0x00007fb1cc7fa433 google::LogMessage::SendToLog() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xd433)
#9 0x00007fb1cc7f815b google::LogMessage::Flush() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xb15b)
#10 0x00007fb1cc7fae1e google::LogMessageFatal::~LogMessageFatal() (/usr/lib/x86_64-linux-gnu/libglog.so.0+0xde1e)
#11 0x00000000025cd9ca glow::detail::exitOnError(char const*, unsigned long, glow::detail::GlowError) (./model-compiler+0x25cd9ca)
#12 0x00000000005ff7db glow::ONNXModelLoader::ONNXModelLoader(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, llvm::ArrayRef<char const*>, llvm::ArrayRef<glow::Type const*>, glow::Function&, glow::detail::GlowError*, bool) (./model-compiler+0x5ff7db)
#13 0x0000000000454540 main (./model-compiler+0x454540)
#14 0x00007fb1cbba2830 __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x20830)
#15 0x00000000004fb909 _start (./model-compiler+0x4fb909)
Aborted (core dumped)

[mciprian13]

Can you please provide the ONNX model to debug it? I tried to generate the model using the Python code you suggested but the code crashes (you did not mentioned what Python version to use, what are the dependencies of the Python packages, etc) so it would be much simpler if you could provide the final model in ONNX format.

[ponnamsairam]

lstm_new1.onnx.zip
@mciprian13

python version:3.7.2
protobuf version:2.6.1