I am using tensorflow 2.1.0 and transformers 2.5.1 on MacOS with python 3.7. I am bulding a Keras model using TFBertForSequenceClassification:
model = TFBertForSequenceClassification.from_pretrained('bert-base-cased',
num_labels=number_label)
model.compile(optimizer=optimizer,
loss=loss,
metrics=[metric])
and I can explore the structure:
model.summary()
and we can see the following
Model: "tf_bert_for_sequence_classification"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
bert (TFBertMainLayer) multiple 108310272
_________________________________________________________________
dropout_37 (Dropout) multiple 0
_________________________________________________________________
classifier (Dense) multiple 1538
=================================================================
Total params: 108,311,810
Trainable params: 108,311,810
Non-trainable params: 0
As we can see from above we only see the 3 main layers and we don't have access to more details.
After fitting the model (not sure why but if we don't do that the input and outpu variable will be empty) we can access:
model.inputs
{'attention_mask': <tf.Tensor 'attention_mask:0' shape=(None, 128) dtype=int32>,
'input_ids': <tf.Tensor 'input_ids:0' shape=(None, 128) dtype=int32>,
'token_type_ids': <tf.Tensor 'token_type_ids:0' shape=(None, 128) dtype=int32>}
model.outputs
[<tf.Tensor 'tf_bert_for_sequence_classification/Identity:0' shape=(None, 2) dtype=float32>]
This is a good start and now I would like to explore the keras layers:
model.layers
[<transformers.modeling_tf_bert.TFBertMainLayer at 0x1a415fd7d0>,
<tensorflow.python.keras.layers.core.Dropout at 0x1a445c3550>,
<tensorflow.python.keras.layers.core.Dense at 0x1a445c3890>]
but now if I try to get more info, it is always empty:
for layer in model.layers:
print(layer.name, layer._inbound_nodes, layer._outbound_nodes)
bert [] []
dropout_37 [] []
classifier [] []
I tried some other method like inbound_nodes but it is always empty!
Is there some way to be able to inspect in more details the layer of some complicated model like BERT ? What is the reason we got empty information ?
I also tried:
tf.keras.utils.plot_model(model,
'model.png',
show_shapes=True)
but what I get is not very hepfull:
Same using TensorBoard that give me one graph and many disconnected elements
Yes as an option, I good look at the code directly: code but I tought it will be possible programatically to inspect the structure of such complex model.