Model A not suppose to continue what Model B has trained

[ammarsaf]

Issues

I trained 2 model with Tensorflow. Say Model A and Model B. Both model has been trained within a single notebook.

After model A training ended, I trained Model B. What weird is, Model B metrics performance looks like continuing what Model A has done. I tried to oppose my hypothesis, by running Model B first, then Model A. But then its showed the similar pattern when Model A will continue what Model B has done (the metrics graph also change as I swap the training order)

How could I solve this problem?

Graph

The first model train graph metrics (if Model A is first, then would be Model A graph, hence if Model B train first)

The later model graph metrics (if Model B is later, then this will be the graph, hence if Model A train later)

The different between the model is, Model B have extra input layers compare to Model A. All layer structures is the same. As the input layer has an extra layer, hence the dataset pipeline would be extra too.

Progress

What I have tried (but failed):

• Encapsulated layers within function
• Used different variable name
• Instantiated different loss and optimizer respectively
• Set seed
• Cleared session using k.clear_session()
• Separated dataset pipeline using tf.data.Dataset

Current solution

• What I did was separated both model on different notebook and train it as isolated (not sharing the kernel). And it worked. Both model showed kinda similar pattern like the first graph.

Code

This is the code
Model A

from tensorflow.keras import layers
# 1. Token inputs
def build_model_A():
    token_inputs_A = layers.Input(shape=(1,), dtype="string")
    text_vectors = text_vectorizer(token_inputs_A)
    token_embeddings = token_embed(text_vectors) 
   
    x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(token_embeddings) 
    token_outputs = layers.Bidirectional(layers.LSTM(64))(x) 
    token_model = tf.keras.Model(inputs=token_inputs_A
                                 outputs=token_outputs)

    # 2. input 2
    inputs_2 = layers.Input(shape=(16,), dtype=tf.int32)
    x = layers.Dense(128, activation="relu")(inputs_2)
    model_2 = tf.keras.Model(inputs=input_2,
                                       outputs=x)

    # 4. Combine all models
    combined_embeddings_A = layers.Concatenate(name="hybrid_embedding")([token_model.output,
                                                                       mode_2.output])
    combined_embeddings = layers.Dropout(0.3)(combined_embeddings_A)

    # 5. Create output layer
    output_layer_A = layers.Dense(num_classes, activation="softmax", name="output_layer")(combined_embeddings)

    # 6. Put together model
    model_A = tf.keras.Model(inputs=[model_2.input, token_model.input, model_3.input],
                             outputs=output_layer_A, name="model_A")
    return model_A
model_A = build_model_A()
model_A_loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.2)
model_A_op = tf.keras.optimizers.Adam()
model_A.compile(
        loss=model_A_loss,
        optimizer=model_A_op,
        metrics=["accuracy"]
)
model_A_history = model_A.fit(
        train_A_dataset,
        validation_data= test_A_dataset,
        epochs=10
)

Model B

from tensorflow.keras import layers
# 1. Token inputs
def build_model_B():
    token_inputs_B = layers.Input(shape=(1,), dtype="string")
    text_vectors = text_vectorizer(token_inputs_B)
    token_embeddings = token_embed(text_vectors) 
   
    x = layers.Bidirectional(layers.LSTM(64, return_sequences=True))(token_embeddings) 
    token_outputs = layers.Bidirectional(layers.LSTM(64))(x) 
    token_model = tf.keras.Model(inputs=token_inputs_B
                                 outputs=token_outputs)

    # 2. input 2
    inputs_2 = layers.Input(shape=(16,), dtype=tf.int32)
    x = layers.Dense(128, activation="relu")(inputs_2)
    model_2 = tf.keras.Model(inputs=input_2,
                                       outputs=x)

    # 3. input 3
    inputs_3 = layers.Input(shape=(2, ), dtype=tf.int32)
    x = layers.Dense(128, activation="relu")(inputs_3 )
    model_3 = tf.keras.Model(inputs=inputs_3,
                                  outputs=x)

    # 4. Combine all models
    combined_embeddings_B= layers.Concatenate(name="hybrid_embedding")([token_model.output,
                                                                       mode_2.output,
                                                                       model_3.output])
    combined_embeddings = layers.Dropout(0.3)(combined_embeddings_B)

    # 5. Create output layer
    output_layer_B = layers.Dense(num_classes, activation="softmax", name="output_layer")(combined_embeddings)

    # 6. Put together model
    model_B = tf.keras.Model(inputs=[model_2.input, token_model.input, model_3.input],
                             outputs=output_layer_B, name="model_B")
    return model_B
model_B = build_model_B()
model_B_loss = tf.keras.losses.CategoricalCrossentropy(label_smoothing=0.2)
model_B_op = tf.keras.optimizers.Adam()
model_B.compile(
        loss=model_B_loss,
        optimizer=model_B_op,
        metrics=["accuracy"]
)
model_B_history = model_B.fit(
        train_B_dataset,
        validation_data= test_B_dataset,
        epochs=10
)

[ammarsaf]

Solved ✅ (edited)

Okay I have solved this problem (kudos to my colleague that raise about State issue). In this code, I shared the same layers.Embedding object for both model, which is the culprit of the issues.

To solve this, I need to instantiate new layers.Embedding object, for every model that I want to train. This is because by instantiate new layers.Embedding, it will reset the state of the weight of the object, and not using the previous one.

Note

@mrdbourke I think there is an issue within your NLP course. This is because, in the Tensorflow NLP learning series, you only used a single layers.Embedding object for every model that you trained. Hence, the next model that will be trained, will used the "learned" weight from the previous one, and giving an illusion of the the model has better performance from the previous one. This is my hypothesis, however I already tested 2 times and I can confirm on it, but you can verify me. I am new and willing to learn.