Getting the test score after restoring a pretrained model

[JayMan91]

When I load the saved Lightning model and test on test data, I saw different epochs are running starting from 0, suggesting the model is being trained once again. Please suggest how to get the test score on the test data.
Let me illustrate in detail. I have a Lightning model like the following

class  MyLightningModule(pl.LightningModule):
    def __init__(self,loss_fn, lr=1e-1):
        super().__init__()
        self.loss_fn = loss_fn
        self.lr = lr
        self.save_hyperparameters("lr")
    
    @staticmethod
    def add_model_specific_args(parent_parser):
        parser = parent_parser.add_argument_group("SemanticPO")
        parser.add_argument("--lr", type=float, default=0.1)
        return parent_parser
    def forward(self,x):
        return self.layer1(x)   
    def training_step(self, batch, batch_idx):
        x,y,sol = batch
        y_hat =  self(x)
        loss = self.loss_fn(y,y_hat)
        self.log("train_loss",loss, prog_bar=True, on_step=True, on_epoch=True )
        return loss
    def validation_step(self, batch, batch_idx):
        x,y,sol = batch
        y_hat =  self(x)
        val_loss=  self.loss_fn(y,y_hatl)
        self.log("val_loss", val_loss, prog_bar=True, on_step=True, on_epoch=True, )
        return val_loss
    def test_step(self, batch, batch_idx):
        # Here we just reuse the validation_step for testing
        return self.validation_step(batch, batch_idx)
    def configure_optimizers(self):
        optimizer = torch.optim.Adam(self.parameters(), lr=self.lr)
        return optimizer 

I am using three separate data for training. testing and validation.

train_dl = DataLoader(train_df, batch_size= 64)
valid_dl = data_utils.DataLoader(valid_df, batch_size= 64)
test_dl = data_utils.DataLoader(test_df, batch_size= 64)

First, I train and validate in the following manner

model = MyLightningModule(loss_fn= Myloss)
checkpoint_callback = ModelCheckpoint(
    monitor="val_loss",
    dirpath="ckpt_dir/",
    filename="mymodel-{epoch:02d}-{val_loss:.2f}",
    save_top_k=2, mode="min",
)
trainer = pl.Trainer(max_epochs= 25, callbacks=[checkpoint_callback])
trainer.fit(model, train_dl, valid_dl)

The training is complete and the checkpoints are saved as expected.
Then I want to restore the saved model and test it on test data. This is done on a separate file like this

model = MyLightningModule.load_from_checkpoint('ckpt_dir/mymodel-epoch=15-val_loss=44.31.ckpt')
test_dl = data_utils.DataLoader(test_df, batch_size= 64)
trainer = pl.Trainer()
result = trainer.test(model,
ckpt_path='ckpt_dir/mymodel-epoch=15-val_loss=44.31.ckpt',dataloaders=test_dl)
print(result)

I expect it to just test on test data and display the result. But I see the model is running once again for a number of epochs.
May be I am doing something wrong. Please suggest the best way to obtain the test result after restoring a saved model.

pytorch-lightning==1.4.9

[rohitgr7]

your code looks correct. Just tested on 1.4.9 it's not running from scratch when you call trainer.test(). Can you share a minimal example to reproduce your issue?

[JayMan91]

Thank you. One last question is do I have to provide the ckpt_path again in trainer.test. Or only loading the model from checkpoint would do the job?

[rohitgr7]

if you provide ckpt_path, then it will just do this inside:

MyLightningModule.load_from_checkpoint(ckpt_path)

if you don't provide the ckpt_path then it won't load any weights from any existing checkpoint and

1. if you provide a model, then it will use that instance
2. if you don't provide a model in trainer.test then it will use the latest updated weights that are present in the model after training.

so in your case since you are loading it by yourself then doing trainer.test(model) would work just fine.