How to evaluate distribution based metrics after the training is finished?

[ghost]

Hi,
This is a really cool tool, I was looking for something similar for quite some time.
I would like to know how to evaluate the distribution-based metrics, since I have finished the training I only have the generated images and the real images available?
The second question it is not very related to the current topic, but I would appreciate it if you answer it, does the tool provide the output with plot graph lines?

Thank you

[snk4tr]

Hi, @Ghaleb-alnakhlani.

Your first question is not quite clear to me. DB metrics are used to compute the distance between two distributions of data. In your case, these two distributions are presumably defined by two sets of images: generated and real ones. To compute some DB metric score (e.g. FID), you first need to compute features from these images either using your own feature extractor or one of the extractors that are provided in the library. For the latter, each DB metric has the compute_feats methods designed to help you to do that. After that, you can pass these features to the metric class to get the resulting score as shown in the example script. This can be done regardless of training status.

Regarding your second question, we do not provide any visualisations and we do not have any plans to provide this functionality in any forceable future.

[snk4tr]

Hi @Ghaleb-alnakhlani.

In your case, the process should look the following way:

1. Take your pix2pix model. I may be a fully trained or a model on one of the intermediate training steps. What to take depends on what you are trying to achieve and verify.
2. Select a metric. For the sake of example, let's consider FID.
3. Read the paper on the metric to better understand how it is computed. In case of FID, it first requires you to compute features from images. Therefore, we are doing the next step.
4. Compute features from images. Features are computed using some network other than your pix2pix model. This network is called feature extractor. According to the paper, for FID it is InceptionNet. To compute features, use the compute_feats method as it is done here.
5. Use the features obtained on the previous step to compute the FID score. Here is an example.

I hope it helps.

[ghost]

This is perfect, it is clear now thank you

[ghost]

Hi,
I have run the evaluation test with this code

import os
import piq
import torch
import torchvision.transforms as transforms
from torch.utils.data import DataLoader
from piq.feature_extractors import InceptionV3
from PIL import Image


def getFilePaths(path):
    # read a folder, return the complete path
    ret = []
    for root, dirs, files in os.walk(path):
        for filespath in files:
            ret.append(os.path.join(root, filespath))
    ret.sort()
    return ret


class DataProcess(torch.utils.data.Dataset):
    def __init__(self, s_src, img_w, img_h):
        super(DataProcess, self).__init__()
        self.img_w = img_w
        self.img_h = img_h

        self.img_transform = transforms.Compose([
            transforms.Resize((self.img_h, self.img_w)),
            transforms.ToTensor(),  # transform to [0, 1]
        ])

        self.f_srcs = getFilePaths(s_src)

    def __getitem__(self, index):
        src = Image.open(self.f_srcs[index])
        t_src = self.img_transform(src.convert('RGB'))

        return {
            'images': t_src,
        }

    def __len__(self):
        return len(self.f_srcs)


if __name__ == '__main__':   
    s_1 = ('./datasets/name/train_B')
    s_2 = ('./results/name/synthesized')
    
    set_1 = DataProcess(s_1, 512, 512)
    set_2 = DataProcess(s_2, 512, 512)
    
    loader_1 = torch.utils.data.DataLoader(set_1, batch_size=1, shuffle=False)
    loader_2 = torch.utils.data.DataLoader(set_2, batch_size=1, shuffle=False)
    
    fid_metric = piq.FID()
    model = InceptionV3()
    feat_1 = fid_metric.compute_feats(loader_1, model)
    feat_2 = fid_metric.compute_feats(loader_2, model)
    fid = fid_metric.compute_metric(feat_1, feat_2)
    print(f'====> fid: {fid}')

And this is the result that I have got, which is reasonable from my observation.

fid: 19.67474915673438

After that I have tested the pytorch-fid from this repo.

!python -m pytorch_fid /content/drive/MyDrive/pix2pixHD/datasets/name/train_B/ /content/drive/MyDrive/pix2pixHD/results/name/synthesized/ --batch-size 1

And this the result FID: 6.98900379066589

The third method that I have tested is from this repo clean-fid, I have used the leagacy-mode which is equal the above pytorch-fid.

from cleanfid import fid
score_clean = fid.compute_fid("/content/drive/MyDrive/pix2pixHD/datasets/name/train_B/", "/content/drive/MyDrive/pix2pixHD/results/name/synthesized/", mode="legacy_pytorch", batch_size=1, num_workers=0)
print(f"clean-fid score is {score_clean:.3f}")

And this is the result that I have got clean-fid score is 7.205

So the question is which one of these results is correct? And why I have different results from testing the same dataset and using the same model InceptionV3?

[snk4tr]

Hi @Ghaleb-alnakhlani.

This question looks irrelevant to the initial topic of this discussion. Please mark this question answered and let's discuss the new question in the issue #320 you just opened.

[ghost]

Hi @snk4tr sure thank you.