Creating a custom dataset

[YanivDorGalron]

Hey.

How should a custom dataset be implemented in order to integrate well with the overall pipeline?

Should I create those files? what format or information it must include?

Thanks in advance,
Yaniv

[shenyangHuang]

Hi Yaniv,

Thanks for your interest in our work. Feel free to add your custom dataset in your local fork if you prefer.

Here are a few tips:

• add a new folder in tgb/datasets with your dataset name say tgbl_cool. Add the edgelist file you have, for example see the format of other *_edgelist.csv file in this case tgbl-review_edgelist_v2.csv. make sure you name it as tgbl-cool_edgelist.csv.
• provide an entry for tgbl-cool in DATA_URL_DICT, DATA_VERSION_DICT, DATA_EVAL_METRIC_DICT in tgb/utils/info.py
• In tgb/utils/pre_process.py add a new function to specify how you would process your data from edgelist, along with node feat etc.
• In tgb/linkproppred/dataset.py, you need to add logic for dataset processing in generate_processed_files() function see line 216.
• (optional) for link property prediction datasets, generate negative samples for tgbl_cool, copy an example negative sample generation script such as tgbl-wiki_neg_generator.py and modify it as needed. put the script into tgb/datasets/tgbl_cool dataset and run it to generate the negative samples.
• (optional) test your script by adding a TGN example to a new folder (tgbl-cool) under examples/linkproppred. reuse TGN examples from other folders.

with the above steps, you should be able to add any custom local dataset to your locally installed TGB version.
Hope that answers your questions.

Best,
Andy

[YanivDorGalron]

Hi Andy,

Thanks for the detailed response!

Just to make sure - when dealing with node classification dataset the changes are almost the same but with addition of:

• Also provide an entry for tgbn-cool in DATA_NUM_CLASSES in tgb/utils/info.py
• In tgb/nodeproppred/dataset.py, you need to add logic for dataset processing in generate_processed_files() function see line 235

In addition, can you please explain more on the format of the csv files? more specifically:

1. If we are dealing with non-bipartite graph which is also not directed - should I reverse the columns in order to fully capture all edges and label all nodes? (It seems to me that only the 'user' list is labeled and the 'items' list does not.)
2. Is there an example for node classification csv file without node features?
3. It also seems that every dataset csv file is in different format, some csv's have different number of nodes at the node.csv compared to the edge.csv, for example - tgbn-genre_node_labels.csv and tgbn-genre_edgelist.csv where the first contains 974 unique users and the second contains 992 unique user.

Thank in advance!
Yaniv

[shenyangHuang]

Hi Yaniv,

Thanks for the follow-up questions, sorry for missing the thread earlier.

• yes, you need to provide an entry in DATA_NUM_CLASSES
• yes, you need to add an if statement to refer to your data preprocessing function in utils/preprocess.py

For your questions:

• bipartite and non-bipartite only matter for the preprocessing function, you can write your processing function to process the data as bipartite or non-bipartite, see examples in utils/preprocess.py. The csv file format would be the same.
• actually csv files only has edge features (in tgbn-genre_edgelist.csv) and has dynamic node label (in tgbn-genre_node_labels.csv).
• tgbn-genre_edgelist.csv contains all the edges and edge features. tgbn-genre_node_labels.csv contains the node labels for the nodes. Node labels would capture the behavior of users over a week. Therefore it is possible that for some new users that appear in the last week to not have node labels yet.