ask for a tutorial: training vit with tfimm

[watsonw]

In short, is there any tutorial or sample can work for vit model training?

I'm working on training vit model throught tfimm with modified config.

Here is what I do to cifar10.py example, model=ModelConfig(...) is replaced with cfg returned from vit_base_patch16_224，others are keep no changed. see source file cifat10.py#L29

But in this way, an error raises with to_cls_format where flat ViTConfig needs to be constructed back from dict, which says

TypeError: __init__() got an unexpected keyword argument 'name'

Since thecfg_serializable decorator only makes ModelFactory work for <class 'tfimm.train.model.ModelConfig'>, cfgs below can not be changed back to ViTConfig('name'='vit_base_patch16_224', ...).

{'model': {'name': 'vit_base_patch16_224', 'url': '[timm]', 'nb_classes': 1000, 'in_channels': 3, 'input_size': (224, 224), 'patch_layer': 'patch_embeddings', 'pat
ch_nb_blocks': (), 'patch_size': 16, 'embed_dim': 768, 'nb_blocks': 12, 'nb_heads': 12, 'mlp_ratio': 4.0, 'qkv_bias': True, 'representation_size': None, 'distilled
': False, 'drop_rate': 0.0, 'attn_drop_rate': 0.0, 'drop_path_rate': 0.0, 'norm_layer': 'layer_norm_eps_1e-6', 'act_layer': 'gelu', 'interpolate_input': False, 'cr
op_pct': 0.875, 'interpolation': 'bicubic', 'mean': (0.5, 0.5, 0.5), 'std': (0.5, 0.5, 0.5), 'first_conv': 'patch_embed/proj', 'classifier': 'head'}, 'model_class'
: 'ModelFactory', 'optimizer': {'lr_schedule': {'lr': 0.001, 'lr_decay_rate': 0.8, 'lr_decay_frequency': 1, 'staircase': True}, 'lr_schedule_class': 'LRExponential
DecayFactory', 'lr_warmup': -1, 'optimizer': 'sgd', 'betas': (0.9, 0.999), 'clipnorm': -1.0, 'clipvalue': -1.0, 'epsilon': 1e-07, 'rho': 0.95, 'initial_accumulator
_value': 0.1}, 'optimizer_class': 'OptimizerFactory', 'binary_loss': False, 'weight_decay': 0.01, 'mixed_precision': False, 'save_input_dtype': 'float32'}

Maybe I take it all in the wrong way! :(

It would be appreciate if you can give a tutorial on how to train vit model, thanks alot.

[martinsbruveris]

Sorry for the late reply. I understand the confusion. There are two separate ModelConfig classes. One that comes from the model part of the library, i.e., anything to do with tfimm.create_model. And there is another in the training framework. They are not interchangeable.

The training framework is a bit rough around the edges at the moment....