Swin-L pretrained checkpoints used

[vadimkantorov]

Hi @dbofseuofhust, @vaesl!

Can't find in the code the URLs to the ImageNet-pretrained Swin-L. Which checkpoints did you use? https://github.com/microsoft/Swin-Transformer provides many different ones.

Could you please publish a config for training using Swin-L?

Are your modifications to swin_transformer.py upstreamed anywhere?

I also wonder, have you tried other Swin backbones like Swin-S or Swin-B? ESViT repo publishes some self-sup trained Swin, but they are only for Swin-S/T/B: https://github.com/microsoft/esvit ...

Thank you!

[vaesl]

I am sorry for the late reply. We use this checkpoint (https://github.com/SwinTransformer/storage/releases/download/v1.0.0/swin_large_patch4_window7_224_22kto1k.pth), which is pretrained on ImageNet22k and finetuned on ImageNet1k with 224x224 input size. The training config for Swin-L is listed as follows. We have not tried on other Swin backbones.

EXP_DIR=exps/solq.swin.large
python3 -m torch.distributed.launch --nproc_per_node=8
--use_env main.py
--meta_arch solq
--backbone 'swin_l_p4w7'
--with_vector
--with_box_refine
--masks
--batch_size 4
--vector_hidden_dim 1024
--vector_loss_coef 3
--output_dir ${EXP_DIR}
--hidden_dim 384
--checkpoint
--pretrained "./pretrained/swin_large_patch4_window7_224_22kto1k.pth" \