Implementation on CRIS and ETRIS

[Kimsure]

Can you release the code that uses your Maskris strategy on the CRIS and ETRIS models? As shown in your paper Tab. 2, your strategy also works on other RIS methods. How to apply your strategy based on CRIS and ETRIS models?

[gaviotas]

Sorry for the late reply, and thanks so much for your interest in our work!
We’re currently on a tight schedule, so we can’t release the integrated code right away — but here's how you can apply the strategy.

You can simply apply MaskRIS to CRIS and ETRIS by introducing masked image and sentence inputs, and using a two-pass forward with distillation loss. The key changes are:

1. Generate masked inputs

   Add the following to get image_masked and sentences_masked:
   

   maskris/training.py

   Lines 19 to 28 in 01b54d5

   	image, targets, sentences, sentences_masked, attentions = data
   	image, sentences, attentions = image.cuda(non_blocking=True), \
   	sentences.cuda(non_blocking=True), \
   	attentions.cuda(non_blocking=True)
   	sentences_masked = sentences_masked.cuda(non_blocking=True)
   	# Image masking for MaskRIS
   	image_masked = patchify(image, args.img_patch_size)
   	image_masked = random_masking(image_masked, args.img_mask_ratio)
   	image_masked = unpatchify(image_masked, args.img_patch_size)

   For sentences_masked, refer to the masking logic used in our dataset loader:
   

   maskris/data/dataset_refer_bert.py

   Lines 130 to 138 in 01b54d5

   	class ReferDatasetMasking(data.Dataset):
   	def __init__(self,
   	args,
   	image_transforms=None,
   	target_transforms=None,
   	split='train',
   	eval_mode=False,
   	txt_mask_ratio=0.0,
   	txt_mask_ratio_sub=(0.8, 0.1)):

2. Two forward passes with distillation
   First, run the model with original inputs. Then, run a second pass with image_masked and sentences_masked, using soft targets from the first output:
   

   maskris/training.py

   Lines 38 to 51 in 01b54d5

   	sentences_masked = sentences_masked.squeeze(1) # [B, N_l]
   	# first forward pass
   	with torch.cuda.amp.autocast():
   	loss_dict, logit = model(image, sentences, l_mask=attentions, targets=targets, return_probs=True)
   	loss_scaler(loss_dict['total_loss'] * 0.5, optimizer, clip_grad=clip_grad,
   	parameters=model.parameters(), update_grad=False)
   	# second forward pass
   	with torch.cuda.amp.autocast():
   	targets_masked = {'mask': F.softmax(logit.detach(), dim=1)}
   	loss_dict_masked = model(image_masked, sentences_masked, l_mask=attentions, targets=targets_masked)
   	grad_norm = loss_scaler(loss_dict_masked['total_loss'] * 0.5, optimizer, clip_grad=clip_grad,
   	parameters=model.parameters(), update_grad=True)

   In short, once you obtain sentences_masked and image_masked, just feed them into the model and apply a distillation loss using F.softmax(logit) from the first pass.

Hope this helps! Let us know if you have any further questions.

[Kimsure]

Thanks for your reply! I will try your suggestions in a few weeks