in XNOR or BWN, convolution should avoid any multiplication

[haqishen]

But you did use multiplication in your implementation.

[rarilurelo]

Yes, so it's not fast.
It only simulates results.

If you want a fast implementation, you have to control bit level implementation on GPU.
theano: pycuda
ex. (https://github.com/MatthieuCourbariaux/BinaryNet/blob/master/Run-time/binary_ops.py)
or
chainer: cupy
ex. (https://github.com/rarilurelo/cupy_for_binarynet/blob/master/binarynet.py)