Why separable convolution implemented by tensorflow is slower than the normal convolution?

I have tested the speed of separable_conv2d and normal conv2d implemented in TF, it seems the only depthwise_conv2d is faster than the normal conv2d but the performance of the dw_conv2d is poor obviously.

The separable_conv2d mentioned in MobileNet, its FLOPs is 1/9 of the normal conv when the kernel_size=3, but considering the Memory Access Cost the separable one cannot be 9 times faster than the normal one but in my experiment, the separable one is too much slower.

I modeling the experiment like this separable_conv2d is too slow. In this experiment, the separable_conv2d seems faster than the normal one when the depth_multiply=1, but when I use tf.nn to implement it as follows:

IMAGE_SIZE= 512
REPEAT = 100
KERNEL_SIZE = 3
data_format = 'NCHW'
#CHANNELS_BATCH_SIZE =  2048 # channe# ls * batch_size
os.environ["TF_CPP_MIN_LOG_LEVEL"] = "3"

def normal_layers(inputs, nfilter, name=''):
    with tf.variable_scope(name, reuse=tf.AUTO_REUSE):
         shape = inputs.shape.as_list()
         in_channels = shape[1]
         filter = tf.get_variable(initializer=tf.initializers.random_normal,
                                  shape=[KERNEL_SIZE, KERNEL_SIZE, 
                                   in_channels, nfilter], name='weight')
         conv = tf.nn.conv2d(input= inputs, filter=filter, strides= 
                           [1,1,1,1],padding='SAME',data_format=data_format, 
                           name='conv')
    return conv
def sep_layers(inputs, nfilter, name=''):
    with tf.variable_scope(name, reuse=tf.AUTO_REUSE):
         shape= inputs.shape.as_list()
         in_channels = shape[1]
         dw_filter=
         tf.get_variable(initializer=tf.initializers.random_normal,
                         shape=[KERNEL_SIZE, KERNEL_SIZE, 
                         in_channels, 1], name='dw_weight')

         pw_filter = 
         tf.get_variable(initializer=tf.initializers.random_normal,
                                shape=[1,1,in_channels, nfilter], 
                         name='pw_weight')          
         conv = tf.nn.depthwise_conv2d_native(input=inputs,
                                              filter=dw_filter,
                                              strides=[1,1,1,1],
                                              padding='SAME',
                                              data_format=data_format)
         conv = tf.nn.conv2d(input=conv,
                             filter=pw_filter,
                             strides=[1,1,1,1],
                             padding='SAME',
                             data_format=data_format)
    return conv  

Each layer run in 100 times,
different from the link is I set the batch_size as a constant 10,
and the channels is in [32, 64, 128],
inputs is[batch_size, channels, img_size, img_size] and the duration of them as follows:

Channels: 32
Normal Conv 0.7769527435302734s, Sep Conv 1.4197885990142822s
Channels: 64
Normal Conv 0.8963277339935303s, Sep Conv 1.5703468322753906s
Channels: 128
Normal Conv 0.9741833209991455s, Sep Conv 1.665834665298462s 

It seems that when batch_size is a constant only change channel the time cost of the normal one and the separable one is growing gradually.

And when setting batch_size * channels as a constant
Inputs shape as [CHANNELS_BATCH_SIZE // channels, channels, imgsize, imgsize]

Channels: 32
Normal Conv 0.871959924697876s, Sep Conv 1.569300651550293s
Channels: 64
Normal Conv 0.909860372543335s, Sep Conv 1.604109525680542s
Channels: 128
Normal Conv 0.9196009635925293s, Sep Conv 1.6144189834594727s

What confuses me is the result is different from the result of the link above: the time cost of sep_conv2d does not have obvious change.

My Questions are:

1. What makes it different between the link above and the experiment by myself?
2. I am a newbie, so is there something wrong in my code to implement the separable_conv2d?
3. How to implement the separable_conv2d can be faster than the normal one in TF or in Pytorch?

Any help would be appreciated. Thank in advance.