Gstreamer gst_buffer_pool_acquire_buffer function is slow on ARM

My Gstreamer version is 1.17, cross compiled using instructions from here.

Here is my gstreamer pipeline,

appsrc name=framesrc0 do-timestamp=true format=time ! video/x-raw,width=640,height=480,framerate=30/1,format=NV12 ! queue ! x264enc ! queue ! h264parse ! mpegtsmux ! filesink name=mysink location=./myfile.ts 

I feed NV12 frames to appsrc using the below function (6404801.5 = 460800 bytes)

    bool BelGst::FeedData0(uint8_t *buf, uint32_t len)
    {
        GstFlowReturn ret;
        GstBuffer *buffer;
        GstMapInfo info;
        timespec ts_beg, ts_end;
        uint32_t time_ms;
    
        clock_gettime(CLOCK_MONOTONIC, &ts_beg);
        ret = gst_buffer_pool_acquire_buffer (pool0, &buffer, NULL);
        if (G_UNLIKELY (ret != GST_FLOW_OK)) {
            cout <<  "BufferPool pool0 failed" << endl;
            return FALSE;
        }
        clock_gettime(CLOCK_MONOTONIC, &ts_end);
    
        time_ms = (ts_end.tv_sec - ts_beg.tv_sec)*1000 + (ts_end.tv_nsec - ts_beg.tv_nsec) / 1e6;
    
        cout << "Buffer pool acquire time = " << time_ms << "ms" << endl;
    
        /* Set its timestamp and duration */
        GST_BUFFER_TIMESTAMP(buffer) = timestamp0;
        GST_BUFFER_DURATION(buffer) = gst_util_uint64_scale(1, GST_SECOND, 30);
        GST_BUFFER_OFFSET(buffer) = offset0++; 
    
        timestamp0 += GST_BUFFER_DURATION(buffer);
    
        gst_buffer_map(buffer, &info, GST_MAP_WRITE);
        memcpy(info.data, buf, len);
        gst_buffer_unmap(buffer, &info);
    
        g_signal_emit_by_name(app_source0, "push-buffer", buffer, &ret);
    
        gst_buffer_unref(buffer);
    
        return TRUE;
    }

I've setup the buffer pool as shown below,

    void BufferPoolSetup(GstBufferPool *&pool)
    {
        GstStructure *config;
        int size, min, max;
        GstCaps *caps;
    
        pool = gst_buffer_pool_new();
    
        /* get config structure */
        config = gst_buffer_pool_get_config(pool);
    
        size = 640*480*1.5;
        min = 1;
        max = 4;
    
        caps = gst_caps_from_string("video/x-raw");
    
        /* set caps, size, minimum and maximum buffers in the pool */
        gst_buffer_pool_config_set_params (config, caps, size, min, max);
        gst_caps_unref(caps);
    
        gst_buffer_pool_set_config (pool, config);
    
        /* and activate */
        gst_buffer_pool_set_active (pool, TRUE);
    
        return;
    }

When I run the pipeline, I see that the function gst_buffer_pool_acquire_buffer is taking somewhere between 20ms to 60ms. Could someone point if there is something wrong in my approach? Am I missing something?