How to drag the object outside the viewport width in HTML Canvas?

I am not sure if I'm asking this question correctly. Here is the scenario, the width of Canvas is around 20,000px. I need to drag the timeline cursor on x-axis. I am using mousedown and mousemove event to drag, but whenever I drag the cursor to extreme right and scroll the canvas and try to drag again, it places the cursor in the beginning. I suspects that the position x and y of mousedown event is relative to viewport, that's why it places the cursor in the beginning. I have looked around the internet but doesn't found any solution to it.