How to get the position of a rectangle in a canvas after transforming the canvas

Question

I am trying to draw a rectangle with a label slightly above it on a canvas. The x, y, width, and height were generated to be around an object that was detected using the coco-ssd model in tensorflow. The problem is that the coordinates generated by the coco-ssd model in tensorflow is relative to a different origin from the canvas itself. More specifically, the origin for the coco-ssd model is at the top right corner, and the origin for the canvas is at the top left corner.

I am able to move the origin of the canvas, but not the model's origin (That I know of). To move the canvas' origin, I translated the canvas to the right 410px, 4px smaller than the width of the canvas, and then reflected it horizontally. This draws the rectangle at the correct position. If I were to also create the text at this point it would be inverted and unreadable (but at the proper position). If it were possible to get the x and y position of the rectangle after translating the canvas back left 410px and reflecting it horizontally once more I could easily use those coordinates to fill in the text at the proper position. From what I have learned about canvas, this is not possible. (Please correct me if im wrong)

Another solution I considered would be to use the x position generated and to apply this formula, -x+xLim, where xLim is the largest possible value of x. The problem here is that obtaining xLim is not possible either, it is not static, and it will change depending on the distance away from the detected object. I know this from trying to obtain what xLim could be by simply positioning the object on the leftmost side of the screen. (The largest value of x that is currently viewable with respect to the coco-ssd model's origin) Keep in mind, that if I create distance from the object, the value of x on the leftmost side of the screen will increase. If I were able to somehow grab the largest x value that is actively viewable on the canvas then this would be another viable solution.

Here is the function in-charge of drawing to the canvas.

export const drawRect = (x, y, width, height, text, ctx, canvas) => {
    ctx.clearRect(0, 0, canvas.current.width, canvas.current.height);
    ctx.transform(-1, 0, 0, 1, 410, 0);

        //draw rectangle
        ctx.beginPath();        
        const r = ctx.rect(x,y,width,height)
        ctx.stroke()

        //draw text
        ctx.save();
        ctx.scale(-1,1);
        ctx.translate(-410, 0)
        //update x and y to point to where the rectangle is currently
        ctx.fillText(text,x,y-5)
        ctx.stroke()
        ctx.restore()
    })

I feel heavily limited by the API available to react native and I hope that there is something I've simply overlooked. I've spent lots of time trying to resolve this issue and have found many relatable questions on stack overflow, but none of them gave insight as to how to solve this problem with so many unknown variables.

Here are some images to provide a visual of the issue at hand.

Leftmost

Rightmost

Without Restoring canvas to its original origin

SUMMARY:

The origin for the coco-ssd model is at the top right corner, and the origin for the canvas is at the top left corner.

I need to

A.)Somehow grab the largest x value that is actively viewable on the canvas

OR

B.) get the x and y position of the rectangle after translating the canvas back left 410px and reflecting it horizontally once more

This is in a react native expo environment

Public repo: https://github.com/dpaceoffice/MobileAppTeamProject

user_6486713 · Answer

The transforms that you are using seem unnecessary. Here is a simple proof of concept using the embed code for coco-ssd and an example on manipulating videos:

The following code does not use any transforms and everything works as expected. The video, canvas and model all use the same coordinate system:

(0, 0) is top left corner of the frame
(width, 0) is top right
(0, height) is bottom left
(width, height) is bottom right

And the rectangle (x, y, width, height) also use the familiar coordinates everywhere:

(x, y) is top left
(x + width, y) is top right
(x, y + height) is bottom left
(x + width, y + height) is bottom right

The only thing that is a bit different, is the context.fillText(text, x, y) method:

(x, y) is bottom left corner of the text. However, this is actually nice, because we just can draw the rectangle and the text with the same coordinates and the text will sit directly above the rectangle.
(x, y - fontSize) is usually the top left corner
(x, y + fontSize) is the coordinate for the next row.

If you want to place the text at a different position, the context.measureText(text) might be of interest. It returns a TextMetrics object. Usually, the .width property of this object is of most interest:

(x + context.measureText(text).width, y) is the bottom right corner of the text.

const processor = {}

processor.doLoad = function (model)
{
    const video = document.getElementById("video")
    this.video = video

    this.canvas = document.getElementById("canvas")
    this.context = this.canvas.getContext("2d")

    this.model = model

    video.addEventListener("play", () => {
        this.canvas.width = video.videoWidth
        this.canvas.height = video.videoHeight
        this.timerCallback()
    }, false)
}

processor.timerCallback = async function ()
{
    if (this.video.paused || this.video.ended)
        return
    await this.computeFrame()
    setTimeout(() => this.timerCallback(), 0)
};

processor.computeFrame = async function ()
{
    // detect objects in the image.
    const predictions = await this.model.detect(this.video)

    const context = this.context
    // draws the frame from the video at position (0, 0)
    context.drawImage(this.video, 0, 0)

    context.strokeStyle = "red"
    context.fillStyle = "red"
    context.font = "16px sans-serif"
    for (const { bbox: [x, y, width, height], class: _class, score } of predictions)
    {
        // draws a rect with top-left corner of (x, y)
        context.strokeRect(x, y, width, height)
        // writes the class directly above (x, y), outside the rectangle
        context.fillText(_class, x, y)
        // writes the class directly below (x, y), inside the rectangle
        context.fillText(score.toFixed(2), x, y + 16)
    }
}

// Load the model.
const model = cocoSsd.load()
model.then(model => processor.doLoad(model))

As it seems, that you want to draw the image into some part of the canvas, here is some code that showcases how to correctly use the transforms. There are two options here:

Use the transforms (1x translate + 1x scale), so that you can use the coordinate system from the image. Note, that it does not require negative scales. The drawing is handled by the browser. You have to correct the font and line widths for the scaling.
Also use transforms (1x translate + 1x scale). Then restore the canvas, and transform the points manually. This involves some further math to transform the points. However, the upside is, that you don't have to correct the font and line widths.

const image = document.createElement("canvas")
image.width = 1200
image.height = 600

const image_context = image.getContext("2d")
image_context.fillStyle = "#4af"
image_context.fillRect(0, 0, 1200, 600)

const circle = (x, y, r) =>
{
    image_context.beginPath()
    image_context.arc(x, y, r, 0, Math.PI*2)
    image_context.fill()
}

image_context.fillStyle = "#800"
image_context.fillRect(500-40/2, 400, 40, 180)

image_context.fillStyle = "#080"
circle(500, 400, 100)
circle(500, 300, 70)
circle(500, 220, 50)

const prediction = { bbox: [500-100, 220-50, 100*2, (400+180)-(220-50)], class: "tree", score: 0.42 } // somehow get a prediction

const canvas = document.getElementById("canvas")
const context = canvas.getContext("2d")

// we want to draw the big image into a smaller area (x, y, width, height)
const x = 50
const y = 80
const width = 220
const height = width * (image.height / image.width)

// debug: mark the area that we want to draw the image into
{
    context.save() // save context, so that stroke properties can be restored
    context.lineWidth = 5
    context.strokeStyle = "red"
    context.setLineDash([10, 10])
    context.strokeRect(x, y, width, height)
    context.restore()
}

{
    // save context, before doing any transforms (even before setTransform)
    context.save()

    // Move top left corner to (x, y)
    context.translate(x, y)

    // This is the scale factor, it should be less than one, because the image is bigger that the target area. The idea is to increase the scale by the target area width, and then decrease the scale by the image width.
    const f = width / image.width
    context.scale(f, f)

    // Draws the image, note that the coordinates are just (0, 0) without scaling.
    context.drawImage(image, 0, 0)

    // option 1: draw the prediction using the native transforms
    if (true)
    {
        context.strokeStyle = "red"
        context.fillStyle = "red"
        context.lineWidth = 1 / f // linewidth and font-size has to be adjusted by scaling
        const fontSize = 16 / f
        context.font = fontSize.toFixed(0) + "px sans-serif"
        const [p_x, p_y, p_width, p_height] = prediction.bbox
        context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
        context.fillText(prediction.class, p_x, p_y) // draw the text
        context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text
    }

    const matrix = context.getTransform() // save transform for option 2, needs to be done before restore()

    context.restore()

    // option 2: draw the prediction by manually transforming the corners
    if (false)
    {
        context.save()

        context.strokeStyle = "red"
        context.fillStyle = "red"
        context.lineWidth = 1
        const fontSize = 16
        context.font = fontSize + "px sans-serif"

        let [p_x, p_y, p_width, p_height] = prediction.bbox
        // manually transform corners
        const topleft = matrix.transformPoint(new DOMPoint(p_x, p_y))
        const bottomright = matrix.transformPoint(new DOMPoint(p_x + p_width, p_y + p_height))
        p_x = topleft.x
        p_y = topleft.y
        p_width = bottomright.x - topleft.x
        p_height = bottomright.y - topleft.y

        context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
        context.fillText(prediction.class, p_x, p_y) // draw the text
        context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text

        context.restore()
    }
}

Now, if you need to mirror the image using the transforms, you just have to make sure to mirror it back before writing the text.

const image = document.createElement("canvas")
image.width = 1200
image.height = 600

const image_context = image.getContext("2d")
image_context.fillStyle = "#4af"
image_context.fillRect(0, 0, 1200, 600)

const circle = (x, y, r) =>
{
    image_context.beginPath()
    image_context.arc(x, y, r, 0, Math.PI*2)
    image_context.fill()
}

image_context.fillStyle = "#800"
image_context.fillRect(500-40/2, 400, 40, 180)

image_context.fillStyle = "#080"
circle(500, 400, 100)
circle(500, 300, 70)
circle(500, 220, 50)

const prediction = { bbox: [500-100, 220-50, 100*2, (400+180)-(220-50)], class: "tree", score: 0.42 } // somehow get a prediction

const canvas = document.getElementById("canvas")
const context = canvas.getContext("2d")

// we want to draw the big image into a smaller area (x, y, width, height)
const x = 50
const y = 80
const width = 220
const height = width * (image.height / image.width)

// debug: mark the area that we want to draw the image into
{
    context.save() // save context, so that stroke properties can be restored
    context.lineWidth = 5
    context.strokeStyle = "red"
    context.setLineDash([10, 10])
    context.strokeRect(x, y, width, height)
    context.restore()
}

{
    // save context, before doing any transforms (even before setTransform)
    context.save()

    // Move top left corner to (x, y)
    context.translate(x, y)

    // This is the scale factor, it should be less than one, because the image is bigger that the target area. The idea is to increase the scale by the target area width, and then decrease the scale by the image width.
    const f = width / image.width
    context.scale(f, f)

    // mirror the image before drawing it
    context.scale(-1, 1)
    context.translate(-image.width, 0)

    // Draws the image, note that the coordinates are just (0, 0) without scaling.
    context.drawImage(image, 0, 0)

    // option 1: draw the prediction using the native transforms
    if (true)
    {
        const [p_x, p_y, p_width, p_height] = prediction.bbox
        // move to correct position and only then undo the mirroring
        context.save()
        context.translate(p_x + p_width, p_y) // move to top "right" (that is now actually at the left, due to mirroring)
        context.scale(-1, 1)

        context.strokeStyle = "red"
        context.fillStyle = "red"
        context.lineWidth = 1 / f // linewidth and font-size has to be adjusted by scaling
        const fontSize = 16 / f
        context.font = fontSize.toFixed(0) + "px sans-serif"
        context.strokeRect(0, 0, p_width, p_height) // draw the prediction
        context.fillText(prediction.class, 0, 0) // draw the text
        context.fillText(prediction.score.toFixed(2), 0, 0 + fontSize) // draw the text
        context.restore()
    }

    const matrix = context.getTransform() // save transform for option 2, needs to be done before restore()

    context.restore()

    // option 2: draw the prediction by manually transforming the corners
    if (false)
    {
        context.save()

        context.strokeStyle = "red"
        context.fillStyle = "red"
        context.lineWidth = 1
        const fontSize = 16
        context.font = fontSize + "px sans-serif"

        let [p_x, p_y, p_width, p_height] = prediction.bbox
        // manually transform corners, note that compared to previous snippet, topleft now uses the top right corner of the rectangle
        const topleft = matrix.transformPoint(new DOMPoint(p_x + p_width, p_y))
        const bottomright = matrix.transformPoint(new DOMPoint(p_x, p_y + p_height))
        p_x = topleft.x
        p_y = topleft.y
        p_width = bottomright.x - topleft.x
        p_height = bottomright.y - topleft.y

        context.strokeRect(p_x, p_y, p_width, p_height) // draw the prediction
        context.fillText(prediction.class, p_x, p_y) // draw the text
        context.fillText(prediction.score.toFixed(2), p_x, p_y + fontSize) // draw the text

        context.restore()
    }
}

How to get the position of a rectangle in a canvas after transforming the canvas

1 Answers