screenPointToWorld in processing

user_1022707 0
Viewed 149

I would like to draw where something where the mouse is, after a set of transformations. It is not about the drawing itself, it's getting the point in xyz when a certain transformation is applied. For this post I just minified the problem to a bare minimum. This is the code that I would like to get to work:

void setup() {
  size(600, 600, P3D);
}


public void draw() {
    background(0);
    translate(width/2, height/2, -100);
    pushMatrix();
    float rot_y = map(mouseX, 0, width, 0, TWO_PI);
    rotateY(rot_y);
    int dim = min(width, height) / 2;
    noFill();
    stroke(255);
    box(dim);
    

    float z = 0; // or ((PGraphicsOpenGL)g).cameraNear;?
    PVector pos = screenPointToWorld(mouseX, mouseY, z);
    translate(pos.x, pos.y, pos.z);
    box(5); // should appear where the mouse is!

    popMatrix();
}

This is simulair to the screenPointToWorld from unity: https://docs.unity3d.com/ScriptReference/Camera.ScreenToWorldPoint.html

I did find this, which helped me along the way: https://answers.unity.com/questions/1293942/calculation-behind-camerascreentoworldpoint.html

Those are my 2 attempts:

PVector screenPointToWorld(PVector sp) {
    PGraphicsOpenGL _g = ((PGraphicsOpenGL)g);

    // cam.worldToCameraMatrix?? Not sure about this one
    PMatrix3D final_matrix = new PMatrix3D(_g.modelview);
    final_matrix.apply(_g.projection);
    final_matrix.invert();

    float[] in = new float[4];

    in[0] = sp.x;
    in[1] = sp.y;
    in[2] = sp.z;
    in[3] = 1.0f;  

    /* Map x and y from window coordinates */
    // in[0] = (in[0] - viewport[0]) / viewport[2];
    // in[1] = (in[1] - viewport[1]) / viewport[3];
    in[0] = in[0] / width;
    in[1] = in[1] / height;

    /* Map to range -1 to 1 */
    in[0] = in[0] * 2 - 1;
    in[1] = in[1] * 2 - 1;
    in[2] = in[2] * 2 - 1;

    float[] out = new float[4];

    final_matrix.mult(in, out);

    out[0] /= out[3];
    out[1] /= out[3];
    out[2] /= out[3];
    
    return new PVector(out[X], out[Y], out[Z]);
}

PVector screenPointToWorld2(PVector sp) {
    PGraphicsOpenGL _g = ((PGraphicsOpenGL)g);

    // cam.worldToCameraMatrix?? Not sure about this one
    PMatrix3D worldToCameraMatrix = new PMatrix3D(_g.camera);
    worldToCameraMatrix.invert();

    PMatrix3D world2Screen = new PMatrix3D(_g.projection);
    world2Screen.apply(worldToCameraMatrix);
    
    PMatrix3D screen2World = new PMatrix3D(world2Screen);
    screen2World.invert();

    float[] inn = new float[4];

    inn[0] = 2.0f * (sp.x / width) - 1.0f;
    inn[1] = 2.0f * (sp.y / height) - 1.0f;
    inn[2] = _g.cameraNear;
    inn[3] = 1.0f;  

    float[] pos = new float[4];
    screen2World.mult(inn, pos);

    int X = 0;
    int Y = 1;
    int Z = 2;
    int W = 3;

    pos[W] = 1.0f / pos[W];

    pos[X] *= pos[W];
    pos[Y] *= pos[W];
    pos[Z] *= pos[W];

    return new PVector(pos[X], pos[Y], pos[Z]);
}

But they are incorrect, and I don't have the knowledge to figure out why. I hope someone can help.

UPDATE:

I can draw a box on the mouse position if I want, that is not the problem. The goal is to get screenPointToWorld working as expected. This is part to a way larger program that would not fit in a stackoverflow post. I'm well aware of pushMatrix and popMatrix and how to use them. But they are no use in this case.

processing math matrix
3 Answers

The following runs on my Mac. I used the second function screenPointToWorld that you posted. I didn't do much to the code except to change the x,y,z coordinates to a PVector so that it could be passed to screenPointToWorld. I see a three dimensional box that is rotated by the mouse.

PVector v;

void setup() {
  size(600, 600, P3D);
}

PVector screenPointToWorld(PVector sp) {
  PGraphicsOpenGL _g = ((PGraphicsOpenGL)g);

  // cam.worldToCameraMatrix?? Not sure about this one
  PMatrix3D worldToCameraMatrix = new PMatrix3D(_g.camera);
  worldToCameraMatrix.invert();

  PMatrix3D world2Screen = new PMatrix3D(_g.projection);
  world2Screen.apply(worldToCameraMatrix);

  PMatrix3D screen2World = new PMatrix3D(world2Screen);
  screen2World.invert();

  float[] inn = new float[4];

  inn[0] = 2.0f * (sp.x / width) - 1.0f;
  inn[1] = 2.0f * (sp.y / height) - 1.0f;
  inn[2] = _g.cameraNear;
  inn[3] = 1.0f;

  float[] pos = new float[4];
  screen2World.mult(inn, pos);

  int X = 0;
  int Y = 1;
  int Z = 2;
  int W = 3;

  pos[W] = 1.0f / pos[W];

  pos[X] *= pos[W];
  pos[Y] *= pos[W];
  pos[Z] *= pos[W];

  return new PVector(pos[X], pos[Y], pos[Z]);
}

public void draw() {
  background(0);
  translate(width/2, height/2, -100);
  pushMatrix();
  float rot_y = map(mouseX, 0, width, 0, TWO_PI);
  rotateY(rot_y);
  int dim = min(width, height) / 2;
  noFill();
  stroke(255);
  box(dim);

  float z = 0; // or ((PGraphicsOpenGL)g).cameraNear;?
  v = new PVector(mouseX, mouseY, z);
  PVector pos = screenPointToWorld(v);
  translate(pos.x, pos.y, pos.z);
  box(5); // should appear where the mouse is!

  popMatrix();
}

This revision reverses the order of the boxes; the little one is now called for first. Translations are cumulative so if it is first in line it will be locked to the mouse coordinates. Not sure if this is what you hoped to achieve, but your initial code appears to do no more than seek to change the coordinates for the little box. If that is the case you likely don't need the screenPointToWorld function.

void setup() {
  size(800, 800, P3D);
}

void draw() {
  background(0);
  translate(mouseX, mouseY);
  box(5);
  translate(200,200);
  float rot_y = map(mouseX, 0, width, 0, TWO_PI);
  rotateY(rot_y);
  noFill();
  stroke(255);
  box(width/4);
}
user_128621110

That's what pushMatrix() and popMatrix() are for. They let you apply and then remove a transformation. If you balance push and pop properly you'll end up back at screen coordinates:

public void draw() {
    background(0);
    
    pushMatrix(); // push a new transformation matrix for the big box
      translate(width/2, height/2, -100);
      float rot_y = map(mouseX, 0, width, 0, TWO_PI);
      rotateY(rot_y);
      int dim = min(width, height) / 2;
      noFill();
      stroke(255);
      box(dim);
    popMatrix(); // done with the big box--revert to screen coords

    translate(mouseX, mouseY); // move to the mouse point to draw the small box
    box(5); 
}
user_4697550

Make it difficult if you want, but it's this easy.

void setup() {
  size(600, 600, P3D);
}

public void draw() {
    background(0);
    translate(width/2, height/2, -100);
    float rot_y = map(mouseX, 0, width, 0, TWO_PI);
    rotateY(rot_y);
    int dim = min(width, height) / 2;
    noFill();
    stroke(255);
    box(dim);
    // Reverse rotation and first translation
    rotateY(-rot_y);
    translate(-width/2, -height/2, 100);
    // Set a new translation so small box will follow cursor
    translate(mouseX,mouseY);
    box(5); // should appear where the mouse is!
}
user_128621110
Related