How to get my selection box to move properly using PySide2?

It's trickier than it seems because you can't simply set the position of the rubber band to the cursor, as it would pop it instead of translating it like one would expect.

So instead I opted to use the rubber band's geometry as it was more consistent no matter what direction the user drags in. The idea is when the user pressed the alt key, we save the mouse's current position so that we can calculate a delta as the mouse moves around. With this delta we can move the geometry and set it without getting any pops. It's also possible that the user may repeatedly press and release alt to size the rubber band, move it, resize it, move it, and so on.. So we have to take that into account otherwise it will pop.

Here's a full example that seems to work. Hopefully the comments make it easy to follow the logic:

from PySide2 import QtCore, QtGui, QtWidgets


class Ui_Form(QtWidgets.QWidget):

    def __init__(self):
        super(Ui_Form, self).__init__()
        self.rubberBand = QtWidgets.QRubberBand(QtWidgets.QRubberBand.Rectangle, self)

        # Create variables that will be handling the moving logic.
        self.sourceGeo = None
        self.altPoint = None
        self.delta = None

    def mousePressEvent(self, event):        
        if event.button() == QtCore.Qt.LeftButton:
            self.origin = event.pos()
            self.drag_selection = QtCore.QRect(self.origin, QtCore.QSize())
            self.rubberBand.setGeometry(self.drag_selection)
            self.rubberBand.show()

    # Must implement this event to detect when alt gets released
    def keyReleaseEvent(self, event):
        if event.key() == QtCore.Qt.Key_Alt:
            if self.delta is not None:
                # This is important: Add delta to origin so that it shifts it over.
                # This is needed if the user repeatedly pressed alt to move it, otherwise it would pop.
                self.origin += self.delta

                # Reset the rest of the variables.
                self.sourceGeo = None
                self.altPoint = None
                self.delta = None

    def mouseMoveEvent(self, event):
        if event.modifiers() == QtCore.Qt.AltModifier:
            # Get the point where alt is pressed and the selection's current geometry.
            if self.altPoint is None:
                self.sourceGeo = self.rubberBand.geometry()
                self.altPoint = event.pos()

            self.delta = event.pos() - self.altPoint  # Calculate difference from the point alt was pressed to where the cursor is now.
            newGeo = QtCore.QRect(self.sourceGeo)  # Create a copy
            newGeo.moveTopLeft(self.sourceGeo.topLeft() + self.delta)  # Apply the delta onto the geometry to move it.
            self.rubberBand.setGeometry(newGeo)  # Move the selection!
        else:
            self.drag_selection = QtCore.QRect(self.origin, event.pos()).normalized()
            self.rubberBand.setGeometry(self.drag_selection)

    def mouseReleaseEvent(self, event):
        if event.button() == QtCore.Qt.LeftButton:
            self.rubberBand.hide() 


inst = Ui_Form()
inst.show()

To do that, you need to create an offset !

Here is the method (in order to do what you expected) :

    def mouseMoveEvent(self, event):
        if event.modifiers() == QtCore.Qt.AltModifier:

            mouse_x = event.pos().x()
            mouse_y = event.pos().y()

            rubber_band_w = self.rubberBand.size().width()
            rubber_band_h = self.rubberBand.size().height()

            self.rubberBand.move(QtCore.QPoint(mouse_x - rubber_band_w, mouse_y - rubber_band_h))

        else:
            self.drag_selection = QtCore.QRect(self.origin, event.pos()).normalized()
            self.rubberBand.setGeometry(self.drag_selection)

I decompose the process with the mouse_x, mouse_y, rubber_band_w, rubber_band_h in order to simplify the comprehension, but you can refacto this.

EDIT : Thanks to the comment, I realize that I misunderstand the problem. Indeed this solution works only for dragging towards the bottom-right (like 2nd picture). I think that in order to achieve this, You just need to compare the mouse position and the rubberband in order create an offset in consequences.