Delayed messages are delivered in the same order they are sent in RabbitMQ

I am trying to implement a rabbitmq delayed message queue (without the plugin). And I am experiencing some weird issues. Here is my current setup (not actual names, btw)

1 exchange "Exchange"

2 queues: The final destination queue "Queue" and the delayed queue "Delayed"

Both queues are bound to "Exchange"

Here is how I published the message

err := confirmModeChannel.Publish(
  "Exchange", // exchange
  "Delayed",   // routing key
  false,        // mandatory
  false,
  amqp.Publishing{
    Body:         payload,
    Expiration:   expiration,
  })

(the publishing confirm is handled but not shown here)

And how the delayed channel is defined

channel.QueueDeclare(
    "Delayed",             // name
    true,             // durable
    false,            // delete when unused
    false,            // exclusive
    false,            // no-wait
    map[string]interface{}{
      "x-dead-letter-exchange": "Exchange",
      "x-dead-letter-routing-key": "Queue"
    }, // arguments
  )

and how it's bound to the exchange

channel.QueueBind(
    "Delayed",    // queue name
    "Delayed",    // routing key
    "Exchange",    // exchange
    false,
    nil,
)

The exchange is properly defined as "direct"

The problem is that if I consume "Queue", and publish some messages to "Delayed", the messages arrive in the same order as they are sent, but delayed until the largest expiration. For example

send message 1 with 3s delay
send message 2 with 1s delay

and 3s later, they are delivered

received message 1 at [timestamp1]
received message 2 at [timestamp2]

timestamp1 and timestamp2 is almost the same (some nanosecond different) I have no ideas where did I mess up? I expect the messages to arrive in this order

received message 2 at [timestamp2]
received message 1 at [timestamp1]

with timestamp2 is about 2s apart from timestamp1

I'm using

• golang 1.14

• rabbitmq:3.8.3-management-alpine running in docker

• github.com/streadway/amqp v0.0.0-20200108173154-1c71cc93ed71