PostgresSQL connection timeout when trying to connect from remote

I have a cloud server which I'm connected to, this server has psql client application installed. On my local machine I have docker and I launched a container with the following docker-compose.yml :

version: '3.8'

services:
  
  timescale:
    container_name: timescaledb
    build: 
      context: .
      dockerfile: postgres.Dockerfile
    ports:
      - "5432:5432"
    volumes:
      - timescale-volume:/home/postgres/pgdata/data
      - timescale-volume:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=password
    networks:
      - trade-net
networks:
  trade-net:
    name: trade-net

volumes:
  timescale-volume:
    name: timescale-volume

This means Postgres is being exposed at port 5432 on my local machine.

docker ps prints:

f719eac32ebc   ark-sql-timescale   "docker-entrypoint.s…"   22 minutes ago   Up 22 minutes   0.0.0.0:5432->5432/tcp   timescaledb

my pg_hba.conf looks like that:

# IPv4 local connections:                                           
host    all             all             127.0.0.1/32            trust
host    all             all             0.0.0.0/0               md5   
# IPv6 local connections:                                            
host    all             all             ::1/128                 trust
host    all             all             ::/0                    md5                                                 
# Allow replication connections from localhost, by a user with the   
# replication privilege.                                             
local   replication     all                                     trust 
host    replication     all             127.0.0.1/32            trust
host    replication     all             ::1/128                 trust

Thus I'm allowing all IPs both on IPv4 and IPv6 with a password connection.

My postgresql.conf file is also listening on all ports:

listen_addresses = '*'

Now from my remote server I try to connect from the psql client into psotgresql which is running via docker on my local machine with:

psql -h my_local_public_ip -p 5432 -U postgres

But the connection hangs and times out. I'm not sure what I am still doing wrong here.