Environment variables are not updating after changes in docker-compose

I try to deploy mysql server on docker and when I try to change enviroment variables in my docker-compose.yml file they just don't change when I do docker-compose up. I have noticed, that when I will remove container using docker container remove <process ID> and again try to docker-compose up it will works. If I will change name of the service in docker-compose.yml it will work too. Do I have always to remove stopped containers to restart enviroments variables or am I doing something wrong?

My docker-compose.yml file:

version: '3.3'
services:
  db:
    image: mysql:5.7
    restart: always
    environment:
      MYSQL_DATABASE: 'db'
      MYSQL_USER: 'user'
      MYSQL_PASSWORD: 'password1'
      MYSQL_ROOT_PASSWORD: 'password1'
    ports:
      - '3306:3306'
    expose:
      - '3306'