I've already setup the ssh key, but VSCode keeps asking for password

It was a problem with the config file.
The VSCode needs the "absolute" path.

In case of MacOS, ssh-copy-id seems to only copy the absolute path relative to the user.
In other words, it omits "/Users/username" before "/.ssh".

Adding "/Users/username" in the IdentityFile attribute in .ssh/config solved the problem.

Check if this microsoft/vscode-remote-release issue 2518 applies:

You should be able to get out of this state by deleting the file (on the remote server side, as sudo root) in the log, /home/#####/.vscode-server/bin/78a4c91400152c0f27ba4d363eb56d2835f9903a/vscode-remote-lock.#####.78a4c91400152c0f27ba4d363eb56d2835f9903a (with unlink) or running the command "Kill VS Code Server on Host..."

If it happens again, you might try setting remote.SSH.useFlock.

The exact command to run in the command palette (View->Command Palette) is:

Remote-SSH: Kill VS Code Server on Host...

Also:

In my case, deleting entire ~/.vscode-server directory after connecting to the container through ssh using terminal worked.
(Deleting only ~/.vscode-server/bin did not work.)

The OP sukrama confirms in the comments having solved the issue

It was a problem with ssh key path in config file.

Here's a quick and handy fix: You do not have to delete the entire .vscode-server folder each time! The problem seems to be a file named 'vscode-remote-lock...'. It can be located inside a folder in ~/.vscode-server/bin/ . That file gets created at each ssh login through vscode. Run the following script on the remote host. It deletes that file whenever it is created:

while true
do
  if ls /home/<your-username>/.vscode-server/bin/*/vscode-remote-lock.<your-username>.* 1> /dev/null 2>&1; then
    find /home/<your-username>/.vscode-server/bin/*/ -name vscode-remote-lock.<your-username>.* -delete
    echo "Killed the troublemaker! ^_^"
  fi
done

The file names and the folder names may differ from machine to machine. So find the names on your machine and paste them in the script. Then run the script and you're good to go.

VSCode in my Windows machine was asking for password even with my key correctly configured (it works from the terminal).

My problem was that VSCode was choosing a wrong user. I was using a host configured in my ssh config file, and VSCode was setting the user as DOMAIN\user instead of user. I solved it configuring the correct user in my .ssh/config file:

Host dados
    HostName vrt1234
    User xxxxx

I had to use UseKeychain yes in my ~/.ssh/config file.

The config file looks like this:

Host server.tld
  HostName server.tld
  User user
  UseKeychain yes
  IdentityFile  ~/.ssh/key

You have to enter ssh-add -K ~/.ssh/key to add your passphrase to KeyChain first.

In case you're having this problem in Windows, keep in mind that the public/private keys that you might use to connect to a remote machine from WSL aren't the same ones that VS Code will use to connect from Windows. You need to create a separate public/private key pair for Windows, and export that private key to the remote server too.

From VS Code remote debug tips and tricks:

In a Powershell window, create a public/private key pair just as you would in a Linux terminal:

ssh-keygen -t rsa -b 4096

Then export it to the remote server:

export USER_AT_HOST="your-user-name-on-host@hostname"
export PUBKEYPATH="$HOME/.ssh/id_rsa.pub"

ssh $USER_AT_HOST "powershell New-Item -Force -ItemType Directory -Path \"\$HOME\\.ssh\"; Add-Content -Force -Path \"\$HOME\\.ssh\\authorized_keys\" -Value '$(tr -d '\n\r' < "$PUBKEYPATH")'"

Make sure you can connect via passwordless SSH via PowerShell.

Finally, in VS Code. press Ctrl+Shift+P to open the command palette and select "Remote-SSH: Open SSH Configuration File..." and edit the config file like so:

Host [convenient name]
    HostName [hostname]
    User [username]
    IdentityFile C:\Users\[username]\.ssh\id_rsa*

Then when you run "Remote-SSH: Connect to Host..." in VS Code and choose the host above, it should connect without prompting for a password.

In case this helps someone, i had a similar issue where VSC was asking for a password (instead of a passphrase). I noticed that my key was on a network drive and it looks like VSC cannot read it there. I moved it to a local file (C:) and it worked.

Not enough rep to comment, but if you followed the steps from this Stack Overflow post and are still running into issues, your VSCode Remote-SSH config file path may not be set.

Make sure that the setting remote.SSH.configFile is set to ~/.ssh/config.

For me it was that my public auth ssh was not working and my home directory permissions were the problem. I had to remove group and other write permissions to my home directory and then everything worked:

chmod go-w ~/