VSCode Remote-SSH extension through SSH gateway?

Question

I need to access several hosts that are behind an SSH gateway. The way the gateway works is:

I upload my public key to a web portal
I get an FQDN and port for SSH access
My client connects with the gateway, which then connects me to the specific host I'm trying to connect to based on the port I came in on

My SSH key is never actually deployed to the remote host itself. It's held by the gateway which has its own keypair that is distributed to the remote host--I do not have access to this private key.

With default settings, here's the logs:

[07:25:44.335] Log Level: 2
[07:25:44.337] remote-ssh@0.65.7
[07:25:44.337] darwin arm64
[07:25:44.338] SSH Resolver called for "ssh-remote+HOSTNAME", attempt 1
[07:25:44.338] "remote.SSH.useLocalServer": true
[07:25:44.338] "remote.SSH.path": undefined
[07:25:44.338] "remote.SSH.configFile": undefined
[07:25:44.338] "remote.SSH.useFlock": true
[07:25:44.338] "remote.SSH.lockfilesInTmp": false
[07:25:44.338] "remote.SSH.localServerDownload": auto
[07:25:44.339] "remote.SSH.remoteServerListenOnSocket": false
[07:25:44.339] "remote.SSH.showLoginTerminal": false
[07:25:44.339] "remote.SSH.defaultExtensions": []
[07:25:44.339] "remote.SSH.loglevel": 2
[07:25:44.339] SSH Resolver called for host: HOSTNAME
[07:25:44.339] Setting up SSH remote "HOSTNAME"
[07:25:44.341] Acquiring local install lock: /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-remote-ssh-HOSTNAME-install.lock
[07:25:44.349] Looking for existing server data file at /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:25:44.349] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:25:44.351] Install and start server if needed
[07:25:44.354] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:25:44.354] Checking ssh with "ssh -V"
[07:25:44.358] > OpenSSH_8.1p1, LibreSSL 2.7.3

[07:25:44.359] askpass server listening on /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-a4f2481dc32265d4aa0d1997d55a93523914f69d.sock
[07:25:44.360] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-04d1b772731e5c58d6a1e3212e8ab358ab99d39c.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52496","-o","ConnectTimeout=15","HOSTNAME","bash"],"dataFilePath":"/Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json"}
[07:25:44.360] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-a4f2481dc32265d4aa0d1997d55a93523914f69d.sock"}
[07:25:44.360] Spawned 6028
[07:25:44.411] > local-server-1> Spawned ssh, pid=6034
[07:25:44.413] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[07:25:44.615] stderr> debug1: Server host key: ssh-rsa SHA256:uvobnEougyevd5GyZzHzqTJ2C/vx2RgT1agdqvlXZRc
[07:25:44.743] stderr> This is a private system; unauthorized access is prohibited. Disconnect immediately if you have not been explicitly authorized to access this system.
[07:25:45.714] stderr> Authenticated to 54.189.187.155 ([54.189.187.155]:47000).
[07:25:46.440] > ready: 9df4e9b85f9c
[07:25:46.496] > Linux 5.3.0-1017-aws #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020
[07:25:46.496] Platform: linux
[07:25:46.564] > 9df4e9b85f9c: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/vscode-remote-lock.ubuntu.379476f0e13988d90fab105c5c19e7abc8b1dea8
[07:25:46.588] > Installing to /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8...
> 9df4e9b85f9c%%1%%
> Downloading with wget
[07:25:48.464] > Download complete
> 9df4e9b85f9c%%2%%
> tar --version:
> tar (GNU tar) 1.29
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later .
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Written by John Gilmore and Jay Fenlason.
[07:25:49.883] > Checking /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log and /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.pid for a running server
> WARNING: VS Code Server is running but its logfile is missing. Don't delete the VS Code Server manually, run the command 'Uninstall VS Code Server'.
> Starting server with command... /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown  --port=0  &> "/home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log" < /dev/null
> printenv:
>     SSH_CONNECTION=10.1.1.254 45382 10.1.1.4 22
>     LANG=C.UTF-8
>     XDG_SESSION_ID=74
>     USER=ubuntu
>     PWD=/home/ubuntu
>     HOME=/home/ubuntu
>     SSH_CLIENT=10.1.1.254 45382 22
>     MAIL=/var/mail/ubuntu
>     SHELL=/bin/bash
>     SHLVL=3
>     VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
>     LOGNAME=ubuntu
>     XDG_RUNTIME_DIR=/run/user/1000
>     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
>     _=/usr/bin/printenv
>     OLDPWD=/home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8
> Spawned remote server: 19082
[07:25:49.917] > Waiting for server log...
[07:25:49.941] > Waiting for server log...
[07:25:49.989] >  
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>  
> Checking server status on port 43643 with wget
[07:25:50.158] > 9df4e9b85f9c: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==43643==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==0==
> downloadTime==1868==
> installTime==1402==
> extInstallTime====
> serverStartTime==88==
> connectionToken==1111aaa1-11a1-111a-11a1-1a111a111a11==
> 9df4e9b85f9c: end
[07:25:50.158] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==43643==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==0==
downloadTime==1868==
installTime==1402==
extInstallTime====
serverStartTime==88==
connectionToken==1111aaa1-11a1-111a-11a1-1a111a111a11==

[07:25:50.159] Remote server is listening on 43643
[07:25:50.159] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"port":43643},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"1111aaa1-11a1-111a-11a1-1a111a111a11"},"downloadTime":1868,"installTime":1402,"serverStartTime":88,"installUnpackCode":"success"}
[07:25:50.160] Persisting server connection details to /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-HOSTNAME-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:25:50.162] Starting forwarding server. localPort 52500 -> socksPort 52496 -> remotePort 43643
[07:25:50.162] Forwarding server listening on 52500
[07:25:50.163] Waiting for ssh tunnel to be ready
[07:25:50.164] Tunneled 43643 to local port 52500
[07:25:50.164] Resolved "ssh-remote+HOSTNAME" to "127.0.0.1:52500"
[07:25:50.165] [Forwarding server 52500] Got connection 0
[07:25:50.171] ------




[07:25:50.198] [Forwarding server 52500] Got connection 1
[07:25:50.220] Failed to set up socket for dynamic port forward to remote port 43643: Socket closed. Is the remote port correct?
[07:25:50.245] Failed to set up socket for dynamic port forward to remote port 43643: Socket closed. Is the remote port correct?

I ran "rm -rf ~/.vscode-server", then I then followed the instructions under "Connecting to systems that dynamically assign machines per connection" on this page: https://code.visualstudio.com/docs/remote/troubleshooting

Here's my ~/.ssh/config:

Host ubuntu
  HostName 
  Port 47000
  ControlMaster auto
  ControlPath  ~/.ssh/sockets/%r@%h:%p
  ControlPersist  600

And here's the remote server's sshd_config (I'm using a Mac so ControlMaster should work):

#   $OpenBSD: sshd_config,v 1.101 2017/03/14 07:19:07 djm Exp $

# This is the sshd server system-wide configuration file.  See
# sshd_config(5) for more information.

# This sshd was compiled with PATH=/usr/bin:/bin:/usr/sbin:/sbin

# The strategy used for options in the default sshd_config shipped with
# OpenSSH is to specify options with their default value where
# possible, but leave them commented.  Uncommented options override the
# default value.

#Port 22
#AddressFamily any
#ListenAddress 0.0.0.0
#ListenAddress ::

#HostKey /etc/ssh/ssh_host_rsa_key
#HostKey /etc/ssh/ssh_host_ecdsa_key
#HostKey /etc/ssh/ssh_host_ed25519_key

# Ciphers and keying
#RekeyLimit default none

# Logging
#SyslogFacility AUTH
#LogLevel INFO

# Authentication:

#LoginGraceTime 2m
#PermitRootLogin prohibit-password
#StrictModes yes
#MaxAuthTries 6
#MaxSessions 10

#PubkeyAuthentication yes

# Expect .ssh/authorized_keys2 to be disregarded by default in future.
#AuthorizedKeysFile .ssh/authorized_keys .ssh/authorized_keys2

#AuthorizedPrincipalsFile none

#AuthorizedKeysCommand none
#AuthorizedKeysCommandUser nobody

# For this to work you will also need host keys in /etc/ssh/ssh_known_hosts
#HostbasedAuthentication no
# Change to yes if you don't trust ~/.ssh/known_hosts for
# HostbasedAuthentication
#IgnoreUserKnownHosts no
# Don't read the user's ~/.rhosts and ~/.shosts files
#IgnoreRhosts yes

# To disable tunneled clear text passwords, change to no here!
PasswordAuthentication no
#PermitEmptyPasswords no

# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no

# Kerberos options
#KerberosAuthentication no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
#KerberosGetAFSToken no

# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
#GSSAPIStrictAcceptorCheck yes
#GSSAPIKeyExchange no

# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication.  Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
AllowStreamLocalForwarding yes
#AllowAgentForwarding yes
AllowTcpForwarding yes
#GatewayPorts no
X11Forwarding yes
#X11DisplayOffset 10
#X11UseLocalhost yes
#PermitTTY yes
PrintMotd no
#PrintLastLog yes
#TCPKeepAlive yes
#UseLogin no
#PermitUserEnvironment no
#Compression delayed
#ClientAliveInterval 0
#ClientAliveCountMax 3
#UseDNS no
#PidFile /var/run/sshd.pid
#MaxStartups 10:30:100
#PermitTunnel no
#ChrootDirectory none
#VersionAddendum none

# no default banner path
#Banner none

# Allow client to pass locale environment variables
AcceptEnv LANG LC_*

# override default of no subsystems
Subsystem   sftp    /usr/lib/openssh/sftp-server

# Example of overriding settings on a per-user basis
#Match User anoncvs
#   X11Forwarding no
#   AllowTcpForwarding no
#   PermitTTY no
#   ForceCommand cvs server

After attempting to reconnect, these are the logs:

[07:35:52.712] Log Level: 2
[07:35:52.714] remote-ssh@0.65.7
[07:35:52.714] darwin arm64
[07:35:52.715] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:35:52.715] "remote.SSH.useLocalServer": true
[07:35:52.715] "remote.SSH.path": undefined
[07:35:52.715] "remote.SSH.configFile": undefined
[07:35:52.715] "remote.SSH.useFlock": true
[07:35:52.715] "remote.SSH.lockfilesInTmp": false
[07:35:52.715] "remote.SSH.localServerDownload": auto
[07:35:52.715] "remote.SSH.remoteServerListenOnSocket": false
[07:35:52.716] "remote.SSH.showLoginTerminal": false
[07:35:52.716] "remote.SSH.defaultExtensions": []
[07:35:52.716] "remote.SSH.loglevel": 2
[07:35:52.716] SSH Resolver called for host: ubuntu
[07:35:52.716] Setting up SSH remote "ubuntu"
[07:35:52.717] Acquiring local install lock: /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-remote-ssh-ubuntu-install.lock
[07:35:52.726] Looking for existing server data file at /Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-ubuntu-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json
[07:35:52.726] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:35:52.728] Install and start server if needed
[07:35:52.730] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:35:52.730] Checking ssh with "ssh -V"
[07:35:52.733] > OpenSSH_8.1p1, LibreSSL 2.7.3

[07:35:52.734] askpass server listening on /var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-53beece063b2e0e2b4c364baee37f2170d54995c.sock
[07:35:52.735] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-ef52c1a9145be571fa987d52436563f7df923b30.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52611","-o","ConnectTimeout=15","ubuntu"],"dataFilePath":"/Users/josh/Library/Application Support/Code/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-ubuntu-379476f0e13988d90fab105c5c19e7abc8b1dea8-0.65.7/data.json"}
[07:35:52.735] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/_9/vfmchjgd5gl2q6s8p3y1pmg80000gn/T/vscode-ssh-askpass-53beece063b2e0e2b4c364baee37f2170d54995c.sock"}
[07:35:52.735] Spawned 6261
[07:35:52.786] > local-server-1> Spawned ssh, pid=6267
[07:35:52.788] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[07:35:52.973] stderr> debug1: Server host key: ssh-rsa SHA256:uvobnEougyevd5GyZzHzqTJ2C/vx2RgT1agdqvlXZRc
[07:35:53.108] stderr> This is a private system; unauthorized access is prohibited. Disconnect immediately if you have not been explicitly authorized to access this system.
[07:35:53.879] stderr> Authenticated to HOSTNAME ([HOSTNAME]:47000).
[07:35:54.565] > -- UDF SSH ---------------------
> Deployment: f5 CIS for k8s w calico (w lab guide)
> Component:  kube-master
> --------------------------------
> Welcome to Ubuntu 18.04.4 LTS (GNU/Linux 5.3.0-1017-aws x86_64)
> 
>  * Documentation:  https://help.ubuntu.com
>  * Management:     https://landscape.canonical.com
>  * Support:        https://ubuntu.com/advantage
> 
>   System information as of Wed Aug 11 14:35:54 UTC 2021
> 
>   System load:     0.16               IP address for eth0:       10.1.1.4
>   Usage of /:      31.3% of 19.32GB   IP address for eth1:       10.1.20.20
>   Memory usage:    17%                IP address for docker0:    172.17.0.1
>   Swap usage:      0%                 IP address for tunl0:      192.168.106.0
>   Processes:       232                IP address for tailscale0: 100.71.141.61
>   Users logged in: 1
> 
>  * Super-optimized for small spaces - read how we shrank the memory
>    footprint of MicroK8s to make it the smallest full K8s around.
> 
>    https://ubuntu.com/blog/microk8s-memory-optimisation
> 
>   Get cloud support with Ubuntu Advantage Cloud Guest:
>     http://www.ubuntu.com/business/services/cloud
> 
>  * Canonical Livepatch is available for installation.
>    - Reduce system reboots and improve kernel security. Activate at:
>      https://ubuntu.com/livepatch
> 
> 187 packages can be updated.
> 135 updates are security updates.
> 
> New release '20.04.2 LTS' available.
> Run 'do-release-upgrade' to upgrade to it.
> 
> 
> Last login: Wed Aug 11 14:24:27 2021 from 10.1.1.254
[07:35:54.588] > echo 'ready: c81f6e315e44'
[07:35:54.613] > ubuntu@ip-10-1-1-4:~$ echo 'ready: c81f6e315e44'
> ready: c81f6e315e44
> ubuntu@ip-10-1-1-4:~$ 
[07:35:54.638] > uname -rsv
> Linux 5.3.0-1017-aws #18~18.04.1-Ubuntu SMP Wed Apr 8 15:12:16 UTC 2020
> ubuntu@ip-10-1-1-4:~$ 
[07:35:54.638] Platform: linux
[07:35:54.690] > bash
> ubuntu@ip-10-1-1-4:~$ 
[07:35:54.691] > function mainScriptWrapper {
> >  
> > echo "c81f6e315e44: running"
> > 
> v> #
> > # Setup
> sion"
>   f> #
> > VSC_TMP="${XDG_RUNTIME_DIR:-`echo /tmp`}"
> > COMMIT_ID="379476f0e13988d90fab105c5c19e7abc8b1dea8"
> > EXTENSIONS=""
> > TELEMETRY=""
> > export VSCODE_AGENT_FOLDER=$HOME/.vscode-server
> > 
[07:35:54.694] > ALLOW_CLIENT_DOWNLOAD="1"
> > FORCE_CLIENT_DOWNLOAD="0"
> > now=`date +%s`
> > SERVER_LISTEN_FLAG=--port=0
> > UNPACK_RESULT=
> > DID_LOCAL_DOWNLOAD=0
> > DOWNLOAD_TIME=
> ed > INSTALL_TIME=
> > EXT_INSTALL_TIME=
> icen> SERVER_START_TIME=
> > CONNECTION_TOKEN=
[07:35:54.713] > > 
> > VSCH_DIR="$VSCODE_AGENT_FOLDER/bin/$COMMIT_ID"
> > VSCH_LOGFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.log"
> > VSCH_PIDFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.pid"
> > VSCH_TOKENFILE="$VSCODE_AGENT_FOLDER/.$COMMIT_ID.token"
> > 
> > fail_with_exitcode() {
> 
>       fi
> > 
[07:35:54.736] > echo "c81f6e315e44: start"
> > echo "exitCode==$1=="
> > 
[07:35:54.761] > echo_common_results
> [[> 
[07:35:54.785] > echo "c81f6e315e44: end"
> > 
[07:35:54.786] Received install output: "
> echo "exitCode==$1=="
> echo_common_results
[[> echo "
[07:35:54.788] Failed to parse remote port from server output
[07:35:54.789] Terminating local server
[07:35:54.791] Resolver error: Error: 
    at Function.Create (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:64597)
    at Object.t.handleInstallOutput (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:63240)
    at Object.e [as tryInstallWithLocalServer] (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:387511)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:294411
    at async Object.t.withShowDetailsEvent (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:406401)
    at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:386050
    at async E (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:382648)
    at async Object.t.resolveWithLocalServer (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:385666)
    at async Object.t.resolve (/Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:1:295808)
    at async /Users/josh/.vscode/extensions/ms-vscode-remote.remote-ssh-0.65.7/out/extension.js:127:110656
[07:35:54.798] ------

The final thing I attempted was setting "remote.SSH.remoteServerListenOnSocket": true in the VScode Remote SSH settings. I also already had "AllowStreamLocalForwarding yes" in the sshd_config.

[07:40:37.993] Log Level: 2
[07:40:37.994] remote-ssh@0.65.7
[07:40:37.995] darwin arm64
[07:40:37.995] SSH Resolver called for "ssh-remote+ubuntu", attempt 1
[07:40:37.995] "remote.SSH.useLocalServer": false
[07:40:37.995] "remote.SSH.showLoginTerminal": false
[07:40:37.996] "remote.SSH.remotePlatform": {"10.10.20.201":"macOS","100.127.23.80":"linux","4b7cb23c-b5a6-4459-b9e7-940a9c7fe875.access.udf.f5.com":"linux","ds918":"linux"}
[07:40:37.996] "remote.SSH.path": undefined
[07:40:37.996] "remote.SSH.configFile": undefined
[07:40:37.996] "remote.SSH.useFlock": true
[07:40:37.996] "remote.SSH.lockfilesInTmp": false
[07:40:37.996] "remote.SSH.localServerDownload": auto
[07:40:37.996] "remote.SSH.remoteServerListenOnSocket": true
[07:40:37.996] "remote.SSH.showLoginTerminal": false
[07:40:37.996] "remote.SSH.defaultExtensions": []
[07:40:37.996] "remote.SSH.loglevel": 2
[07:40:37.996] SSH Resolver called for host: ubuntu
[07:40:37.996] Setting up SSH remote "ubuntu"
[07:40:38.005] Using commit id "379476f0e13988d90fab105c5c19e7abc8b1dea8" and quality "stable" for server
[07:40:38.007] Install and start server if needed
[07:40:39.167] PATH: /opt/homebrew/bin:/opt/homebrew/sbin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/Wireshark.app/Contents/MacOS
[07:40:39.167] Checking ssh with "ssh -V"
[07:40:39.171] > OpenSSH_8.1p1, LibreSSL 2.7.3

[07:40:39.173] Running script with connection command: ssh -T -D 52660 -o ConnectTimeout=15 ubuntu bash
[07:40:40.134] > 786dc09e271f: running
> Acquiring lock on /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/vscode-remote-lock.ubuntu.379476f0e13988d90fab105c5c19e7abc8b1dea8
> Installing to /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8...
> 786dc09e271f%%1%%
> Downloading with wget
[07:40:40.135] Got some output, clearing connection timeout
[07:40:42.399] > Download complete
> 786dc09e271f%%2%%
> tar --version:
> tar (GNU tar) 1.29
> Copyright (C) 2015 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later .
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.
> 
> Written by John Gilmore and Jay Fenlason.
[07:40:43.749] > Checking /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log and /home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.pid for a running server
> WARNING: VS Code Server is running but its logfile is missing. Don't delete the VS Code Server manually, run the command 'Uninstall VS Code Server'.
> Starting server with command... /home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8/server.sh --start-server --host=127.0.0.1 --enable-remote-auto-shutdown  --socket-path=/run/user/1000/vscode-ssh-remote-server-sock-1628692840  &> "/home/ubuntu/.vscode-server/.379476f0e13988d90fab105c5c19e7abc8b1dea8.log" < /dev/null
> printenv:
>     SSH_CONNECTION=10.1.1.254 10190 10.1.1.4 22
>     LANG=C.UTF-8
>     XDG_SESSION_ID=76
>     USER=ubuntu
>     PWD=/home/ubuntu
>     HOME=/home/ubuntu
>     SSH_CLIENT=10.1.1.254 10190 22
>     MAIL=/var/mail/ubuntu
>     SHELL=/bin/bash
>     SHLVL=2
>     VSCODE_AGENT_FOLDER=/home/ubuntu/.vscode-server
>     LOGNAME=ubuntu
>     XDG_RUNTIME_DIR=/run/user/1000
>     PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
>     _=/usr/bin/printenv
>     OLDPWD=/home/ubuntu/.vscode-server/bin/379476f0e13988d90fab105c5c19e7abc8b1dea8
> Spawned remote server: 4972
[07:40:43.773] > Waiting for server log...
[07:40:43.798] > Waiting for server log...
[07:40:43.860] >  
> *
> * Reminder: You may only use this software with Visual Studio family products,
> * as described in the license (https://go.microsoft.com/fwlink/?linkid=2077057)
> *
>  
> Checking server status on socket /run/user/1000/vscode-ssh-remote-server-sock-1628692840 with curl
[07:40:44.016] > 786dc09e271f: start
> SSH_AUTH_SOCK====
> DISPLAY====
> webUiAccessToken====
> listeningOn==/run/user/1000/vscode-ssh-remote-server-sock-1628692840==
> osReleaseId==ubuntu==
> arch==x86_64==
> tmpDir==/run/user/1000==
> platform==linux==
> unpackResult==success==
> didLocalDownload==0==
> downloadTime==2266==
> installTime==1329==
> extInstallTime====
> serverStartTime==84==
> connectionToken==111a1111-a111-1a11-1111-11111aaa111a==
> 786dc09e271f: end
[07:40:44.016] Received install output: 
SSH_AUTH_SOCK====
DISPLAY====
webUiAccessToken====
listeningOn==/run/user/1000/vscode-ssh-remote-server-sock-1628692840==
osReleaseId==ubuntu==
arch==x86_64==
tmpDir==/run/user/1000==
platform==linux==
unpackResult==success==
didLocalDownload==0==
downloadTime==2266==
installTime==1329==
extInstallTime====
serverStartTime==84==
connectionToken==111a1111-a111-1a11-1111-11111aaa111a==

[07:40:44.018] Remote server is listening on /run/user/1000/vscode-ssh-remote-server-sock-1628692840
[07:40:44.018] Parsed server configuration: {"serverConfiguration":{"remoteListeningOn":{"socketPath":"/run/user/1000/vscode-ssh-remote-server-sock-1628692840"},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/1000","platform":"linux","connectionToken":"111a1111-a111-1a11-1111-11111aaa111a"},"downloadTime":2266,"installTime":1329,"serverStartTime":84,"installUnpackCode":"success"}
[07:40:44.022] Running script with connection command: ssh -T -L 127.0.0.1:52664:/run/user/1000/vscode-ssh-remote-server-sock-1628692840 -o ConnectTimeout=15 ubuntu bash
[07:40:44.022] Spawning tunnel with: ssh -T -L 127.0.0.1:52664:/run/user/1000/vscode-ssh-remote-server-sock-1628692840 -o ConnectTimeout=15 ubuntu bash << 'EOSSH'
echo SSH_AUTH_SOCK=$SSH_AUTH_SOCK && echo -e 'Connected to SSH Host - Please do not close this terminal' && while true; do sleep 180; echo -n ' '; done
EOSSH
[07:40:44.925] > SSH_AUTH_SOCK=
> Connected to SSH Host - Please do not close this terminal
[07:40:44.925] Spawned SSH tunnel between local port 52664 and remote target /run/user/1000/vscode-ssh-remote-server-sock-1628692840
[07:40:44.926] Waiting for ssh tunnel to be ready
[07:40:44.928] Tunneled /run/user/1000/vscode-ssh-remote-server-sock-1628692840 to local port 52664
[07:40:44.928] Resolved "ssh-remote+ubuntu" to "127.0.0.1:52664"
[07:40:44.935] ------

I didn't see any errors in the above and the status bar in VScode says I'm connected. But I can't open a terminal window nor open any folders on the remote host. I got these toast notifications:

Could not fetch remote environment 

Failed to connect to the remote extension host server (Error: WebSocket close with status code 1006)

Anything else I haven't tried that I should? Is there a way I can get more verbose logs here in case this info isn't enough to determine what the problem is?

VSCode Remote-SSH extension through SSH gateway?

0 Answers