Interactive GPU session through VScode

Author

Romain Ligneul

Warning

The solution implemented by this tutorial does not allow launching SLURM jobs from within the VS code session on the CRNL cluster. But it will allow you to use the GPUs interactively. Drop me at message at romain.ligneul@inserm.fr if you really need to launch slurm jobs from VS code (instead of launching them from Jupyter hub). A solution exists but it should be used mindfully.

VScode is a very potent IDE that is used by many people, including at the CRNL.

However, the straightforward usage of VScode on the cluster poses a big collective problem. Indeed, when you open a SSH session with VScode, you arrive in the so-called “login node” (currently node14 on the new server, and node12 on the old one). This node is not supposed to perform any computation, but instead to organise input-output, ssh connections, etc.

Even if they do most of their computations using jobs submitted to SLURM, VScode users (I am amongst them) can still make (made!) the entire infrastructure crash by overloading the “login node”. This is because every VScode user will open some notebook, load some data, call slurm from that same login node. So, when 10 or 20 of them are connected at the same time…

💥

Fortunately, a simple solution exists and this tutorial will guide you through it.

1. Connect to the cluster through SSH and add a SSH key (only once)

Follow the instructions of the dedicated tutorial to avoid having to constantly enter your CRNL password when using VS code.

2. Install the remote tunnel extension (only once)

On your local computer (either at home or in the lab), install the Remote - Tunnel extension of VS code.

3. Download VS code on the cluster (only once)

Using SSH, log into the terminal of the cluster with the following:

ssh cluster_username@CRNLcluster

Note

Adjust “cluster_username” to match your credentials. If you did not complete the SSH tutorial, replace CRNLcluster by: 10.69.168.93

Then, run the following lines:

wget --user-agent=Mozilla --content-disposition -E -c "https://code.visualstudio.com/sha/download?build=stable&os=linux-x64" -O vscode.tar.gz
tar -xzf vscode.tar.gz

VScode is now available in our home directory.

4. Create a tunnel to VS code on the cluster from your local computer (only once)

Obviously, in the example below, replace romain.ligneul by own your user name (CRNLcluster should match your own SSH config hostname). You may also adjust the resources requested by srun based on your needs.

Windows users On your local computer, create a file named vscode_tunnel_dividedGPU.bat in a convenient place and copy paste the following lines:

@echo off
set "remote_username=romain.ligneul"
set "remote_hostname=CRNLcluster"
ssh -t %remote_username%@%remote_hostname% "srun --partition=GPU --mem=20G --cpus-per-task=4 --gres=gpu:1g.20gb:1 --time=01:00:00 --pty $SHELL -c '~/VSCode-linux-x64/bin/code tunnel --cli-data-dir=dividedGPU --name=dividedGPU'; bash"

Note, if you use the Notepad to create the file, make your to select the option ‘All files’ in “Save as type”. Otherwise, you won’t be able to have a .bat extension (it would be .bat.txt, which does not work!)

If you need access to the full GPU capacity, you may create a second tunnel .bat file named vscode_tunnel_fullGPU.bat:

@echo off
set "remote_username=romain.ligneul"
set "remote_hostname=CRNLcluster"
ssh -t %remote_username%@%remote_hostname% "srun --partition=GPU --cpus-per-task=8 --mem=40G --gres=gpu:a100:1 --time=01:00:00 --pty $SHELL -c '~/VSCode-linux-x64/bin/code tunnel --cli-data-dir=fullGPU --name=fullGPU'; bash"

Linux users On your local computer, create a file named vscode_cluster_tunnel.sh in a convenient place and copy-paste the following

#!/bin/bash
remote_username="romain.ligneul"
remote_hostname="CRNLcluster"
ssh -t "${remote_username}@${remote_hostname}" "srun --partition=GPU --cpus-per-task=4 --mem=20G --gres=gpu:1g.20gb:1 --time=01:00:00 --pty \$SHELL -c '~/VSCode-linux-x64/bin/code tunnel'; bash"

If you need access to the full GPU capacity, you may create a second tunnel .bat file named vscode_tunnel_fullGPU.sh:

#!/bin/bash
remote_username="romain.ligneul"
remote_hostname="CRNLcluster"
ssh -t "${remote_username}@${remote_hostname}" "srun --partition=GPU --cpus-per-task=8 --mem=40G --gres=gpu:a100:1 --time=01:00:00 --pty \$SHELL -c '~/VSCode-linux-x64/bin/code tunnel'; bash"

Note

Now, each time you will need to open a VS GPU session on the server, you will need to run this .bat or .sh file.

The first time you will run it, you will be asked to authorize Github and to name tunnel as you prefer (for example: vscode_cluster_tunnel). This tunnel name will be reused every time you will launch a new VS code session.

Consider using Github as a tunnel provider rather than Microsoft.

5. Connect to the tunnel from your local VS code (each time)

Run the script vscode_cluster_tunnel.sh or vscode_cluster_tunnel.bat (on Windows, you can just double click on the .bat file). From time to time, you might have to re-authorized your tunnel in Github.

When running this script, the terminal window will ask you to authorize the tunnel through Github (or Microsoft). You may also adjust mem, cpus-per-task, and time (in minutes) based on your needs.

Then, in VS code, either press Ctrl+Shift+P and type “tunnel connect” to select the function “Remote Tunnels: Connect to Tunnel”, or click on the bottom left icon that look like this:

And click “Connect to Tunnel”

Then, select Github and the tunnel name that you have defined just before.

That’s it!

You are now running VS code on a compute node with dedicated ressource. You can mess around, it won’t interfere with the happiness of other users!

Tip

When you stop using VS code, quit or kill the process initiated by vscode_cluster_tunnel (on Windows, just close the corresponding Terminal window) It will automatically interrupt the SLURM job that runs your VS code session, and free resources for your colleagues!

Leave a message

Feel free to comment, make a question, report a bug, etc. If you want to notify a specific user (e.g. to get a faster response), you may include their Github handle in your message (e.g. @romainligneul or @samuelgarcia).

--- title: Interactive GPU session through VScode author: Romain Ligneul toc: true execute: echo: true freeze: false eval: false --- :::{.callout-warning} The solution implemented by this tutorial does not allow launching SLURM jobs from within the VS code session on the CRNL cluster. But it will allow you to use the GPUs interactively. Drop me at message at [romain.ligneul@inserm.fr](romain.ligneul@inserm.fr) if you really need to launch slurm jobs from VS code (instead of launching them from Jupyter hub). A solution exists but it should be used mindfully. ::: [VScode](https://code.visualstudio.com/) is a very potent IDE that is used by many people, including at the CRNL. However, the straightforward usage of VScode on the cluster poses a **big collective problem**. Indeed, when you open a SSH session with VScode, you arrive in the so-called "login node" (currently node14 on the new server, and node12 on the old one). This node is not supposed to perform any computation, but instead to organise input-output, ssh connections, etc. Even if they do most of their computations using jobs submitted to SLURM, VScode users (I am amongst them) can still make (made!) the entire infrastructure crash by overloading the "login node". This is because every VScode user will open some notebook, load some data, call slurm from that same login node. So, when 10 or 20 of them are connected at the same time... 💥 Fortunately, a simple solution exists and this tutorial will guide you through it. ### 1. Connect to the cluster through SSH and add a SSH key (only once) Follow the instructions of the dedicated [tutorial](./help_ssh){target="_blank"} to avoid having to constantly enter your CRNL password when using VS code. ### 2. Install the remote tunnel extension (only once) On your local computer (either at home or in the lab), install the [Remote - Tunnel](https://marketplace.visualstudio.com/items?itemName=ms-vscode.remote-server) extension of VS code. ### 3. Download VS code on the cluster (only once) Using SSH, log into the terminal of the cluster with the following: `ssh cluster_username@CRNLcluster` :::{.callout-note} Adjust "cluster_username" to match your credentials. If you did not complete the SSH [tutorial](./help_ssh){target="_blank"}, replace CRNLcluster by: 10.69.168.93 ::: Then, run the following lines: ```{bash} wget --user-agent=Mozilla --content-disposition -E -c "https://code.visualstudio.com/sha/download?build=stable&os=linux-x64" -O vscode.tar.gz tar -xzf vscode.tar.gz ``` VScode is now available in our home directory. ### 4. Create a tunnel to VS code on the cluster from your local computer (only once) Obviously, in the example below, replace romain.ligneul by own your user name (CRNLcluster should match your own SSH config hostname). You may also adjust the resources requested by `srun` based on your needs. **Windows users** On your local computer, create a file named vscode_tunnel_dividedGPU.bat in a convenient place and copy paste the following lines: ``` @echo off set "remote_username=romain.ligneul" set "remote_hostname=CRNLcluster" ssh -t %remote_username%@%remote_hostname% "srun --partition=GPU --mem=20G --cpus-per-task=4 --gres=gpu:1g.20gb:1 --time=01:00:00 --pty $SHELL -c '~/VSCode-linux-x64/bin/code tunnel --cli-data-dir=dividedGPU --name=dividedGPU'; bash" ``` Note, if you use the Notepad to create the file, make your to select the option 'All files' in "Save as type". Otherwise, you won't be able to have a .bat extension (it would be .bat.txt, which does not work!) If you need access to the full GPU capacity, you may create a second tunnel .bat file named vscode_tunnel_fullGPU.bat: ``` @echo off set "remote_username=romain.ligneul" set "remote_hostname=CRNLcluster" ssh -t %remote_username%@%remote_hostname% "srun --partition=GPU --cpus-per-task=8 --mem=40G --gres=gpu:a100:1 --time=01:00:00 --pty $SHELL -c '~/VSCode-linux-x64/bin/code tunnel --cli-data-dir=fullGPU --name=fullGPU'; bash" ``` **Linux users** On your local computer, create a file named vscode_cluster_tunnel.sh in a convenient place and copy-paste the following ``` #!/bin/bash remote_username="romain.ligneul" remote_hostname="CRNLcluster" ssh -t "${remote_username}@${remote_hostname}" "srun --partition=GPU --cpus-per-task=4 --mem=20G --gres=gpu:1g.20gb:1 --time=01:00:00 --pty \$SHELL -c '~/VSCode-linux-x64/bin/code tunnel'; bash" ``` If you need access to the full GPU capacity, you may create a second tunnel .bat file named vscode_tunnel_fullGPU.sh: ``` #!/bin/bash remote_username="romain.ligneul" remote_hostname="CRNLcluster" ssh -t "${remote_username}@${remote_hostname}" "srun --partition=GPU --cpus-per-task=8 --mem=40G --gres=gpu:a100:1 --time=01:00:00 --pty \$SHELL -c '~/VSCode-linux-x64/bin/code tunnel'; bash" ``` ::: callout-note Now, each time you will need to open a VS GPU session on the server, you will need to run this .bat or .sh file. The first time you will run it, you will be asked to authorize Github and to name tunnel as you prefer (for example: vscode_cluster_tunnel). This tunnel name will be reused every time you will launch a new VS code session. Consider using Github as a tunnel provider rather than Microsoft. ::: ### 5. Connect to the tunnel from your local VS code (each time) Run the script vscode_cluster_tunnel.sh or vscode_cluster_tunnel.bat (on Windows, you can just double click on the .bat file). From time to time, you might have to re-authorized your tunnel in Github. When running this script, the terminal window will ask you to authorize the tunnel through Github (or Microsoft). You may also adjust mem, cpus-per-task, and time (in minutes) based on your needs. Then, in VS code, either press Ctrl+Shift+P and type "tunnel connect" to select the function "Remote Tunnels: Connect to Tunnel", or click on the bottom left icon that look like this: ![](images/paste-1.png) And click "Connect to Tunnel" Then, select Github and the tunnel name that you have defined just before. That's it! You are now running VS code on a compute node with dedicated ressource. You can mess around, it won't interfere with the happiness of other users! ::: callout-tip When you stop using VS code, **quit or kill the process initiated by vscode_cluster_tunnel** (on Windows, just close the corresponding Terminal window) It will automatically interrupt the SLURM job that runs your VS code session, and free resources for your colleagues! ::: {{< include _comment_box.qmd >}}