Networking

Networking#

This section covers some basic concepts and tools for connecting to machines over a network, as is commonly needed when working with HPC systems and web servers at ICHEC.

SSH#

The Secure Shell (SSH) protocol allows cryptographically secure communication over a network, typically used for login and terminal sessions. A server running an SSH implementation (often OpenSSH) will allow connections from am SSH client with suitable credentials. There are several ways to provide credentials, however public-private key pairs are now most typical (as sometimes the only method allowed by the server). Another common use for SSH clients is to set up SSH tunnels (also called port forwarding) whereby connections to a local network port get forwarded over the secure SSH connection to a different network port on the remote side of the connection - this is often used to access web servers on remote systems which are not directly accessible due to firewalls along the route.

Once a connection is established it can be used to open a terminal on the remote machine and view it on your local machine, or to run commands or view and modify remote files.

A bare-mininum ssh connection involes the ssh command and the address of the remote machine:

ssh remote_machine.remote_site.com

If the user of the local shell session is not found on the remote system then a user name will be prompted for, which can be avoided with:

ssh my_user@remote_machine.remote_site.com

Traditionally the server would prompt for a password and if correct would then launch a shell session. Most modern configurations instead opt for a ssh keypair for authentication. To create a keypair:

ssh-keygen -t ed25519

The user will be prompted to enter a passphrase which will be used to encrypt the private key. Do not leave this blank as this will enable anyone who can read this key to use it to login in the same way as a writing a password in a file would. ssh-keygen also prompts for the location and filename for the public and private key - if this is the first and only keypair then the default is usually fine ($HOME/.ssh/id_ed25519 and $HOME/.ssh/id_ed25519.pub) but if creating an additional keypair for a specific use (recommended in certain circumstances) then enter a unique filename to avoid overwriting the main, default keypair.

In order to successfully login to a remote server using ssh keys, the remote server must have a copy of your public ssh key. This can be done by the administrator of that system placing a copy of the key ($HOME/.ssh/id_ed25519.pub) into the default file on the remote server $HOME/.ssh/authorized_keys or in a central directory which that server is configured to lookup.

There are many other configurations and settings that can be supplied to set up the ssh session of the command line, however it can become tedious to remember and type them each time. For this reason use of an ‘ssh config’ file is typical.

SSH Config#

An ‘ssh config’ file, usually stored in ~/.ssh/config allows storing commonly used machines and settings so that lengthy login commands in the terminal can be avoided. Below is an example of a config file:

# We set a default user name 'my_user_name' and easy to remember machine name 'my_machine0'
Host my_machine0
  HostName my_machine0.my_site.com
  User my_user_name

# We forward the user agent - which allows use of certs on this machine in 'jump connections'
Host my_machine1
  HostName my_machine1.my_site.com
  ForwardAgent yes
  User my_user_name
  IdentityFile ~/.ssh/id_ed25519

# We connect to this machine using an ssh 'jump' connection, through another machine
# We also use a non-standard location of the private key 'id_ecdsa' and a non default port 8888
Host my_machine2
  HostName my_machine2.my_site.com
  ProxyJump my_machine1.my_site.com
  User my_user_name
  Port 8888
  IdentityFile ~/.ssh/id_ecdsa

Here we give endpoints like my_machine0.my_site.com an easy to remember name my_machine0, so we can just do ssh my_machine0 to log in.

For my_machine1 we want to be able to use it to log into my_machine2, we use ‘ForwardAgent yes’ to forward the credentials we are passing my_machine1 onto my_machine2. This avoids having to put our SSH keys on my_machine2.

For my_machine2 we specify a ProxyJump via my_machine1, so now we can just do ssh my_machine2.

Notice for my_machine2 it wants a different user name and non-standard port for connection. It is also using a different SSH key.

This config file extends to other ssh related tools, like scp, so you can copy files directly to my_machine2 without the explicit -J jump through calliope.

There are many other useful settings for the config (e.g. see below for using the system keychain to avoid having to continuosly give a password). If you find others useful you can add them here.

SSH Agent and Keychain#

To avoid needing to enter a password for your SSH key each time you can add it to the ssh-agent process:

ssh-add ~/.ssh/id_ed25519

or on Mac

ssh-add --apple-use-keychain ~/.ssh/id_ed25519

For the credentials to persist on restart you can add this option to ~/.ssh/config:

Host *
  UseKeychain yes
  AddKeysToAgent yes

You can narrow down the wildcard ‘*’ if you only want it for certain hosts.

File Transfer#

scp#

The Secure cp tool should be available on most systems.

You can use it to copy files to and from a remote. To copy to a remote:

scp $LOCAL_FILE_PATH $REMOTE_ADDR:$REMOTE_FILE_PATH

e.g.

scp my_file.dat my_machine2.my_site.com:~/my_file_on_my_machine2.dat

To copy from a remote:

scp $REMOTE_ADDR:$REMOTE_FILE_PATH $LOCAL_FILE_PATH 

e.g.

scp my_machine2.my_site.com:~/my_file_on_my_machine2.dat my_file.dat

rsync#

Rsync is a tool for performing incremental copies of files and directories and can use ssh as a connection mechanism to do this between servers. Rsync is more feature rich than cp/scp but can be slower in some scenarios as it makes an index of the transfer before performing it. However this enables it to only copy new files that haven’t already been copied over if you run another rsync later on.

You can copy directories while excluding some with:

rsync -av -e ssh --exclude dir_relative_to_src_dir src_dir dst_dir

Port Forwarding#

SSH port forwarding (or tunnelling) can be thought of as a very simple VPN in that network connections to specific local ports can be passed through a SSH connection to a remote host and port and thereby bypass the normal network routing which may involve several firewalls along the path. For example, a webserver running on port 443 of HostB iwhich is not accessible via the Internet could be accessed on a browser on HostA at https://localhost:8080 by first starting a local port forwarding ssh tunnel on HostA:

ssh -N -L8080:localhost:443 HostB.ichec.ie

Graphics Over Network#

X11 Forwarding#

To work with X11 forwarding on Mac XQuartz needs to be installed and running.

brew install xquartx
xquartz

Remote Desktop#

Remote Desktop is a Windows technology for remote graphical access to the Windows desktop. Some useful resources:

Forward Remote Desktop through linux server