SSH Access to VM-like Containers

Paweł Kozubal

Paweł Kozubal

Software Developer @ Husarnet

Docker Containers provide some level of isolation, but unfortunately, not enough if you want to give full access to someone who you do not trust.

🔐 In this blog post we will show you a much safer alternative: Kata Containers that provide VM-like isolation for your container, while keeping the simplicity of Docker ecosystem.

💡 As an example use-case we will configure a VPN client and SSH server inside Kata Container, that can be securely exposed to untrusted users of your service. Doing the same with pure Docker would be very risky.

Thanks to Husarnet P2P VPN used in this example you can provide a secure and quick access to isolated containers in an easy way.

Kata vs runc#

Kata Containers vs Docker

Docker is a very extensible tool. One of it's building blocks is called "runtime". Runtime is responsible for actually running the containers. The default runtime is called runC and it uses Linux namespaces and cgroups in order to isolate the container from the rest of the system. This is a relatively thin layer of separation implemented in the Linux kernel that tries to maximize the performance, minimize the startup times while providing as much security as possible. This is fine for most of the cases but it shouldn't be considered secure when running untrusted apps. An alternative designed for such cases is for example Kata Containers presented in this article. The main difference here is that each of the containers is being run in a separate virtual machine (so called MicroVM). The main benefit of this is a much greater separation between your container and the host - thus increased security. What's more - Kata Containers is designed to be a drop-in replacement for runc so using it is super easy - which we will show in this blog post.

What we're going to do?#

In this blog post we're going to:

  • install Docker (you can skip this step if you already have Docker installed)
  • install Kata runtime, make it able to run VPN inside and let your Docker know about alternative runtime
  • clone and configure the example so you actually use it
  • start the containwe using two different methods so you can use your favourite for production use
info

Tested on a clear installation of the following host system:

Operating System: Ubuntu 20.04.1 LTS
Kernel: Linux 5.4.0-62-generic
Architecture: x86-64

Install Docker#

If you have Docker already installed - you can skip this step.

The official instruction is the best tutorial but here's a quick rundown for you:

sudo -E apt-get -y install apt-transport-https ca-certificates software-properties-common && \
curl -sL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
arch=$(dpkg --print-architecture) && \
sudo -E add-apt-repository "deb [arch=${arch}] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" && \
sudo -E apt-get update && \
sudo -E apt-get -y install docker-ce docker-compose
sudo systemctl daemon-reload
sudo systemctl restart docker

Get and prepare the Kata containers runtime#

Even if you have Kata Containers already installed do NOT skip this step. Kata's kernel needs some extra kernel options in order to be able to run Husarnet VPN Client.

As this step is a little bit cumbersome we've provided a script that'll do all the necessary configuration steps for you.

Begin with cloning the GitHub repository.

git clone https://github.com/husarnet/kata-example.git

If you do not have git installed, add it with:

sudo apt-get install -y git

Then - run our script. It won't ask you for any additional info (except maybe for sudo password ;-P) and it won't override any of your existing Docker defaults. Kata runtime will be installed and added as a new runtime to Docker. Kernel in Kata Containers will be rebuilt (this step may take a couple of minutes!) and installed as it's default.

sudo ./prepare-host.sh

…and voilà! Your host is now ready to run Husarnet Client in Kata Container.

Install Husarnet Client on the host#

It's not necessary for production usage, but in order to test this example, you'll need an access to a host with Husarnet Client installed and running. The easiest way would be to install it on the same host you're running Docker on.

Use our getting started on Linux tutorial in order to do that.

Start the example#

Using docker-compose#

Create .env file#

…and specify Husarnet JoinCode and hostname there. The file should look something like this:

JOINCODE=fc94:b01d:1803:8dd8:3333:2222:1234:1111/xxxxxxxxxxxxxxxxx
HOSTNAME=my-container-1

You will find your JoinCode at https://app.husarnet.com
-> Click on the desired network
-> Add element button
-> Join code tab

Build an image and start a container#

sudo docker-compose up

Skip the docker run method and go to the result paragraph.

Using docker run#

Build an image#

Make sure init-container.sh is executable. If not:

sudo chmod +x init-container.sh

Then build an image:

sudo docker build -t kata-example .

Get your JoinCode#

You will find your JoinCode at https://app.husarnet.com
-> Click on the desired network
-> Add element button
-> Join code tab

…and change the JOINCODE variable in the listing below to your JoinCode.

Start a container#

Execute in a Linux terminal:

sudo docker run --rm -it \
--env HOSTNAME='my-container-1' \
--env JOINCODE='fc94:b01d:1803:8dd8:3333:2222:1234:1111/xxxxxxxxxxxxxxxxx' \
--volume husarnet-ssh-v:/var/lib/husarnet \
--volume husarnet-ssh-keys:/srv/sshd \
--device /dev/net/tun \
--cap-add NET_ADMIN \
--sysctl net.ipv6.conf.all.disable_ipv6=0 \
--runtime kata-runtime \
kata-example
tip

Note that we provide --runtime kata-runtime while starting a container, not before.

If you remove this line, the container will run based on the default runtime for Docker - runc. So basically you use Kata almost the same way, as standard Docker. You can even run both types of containers on the same host.

That's a great featue: you can use kata-runtime only for containers needing an extra level of isolation, that however means also utilizing a little bit more resources of your host system. A "trusted containers" that don't need deep isolation can run the old way, being a little bit faster than the same container running on Kata runtime.

Result#

After running a container you should see a log like this:

⏳ [1/2] Initializing Husarnet Client:
waiting...
waiting...
waiting...
waiting...
success
🔥 [2/2] Connecting to Husarnet network as "my-container-1":
[693849] joining...
[695852] joining...
done
*******************************************
💡 Tip
To SSH your container execute in a new terminal session:
ssh johny@fc94:2106:8423:68d7:695e:f6cf:8130:c059
(default password is "johny" as well)
*******************************************

The container is up and running. This example uses SSH as the "main application" but you can change it to whatever you want.

You can now access it from other machines in the same Husarnet network using standard SSH client:

  • Husarnet IPv6 Address
ssh johny@fc94:2106:8423:68d7:695e:f6cf:8130:c059
  • Hostname provided as --env in docker run or in .env file
ssh johny@my-container-1

Remarks#

If you like this example and want to run this in production, here are some tips that you may find helpful.

tip

prepare-host.sh script - performs a pretty standard installation of Kata Containers, the only tricky part is where it adds extra modules to the kernel. Look for enable_option function usages in that file to get the current list of required modules.

tip

init-container.sh script - this file shows:

  • which processes have to be running in the background (husarnet daemon)
  • how to delay starting your application until the Husarnet Client is ready
  • how to get the IPv6/Husarnet address of your container
tip

docker-compose.yml file

The most important part here is runtime: kata-runtime. If you've manually installed kata as the default runtime - it's not necessary, but we'd still advise to be explicit here.

In order to preserve the IPv6/Husarnet address between runs of the container you need to persistently store /var/lib/husarnet hence the volume.

/etc/sshd volume is just a sugar to albo persist SSH keys in case you'd really like to run SSH in the container.

The way you specify environment variables is completely up to you. Just remember, in case you store your confings in some kind of version control system, that JoinCode is a sensitive data and should be treated as a secret - it gives the access to the whole specific network.

sysctls, cap NET_ADMIN and access to /dev/net/tun are all required in most of the cases.

tip

Dockerfile

Pretty much everything here is up to you, as long as you:

  • install Husarnet Client
  • make ip6tables use ip6tables-legacy as it's what we're configuring the kernel for in the script above. If your OS in the container needs a different ip6tables backend - please configure the kernel accordingly
  • start husarnet daemon in the background as we've mentioned above.

Summary#

In this article you've learned:

  • how to run untrusted containers in a secure way with Kata Containers
  • how to provide a secure, remote SSH access to the Kata Containers even if you are being behind NAT or firewall

We hope you will find this article helpful.

We would be happy to discuss the topic covered in this blog post on Husarnet Community Forum.