![You are currently viewing How to run Firecracker microVMs with vmsan on Incus](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="765" height="447"><rect fill-opacity="0"/></svg>)
Incus is a manager for virtual machines and system containers.
A virtual machine (VM) is an instance of an operating system that runs on a computer, along with the main operating system. A virtual machine uses hardware virtualization features for the separation from the main operating system. With virtual machines, a complete operating system boots up in them.
Incus, to run VMs, uses KVM as a Type 1 hypervisor, and QEMU as a Type 2 hypervisor and emulator.
On the other hand, vmsan only uses KVM as a Type 1 hypervisor. This means that it has far less features, but it is very lightweight.
In this post we have a look at vmsan by running it in an Incus virtual machine.
Table of Contents
- Prerequisites
- Setting up vmsan
- Using vmsan
- vmsan connect to get a shell into a microVM
- Running nginx in a microVM
- Conclusion
Prerequisites
You need to
- have Incus installed and initialized
- have configured Incus to run virtual machines
- have configured your system for nested virtualization
For the last two points, check out this post,
Setting up vmsan
We launch an Incus VM and in there we install vmsan. By default, the Incus VM will have 1GiB RAM and 10GiB disk space. When you run incus shell, the instance should have finished the start-up. Otherwise, you get an error as shown below. Just try again and until you get the command succeeds.
$ incus launch --vm images:ubuntu/24.04/cloud microvm
$ incus shell microvm
Error while executing alias expansion: incus exec microvm -- su -l
Error: VM agent isn't currently running // it just means the VM did not finish yet booting.
$ incus shell microvm
root@microvm:~# Then, we follow the instructions at https://vmsan.dev/getting-started/installation to install vmsan. First we install curl because the installation script is a curl pipe script. And we also install iptables because currently it’s not installed automatically by the installation script. This may change in the future. Note that unzip and squashfs-tools are both installed by the installation script, as shown below. We are also prompted to setup an optional Cloudflare tunnel which we do not do in this tutorial for simplicity.
root@microvm:~# apt install -y curl iptables
...
root@microvm:~# curl -fsSL https://vmsan.dev/install | bash
vmsan installer
Firecracker microVM sandbox toolkit
https://github.com/angelorc/vmsan
[info] Installing prerequisites: unzip squashfs-tools...
[ok] Prerequisites installed
[info] Node.js not found — installing Node.js 22 via NodeSource...
...
2026-03-04 18:15:18 - Repository configured successfully.
2026-03-04 18:15:18 - To install Node.js, run: apt install nodejs -y
2026-03-04 18:15:18 - You can use N|solid Runtime as a node.js alternative
2026-03-04 18:15:18 - To install N|solid Runtime, run: apt install nsolid -y
[ok] Node.js v22.22.0 installed
[info] Setting up /root/.vmsan...
[ok] Directories created
[info] Fetching latest Firecracker version...
[info] Downloading firecracker.tgz...
[ok] Firecracker v1.14.2 installed
[info] Downloading vmlinux-6.1...
[ok] Kernel installed (vmlinux-6.1)
[info] Downloading ubuntu-24.04.squashfs...
[info] Converting squashfs to ext4 (this may take a minute)...
[ok] Rootfs installed (ubuntu-24.04.ext4, 1024 MB)
[info] Fetching latest release tag...
[ok] Latest release: v0.1.0-alpha.24
[info] Installing vmsan CLI via npm...
npm warn deprecated node-domexception@1.0.0: Use your platform's native DOMException instead
added 62 packages in 27s
8 packages are looking for funding
run `npm fund` for details
npm notice
npm notice New major version of npm available! 10.9.4 -> 11.11.0
npm notice Changelog: https://github.com/npm/cli/releases/tag/v11.11.0
npm notice To update run: npm install -g npm@11.11.0
npm notice
[ok] vmsan CLI installed (0.1.0-alpha.24)
[info] Downloading vmsan-agent...
[ok] vmsan-agent v0.1.0-alpha.24 installed
[info] Downloading cloudflared...
[ok] cloudflared 2026.2.0 installed
┌─────────────────────────────────────────────────────────────┐
│ Cloudflare Tunnel (optional) │
│ │
│ vmsan can expose VMs via Cloudflare Tunnels. │
│ You need a Cloudflare API token and a domain managed │
│ by Cloudflare. │
└─────────────────────────────────────────────────────────────┘
Configure Cloudflare now? [y/N] N
[info] Skipping Cloudflare configuration
[ok] vmsan environment ready at /root/.vmsan
Firecracker /root/.vmsan/bin/firecracker
Jailer /root/.vmsan/bin/jailer
Kernel /root/.vmsan/kernels/vmlinux-6.1
Rootfs /root/.vmsan/rootfs/ubuntu-24.04.ext4
Agent /root/.vmsan/bin/vmsan-agent
cloudflared /root/.vmsan/bin/cloudflared (not configured)
CLI /usr/bin/vmsan
To configure Cloudflare later, re-run this installer.
root@microvm:~# Finally, we launch a microVM. We use the parameter --connect that gives as a non-root shell into the newly created microVM. The microVM has by default 128MiB RAM and a disk space of 10GiB. The Incus VM is also at 10GiB, which may not good. The default memory size is too little to even run apt get and the default disk space is too much. Don’t worry, we will remove this microVM and create a new one in the next section. What’s important, is that we got a shell into the microVM. It works!
root@microvm:~# vmsan create --connect
◐ Creating VM vm-918c59dc...
◐ Setting up networking...
Network: TAP fhvm0, Host 172.16.0.1, Guest 172.16.0.2
◐ Preparing chroot...
◐ Spawning Firecracker via jailer...
◐ Waiting for API socket...
API socket ready
◐ Starting VM...
VM vm-918c59dc is running (PID: 2116)
╭───────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ VM Created: vm-918c59dc │
│ │
│ Status: running │
│ PID: 2116 │
│ vCPUs: 1 │
│ Memory: 128 MiB │
│ Runtime: base │
│ Disk: 10 GB │
│ │
│ Network: │
│ TAP: fhvm0 │
│ Host: 172.16.0.1 │
│ Guest: 172.16.0.2 │
│ MAC: AA:FC:00:00:00:01 │
│ Policy: allow-all │
│ │
│ Kernel: /root/.vmsan/kernels/vmlinux-6.1 │
│ Rootfs: /root/.vmsan/rootfs/ubuntu-24.04.ext4 │
│ │
│ Socket: /root/.vmsan/jailer/firecracker/vm-918c59dc/root/run/firecracker.socket │
│ Chroot: /root/.vmsan/jailer/firecracker/vm-918c59dc │
│ State: /root/.vmsan/vms/vm-918c59dc.json │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────╯
◐ Waiting for agent to become ready...
Agent is ready. Connecting via PTY shell...
ubuntu@ubuntu-fc-uvm:~$ The microVM gets an IP address from the 176.16.x.y private address range. The hostname of the microVM is ubuntu-fc-uvm, and the name is derived from Ubuntu FireCracker microVirtualMachine.
Let’s list the available microVMs and then remove ’em. In the next section we examine a bit more about these commands.
root@microvm:~# vmsan list
ID STATUS CREATED MEMORY VCPUS RUNTIME TIMEOUT TUNNEL SNAPSHOT
vm-918c59dc running 5 minutes ago 128 MiB 1 base - - -
root@microvm:~# vmsan remove vm-63eade04
ERROR Cannot remove running VM(s): vm-63eade04. Stop them first or use --force (-f).
root@microvm:~# vmsan remove vm-63eade04 --force
◐ Removing vm-63eade04...
Removed vm-63eade04
root@microvm:~# vmsan list
No VMs found.
root@microvm:~#Using vmsan
Here are the available commands for vmsan. We can create microVMs and we can list them. By listing them, we can see the name that was given to them so that we can start, stop and remove them. We can get a shell or run commands in a microVM with connect and exec. We transfer files with upload and download. Finally, with network we change the network policy.
root@microvm:~# vmsan
Firecracker microVM sandbox toolkit (vmsan v0.1.0-alpha.24)
USAGE vmsan [OPTIONS] create|list|ls|start|stop|remove|rm|connect|upload|download|exec|network
OPTIONS
--json Output structured JSON (one event per command)
--verbose Show detailed debug output with wide event tree
COMMANDS
create Create and start a Firecracker microVM
list List all VMs
ls List all VMs
start Start a previously stopped VM
stop Stop one or more running VMs
remove Remove one or more VMs (stops if running, then deletes)
rm Remove one or more VMs (stops if running, then deletes)
connect Connect to a running VM
upload Upload local files to a running VM
download Download a file from a running VM
exec Execute a command inside a running VM
network Update network policy on a running VM
Use vmsan <command> --help for more information about a command.
No command specified.
root@microvm:~# vmsan create to create a microVM
Here are the parameters. Note that if you run without any parameters (like --help), it will plough ahead create a microVM with the default parameters.
By default, a microVM has 1 vCPU, 128MiB memory (you specify here the number only), disk space of 10gb (you specify here the number with gb). If you use the --connect parameter, you get a non-root shell once the microVM is created. The default image is base, which is currently an Ubuntu 24.04.3. Once you launch a microVM, the base image uses 388MB of disk space. In addition, the microVM from the base image occupies about 34MiB of memory.
root@microvm:~# vmsan create --help
Create and start a Firecracker microVM (vmsan create v0.1.0-alpha.24)
USAGE vmsan create [OPTIONS]
OPTIONS
--vcpus="1" Number of vCPUs (default: 1)
--memory="128" Memory in MiB (default: 128)
--kernel Path to kernel image. Auto-detected from kernels/ if not specified.
--rootfs Path to rootfs image. Auto-detected from rootfs/ if not specified.
--from-image Build rootfs from a Docker/OCI image (e.g. ubuntu:latest).
--runtime="base" Runtime label (e.g. python3.13, node22, node22-demo). node22-demo auto-selects node:22 image and serves a branded welcome page. Default: base
--project Project label for grouping VMs
--disk="10gb" Root disk size in GB (default: 10gb)
--timeout Auto-shutdown timeout (e.g. 1h, 30m, 2h30m)
--publish-port Ports to forward to the VM (comma-separated, e.g. 8080,3000)
--snapshot Snapshot ID to restore from
--network-policy="allow-all" Base network mode: allow-all (default), deny-all, or custom. Auto-promoted to custom when domains or CIDRs are provided.
--allowed-domain Domains/patterns to allow (comma-separated). Wildcard * for subdomains.
--allowed-cidr Address ranges to allow (comma-separated CIDR, e.g. 10.0.0.0/8)
--denied-cidr Address ranges to deny (comma-separated CIDR). Takes precedence over all allows.
--no-seccomp Disable seccomp-bpf filter for the Firecracker process.
--no-pid-ns Disable PID namespace isolation for the jailer.
--no-cgroup Disable cgroup resource limits for CPU and memory.
--no-netns Disable per-VM network namespace isolation.
--bandwidth Max bandwidth per VM (e.g., 50mbit, 100mbit). Default: unlimited.
--connect Automatically connect to the VM shell after creation.
--silent Suppress all output
root@microvm:~# We are creating microVM with the following defaults.
root@microvm:~# vmsan create --disk="1gb" --memory="512" --connect
◐ Creating VM vm-4373c955...
◐ Setting up networking...
Network: TAP fhvm0, Host 172.16.0.1, Guest 172.16.0.2
◐ Preparing chroot...
◐ Spawning Firecracker via jailer...
◐ Waiting for API socket...
API socket ready
◐ Starting VM...
VM vm-4373c955 is running (PID: 3987)
╭───────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ VM Created: vm-4373c955 │
│ │
│ Status: running │
│ PID: 3987 │
│ vCPUs: 1 │
│ Memory: 512 MiB │
│ Runtime: base │
│ Disk: 1 GB │
│ │
│ Network: │
│ TAP: fhvm0 │
│ Host: 172.16.0.1 │
│ Guest: 172.16.0.2 │
│ MAC: AA:FC:00:00:00:01 │
│ Policy: allow-all │
│ │
│ Kernel: /root/.vmsan/kernels/vmlinux-6.1 │
│ Rootfs: /root/.vmsan/rootfs/ubuntu-24.04.ext4 │
│ │
│ Socket: /root/.vmsan/jailer/firecracker/vm-4373c955/root/run/firecracker.socket │
│ Chroot: /root/.vmsan/jailer/firecracker/vm-4373c955 │
│ State: /root/.vmsan/vms/vm-4373c955.json │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────╯
◐ Waiting for agent to become ready...
Agent is ready. Connecting via PTY shell...
ubuntu@ubuntu-fc-uvm:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=24.04
DISTRIB_CODENAME=noble
DISTRIB_DESCRIPTION="Ubuntu 24.04.3 LTS"
ubuntu@ubuntu-fc-uvm:~$ df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/root 974M 388M 570M 41% /
ubuntu@ubuntu-fc-uvm:~$ free -h
total used free shared buff/cache available
Mem: 104Mi 34Mi 28Mi 1.7Mi 52Mi 70Mi
Swap: 0B 0B 0B
ubuntu@ubuntu-fc-uvm:~$ su
root@ubuntu-fc-uvm:/home/ubuntu# cd
root@ubuntu-fc-uvm:~# pwd
/root
root@ubuntu-fc-uvm:~# exit
exit
ubuntu@ubuntu-fc-uvm:~$ exit
root@microvm:~# vmsan list to list the microVMs
We run vmsan list to list the available microVMs.
root@microvm:~# vmsan list
ID STATUS CREATED MEMORY VCPUS RUNTIME TIMEOUT TUNNEL SNAPSHOT
vm-4373c955 running 7 minutes ago 512 MiB 1 base - - -
root@microvm:~# vmsan connect to get a shell into a microVM
We run vmsan connect to get a shell into a microVM. We need the ID of the microVM, and we get it by running vmsan list. We can get root by running su. The root account does not have a password, therefore you are not asked for a password for the su command. sudo is not enabled in this image as it is not needed. I am running su --login in order to get a login shell for root.
root@microvm:~# vmsan connect vm-4373c955
ubuntu@ubuntu-fc-uvm:~$ su --login
root@ubuntu-fc-uvm:~# Running nginx in a microVM
Let’s do something useful. We are going to run nginx in a microVM and we will access it from the host. We are going to start from scratch, creating a new microVM.
root@microvm:~# vmsan create --disk="1gb" --memory="512" --publish-port 80 --connect
◐ Creating VM vm-2f796b98...
◐ Setting up networking...
Network: TAP fhvm0, Host 198.19.0.1, Guest 198.19.0.2
◐ Preparing chroot...
◐ Spawning Firecracker via jailer...
◐ Waiting for API socket...
API socket ready
◐ Starting VM...
VM vm-2f796b98 is running (PID: 6942)
╭───────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ VM Created: vm-2f796b98 │
│ │
│ Status: running │
│ PID: 6942 │
│ vCPUs: 1 │
│ Memory: 512 MiB │
│ Runtime: base │
│ Disk: 1 GB │
│ │
│ Network: │
│ TAP: fhvm0 │
│ Host: 198.19.0.1 │
│ Guest: 198.19.0.2 │
│ MAC: AA:FC:00:00:00:01 │
│ Policy: allow-all │
│ Ports: 80 │
│ │
│ Kernel: /root/.vmsan/kernels/vmlinux-6.1 │
│ Rootfs: /root/.vmsan/rootfs/ubuntu-24.04.ext4 │
│ │
│ Socket: /root/.vmsan/jailer/firecracker/vm-2f796b98/root/run/firecracker.socket │
│ Chroot: /root/.vmsan/jailer/firecracker/vm-2f796b98 │
│ State: /root/.vmsan/vms/vm-2f796b98.json │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────╯
◐ Waiting for agent to become ready...
Agent is ready. Connecting via PTY shell...
ubuntu@vm-2f796b98:~$ Note that if we try to run apt install, it will fail because it cannot find the packages. The reason is that this VM has not updated automatically the package list, and we have to perform this manually. lsof -i shows that the Web server has not started yet. We enable and then start the nginx service. We then verify that nginx is now running. Finally, we change the default index.html file to reflect that we are running the Web server in a microVM and all that in an Incus VM.
ubuntu@vm-2f796b98:~$ su -l
root@ubuntu-fc-uvm:~#
root@ubuntu-fc-uvm:~# apt install nginx lsof
Reading package lists... Done
Building dependency tree... Done
E: Unable to locate package nginx
E: Unable to locate package lsof
root@ubuntu-fc-uvm:~# apt update
...
root@ubuntu-fc-uvm:~# apt install -y nginx lsof
...
root@ubuntu-fc-uvm:~# lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 64u IPv4 1507 0t0 TCP *:ssh (LISTEN)
systemd 1 root 65u IPv6 1511 0t0 TCP *:ssh (LISTEN)
vmsan-age 582 root 3u IPv6 1630 0t0 TCP *:9119 (LISTEN)
vmsan-age 582 root 8u IPv6 1953 0t0 TCP 172.16.0.2:9119->10.200.0.1:39920 (ESTABLISHED)
root@vm-f1d2f963:~# systemctl enable nginx
Synchronizing state of nginx.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
Executing: /usr/lib/systemd/systemd-sysv-install enable nginx
root@ubuntu-fc-uvm:~# systemctl start nginx
root@ubuntu-fc-uvm:~# lsof -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
systemd 1 root 64u IPv4 1507 0t0 TCP *:ssh (LISTEN)
systemd 1 root 65u IPv6 1511 0t0 TCP *:ssh (LISTEN)
vmsan-age 582 root 3u IPv6 1630 0t0 TCP *:9119 (LISTEN)
vmsan-age 582 root 8u IPv6 1953 0t0 TCP 172.16.0.2:9119->10.200.0.1:39920 (ESTABLISHED)
nginx 6342 root 5u IPv4 11303 0t0 TCP *:http (LISTEN)
nginx 6342 root 6u IPv6 11304 0t0 TCP *:http (LISTEN)
nginx 6343 www-data 5u IPv4 11303 0t0 TCP *:http (LISTEN)
nginx 6343 www-data 6u IPv6 11304 0t0 TCP *:http (LISTEN)
root@vm-f1d2f963:~# sed -i "s/Welcome to nginx/Welcome to nginx running in a vmsan microMV, which runs in an Incus VM/g" /var/www/html/index.nginx-debian.html
root@ubuntu-fc-uvm:~# We exit to the host and create the Incus proxy device. While exiting, we have a look at the firewall rule that performed the publishing of port 80 (http) to the Incus virtual machine.
root@ubuntu-fc-uvm:~# exit
exit
ubuntu@ubuntu-fc-uvm:~$ exit
exit
root@microvm:~# iptables-nft
...
Chain FORWARD (policy DROP)
ACCEPT tcp -- anywhere 198.19.0.2 tcp dpt:http
...
root@microvm:~# exit
logout
$ Here is a screenshot of the the website, when accessed from the host. We use the IP address of the Incus VM microvm.
Conclusion
We had a quick look into vmsan, a project that manages micro virtual machines that are based on Firecracker. vmsan is a new project and the maintainer is Angelo RC.
vmsan also added support for Cloudflare tunnels. This means that you can expose your microVMs to the Internet using those Cloudflare tunnels. If you own a domain and you have configured it to be managed by Cloudflare, then vmsan allows you to publish your site automatically without needing to enable port forwarding. You would need to create an API key for this on Cloudflare and then feed that API key to vmsan.
At the moment vmsan comes with four images.
- Ubuntu 24.04
- Ubuntu 24.04 with Python 3.13
- Ubuntu 24.04 with Node.js 22 LTS
- Ubuntu 24.04 with Node.js 24 LTS
I would expect that the project will add support for smaller images, such as Alpine. Considering that Firecracker is not QEMU and has fewer requirements for resources (and features!), it would make sense to have tiny VMs.
vmsan itself is a Node.js application.
