Troubleshooting
VM using GRUB not booting after installation
If your newly installed VM does not boot or gets stuck in PXE boot mode, it is possible that the EFI partition of the VM was not added to the boot options of that VM.
This can be adjusted by switching into Boot Options during boot of the VM. This can be done by pressing Esc during startup of the VM. Then switch into the Boot Maintenance Manager and select Boot Options. Then go to Add Boot Option and select the volume that contains the newly installed OS. Then navigate to the corresponding EFI file that can be used to boot the OS. In case of a Debian version for ARM the path would look like this:
/EFI/debian/grubaa64.efi
Then add an appropriate name for that boot option.
Now go to Change Boot Order in the Boot Options and put this option to the top of the list. Now save the changes and restart. The OS should now be able to boot.
Setup VM on Debian on ARM
1. Update Your System
Before starting, ensure your systems package list and installed packages are up-to-date.
sudo apt update
sudo apt upgrade -y2. Install QEMU and Related Packages
Install QEMU along with the necessary packages for ARM virtualization and UEFI support.
sudo apt install -y qemu-system-arm qemu-efi qemu-utilsqemu-system-arm: Provides QEMU binaries for ARM virtualization.
qemu-efi: Contains UEFI firmware for ARM, necessary for booting the VM.
qemu-utils: Includes utilities like qemu-img for managing disk images.
3. Verify KVM Support
Ensure that your system supports KVM and that it’s enabled.
Check for /dev/kvm:
ls -l /dev/kvmIf the output shows the device exists and has appropriate permissions, KVM is available.
Check if KVM modules are loaded:
lsmod | grep kvmYou should see kvm and kvm_arm in the output.
If not loaded, load the KVM module:
sudo modprobe kvmAdd User to KVM Group
Add your current user to the kvm group to run virtual machines without root privileges.
sudo usermod -aG kvm $USERNote: Log out and log back in for the group change to take effect.
4. Create a Virtual Disk Image
Use qemu-img to create a virtual hard disk for your VM.
qemu-img create -f qcow2 debian_arm64.qcow2 8G-f qcow2: Specifies the QCOW2 format, which supports advanced features like compression and snapshots.
debian_arm64.qcow2: The name of your virtual disk file.
8G: Allocates 8 GB of space for the virtual disk.
6. Prepare UEFI Firmware Files
QEMU requires UEFI firmware files to boot ARM virtual machines.
- Copy the UEFI firmware to your working directory:
cp /usr/share/qemu-efi-aarch64/QEMU_EFI.fd .- Create a writable UEFI variable store:
dd if=/dev/zero of=edk2-arm-vars.fd bs=1M count=64In case the firmware is not the right size - this could cause an error when the VM is started later - it has to be concatenated with some 0 Bits. The target is probably that both, firmware and variable store, habe the same size. This, make sure to take care of that.
If the firmware does not have the right size, you can append some data like this:
dd if=/dev/zero bs=1M count=64 of=QEMU_EFI.fd.tmp
dd if=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd of=QEMU_EFI.fd.tmp conv=notrunc
mv QEMU_EFI.fd.tmp QEMU_EFI.fdYou can now confirm the file size using:
ls -lh QEMU_EFI.fd edk2-arm-vars.fd7. Download Debian ARM64 Netinst ISO
Download the Debian netinst ISO for ARM64 architecture:
wget https://cdimage.debian.org/debian-cd/current/arm64/iso-cd/debian-XX.X.X-arm64-netinst.isoReplace XX.X.X with the current version number of Debian.
8. Start the QEMU Virtual Machine
Run the following command to start your VM:
qemu-system-aarch64 \
-machine virt,accel=kvm \
-cpu host \
-m 2048 \
-smp 4 \
-drive if=pflash,format=raw,file=QEMU_EFI.fd,readonly=on \
-drive if=pflash,format=raw,file=edk2-arm-vars.fd \
-drive file=debian_arm64.qcow2,if=virtio \
-cdrom debian-XX.X.X-arm64-netinst.iso \
-netdev user,id=net0 \
-device virtio-net-device,netdev=net0 \
-nographicNote: Ensure that all file paths (like the ISO and firmware files) are correctly specified based on your current directory.
9. Install Debian in the VM
Boot the VM:
The VM should boot from the Debian installer ISO. You will see the Debian installation menu in the QEMU window.
Proceed with Installation:
Follow the on-screen instructions to install Debian as you would on a physical machine.
Remove the Installation Media:
After the installation is complete and before rebooting, remove the CD-ROM from the VM to prevent booting into the installer again.
Close the VM or switch to the QEMU monitor by pressing Ctrl+Alt+2. In the QEMU monitor, remove the CD-ROM:
eject ide1-cd0Switch back to the VM display with Ctrl+Alt+1.
Reboot the VM:
Allow the VM to reboot into the newly installed Debian system.
10. Post-Installation Configuration
Once the VM has booted into Debian update the VM’s system:
sudo apt update
sudo apt upgrade -y11. Running the VM Headless
If you prefer to run the VM without a graphical interface (e.g., if you’re accessing the Raspberry Pi via SSH), you can use the following command:
qemu-system-aarch64 \
-machine virt,accel=kvm \
-cpu host \
-m 2048 \
-smp 4 \
-drive if=pflash,format=raw,file=QEMU_EFI.fd,readonly=on \
-drive if=pflash,format=raw,file=edk2-arm-vars.fd \
-drive file=debian_arm64.qcow2,if=virtio \
-nographic \
-serial mon:stdio \
-netdev user,id=net0,hostfwd=tcp::2222-:22 \
-device virtio-net-device,netdev=net0After booting, you can SSH into the VM using:
ssh -p 2222 user@localhostReplace user with the username you created during the Debian installation.
USB-Device Passthrough
In case the VM is run via virt machine type it is necessary to define a Bis controller that is compatible with virt, sich as qemu-xhci. The qemu-xhci device emulates a USB 3.0 xHCI controller.
To add this device add the following to the QEMU command:
-device qemu-xhci,id=xhciA USB device can be added to the VM as follows:
-device usb-host,bus=xhci.0,vendorid=0x1a86,productid=0x55d4The vendor and product ID can be obtained from the lsusb command which is part of the usbutils package on Debian.
A sample output of lsusb could look like this:
Bus 001 Device 005: ID 0781:5567 SanDisk Corp. Cruzer Blade
Bus 001 Device 006: ID 046d:c534 Logitech, Inc. Unifying Receiver
The ID here shows the vendor and product ID as vendorid:productid.
An exemplary QEMU VM command could therefore look like this:
qemu-system-aarch64 \
-machine virt,accel=kvm \
-cpu host \
-m 2048 \
-smp 4 \
-drive if=pflash,format=raw,file=QEMU_EFI.fd,readonly=on \
-drive file=haos_generic-aarch64-13.1.qcow2,if=virtio \
-netdev tap,id=net1,script=/etc/qemu-ifup \
-device virtio-net-device,netdev=net1,mac=<mac_address> \
-nographic \
-device qemu-xhci,id=xhci \
-device usb-host,bus=xhci.0,vendorid=0x1a86,productid=0x55d4 \
-device usb-host,bus=xhci.0,vendorid=0x1234,productid=0x5678Permission Management for QEMU
Why Not Run QEMU as Root?
Security Risks: Running QEMU as root grants it full system privileges. If QEMU or a guest VM is compromised, an attacker could gain control over the host system.
Least Privilege Principle: It’s a best practice to run applications with the least privileges necessary to perform their functions.
Setting Up Permissions for QEMU
To run QEMU as a non-root user while allowing access to devices like USB, you need to adjust device permissions and user group memberships.
Create a Dedicated User and Group for QEMU (Optional)
You can run QEMU under your regular user account or create a dedicated user and group for QEMU.
Create a User and Group:
sudo groupadd kvm
sudo useradd -g kvm -s /bin/false qemuNote: Replace qemu with the desired username.
Add Yourself to the Group:
sudo usermod -aG kvm $USERLog out and log back in for the group change to take effect.
Adjust Permissions for KVM Devices
The KVM kernel module provides device files that QEMU needs access to, typically /dev/kvm.
Set Group Ownership and Permissions:
sudo chown root:kvm /dev/kvm
sudo chmod 660 /dev/kvmThis grants read and write permissions to users in the kvm group.
Configure Permissions for USB Devices
By default, USB device files under /dev/bus/usb/ may not be accessible to non-root users. You can use udev rules to adjust permissions.
Create udev rules:
sudo vim /etc/udev/rules.d/50-qemu-usb.rulesAdd the following lines:
SUBSYSTEM=="usb", ATTR{idVendor}=="1a86", ATTR{idProduct}=="55d4", MODE="0666"
SUBSYSTEM=="usb", ATTR{idVendor}=="1234", ATTR{idProduct}=="5678", MODE="0666"Reload udev rules:
sudo udevadm control --reload-rules
sudo udevadm triggerAlternative: Grant Access to the Plugdev Group
On some systems, USB devices are accessible to users in the plugdev group.
Add yourself to the plugdev group:
sudo usermod -aG plugdev $USERModify udev rules to set group ownership:
SUBSYSTEM=="usb", ATTR{idVendor}=="1a86", ATTR{idProduct}=="55d4", GROUP="plugdev", MODE="0660"
SUBSYSTEM=="usb", ATTR{idVendor}=="1234", ATTR{idProduct}=="5678", GROUP="plugdev", MODE="0660"Networking Permissions
If you’re using tap interfaces for networking, you need to set permissions accordingly.
- Set Up a Bridge and Tap Interface
- Create a bridge interface and allow your user to create tap devices.
- Configure Polkit or Sudo Rules
Allow user access to the qemu-bridge-helper:
sudo chmod u+s /usr/lib/qemu/qemu-bridge-helperIf additional permissions are required, you might need to adjust sudoers or polkit configurations to allow your user to perform networking operations without full root privileges.
Adjusting File Permissions for Disk Images
Ensure that your user has read and write permissions to the disk image files used by QEMU.
sudo chown $USER:$USER /path/to/your/image.qcow2Running QEMU as a Non-Root User
With the permissions set up, you can run QEMU as your regular user.
Updated QEMU Command
qemu-system-aarch64 \
-machine virt,accel=kvm \
-cpu host \
-m 2048 \
-smp 4 \
-drive if=pflash,format=raw,file=QEMU_EFI.fd,readonly=on \
-drive file=haos_generic-aarch64-13.1.qcow2,if=virtio \
-netdev tap,id=net1,script=/qemu/vm/vm02/qemu-ifup \
-device virtio-net-device,netdev=net1,mac=52:54:00:12:34:57 \
-nographic \
-device qemu-xhci,id=xhci \
-device usb-host,bus=xhci.0,vendorid=0x1a86,productid=0x55d4 \
-device usb-host,bus=xhci.0,vendorid=0x1234,productid=0x5678Run this command as your regular user.
Comparison with bhyve
bhyve: FreeBSD’s bhyve hypervisor typically requires root privileges because it interacts directly with system resources like network interfaces and storage devices.
QEMU: Designed to run as a regular user with proper permissions. It can utilize user-space networking (e.g., slirp) or tap devices with appropriate permissions.
Resize qcow2 image
To resize an existing image for instance to increase the available storage space of the VM by 20GB use the following command:
qemu-img resize <image_name>.qcow2 +20G