2016-02-23 notes

  1. Xvisor on the Raspberry Pi Demo
  2. Xvisor on the Raspberry Pi 2
  3. Xvisor on KVM (Running in the QEMU emulator for RPi)
  4. KVM on the Raspberry Pi 2 (Enabling KVM virtualization for Raspberry Pi 2)

Introduction

Definitions

  • Virtual Machine
    "Efficient, isolated duplicate of a real machine" (Popek and Goldberg)
  • Hypervisor
    Software/firmware ("host") creating and running virtual machines ("guests")
    Rely on vitualizaiton support on the host CPU
  • Emulator
    An emulator fully duplicate a real machine
    Does not rely on virtualization support on the host CPU

Hypervisor

Hypervisor Classification

  1. Type-1, bare metal hypervisors (native)

    Bare machine (or bare metal), in computer parlance, means a computer without its operating system.

    1. Real-time virtualization:
      • industrial automation
      • medicine
      • telecommunications
      • high-performance computing
    2. Security
      • rigorous certifications
      • formal verification procedures

      The hypervisor must isolate them from non-certifiable workloads. (must be small enough)

      • larger (and more "suspicious")
      • segregates
      • devastating the whole idea of isolation
    e.g. VMware ESX/ESXi, Xen, Xvisor, KVM. BHyVe
  2. Type-2, hosted hypervisors
    e.g. VMware workstation, VirtualBox, KVM, BHyVe
Hyperviseur
Type-1 and type-2 hypervisors



jailhouse
A visualization of Linux running-bare metal (a) and under the Jailhouse hypervisor (b) alongside a real-time application. (Image from Yulia Sinitsyna; Tux image from Larry Ewing.)

Xvisor

Xvisor® is an open-source type-1 hypervisor virtualization solution.

  • monolithic
  • light-weight
  • portable
  • flexible

Xvisor supports

  • Full virtualization (primarily)
    unmodified Guest operating systems
  • Paravirtualization (optional)
    such as VirtIO PCI/MMIO devices

Hypervisors can be categorized into three categories based on Host hardware access, CPU virtualization, and Guest IO emulation, as follows:

  1. Complete Monolithic: Complete monolithic hypervisors (e.g. Xvisor) have one common software for Host hardware access, CPU virtualization, and Guest IO emulation.
  2. Partially Monolithic: Partially monolithic hypervisors (e.g. KVM) are usually an extension of general purpose monolithic OS (e.g. Linux®, FreeBSD®, NetBSD®, etc.) to support Host hardware access + CPU virtualization in kernel and support Guest IO emulation from software running in user-space (e.g. QEMU®).
  3. Micro-kernelized: Micro-kernelized hypervisors (e.g. Xen®) are usually light-weight micro-kernels providing basic Host hardware access + CPU virtualization in kernel and for rest it depends on a Managment Guest (e.g. Dom0 of Xen) which provides complete Host hardware access, Management interface, and Guest IO emulation.

"Faster" than KVM and Xen DomU

  • +0.2% DMIPS VS. KVM
  • +0.46% DMIPS VS. Xen DomU
  • +1.2% read-modify-write operations VS. KVM
  • +1.64% read-modify-write operations VS. Xen DomU
xvisor-fig1
System Virtual Machine Model implemented by Xvisor


SettingUpASandbox

  • Source and Build
  • Bootable ISO
  • Building-Coreboot-FILO
  • Hard Disk Image
  • Running the setup

Build source code

  1. Build source from xvisor-x86_64 github.
    sudo aptitude install gcc-multilib git-all cscope iasl cgdb xorriso \
      libncurses5-dev m4 flex bison autoconf expect qemu-system-x86 qemu-utils
    git clone https://github.com/hschauhan/xvisor-x86_64.git
    cd xvisor-x86_64
    git checkout --track -b x86-next origin/x86-next
    Branch x86-next set up to track remote branch x86-next from origin.
    Switched to a new branch 'x86-next'
    make ARCH=x86 x86_64_generic-defconfig
    make
    echo $?
    0
    cd ..
    mkdir -p xvisor-iso/boot/grub
    cp xvisor-x86_64/build/vmm.bin xvisor-iso/boot
    emacs xvisor-iso/boot/grub/grub.cfg
    set timeout=15
    set default=0
    
    menuentry "Xvisor x86_64" {
      multiboot /boot/vmm.bin earlyprint=vga
      boot
    }
    
    grub-mkrescue -o xvisor-bootable.iso xvisor-iso/
    xorriso 1.4.0 : RockRidge filesystem manipulator, libburnia project.
    
      Drive current: -outdev 'stdio:xvisor-bootable.iso'
      Media current: stdio file, overwriteable
      Media status : is blank
      Media summary: 0 sessions, 0 data blocks, 0 data,  278g free
      Added to ISO image: directory '/'='/tmp/grub.iGBw7p'
      xorriso : UPDATE : 311 files added in 1 seconds
      Added to ISO image: directory '/'='/src3/Xvisor/xvisor-iso'
      xorriso : UPDATE : 315 files added in 1 seconds
      xorriso : NOTE : Copying to System Area: 512 bytes from file '/usr/lib/grub/i386-pc/boot_hybrid.img'
      ISO image produced: 4544 sectors
      Written to medium : 4544 sectors at LBA 0
      Writing to 'stdio:xvisor-bootable.iso' completed successfully.
    
    
    ls -l xvisor-bootable.iso
    -rw-r--r-- 1 cloud cloud 9306112 Dec 15 04:42 xvisor-bootable.iso
    kvm -m 2048M -cpu qemu64,+svm,vendor=AuthenticAMD -cdrom xvisor-bootable.iso -boot d
    
  2. Build source form xvisor-0.2.7.tar.gz.
    wget http://xhypervisor.org/tarball/xvisor-0.2.7.tar.gz
    tar zxvf xvisor-0.2.7.tar.gz
    cd xvisor-0.2.7
    make ARCH=x86 x86_64_generic-defconfig
    make menuconfig
    

    Modify following options:

    1. Processor Features  --->
        [*]   AMD-SVM
        [*]   Intel VT-x
      
    2. Hypervisor Options  --->
        <*> Networking Framework
      
    3. Library Options  --->
        (admin)    Default user name
        (12qwaszx) Default Password
      
    4. Device Drivers  --->
        Network Device Support  --->
          [*] Enable Network Device Support
          <*> Ethernet driver support  --->
          < > PHY Device support and infrastructure  --->
      
    5. Save an Alternate Configuration File
      /src3/XVisor/xvisor-0.2.7/build/tmpconf/.config

    make
    echo $?
    0
    

Build bootable iso image

cd ..
mkdir -p xvisor-iso-0.2.7/boot/grub
cp xvisor-0.2.7/build/vmm.bin xvisor-iso-0.2.7/boot
emacs xvisor-iso-0.2.7/boot/grub/grub.cfg
set timeout=15
set default=0

menuentry "Xvisor x86_64" {
  multiboot /boot/vmm.bin earlyprint=vga
  boot
}

grub-mkrescue -o xvisor-bootable-0.2.7.iso xvisor-iso-0.2.7/
xorriso 1.4.0 : RockRidge filesystem manipulator, libburnia project.

  Drive current: -outdev 'stdio:xvisor-bootable-0.2.7.iso'
  Media current: stdio file, overwriteable
  Media status : is blank
  Media summary: 0 sessions, 0 data blocks, 0 data,  278g free
  Added to ISO image: directory '/'='/tmp/grub.xBef8g'
  xorriso : UPDATE : 311 files added in 1 seconds
  Added to ISO image: directory '/'='/src3/Xvisor/xvisor-iso-0.2.7'
  xorriso : UPDATE : 315 files added in 1 seconds
  xorriso : NOTE : Copying to System Area: 512 bytes from file '/usr/lib/grub/i386-pc/boot_hybrid.img'
  ISO image produced: 4544 sectors
  Written to medium : 4544 sectors at LBA 0
  Writing to 'stdio:xvisor-bootable-0.2.7.iso' completed successfully.


ls -l xvisor-bootable*.iso
-rw-r--r-- 1 cloud cloud 9306112 Dec 15 04:54 xvisor-bootable-0.2.7.iso
-rw-r--r-- 1 cloud cloud 9306112 Dec 15 04:42 xvisor-bootable.iso

kvm -m 2048M -cpu qemu64,+svm,vendor=AuthenticAMD -cdrom xvisor-bootable-0.2.7.iso -boot d

"Default Password" default "EQuiN0X"

Create virtual disk(s)

Create a virtual disk and install grub2 onto the mbr.

cd /src3/Xvisor
dd if=/dev/zero of=xvisor-disk.img count=40320
40320+0 records in
40320+0 records out
20643840 bytes (21 MB) copied, 0.0619101 s, 333 MB/s

fdisk -l xvisor-disk.img
Disk xvisor-disk.img: 19.7 MiB, 20643840 bytes, 40320 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Necessary inputs required to complete this step
Number of sectors(s) = 63 (choosing default)
Number of heads(h) = 16
Number of cylinders(c) = C / (s * h)

Thus c = 40320 / (63 * 16) = 40
fdisk xvisor-disk.img

Welcome to fdisk (util-linux 2.27.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x673c0080.

Command (m for help): x

Expert command (m for help): s
Number of sectors (1-63, default 63): 63

Expert command (m for help): h
Number of heads (1-256, default 255): 16

Expert command (m for help): c
Number of cylinders (1-1048576, default 40): 40

Expert command (m for help): r

Command (m for help): w

The partition table has been altered.
Syncing disks.


fdisk xvisor-disk.img
Command (m for help): n
Partition type
p   primary (0 primary, 0 extended, 4 free)
e   extended (container for logical partitions)
Select (default p):
Using default value p

Partition number (1-4, default 1):
Using default value 1

First sector (2048-40319, default 2048):
Using default value 2048

Last sector, +sectors or +size{K,M,G,T,P} (2048-40319, default 40319):
Using default value 40319

Created a new partition 1 of type 'Linux' and of size 18.7 MiB.

Command (m for help): w
The partition table has been altered.
Syncing disks.

fdisk -l xvisor-disk.img
Disk xvisor-disk.img: 19.7 MiB, 20643840 bytes, 40320 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xd7721429

Device           Boot Start   End Sectors  Size Id Type
xvisor-disk.img1       2048 40319   38272 18.7M 83 Linux

sudo modprobe nbd max_part=8
sudo qemu-nbd -c /dev/nbd0 xvisor-disk.img
ls -l /dev/nbd0*
brw-rw---- 1 root disk 43, 0 Dec 15 16:23 /dev/nbd0
brw-rw---- 1 root disk 43, 1 Dec 15 16:23 /dev/nbd0p1

sudo aptitude install dosfstools
sudo mkfs.vfat /dev/nbd0p1
sudo mount /dev/nbd0p1 /mnt/tmp
sudo grub-install --target=i386-pc --root-directory=/mnt/tmp /dev/nbd0
Installing for i386-pc platform.
Installation finished. No error reported.

sudo cp xvisor-0.2.7/build/vmm.{bin,elf} xvisor-0.2.7/build/system.map /mnt/tmp/boot
sudo emacs /mnt/tmp/boot/grub/grub.cfg
set timeout=15
set default=0

menuentry "Xvisor x86_64" {
  multiboot /boot/vmm.bin earlyprint=vga
  boot
}

sync;sync
sudo umount /mnt/tmp
sudo qemu-nbd -d /dev/nbd0
/dev/nbd0 disconnected
ls -l /dev/nbd0*
brw-rw---- 1 root disk 43, 0 Dec 15 16:29 /dev/nbd0
sudo rmmod nbd
kvm -m 2048M -cpu qemu64,+svm,vendor=AuthenticAMD -hda xvisor-disk.img
host info
Host Name           : x86_64_generic
Boot CPU            : 0
Total Online CPUs   : 1
Total VAPOOL        : 16 MB
Total RAM           : 2048 MB

Build guest OS

Preparing to Boot Guest

git clone https://github.com/hschauhan/xvisor-seabios.git
cd xvisor-seabios
cp config-xvisor .config; make
Version: -20160120_222354-Suhu-Archangel
Fixed space: 0xe05b-0x10000  total: 8101  slack: 5  Percent slack: 0.1%
16bit size:           22400
32bit segmented size: 0
32bit flat size:      41792
32bit flat init size: 0
Lowmem size:          2144
f-segment var size:   1600
  Linking out/rom16.o
  Stripping out/rom16.strip.o
  Linking out/rom32seg.o
  Stripping out/rom32seg.strip.o
  Linking out/rom.o
  Prepping out/bios.bin
Total size: 67936  Fixed: 65799  Free: 63136 (used 51.8% of 128KiB rom)

Note: You have to "enforce" locale:Â export LC_ALL=en_US.utf-8 if keyboard mapping is else

echo $?
0
cd ..
dd if=/dev/zero of=xvisor-guest.img bs=1M count=32
fdisk xvisor-guest.img
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0xc667de96.

Command (m for help): n
Partition type
p   primary (0 primary, 0 extended, 4 free)
e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 1
First sector (2048-65535, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-65535, default 65535):

Created a new partition 1 of type 'Linux' and of size 31 MiB.

Command (m for help): w
The partition table has been altered.
Syncing disks.

sudo modprobe nbd max_part=8
sudo qemu-nbd -c /dev/nbd0 xvisor-guest.img
sudo mkfs.vfat /dev/nbd0p1
sudo mount /dev/nbd0p1 /mnt/tmp
sudo cp xvisor-seabios/out/bios.bin /mnt/tmp
sudo umount /mnt/tmp
sudo qemu-nbd -d /dev/nbd0
mkdir /src3/Xvisor/network-xvisor
screen -S xvisor -d -m kvm -m 2048M \
  -cpu qemu64,+svm,vendor=AuthenticAMD \
  -curses -monitor unix:/src3/Xvisor/network-xvisor/MonSock,server,nowait \
  -hda xvisor-disk.img \
  -hdb xvisor-guest.img
blockdev list
--------------------------------------------------------------------------------
 Name             Parent           Start LBA        Blk Sz      Blk Cnt
--------------------------------------------------------------------------------
 hda0             ---              0                512         514
 hda1             ---              0                512         514
 hda0p0           hda0             2048             512         514
 hda2             ---              0                2048        514
 hda1p0           hda1             2048             512         514
--------------------------------------------------------------------------------

vfs ls /
drwxrwxrwx          0 Dec 15 2015 08:30:15 boot/
vfs mkdir /hda1p0
Error: command vfs failed (code -16)
vfs ls /
drwxrwxrwx          0 Dec 15 2015 08:30:15 boot/
drwxrwxrwx          0 Dec 01 2097 00:02:22 hda1p0/
vfs mount hda1p0 /hda1p0
Trying: ext4 fat
Mounted hda1p0 using fat at /hda1p0
vfs ls /hda1p0
-rwxrwxrwx     131072 Dec 15 2015 08:59:28 bios.bin
total 1
vfs host_load 0xc0e0000 /hda1p0/bios.bin
host: Loaded 0xc0e0000 with 131072 bytes
guest kick guest0
Failed to find guest
Error: command guest failed (code -3)
guest list
-------------------------------------------------------------------------------
 ID     Name              Endianness    Device Path
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------

vfs umount /hda1p0
Unmount successful
vfs umount /
Unmount successful
shutdown
Stopping Hypervisor Timer
Issuing System Shutdown
Error: shutdown failed (error -1)

echo "quit" | sudo socat - unix-connect:/src3/Xvisor/xvisor/MonSock
QEMU 2.4.0 monitor - type 'help' for more information
(qemu) quit

Building Coreboot and FILO

# sudo aptitude install libgmp3-dev gmpc-dev libmpfr-dev libmpc-dev texinfo
git clone http://review.coreboot.org/p/coreboot
cd coreboot
git submodule update --init --checkout
make help
*** coreboot platform targets ***
  Use "make [target] V=1" for extra build debug information
  all                   - Build coreboot
  clean                 - Remove coreboot build artifacts
  distclean             - Remove build artifacts and config files
  doxygen               - Build doxygen documentation for coreboot
  what-jenkins-does     - Run platform build tests (Use CPUS=# for more cores)
  printall              - print makefile info for debugging
  lint / lint-stable    - run coreboot lint tools (all / minimal)
  gitconfig             - set up git to submit patches to coreboot
  ctags / ctags-project - make ctags file for all of coreboot or current board
  cscope / cscope-project - make cscope.out file for coreboot or current board

*** Kconfig Targets ***
  config          - Update current config utilising a line-oriented program
  nconfig         - Update current config utilising a ncurses menu based program
  menuconfig      - Update current config utilising a menu based program
  xconfig         - Update current config utilising a QT based front-end
  gconfig         - Update current config utilising a GTK based front-end
  oldconfig       - Update current config utilising a provided .config as base
  silentoldconfig - Same as oldconfig, but quietly, additionally update deps
  defconfig       - New config with default answer to all options
  savedefconfig   - Save current config as ./defconfig (minimal config)

*** Toolchain targets ***
  crossgcc        - Build coreboot cross-compilers for all platforms
  crosstools      - Build coreboot cross-compiler and GDB for all platforms
  crossgcc-clean  - Remove all built coreboot cross-compilers
  iasl            - Build coreboot IASL compiler (built by all cross targets)
  clang           - Build coreboot clang compiler
  test-toolchain  - Reports if toolchain components are out of date
  crossgcc-ARCH   - Build cross-compiler for specific architecture
  crosstools-ARCH - Build cross-compiler with GDB for specific architecture
  ARCH can be "i386", "x64", "arm", "aarch64", "mips", "riscv", or "power8"
  Use "make [target] CPUS=#" to build toolchain using multiple cores

make crossgcc-x64
Target architecture is now x86_64-elf
echo $?
0
make menuconfig
Mainboard  --->
  ROM chip size (256 KB)  --->  512 KB
  Mainboard model (QEMU armv7 (vexpress-a9))  --->  QEMU x86 i440fx/piix4 (aka qemu -M pc)
Payload  --->
  Add a payload (SeaBIOS)  --->  An ELF executable payload
  (payload.elf) Payload path and filename (NEW)
  filo.elf
Save
Exit

cd payloads
git clone http://review.coreboot.org/p/filo
cd filo
make menuconfig
Save
Exit
Select "Include Multiboot Header"
Interface Options  --->
  Deselect the Grub interface and deselect the autoboot.
Filesystems  --->
  Enable FAT support.
Drivers  --->
  Disable USB Stack 
  Enable PCI support.
  Use libpayload's storage drivers 
Save
Exit

make
echo $?
0
cp build/filo.elf ../../
cd ../../
make
echo $?
0
ls -l build/coreboot.rom
-rw-r--r-- 1 cssu cssu 524288 Jan 22 17:01 build/coreboot.rom
fdisk -l xvisor-guest.img
Disk xvisor-guest.img: 32 MiB, 33554432 bytes, 65536 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x71c9dc1c

Device            Boot Start   End Sectors Size Id Type
xvisor-guest.img1       2048 65535   63488  31M 83 Linux
sudo mount -o loop,offset=`expr 512 '*' 2048` xvisor-guest.img /mnt/tmp
sudo cp coreboot/build/coreboot.rom /mnt/tmp
sudo umount /mnt/tmp
mkdir /src3/Xvisor/network-xvisor
screen -S xvisor -d -m kvm -m 2048M \
  -cpu qemu64,+svm,vendor=AuthenticAMD \
  -curses -monitor unix:/src3/Xvisor/network-xvisor/MonSock,server,nowait \
  -hda xvisor-disk.img \
  -hdb xvisor-guest.img

Boot up guest OS

vfs mount hda1p0 /hda1p0
Trying: ext4 fat
Mounted hda1p0 using fat at /hda1p0
vfs ls /hda1p0
-rwxrwxrwx     131072 Jan 20 2016 22:29:28 bios.bin
-rwxrwxrwx     524288 Jan 25 2016 20:06:19 coreboot.rom
total 2
devtree node copy /guests guest0 /templates/x86_64_generic
guest create guest0
(amd_setup_vm_control:443) IOPM Base physical address: 0x1f5000
(i440fx_emulator_probe:292) i440fx_emulator_probe: 1 busses on this controller.
(i440fx_emulator_probe:317) Success.
Error: failed to create guest0
Error: command guest failed (code -1)

echo 'quit' | sudo  socat - unix-connect:/src3/Xvisor/network-xvisor/MonSock

Docs, tools and scripts

ls -l docs/x86
total 44
-rw-r--r-- 1 cssu cssu 10565 Nov 14 20:27 bios_map.txt
-rw-r--r-- 1 cssu cssu  1502 Nov 14 20:27 Building-Coreboot-FILO.txt
-rw-r--r-- 1 cssu cssu  5090 Nov 14 20:27 grub-on-virtual-disk-howto.txt
-rw-r--r-- 1 cssu cssu    68 Nov 14 20:27 vmm.gdb32
-rw-r--r-- 1 cssu cssu    75 Nov 14 20:27 vmm.gdb64
-rw-r--r-- 1 cssu cssu  4758 Nov 14 20:27 x86_64_generic.txt
-rw-r--r-- 1 cssu cssu   279 Nov 14 20:27 x86_TODO

ls -l tests/x86
total 52
drwxr-xr-x 3 cssu cssu  4096 Nov 14 20:27 bios
-rwxr-xr-x 1 cssu cssu   853 Nov 14 20:27 create_hdd_partitions.expt
-rw-r--r-- 1 cssu cssu   266 Nov 14 20:27 guest_init.cmd
-rw-r--r-- 1 cssu cssu   279 Nov 14 20:27 hdd.layout
-rw-r--r-- 1 cssu cssu 10302 Nov 14 20:27 lomount.c
-rwxr-xr-x 1 cssu cssu 12924 Nov 14 20:27 make_x86_setup
drwxr-xr-x 3 cssu cssu  4096 Nov 14 20:27 os
drwxr-xr-x 2 cssu cssu  4096 Nov 14 20:27 scripts

References

  1. Bare machine
  2. Hypervisor
  3. Xvisor
  4. Xvisor github
  5. Xvisor-x86
  6. DesignDoc
  7. jailhouse
  8. jailhouse github
  9. GCC Cross-Compiler
  10. Build HOWTO