Using VLANS with XEN -------------------- Paul Koppel Electronic Radiology Laboratory Mallinckrodt Institute of Radiology Washington University School of Medicine St. Louis, MO 63110 There have been a number of recent articles explaining how to use VLANS with Xen. Based on these postings - here is a more complete explanation covering from start to finish: - setting up the VLANS in the base OS (CentOS 5.2) - modifying the Xen networking scripts - creating a para-virtualized guest (CentOS 5.2) - a few comments about creating a full hardware-virtualized guest Several articles proved to be extremely useful in getting everything working at the Electronic Radiology Laboratory: - http://mike.neir.org/wiki/articles/CentOS-5_Xen - http://wiki.centos.org/HowTos/Xen/InstallingCentOSDomU Here are the details: 1. Setting up the VLANS in the Base OS (CentOS 5.2) --------------------------------------------------- The VLAN numbers that were used are: 100 = DMZ (contains subnet 192.168.100.0/24) 101 = INSIDE (contains subnet 192.168.101.0/24) An IP that was used to test on the DMZ VLAN is: 192.168.100.51 We used the INSIDE network (VLAN 101) as the management network. The para-virtualized CentOS guest will be running on the DMZ network (VLAN 100). Here are the networking scripts at the base OS that were modified. In the "network" file, notice the addition of a GATEWAY address and GATEWAYDEV pointing to eth0.101: [root@xentest koppel]# more /etc/sysconfig/network NETWORKING=yes NETWORKING_IPV6=no HOSTNAME=xentest.example.com GATEWAY=192.168.101.254 GATEWAYDEV=eth0.101 Here are the definitions of the VLAN interfaces - note that BOOTPROTO is none for these; there is no HWADDR address; and that there is an IP address for VLAN 101 but not for VLAN 100: [root@xentest koppel]# more /etc/sysconfig/network-scripts/ifcfg-eth0.101 # Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet DEVICE=eth0.101 BOOTPROTO=none #HWADDR=00:11:22:32:6C:23 IPADDR=192.168.101.69 NETMASK=255.255.255.0 ONBOOT=yes VLAN=yes [root@xentest koppel]# more /etc/sysconfig/network-scripts/ifcfg-eth0.100 # Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet DEVICE=eth0.100 BOOTPROTO=none #HWADDR=00:11:22:32:6C:23 #IPADDR=192.168.100.51 NETMASK=255.255.255.0 ONBOOT=yes VLAN=yes For eth0, notice that ONBOOT is set to no, but it does have a HWADDR defined: [root@xentest koppel]# more /etc/sysconfig/network-scripts/ifcfg-eth0 # Broadcom Corporation NetXtreme II BCM5708 Gigabit Ethernet DEVICE=eth0 #BOOTPROTO=static HWADDR=00:11:22:32:6C:23 #IPADDR=192.168.101.69 #NETMASK=255.255.255.0 ONBOOT=no TYPE=Ethernet With this configuration, we could secure shell to 192.168.101.69 on the INSIDE network, and do DNS lookups. By disabling this address in ifcfg-eth0.101, and enabling 192.168.100.51 in eth0.100, we could also secure shell to 192.168.100.51 and do DNS lookups after restarting the network services (without any errors): [root@xentest koppel]# /etc/init.d/network restart Shutting down interface eth0.100: Removed VLAN -:eth0.100:- [ OK ] Shutting down interface eth0.101: Removed VLAN -:eth0.101:- [ OK ] Shutting down interface eth0: [ OK ] Shutting down loopback interface: [ OK ] Disabling IPv4 packet forwarding: net.ipv4.ip_forward = 0 [ OK ] Bringing up loopback interface: [ OK ] Bringing up interface eth0.100: Added VLAN with VID == 100to IF -:eth0:- [ OK ] Bringing up interface eth0.101: Added VLAN with VID == 101 to IF -:eth0:- [ OK ] The main purpose of these tests were to verify that the basic VLAN networking worked as expected - outside of Xen. Once the Xen networking scripts were modified, it was not possible to cleanly restart the network service – which was also noted in the first reference. 2. Modifying the Xen Networking Scripts A. Modify the network-bridge Script The default Xen networking scripts that come with CentOS 5 need to be modified to work with Xen virtual machines. The first change bypasses the broken ifdown call that destroys the VLAN interface when the bridge is created. [root@xentest koppel]# cd /etc/xen/scripts [root@xentest koppel]# cp -p network-bridge network-bridge-dist Change (around line 78) from: [ -f "/sys/class/net/$1/bonding/slaves" ] to: [ -f "/sys/class/net/$1/bonding/slaves" ] || [ -f /proc/net/vlan/$1 ] By copying the original to "dist", it is easy to review your changes at a later date (and also go back to where you started if necessary): [root@xentest scripts]# diff network-bridge-dist network-bridge 79c79 < [ -f "/sys/class/net/$1/bonding/slaves" ] --- > [ -f "/sys/class/net/$1/bonding/slaves" ] || [ -f /proc/net/vlan/$1 ] B. Modify xend-config.sxp The second change involves modifying the xend configuration file to use a new network initialization script. [root@xentest scripts]# cd /etc/xen [root@xentest scripts]# cp -p xend-config.sxp xend-config.sxp-dist Change the network-script configuration directive to the new script name - such as network-multi-bridge. After the modification: [root@xentest xen]# diff xend-config.sxp-dist xend-config.sxp 91c91 < (network-script network-bridge) --- > (network-script network-multi-bridge) C. Create Script network-multi-bridge This script uses the two VLAN interfaces previously defined. [root@xentest scripts]# cd /etc/xen/scripts [root@xentest scripts]# more network-multi-bridge #!/bin/sh dir=$(dirname "$0") "$dir/network-bridge" "$@" vifnum=0 netdev=eth0 bridge="xbr_trunk" /sbin/ifup eth0.100 /sbin/ifup eth0.101 "$dir/network-bridge" "$@" vifnum=1 bridge="xbr_v100" netdev="eth0.100" "$dir/network-bridge" "$@" vifnum=2 bridge="xbr_v101" netdev="eth0.101" For reference, here are the networking interfaces now: [root@xentest scripts]# /sbin/ifconfig -a eth0 Link encap:Ethernet HWaddr 00:11:22:32:6C:23 inet6 addr: fe80::211:22ff:fe32:6c23/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:302333 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:26546110 (25.3 MiB) TX bytes:468 (468.0 b) eth0.100 Link encap:Ethernet HWaddr 00:11:22:32:6C:23 inet6 addr: fe80::211:22ff:fe32:6c23/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:442743 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:28957395 (27.6 MiB) TX bytes:468 (468.0 b) eth0.101 Link encap:Ethernet HWaddr 00:11:22:32:6C:23 inet addr:192.168.101.69 Bcast:192.168.101.255 Mask:255.255.255.0 inet6 addr: fe80::211:22ff:fe32:6c23/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:585869 errors:0 dropped:0 overruns:0 frame:0 TX packets:211493 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:44139275 (42.0 MiB) TX bytes:845155397 (806.0 MiB) lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope:Host UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:148889 errors:0 dropped:0 overruns:0 frame:0 TX packets:148889 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:852414208 (812.9 MiB) TX bytes:852414208 (812.9 MiB) peth0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:2203720 errors:0 dropped:0 overruns:0 frame:0 TX packets:1076244 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1125220455 (1.0 GiB) TX bytes:911581156 (869.3 MiB) Interrupt:16 Memory:f8000000-f8012100 peth0.100Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:1152420 errors:0 dropped:0 overruns:0 frame:0 TX packets:417706 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:1012847362 (965.9 MiB) TX bytes:28202721 (26.8 MiB) peth0.101 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:585881 errors:0 dropped:0 overruns:0 frame:0 TX packets:658576 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:35937745 (34.2 MiB) TX bytes:874665199 (834.1 MiB) sit0 Link encap:IPv6-in-IPv4 NOARP MTU:1480 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) veth3 Link encap:Ethernet HWaddr 00:00:00:00:00:00 BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif0.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:302333 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:468 (468.0 b) TX bytes:26546110 (25.3 MiB) vif0.1 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:6 errors:0 dropped:0 overruns:0 frame:0 TX packets:442743 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:468 (468.0 b) TX bytes:28957395 (27.6 MiB) vif0.2 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:211524 errors:0 dropped:0 overruns:0 frame:0 TX packets:585898 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:845162819 (806.0 MiB) TX bytes:44141201 (42.0 MiB) vif0.3 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:0 (0.0 b) vif5.0 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF inet6 addr: fe80::fcff:ffff:feff:ffff/64 Scope:Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:9838 errors:0 dropped:0 overruns:0 frame:0 TX packets:441257 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:32 RX bytes:522212 (509.9 KiB) TX bytes:33182147 (31.6 MiB) virbr0 Link encap:Ethernet HWaddr 00:00:00:00:00:00 inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0 inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:0 (0.0 b) TX bytes:468 (468.0 b) xbr_trunk Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:301781 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:22288056 (21.2 MiB) TX bytes:0 (0.0 b) xbr_v100 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:442728 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:22758178 (21.7 MiB) TX bytes:0 (0.0 b) xbr_v101 Link encap:Ethernet HWaddr FE:FF:FF:FF:FF:FF UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:272445 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:15262219 (14.5 MiB) TX bytes:0 (0.0 b) 3. Creating a Para-Virtualized Guest (CentOS 5.2) ------------------------------------------------- A. Create an Image File The first step was to create an image file for the domU virtual hard drive. The command to create an 8GB hard drive image - pkvlantest.img in the directory /var/lib/xen/images is: [root@xentest scripts]# dd if=/dev/zero of=/var/lib/xen/images/pkvlantest.img oflag=direct bs=1M count=8192 B. Download initrd.img and vmlinuz For domU installation, two files are necessary - initrd.img and vmlinuz, which can be downloaded from: http://mirror.centos.org/centos/5/os/i386/images/xen/ or http://mirror.centos.org/centos/5/os/x86_64/images/xen/ and put in a known location such as a home directory or /boot. The locations of these two files are referenced in the para-virtualized, guest-installation, configuration file (see below). C. Create an Installation Configuration File Here is the installation file for "pkvlantest" - note that xbr_v100 is being used (for the DMZ network): [root@xentest xen]# more pkvlantest kernel = "/home/koppel/vmlinuz" ramdisk = "/home/koppel/initrd.img" name = "pkvlantest" maxmem = 512 memory = 512 vcpus = 1 on_poweroff = "destroy" on_reboot = "destroy" on_crash = "destroy" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/pkvlantest.img,xvda,w" ] vif = [ "mac=00:16:3e:18:35:76,bridge=xbr_v100" ] D. Start the Installation To launch the domU virtual machine installation: [root@xentest xen]# /usr/sbin/xm create pkvlantest Attach to a text console: [root@xentest xen]# /usr/sbin/xm console pkvlantest Instead of a text console - could also use the virt-manager and attach to the text console that way: [root@xentest xen]# /usr/sbin/virt-manager & or even use vncviewer directtly: [root@xentest xen]# /usr/bin/vncviewer localhost:5900 Note - the 5900 port number may be different if there are other domU's running already. For a para-virtualized system, a network-install using ftp or http is very convenient and possible - simply by using the CentOS installer and pointing to one of the CentOS mirrors. Creating a simple configuration file as above with the correct network properties (bridge=xbr_v100) is, of course, necessary for this to work. Attempts at trying to use the virt-manager program to create a configuration file with the correct network properties were not successful. E. Post Installation The configuration file needs to be modified for non-install use. The lines for the kernel and initrd are gone, and now there is a new line using pygrub: [root@xentest xen]# more pkvlantest bootloader = "/usr/bin/pygrub" name = "pkvlantest" maxmem = 512 memory = 512 vcpus = 1 on_poweroff = "destroy" on_reboot = "restart" on_crash = "restart" vfb = [ "type=vnc,vncunused=1,keymap=en-us" ] disk = [ "tap:aio:/var/lib/xen/images/pkvlantest.img,xvda,w" ] vif = [ "mac=00:16:3e:18:35:76,bridge=xbr_v100" ] 4. Creating a Full Hardware-Virtualized Guest --------------------------------------------- For a full hardware virtualized system, one could use the virt-manager program and point to an ISO file (CentOS-5.2-i386-bin-DVD.iso) to create a configuration file, virtual hard drive, and install successfully. After the install, simply change the bridge statement from xbr_trunk to xbr_v100in the generated configuration file. This approach will work for full- hardware virtualized systems because it uses an iso file for the guest operating system installation - not the network. 5. Acknowledgements ------------------- The two URLs mentioned above were extremely helpful in making all of this work. -- pk