作者：张华 发表于：2023-03-10
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明

问题

想创建一个local local test bed, 用来方便做各种云实验，如openstack, k8s, ovn, lxd等实验，限制条件为：

• 只有一台物理机，且为单网卡
• 尽量使用各种cache来应付特色网络

apt cache

首先就是apt cache:

sudo apt install apt-cacher-ng -y
echo 'PassThroughPattern: .*' |sudo tee -a /etc/apt-cacher-ng/acng.conf
sudo systemctl restart apt-cacher-ng.service && sudo systemctl enable apt-cacher-ng.service
du -sh /var/cache/apt-cacher-ng/
#vim /var/lib/dpkg/info/apt-cacher-ng.postinst
#dpkg --configure apt-cacher-ng#change the dir from /var/cache/apt-cacher-ng/ to /mnt/udisk/apt-cacher-ng
cat << EOF |sudo tee -a /etc/fstab
#use blkid to see uuid
UUID="d63d7251-ec3d-4ef5-aa92-f3d4c480f20c" /mnt/udisk   ext4    defaults    0  2
EOF
mkfs.ext4 -F -L udisk /dev/sdb1
mkdir /mnt/udisk/apt-cacher-ng
chown -R apt-cacher-ng:apt-cacher-ng /mnt/udisk/apt-cacher-ng
sudo sed -i 's/CacheDir: \/var\/cache\/apt-cacher-ng/CacheDir: \/mnt\/udisk\/apt-cacher-ng/g' /etc/apt-cacher-ng/acng.conf
du -sh /mnt/udisk/apt-cacher-ng#Use apt cache proxy
echo 'Acquire::http::Proxy "http://proxy:3142";' | sudo tee /etc/apt/apt.conf.d/01acng

pip mirror

#use pip mirror, or use this instead: PYPI_ALTERNATIVE_URL=http://mirrors.aliyun.com/pypi/simple
mkdir -p ~/.pip
cat << EOF |tee ~/.pip/pip.conf
[global]
trusted-host=mirrors.aliyun.com
index-url = http://mirrors.aliyun.com/pypi/simple
disable-pip-version-check = true
timeout = 120
EOF

image mirror

注：下列的sstream-mirror不知为什么在特色网络下始终都是0%, 待查

• archive.ubuntu.com, 可用 http://mirrors.clooud.tencent.com/ubuntu 代替
• ports.ubuntu.com, 可用http://ports.ubuntu.com/ubuntu-ports 代替

maas.io与cloud-images.ubuntu.com自己做mirror, 方法如下：

#https://blog.csdn.net/quqi99/article/details/78456909
sudo apt -y install simplestreams -y
#for cloud-images.ubuntu.com
sudo sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg --progress --max=1 --path=streams/v1/index.json \https://cloud-images.ubuntu.com/releases/ /images/simplestreams 'arch=amd64' 'release~(jammy)' \'ftype~(lxd.tar.xz|squashfs|root.tar.xz|root.tar.gz|disk1.img|.json|.sjson)'
#for images.maas.io
sudo sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg https://images.maas.io/ephemeral-v3/stable \/images/simplestreams 'arch=amd64' 'release~(jammy)' --max=1 --progress
sudo sstream-mirror --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg https://images.maas.io/ephemeral-v3/stable \/images/simplestreams 'os~(grub*|pxelinux)' --max=1 --progress

然后解决密钥:

#https://goharbor.io/docs/2.6.0/install-config/configure-https/
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -sha512 -days 3650 -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=node1.lan" -key ca.key -out ca.crt
openssl genrsa -out node1.lan.key 4096
openssl req -sha512 -new -subj "/C=CN/ST=Beijing/L=Beijing/O=example/OU=Personal/CN=node1.lan" -key node1.lan.key -out node1.lan.csr
#complies with the Subject Alternative Name (SAN) and x509 v3 extension requirements to avoid 'x509: certificate relies on legacy Common Name field, use SANs instead'
cat > v3.ext <<-EOF
authorityKeyIdentifier=keyid,issuer
basicConstraints=CA:FALSE
keyUsage = digitalSignature, nonRepudiation, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names[alt_names]
DNS.1=node1.lan
DNS.2=node1
DNS.3=hostname
EOF
openssl x509 -req -sha512 -days 3650 -extfile v3.ext -CA ca.crt -CAkey ca.key -CAcreateserial -in node1.lan.csr -out node1.lan.crt
#for docker, the Docker daemon interprets .crt files as CA certificates and .cert files as client certificates.
openssl x509 -inform PEM -in node1.lan.crt -out node1.lan.cert

设置nginx为https, 另外，由于上面使用了一个新目录/images/simplestreams作为root，那需要将/etc/nginx/nginx.conf中添加’user root;'来避免权限问题

$ cat /etc/nginx/sites-available/default
server {listen 443 ssl http2;listen [::]:443 ssl http2;server_name node1.lan;ssl_certificate /home/hua/ca/node1.lan.crt;ssl_certificate_key /home/hua/ca/node1.lan.key;#ssl_protocols TLSv1.2;ssl_prefer_server_ciphers on; location / {root /images/simplestreams;index index.html;}  
}

测试:

curl --resolve node1.lan:443:192.168.99.235 --cacert ~/ca/ca.crt https://node1.lan:443/streams/v1/index.json
sudo cp ~/ca/ca.crt /usr/local/share/ca-certificates/ca.crt
sudo chmod 644 /usr/local/share/ca-certificates/ca.crt
sudo update-ca-certificates --fresh
curl --resolve node1.lan:443:192.168.99.235 https://node1.lan:443/streams/v1/index.json

物理机网络设计

因为只有一台机器node1, 只有一个网卡eno1:

• br-eth0, 虽然lxd也支持ovs bridge, 但为了更方便使用wol，更稳定的management网络，我们还是决定使用用这个仅有的网卡eno1创建一个linux bridge (br-eth0), 这样br-eth0足够做各种lxd实验了
• br-data, 若要做openstack实验，还需要创建一个ovs bridge (br-data), 只用node1这一台物理机做实验它并不需要物理网卡，若今后还想加物理机，可以创建一对linux peers来连接br-eth0与br-data，这linux peers的一端加入br-data即可。另外，即使要创建多机环境，用vagrant, lxd很多方案可以解决，并没有加多物理机的需求

下列netplan配置创建了br-eth0，也让br-eth0支持wol通过魔术包唤醒，也创建了一个没有dhcp的br-maas用于maas实验

cat << EOF |sudo tee /etc/netplan/90-local.yaml
network:version: 2renderer: networkdethernets:eno1:dhcp4: nomatch:macaddress: f8:32:e4:be:87:cdwakeonlan: truebridges:br-eth0:dhcp4: yesinterfaces:- eno1#Use 'etherwake F8:32:E4:BE:87:CD' to wol in bridgemacaddress: f8:32:e4:be:87:cdbr-maas:#br-maas without dhcp enabled so it's for maasdhcp4: falseaddresses:- 192.168.9.1/24routes:- to: defaultvia: 192.168.99.1nameservers:addresses:- 192.168.99.1
EOF
sudo netplan generate
sudo netplan apply

使用netplan的配置是想运行一些post script hook时不方便, 未测试下面使用networkd-dispatcher hook的曲线救国方法.

sudo systemctl stop NetworkManager.service
sudo systemctl disable NetworkManager.service
sudo systemctl stop NetworkManager-wait-online.service
sudo systemctl disable NetworkManager-wait-online.service
sudo systemctl stop NetworkManager-dispatcher.service
sudo systemctl disable NetworkManager-dispatcher.service
sudo apt install netplan.io openvswitch-switch -y
sudo apt install -y networkd-dispatcher -y
cat << EOF |sudo tee /etc/networkd-dispatcher/off.d/start.sh
#!/bin/bash -e
#IFACE='eno1'
if [ \$IFACE = "eno1" -o \$IFACE = "br-eth0" ]; thenif ip link show eno1 | grep "state DOWN" > /dev/null && !(arp -ni br-data | grep "ether" > /dev/null); thendate > /tmp/start.txt;/usr/bin/ovs-vsctl --may-exist add-port br-eth0 eno1ip l add name veth-br-eth0 type veth peer name veth-exip l set dev veth-br-eth0 upip l set dev veth-ex upip l set veth-br-eth0 master br-eth0fi
fi
EOF
cat << EOF |sudo tee /etc/networkd-dispatcher/routable.d/stop.sh
#!/bin/bash -e
if [ \$IFACE = "eno1" -o \$IFACE = "br-eth0" ]; thenif ip link show eno1 | grep "state UP" > /dev/null || arp -ni br-data | grep "ether" > /dev/null; thendate > /tmp/stop.txt;systemctl stop hostapd;fi
fi
EOF
sudo chmod +x /etc/networkd-dispatcher/off.d/start.sh
sudo chmod +x /etc/networkd-dispatcher/off.d/stop.sh

直接创建ovs-bridge的方法如下，但我们的设计并没有使用ovs-bridge的需求：

auto br-eth0
allow-ovs br-eth0
iface br-eth0 inet static
pre-up /usr/bin/ovs-vsctl -- --may-exist add-br br-eth0
pre-up /usr/bin/ovs-vsctl -- --may-exist add-port br-eth0 eno1address 192.168.99.125gateway 192.168.99.1network 192.168.99.0netmask 255.255.255.0broadcast 192.168.99.255
ovs_type OVSBridge
ovs_ports eno1#sudo ip -6 addr add 2001:2:3:4500:fa32:e4ff:febe:87cd/64 dev br-eth0
iface br-phy inet6 static
pre-up modprobe ipv6
address 2001:2:3:4500:fa32:e4ff:febe:87cd
netmask 64
gateway 2001:2:3:4500::1auto eno1
allow-br-phy eno1
iface eno1 inet manual
ovs_bridge br-eth0
ovs_type OVSPort

使用Networkmanager来代替netplan的方法容易支持post script hook:

root@node1:~# cat /etc/network/interfaces
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet manual
auto br-eth0
iface br-eth0 inet staticaddress 192.168.99.124/24gateway 192.168.99.1bridge_ports eth0dns-nameservers 192.168.99.1bridge_stp onbridge_fd 0bridge_maxwait 0up echo -n 0 > /sys/devices/virtual/net/$IFACE/bridge/multicast_snooping
# for stateless it's 'inet6 auto', for stateful it's 'inet6 dhcp'
iface br-eth0 inet6 auto#iface eth0 inet6 static#address 2001:192:168:99::135                                                                                            #gateway 2001:192:168:99::1#netmask 64# use SLAAC to get global IPv6 address from the router# we may not enable ipv6 forwarding, otherwise SLAAC gets disabled# sleep 5 is due a bug and 'dhcp 1' indicates that info should be obtained from dhcpv6 server for statelessup echo 0 > /proc/sys/net/ipv6/conf/$IFACE/disable_ipv6up sleep 5autoconf 1accept_ra 2dhcp 1

devstack实验

单机实验，br-data不用veth-ex也行，br-ex用veth-ex也行(这时就可以当br-ex用了 ）

• OVS_PHYSICAL_BRIDGE=br-data将用于这里(ovs-vsctl set open . external-ids:ovn-bridge-mappings=physnet1:br-data)
• 如果它想要访问外网的话，可以再设置PUBLIC_INTERFACE=veth-ex， 其中veth-ex为br-eth0(linux bridge)与br-data(ovs bridge)之间的linux peers
• PUBLIC_INTERFACE也就是br-ex, 也可以让它使用br-data

配置如下，未测试, 仅供参考：

cat << EOF |tee local.conf
[[local|localrc]]
#make rabbitmq-server to run well
#echo '10.0.1.1 node1' |sudo tee -a /etc/hosts
#sudo pip install --upgrade setuptools
#when USE_VENV=True and hitting pip issue, eg: install_ipip.sh related issues, can try:
#find /bak/openstack -name '*.venv' |xargs rm -rf {}
#https://docs.openstack.org/devstack/latest/configuration.html
#TARGET_BRANCH=stable/zed
#PYPI_ALTERNATIVE_URL=http://mirrors.aliyun.com/pypi/simple
sudo ovs-vsctl show
sudo ip l add name veth-br-eth0 type veth peer name veth-ex >/dev/null 2>&1
sudo ip l set dev veth-br-eth0 up
sudo ip l set dev veth-ex up
sudo ip l set veth-br-eth0 master br-eth0
sudo ovs-vsctl --may-exist add-br br-data
sudo ovs-vsctl --may-exist add-port br-data veth-ex
sudo ip addr add 10.0.1.1/24 dev br-data >/dev/null >/dev/null 2>&1
USE_VENV=False
OFFLINE=False
DEST=/bak/openstack
PUBLIC_INTERFACE=veth-ex
OVS_PHYSICAL_BRIDGE=br-data
PUBLIC_BRIDGE=br-data
HOST_IP=10.0.1.1
FIXED_RANGE=10.0.1.0/24
NETWORK_GATEWAY=10.0.1.1
PUBLIC_NETWORK_GATEWAY=192.168.99.1
FLOATING_RANGE=192.168.99.0/24
Q_FLOATING_ALLOCATION_POOL=start=192.168.99.240,end=192.168.99.249
disable_service tempest
disable_service horizon
disable_service memory_tracker
ADMIN_PASSWORD=password
DATABASE_PASSWORD=\$ADMIN_PASSWORD
RABBIT_PASSWORD=\$ADMIN_PASSWORD
SERVICE_PASSWORD=\$ADMIN_PASSWORD
IP_VERSION=4
SYSLOG=False
VERBOSE=True
LOGFILE=\$DEST/logs/stack.log
ENABLE_DEBUG_LOG_LEVEL=False
SCREEN_LOGDIR=\$DEST/logs
LOG_COLOR=False
LOGDAYS=5
Q_USE_DEBUG_COMMAND=False
WSGI_MODE=mod_wsgi
KEYSTONE_USE_MOD_WSGI=False
NOVA_USE_MOD_WSGI=False
CINDER_USE_MOD_WSGI=False
MYSQL_GATHER_PERFORMANCE=False
DOWNLOAD_DEFAULT_IMAGES=False
IMAGE_URLS="http://download.cirros-cloud.net/0.6.1/cirros-0.6.1-x86_64-disk.img"
heartbeat_timeout_threshold=7200
#GIT_BASE=http://git.trystack.cn
EOF

安装lxd

配置lxd默认创建两块网卡:

• eth0: br-eth0
• eth1: lxdbr0 with dhcp

对于maas lxd 容器，可能还需要一块没有dhcp的网卡(上面netplan中创建了br-maas), 可这样用它：lxc config device add maas eth2 nic name=eth2 nictype=bridged parent=br-maas

sudo snap install lxd --classic
sudo usermod -aG $USER lxd
sudo chown -R $USER ~/.config/
export EDITOR=vim
# MUST NOT use sudo, so must cd to home dir to run it
cd ~ && lxd init --auto
#lxc network set lxdbr0 ipv4.address=10.10.10.1/24
#lxc network set lxdbr0 ipv6.address none#Change the default storage
lxc profile device remove default root
lxc storage delete default
cat << EOF | sudo tee -a /etc/fstab
#mount -o bind /images/lxd /var/snap/lxd/common/lxd/storage-pools
/var/snap/lxd/common/lxd/storage-pools /images/lxd none bind 0 0
EOF
mkdir /images/lxd && sudo mount -a
sudo systemctl restart snap.lxd.daemon
lxc storage create default dir && lxc storage show default
lxc profile device add default root disk path=/ pool=default
lxd sql global "SELECT * FROM storage_pools_config"#Use br-data for lxd containers
cat << EOF |tee /tmp/default.yaml
config:boot.autostart: "true"linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tablessecurity.nesting: "true"security.privileged: "true"
description: ""
devices:eth0:name: eth0nictype: bridgedparent: br-datatype: niceth1:mtu: "9000"name: eth1nictype: bridgedparent: lxdbr0type: nickvm:path: /dev/kvmtype: unix-charmem:path: /dev/memtype: unix-charroot:path: /pool: defaulttype: disktun:path: /dev/net/tuntype: unix-char
name: default
EOF
cat /tmp/default.yaml |lxc profile edit defaultwget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz
wget https://cloud-images.ubuntu.com/releases/22.04/release/ubuntu-22.04-server-cloudimg-amd64.squashfs
lxc image import ./ubuntu-22.04-server-cloudimg-amd64-lxd.tar.xz ./ubuntu-22.04-server-cloudimg-amd64.squashfs --alias jammy
lxc image listlxc launch jammy maas
lxc config show maas --expanded
lxc exec maas bash

注：上面的配置在安装maas snap版本时会报错：security profiles (cannot setup udev for snap “maas”: cannot reload udev rules: exit status 1
继续使用’lxc profile edit default’来加入：

#https://discourse.maas.io/t/install-with-lxd/757/2
config:raw.lxc: |-lxc.mount.auto=sys:rwlxc.cgroup.devices.allow = c 10:237 rwmlxc.apparmor.profile = unconfinedlxc.cgroup.devices.allow = b 7:* rwm

若容器里如果上不了网，如无法访问api.snapcraft.io，是因为lxd容易默认使用了eth1上的dns=10.10.10.1，下面的配置可让eth0, eth1, eth2都默认使用dns=192.168.99.1来避免特色网络对api.snapcraft.io的污染

lxc exec maas bash
cat << EOF |sudo tee /etc/netplan/50-cloud-init.yaml
#make 192.168.99.1 as default dns instead of 10.10.10.1
network:version: 2renderer: networkdethernets:eth0:dhcp4: falseaddresses:- 192.168.99.221/24routes:- to: defaultvia: 192.168.99.1nameservers:addresses:- 192.168.99.1eth1:dhcp4: truenameservers:addresses:- 192.168.99.1eth2:dhcp4: falseaddresses:- 192.168.9.3/24nameservers:addresses:- 192.168.9.3
EOF
#In systemd 239 systemd-resolve has been renamed to resolvectl
resolvectl status
cat /run/systemd/netif/leases/*
nslookup api.snapcraft.io

安装maas

sudo snap install maas --channel=3.3/stable
sudo apt install -y postgresql
sudo -iu postgres psql -d template1 -U postgres
CREATE USER maas WITH ENCRYPTED PASSWORD 'password';
CREATE DATABASE maasdb;
GRANT all privileges on database maasdb to maas;
\c maasdb
cat << EOF | sudo tee -a /etc/postgresql/14/main/pg_hba.conf
host    maas    maasdb  0/0     md5
EOF
#This maas container has 3 IPs: eth0=192.168.99.221 eth1=10.10.10.238 eth2=192.168.9.3
sudo /snap/bin/maas init region+rack --maas-url http://192.168.99.221:5240/MAAS --database-uri "postgres://maas:password@localhost/maasdb"
sudo /snap/bin/maas createadmin --username admin --password password --email admin@example.com --ssh-import lp:zhhuabj
sudo /snap/bin/maas apikey --username admin |tee ~/admin-api-key
sudo /snap/bin/maas status
#login into http://192.168.99.221:5240/MAAS/r/
#change mirror: http://mirrors.cloud.tencent.com/ubuntu/ for http://archive.ubuntu.com/ubuntu
#change mirror: https://mirrors.tuna.tsinghua.edu.cn/ubuntu-ports/ for http://ports.ubuntu.com/ubuntu-ports
apikey=$(sudo maas apikey --username admin)
maas login admin http://127.0.0.1:5240/MAAS $apikey
maas root boot-source update 1 url=https://node1.lan:443 keyring_filename=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg
maas admin boot-resources import

创建openstack实验环境

在单机上创建openstack实验环境的方法有：

• juju/charm + lxd: 直接部署在lxd容器中，好像并发安装时速度较慢, 还是用虚机吧
• 用虚机的话，vagrant是一个选择，multipath也行(但multipath在运行qemu provider时cpu较高)
• 若创建openstack over openstack环境，底层openstack采用devstack或者microstack来搭建，然后使用juju openstack provider来通过juju来在租户之下搭建上层openstack环境。这种情况下，不需要修改现有bundle中的machine配置，这样是为每一个openstack组件都会新启动一个虚机来安装。问题是这一台物理机可能性能无法运行如此多的虚机吧、
• 这台物理机是4核的，那创建三台虚机差不多了，一台做controller, 两台做compute. 在lxd容器中安装maas, 用maas通过pxe来自动安装这三台虚机。然后使用juju maas provider来通过juju管理上层openstack环境，各种控制服务安装在controller虚机的lxd容器里

待续, 目前的问题主要是特色网络造成镜像无法下载，使用sstream-mirror做mirror时也下载不下来。