Kolla安装配置

本文档介绍 Kolla安装配置 的相关内容。

kolla-ansible 多节点部署OpenStack Rocky以及与ceph luminous对接使用： kolla build openstack镜像时访问公司git需要登录vpn： yum -y install openvpn openvpn --daemon --config /opt/hujh201902200927_vpn.ovpn --auth-user-pass /opt/passwd passwd文件自己创建第一行用户名，第二行密码 使用公司yum源： wget http://yum.example.com:9180/repo/Cloud-Example.repo 添加公司pip源加速： [root@sze0-sa-hujh04-10033 kolla]# vim /etc/pip.conf [global] index-url = https://ops-pypi.example.com/root/ops-pypi/+simple/ [install] trusted-host = ops-pypi.example.com 升级内核4.4: yum remove -y kernel kernel-tools kernel-tools-libs kernel-devel && yum install -y kernel-lt kernel-lt-tools kernel-lt-tools-libs kernel-lt-devel kernel-lt-headers rpm -qa |grep kernel grub2-set-default 0 reboot 虚机测试环境： 3controller节点（存储计算融合），双网卡 eth0 做管理网络，eth1（不配置ip地址）做neutron 虚机网络 10. 224.100.51 controller1 //kolla和kolla-ansible项目部署节点 10. 224.100.52 controller2 10. 224.100.53 controller3 keepalived vip 10.224.100.54 1. 系统初始化设置（主机名，/etc/hosts文件，selinux,防火墙，配置部署节点到其他节点互信等） hostnamectl set-hostname controller2 --static yum -y install ntp systemctl enable ntpd.service && systemctl start ntpd.service && systemctl status ntpd.service 2. 所有节点安装相关yum源及python包： wget -O /etc/yum.repo.d/ http://yum.example.com:9180/repo/Cloud-Example.repo yum update -y && reboot yum install python-pip -y pip install -U pip yum install python-devel libffi-devel vim gcc git openssl-devel libselinux-python -y 3，所有节点安装docker： 先用清华的yum源安装： wget -O /etc/yum.repos.d/docker-ce.repo https://download.docker.com/linux/centos/docker-ce.repo sed -i 's+download.docker.com+mirrors.tuna.tsinghua.edu.cn/docker-ce+' /etc/yum.repos.d/docker-ce.repo 执行 yum -y install docker-ce docker -v docker相关配置： mkdir -p /etc/systemd/system/docker.service.d tee /etc/systemd/system/docker.service.d/kolla.conf <<-'EOF' [Service] MountFlags=shared EOF 修改一下docker0的地址： mkdir /etc/docker/ [root@ceph01 ~]# cat /etc/docker/daemon.json {"bip": "10.35.0.1/24"} systemctl daemon-reload && systemctl start docker && systemctl enable docker && systemctl status docker brctl addbr docker0 ip addr add 10.35.0.1/24 dev docker0 ip link set dev docker0 up ip addr add 172.17.0.1/24 dev docker0 4. 安装基础包： 部署节点安装： pip install -U ansible==2.7.10 pip install -U tox pip install -U python-openstackclient (openstack 客户端工具包，后面安装也可以) 所有节点安装：docker python包，jinja2 pip install -U docker==3.7.2 pip install -U Jinja2==2.10.1 查看确认版本： pip freeze urllib3==1.10.2 ansible==2.7.9（部署节点有） Jinja2==2.10 docker==3.7.2 python包固定版本： ansible 2.7.10 chardet 3.0.4 docker 3.7.2 Jinja2 2.10.1 kolla 7.0.3.dev11 /opt/kolla kolla-ansible 7.1.1.dev20 /opt/kolla-ansible PyYAML 5.1 requests 2.21.0 urllib3 1.24.3 5. 在部署节点上（controller1）安装kolla和kolla-ansible（kolla项目用来build OpenStack相关镜像，kolla-ansible用来部署OpenStack） a.安装kolla: 源码安装： cd /opt/ git clone https://hujh:your.email@example.com/openstack/rocky/kolla.git -b stable/rocky cd /opt/kolla pip install -r requirements.txt -r test-requirements.txt -e . （安装依赖环境） tox -egenconfig 生成配置文件 注意：配置修改kolla项目，使用自己的yum源，和定义相关软件包等，参考kolla改动_20190509.txt mkdir /etc/kolla/ cp etc/kolla/kolla-build.conf /etc/kolla/ 设置build镜像的细则： [root@controller1 ~]# vim /etc/kolla/kolla-build.conf [DEFAULT] base = centos namespace = kolla cache = true profile = main push = false retries = 3 install_type = source threads = 8 tag = rocky timeout = 120 rpm_setup_config = Cloud-Example.repo [profiles] main = chrony,cron,fluentd,kolla-toolbox,glance,haproxy,horizon,keepalived,keystone,mariadb,memcached,neutron,nova,rabbitmq,cinder [keystone-base] type = git location = https://example.com/openstack/rocky/keystone.git reference = stable/rocky [glance-base] type = git location = https://hujh:your.email@example.com/openstack/rocky/glance.git reference = stable/rocky [nova-base] type = git location = https://hujh:your.email@example.com/openstack/rocky/nova.git reference = stable/rocky [neutron-base] type = git location = https://hujh:your.email@example.com/openstack/rocky/neutron.git reference = stable/rocky [horizon-base] type = git location = https://hujh:your.email@example.com/openstack/rocky/horizon.git reference = stable/rocky [cinder-base] type = git location = https://hujh:your.email@example.com/openstack/rocky/cinder.git reference = stable/rocky 在build openstack镜像之前，我们要设置好centos7.5 base的镜像： 在部署节点上，启动docker私有仓库： mkdir -p /opt/registry docker run -d -v /opt/registry:/var/lib/registry -p 10.224.100.33:4000:5000 --restart=always --name registry registry:2.3 (1)先拉取centos7.5的镜像： docker pull centos:centos7.5.1804 cd /root vim Dockerfile FROM centos:centos7.5.1804 RUN rm -f /etc/yum.repos.d/* 执行docker build ./ -t centos7.5-1804:rocky 修改docker配置： [root@controller1 ~]# cat /etc/systemd/system/docker.service.d/kolla.conf [Service] MountFlags=shared ExecStart= ExecStart=/usr/bin/dockerd --insecure-registry 10.224.100.33:4000 systemctl daemon-reload && systemctl restart docker (2)推送到私有仓库 docker tag image_uuid 10.224.100.33:4000/kolla/centos7.5-1804:rocky docker push 10.224.100.33:4000/kolla/centos7.5-1804:rocky 执行创建镜像命令： cd /opt/kolla/tools python build.py --base-tag rocky --base-image 10.224.100.33:4000/kolla/centos7.5-1804 --profile main 本次制作的opentack项目相关镜像： chrony,cron,kolla-toolbox,fluentd,glance,haproxy,horizon,keepalived,keystone,mariadb,memcached,neutron,nova,rabbitmq,cinder OpenStack docker镜像build方法： https://github.com/openstack/kolla/blob/stable/rocky/doc/source/admin/image-building.rst 6. build好的镜像，打tag，push到私有registry仓库： a.把build完的OpenStack相关镜像打tag，push到镜像仓库： 例如nova-compute容器镜像： docker tag <images_uuid> 10.224.100.51:4000/kolla/centos-source-nova-compute:rocky docker push 10.224.100.51:4000/kolla/centos-source-nova-compute:rocky b.所有节点配置使用registry仓库： cat /etc/systemd/system/docker.service.d/kolla.conf [Service] MountFlags=shared ExecStart= ExecStart=/usr/bin/dockerd --insecure-registry 10.224.100.51:4000 systemctl daemon-reload && systemctl restart docker && systemctl status docker curl -XGET http://10.224.100.100:4000/v2/kolla/centos-source-kolla-toolbox/tags/list 所有节点都验证一下能否正常访问私有仓库： [root@controller1 ~]# curl http://10.224.100.51:4000/v2/_catalog {"repositories":["centos","kolla/centos-source-base","kolla/centos-source-chrony","kolla/centos-source-cinder-api","kolla/centos-source-cinder-backup","kolla/centos-source-cinder-scheduler","kolla/centos-source-cinder-volume","kolla/centos-source-cron","kolla/centos-source-fluentd","kolla/centos-source-glance-api","kolla/centos-source-glance-base","kolla/centos-source-glance-registry","kolla/centos-source-haproxy","kolla/centos-source-horizon","kolla/centos-source-keepalived","kolla/centos-source-keystone","kolla/centos-source-keystone-fernet","kolla/centos-source-keystone-ssh","kolla/centos-source-kolla-toolbox","kolla/centos-source-mariadb","kolla/centos-source-memcached","kolla/centos-source-neutron-base","kolla/centos-source-neutron-dhcp-agent","kolla/centos-source-neutron-l3-agent","kolla/centos-source-neutron-lbaas-agent","kolla/centos-source-neutron-linuxbridge-agent","kolla/centos-source-neutron-metadata-agent","kolla/centos-source-neutron-openvswitch-agent","kolla/centos-source-neutron-server","kolla/centos-source-nova-api","kolla/centos-source-nova-base","kolla/centos-source-nova-compute","kolla/centos-source-nova-compute-ironic","kolla/centos-source-nova-conductor","kolla/centos-source-nova-consoleauth","kolla/centos-source-nova-libvirt","kolla/centos-source-nova-novncproxy","kolla/centos-source-nova-placement-api","kolla/centos-source-nova-scheduler","kolla/centos-source-nova-ssh","kolla/centos-source-openstack-base","kolla/centos-source-openvswitch-base","kolla/centos-source-openvswitch-db-server","kolla/centos-source-openvswitch-vswitchd","kolla/centos-source-rabbitmq"]} 7. 安装 Kolla-ansible： 源码安装 cd /opt git clone https://gitlab.example.com/openstack/rocky/kolla-ansible.git -b stable/rocky cd /opt/kolla-ansible pip install -r requirements.txt -r test-requirements.txt -e . (安装相关依赖) 拷贝相关配置文件 cp etc/kolla/globals.yml /etc/kolla/ gloabals.yml kolla-ansible全局配置文件 cp etc/kolla/passwords.yml /etc/kolla/ passwords.yml openstack相关服务的密码设置文件 cp ansible/inventory/multinode /etc/kolla/ kolla-ansible 多节点部署的ansible的hosts清单模板 编辑 passwords.yml：设定 OpenStack 服务的各种密码，这里仅设定管理员的登陆密码。 vim /etc/kolla/passwords.yml keystone_admin_password: admin 执行kolla-genpwd 自动生成密码来填充 passwords.yml 8. kolla-ansible部署前准备： a.修改/etc/kolla/globals.yml 设置kolla-ansible部署的全局细则： kolla_base_distro: "centos" kolla_install_type: "source" 使用源码方式 openstack_release: "rocky" 这里是镜像的tag kolla_internal_vip_address: "10.224.100.54" keepalived的vip地址 docker_registry: "10.224.100.51:4000" 指定私有的镜像仓库 docker_namespace: "kolla" 打tag时使用的docker的namespace network_interface: "eth0" 部署管理地址的网卡接口，vip所在的网卡 neutron_external_interface: "eth1" neutron的虚机网络网卡 # neutron_external_interface: "eth1,eth2" 假如有多个，用逗号隔开 neutron_plugin_agent: "linuxbridge" 设置使用linuxbridge,kolla默认使用的是openvswitch enable_openvswitch: "{{ neutron_plugin_agent != 'linuxbridge' }}" keepalived_virtual_router_id: "66" 设置keepalived的id，注意不能与同个局域网内的keepalived的id一样 enable_cinder: "yes" 把cinder打开，默认cinder不部署 enable_cinder_backup: "yes" enable_haproxy: "yes" haproxy默认是使用的，all-in-one单节点要关掉 enable_heat: "no" 关掉OpenStack heat插件 ....... (其他相关定制) b.修改多节点部署的ansible的hosts清单 (其他根据实际情况修改) vim /etc/kolla/multinode [control] controller[1:3] [network] controller[1:3] [inner-compute] [external-compute] controller[1:3] [monitoring] # monitoring01 [storage] 会在storage节点启动cinder_backup和cinder_volume容器 controller[1:3] # storage01 ..... 9. 使用kolla-ansible进行部署： cd /opt/kolla-ansible/tools [root@controller1 tools]# pwd /root/kolla-ansible/tools a.执行： ./kolla-ansible -i /etc/kolla/multinode bootstrap-servers 处理bootstrap servers所需要的依赖 b.进行部署前的precheck,检查环境是否有报错：（主要是网络问题和软件包版本等） ./kolla-ansible -i /etc/kolla/multinode prechecks c.prechecks 过了之后执行部署： ./kolla-ansible -i /etc/kolla/multinode deploy （-t neutron 指定容器） ./kolla-ansible -i /etc/kolla/multinode post-deploy (生成admin的认证环境文件/etc/kolla/admin-openrc.sh) 其他kolla-ansible 可能需要用到的选项： # ./kolla-ansible -i /etc/kolla/multinode reconfigure (当配置改动时，kolla刷配置重启相关容器) # ./kolla-ansible -i /etc/kolla/multinode destroy --yes-i-really-really-mean-it (需要重新部署的时候，摧毁OpenStack集群) 部署完成后查看容器的执行状态： docker ps d.openstack初始化设置（可选） source /etc/kolla/admin-openrc.sh 执行init脚本：/root/kolla-ansible/tools/init-runonce 部署完成之后使用vip登录OpenStack验证： 10. 部署ceph luminous，对接OpenStack：（脚本要根据实际情况修改） 修改脚本ceph_kolla.sh [root@controller1 tools]# cat /root/ceph_kolla.sh # !/bin/bash # ceph对接kolla-ansible部署的OpenStack环境脚本,只需改动ceph pool的pg_num的数量： function rm_osd_pool() { ceph osd pool rm $1 $1 --yes-i-really-really-mean-it } # 按实际情况修改pool的pg_num function create_osd_pool() { ceph osd pool stats images > /dev/null 2>&1 if [[ "$?" != "0" ]]; then ceph osd pool create $1 128 else rm_osd_pool $1 ceph osd pool create $1 128 fi } function make_dir() { if [ ! -f $1 ]; then mkdir -p $1 fi } create_osd_pool images create_osd_pool vms create_osd_pool backups create_osd_pool volumes ceph auth del client.nova ceph auth del client.cinder ceph auth del client.glance ceph auth del client.cinder-backup ceph auth get-or-create client.cinder mon 'allow *' osd 'allow *' mds 'allow *' ceph auth get-or-create client.glance mon 'allow *' osd 'allow *' mds 'allow *' ceph auth get-or-create client.cinder-backup mon 'allow *' osd 'allow *' mds 'allow *' ceph auth get-or-create client.nova mds 'allow *' mon 'allow *' osd 'allow *' make_dir "/etc/kolla/config/glance/" ceph auth get-or-create client.glance > /etc/kolla/config/glance/ceph.client.glance.keyring make_dir "/etc/kolla/config/cinder/cinder-backup/" ceph auth get-or-create client.cinder > /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder.keyring make_dir "/etc/kolla/config/cinder/cinder-volume/" ceph auth get-or-create client.cinder > /etc/kolla/config/cinder/cinder-volume/ceph.client.cinder.keyring make_dir "/etc/kolla/config/cinder/cinder-backup/" ceph auth get-or-create client.cinder-backup > /etc/kolla/config/cinder/cinder-backup/ceph.client.cinder-backup.keyring make_dir "/etc/kolla/config/nova/" ceph auth get-or-create client.nova > /etc/kolla/config/nova/ceph.client.nova.keyring make_dir "/etc/kolla/config/nova/" ceph auth get-or-create client.cinder > /etc/kolla/config/nova/ceph.client.cinder.keyring cp -f /etc/ceph/ceph.conf /etc/kolla/config/nova cp -f /etc/ceph/ceph.conf /etc/kolla/config/glance cp -f /etc/ceph/ceph.conf /etc/kolla/config/cinder cat << EOF > /etc/kolla/config/glance/glance-api.conf [DEFAULT] show_multiple_locations = True show_image_direct_url = True [glance_store] stores = rbd default_store = rbd rbd_store_pool = images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf EOF cat << EOF > /etc/kolla/config/cinder/cinder-volume.conf [DEFAULT] enabled_backends=rbd-1 [rbd-1] rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=cinder backend_host=rbd:volumes rbd_pool=volumes volume_backend_name=rbd-1 volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_secret_uuid = {{ cinder_rbd_secret_uuid }} EOF cat << EOF > /etc/kolla/config/cinder/cinder-backup.conf [DEFAULT] backup_ceph_conf=/etc/ceph/ceph.conf backup_ceph_user=cinder backup_ceph_chunk_size = 134217728 backup_ceph_pool=backups backup_driver = cinder.backup.drivers.ceph backup_ceph_stripe_unit = 0 backup_ceph_stripe_count = 0 restore_discard_excess_bytes = true EOF cat << EOF > /etc/kolla/config/nova/nova-compute.conf [libvirt] # virt_type=qemu # cpu_mode=none # cpu_model=kvm64 images_rbd_pool=vms images_type=rbd images_rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=nova EOF sed_kolla_globals(){ local key_word=$1 local val=$2 local globals_yml=${3:-/etc/kolla/globals.yml} sed -i "s|^#*${key_word}:.*|${key_word}: \"${val}\"|g" $globals_yml } sed_kolla_globals "cinder_backend_ceph" "yes" sed_kolla_globals "cinder_volume_group" "cinder-volumes" sed_kolla_globals "glance_backend_file" "no" sed_kolla_globals "glance_backend_ceph" "yes" sed_kolla_globals "nova_backend_ceph" "yes" 修改完globals.yml之后，执行kolla-ansible reconfigure 使OpenStack相关服务配置生效： 对接完ceph 需要reconfigure cd /opt/kolla-ansible/tools/ ./kolla-ansible -i /etc/kolla/multinode reconfigure 报错：python包urllib3有问题： ImportError: No module named 'requests.packages.urllib3' 解决： pip uninstall urllib3 （1.24.1版本问题） yum -y install python-urllib3 （安装1.10.2版本） 把原来的磁盘清理掉，创建vg： dd if=/dev/urandom of=/dev/sdd bs=512 count=64 pvcreate /dev/sdb vgcreate lvm-sas /dev/sdb openstack server example-resize --flavor 5dd091ee-bcd9-46d4-8e5a-5765699512f6 15c7d646-27ca-42aa-aaeb-a31e6be96ae1 openstack volume create --type sas --size 3 --availability-zone nova:bjdz-cloud-opsnode-A0107-9-1309.example.com@sas sas3 nova boot bjdz-mysql-openstack04 --flavor 29dc9b74-a25c-45fb-b789-23bb271e3e8d --nic net-id=0bfd529c-40bc-475c-be38-133f2e6c7f4e --availability-zone bjdz-az-dmz:bjdz-cloud-opsnode-A0107-8-1308.example.com --block-device id=6cbd4043-ce54-422a-9ce1-56657e1888a8,source=image,dest=volume,device=vda,size=185,bootindex=0,shutdown=remove,volume_type=ssd nova boot bjdz-mysql-openstack05 --flavor 29dc9b74-a25c-45fb-b789-23bb271e3e8d --nic net-id=0bfd529c-40bc-475c-be38-133f2e6c7f4e --availability-zone bjdz-az-dmz:bjdz-cloud-opsnode-A0107-9-1309.example.com --block-device id=6cbd4043-ce54-422a-9ce1-56657e1888a8,source=image,dest=volume,device=vda,size=185,bootindex=0,shutdown=remove,volume_type=ssd 在ceph机器上操作： ceph osd pool create glance 1024 ceph auth get-or-create client.glance mon 'allow r' osd 'allow class-read object_prefix rdb_children, allow rwx pool=images' -o /etc/ceph/ceph.client.glance.keyring mkdir -p /etc/kolla/config/glance/ 把认证文件和ceph.conf 拷贝到/etc/kolla/config/glance/下 cat << EOF > /etc/kolla/config/glance/glance-api.conf [DEFAULT] show_multiple_locations = True show_image_direct_url = True [glance_store] stores = rbd default_store = rbd rbd_store_pool = rbd-images rbd_store_user = glance rbd_store_ceph_conf = /etc/ceph/ceph.conf EOF 修改global.yml里面的配置： "glance_backend_file" "no" "glance_backend_ceph" "yes" [DEFAULT] enabled_backends = sas,ssd [sas] volume_group = lvm-sas volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver volume_backend_name = SAS volume_clear_size = 5 [SSD] volume_group = lvm-ssd volume_driver = cinder.volume.drivers.lvm.LVMISCSIDriver volume_backend_name = SSD volume_clear_size = 5 # source /etc/kolla/admin-openrc.sh # cinder type-create SATA # cinder type-create SSD # cinder type-list + --------------------------------------+------+-------------+-----------+ | ID | Name | Description | Is_Public | + --------------------------------------+------+-------------+-----------+ | 8c1079e5-90a3-4f6d-bdb7-2f25b70bc2c8 | SSD | | True | | a605c569-1e88-486d-bd8e-7aba43ce1ef2 | SAS | | True | + --------------------------------------+------+-------------+-----------+ 设置卷类型的key键值 [root@controller1 tools]# cinder type-key SSD set volume_backend_name=ssd [root@controller1 tools]# cinder type-key SAS set volume_backend_name=sas [root@controller1 tools]# cinder extra-specs-list + --------------------------------------+------+--------------------------------+ | ID | Name | extra_specs | + --------------------------------------+------+--------------------------------+ | 63ccc62c-3df8-41c7-8256-5be492b1f1c1 | SSD | {'volume_backend_name': 'ssd'} | | 7d561912-557d-475f-9811-0a21591c7a8e | SAS | {'volume_backend_name': 'sas'} | + --------------------------------------+------+--------------------------------+ 在kolla中配置cinder ceph多后端: https://www.lijiawang.org/posts/%E5%9C%A8kolla%E4%B8%AD%E9%85%8D%E7%BD%AEcinder%20ceph%E5%A4%9A%E5%90%8E%E7%AB%AF.html#more 加密解密ssl： http://blog.sina.com.cn/s/blog_44ee37cd01016r1h.html 虚机创建： http://blog.itpub.net/18796236/viewspace-1840119/ openstack server set uuid --state active && openstack server uuid reboot --hard 一、常用指令 # ./perccli64 /c0/eall/sall show 查看物理硬盘信息列表 # ./perccli64 /c0/vall 查看虚拟磁盘信息列表，即阵列信息 # ./perccli64 /c0 show preservedCache 查看虚拟磁盘丢失信息 # ./perccli64 /c0/fall show all 查看脱机硬盘信息 # ./perccli64 /c0/v11 delete preservedcache 清除控制器0上的虚拟磁盘11的缓存信息 # ./perccli64 /c0/fall delete 清除外来硬盘配置信息 # ./perccli64 /c0/fall import [preview ] 导入外来硬盘配置 # ./perccli64 /c0 add vd r0 drives=32:10 wb ra 编号为32:10的硬盘做raid0 # ./perccli64 /c0 add vd r5 size=all drives=32:01,32:02,32:03 对应编号3块硬盘做raid5 # ./perccli64 /c0 add vd r1 size=all drives=32:01,32:02 对应编号2块硬盘做raid1