基于本地存储LVM新建虚机方案

date: 2021/12/22

auth: mmwei3

---

一、环境信息如下：

IP	角色	VGS
192.168.162.123	控制+计算+存储	cinder-volumes、cinder-volumes-ssd
192.168.162.124	控制+计算+存储	cinder-volumes、cinder-volumes-ssd
192.168.162.125	控制+计算+存储	cinder-volumes、cinder-volumes-ssd

二、需求方案：

1、虚机(卷启动)+系统盘+数据盘 三者在同一计算节点。

2、虚机可以挂本计算节点的数据盘也可以挂载其他计算节点的数据盘。

3、虚机可以使用本节点上的HDD做系统盘及云盘，也可以使用本节点上的SSD做系统盘及云盘

4、虚机(本地)+数据盘 三者在同一计算节点。

5、虚机冷迁移方案

三、环境部署部分说明

 参考官方文档：https://docs.openstack.org/mitaka/config-reference/block-storage/drivers/lvm-volume-driver.htmlhttps://docs.openstack.org/cinder/latest/admin/blockstorage-lio-iscsi-support.html

1、每个节点新建两个vg：cinder-volumes、cinder-volumes-ssd

# 001节点
[root@rg2-test-control001 ~]# vgsVG                 #PV #LV #SN Attr   VSize   VFreecinder-volumes       1   4   0 wz--n-  <3.64t  <3.30tcinder-volumes-ssd   2   0   0 wz--n- 278.24g 278.24g# 002节点
[root@rg2-test-control002 ~]# vgsVG                 #PV #LV #SN Attr   VSize   VFreecinder-volumes       1   1   0 wz--n-  <3.64t  <3.59tcinder-volumes-ssd   2   0   0 wz--n- 278.24g 278.24g# 003节点
[root@rg2-test-control003 ~]# vgsVG                 #PV #LV #SN Attr   VSize   VFreecinder-volumes       1   1   0 wz--n-  <3.64t  <3.59tcinder-volumes-ssd   2   2   0 wz--n- 278.24g 258.24g

2、修改相关配置文件 (解决方案一)

参考官方文档：
https://docs.openstack.org/kilo/config-reference/content/section_cinder.conf.html
https://docs.openstack.org/mitaka/config-reference/compute/iscsi-offload.html
https://docs.openstack.org/cinder/latest/admin/ts-cinder-config.html
https://docs.openstack.org/cinder/latest/admin/blockstorage-multi-backend.html

① 修改cinder-volume、cinder-api、cinder-scheduler配置

# 三个节点均需修改(或者说cinder-volume节点所在节点都修改)
# 为了保持统一cinder-api、cinder-scheduler、cinder-volume三者均修改# iSCSI target user-land tool to use. tgtadm is default, use
# lioadm for LIO iSCSI support, scstadmin for SCST target
# support, iseradm for the ISER protocol, ietadm for iSCSI
# Enterprise Target, iscsictl for Chelsio iSCSI Target or fake
# for testing. (string value)
#iscsi_helper=tgtadm# Method used to wipe old volumes (string value)
#volume_clear=zero# Size in MiB to wipe at start of old volumes. 0 => all
# (integer value)
#volume_clear_size=0[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf|egrep -w 'enabled_backends|default_volume_type'
enabled_backends = lvm,lvm-ssd
default_volume_type = lvm[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf  | grep 'storage_availability_zone'
storage_availability_zone = rg2-test-control003.ostack.hfb3.iflytek.net (cinder-volume节点的主机名, 即myhostname)[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-volume/cinder.conf  | egrep  -w '\[lvm\]' -A 30
[lvm]
volume_backend_name = lvm
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes1
iscsi_protocol = iscsi
#iscsi_helper = lioadm   #这里不适用lioadm，使用默认的配置 #iscsi_helper=tgtadm,可以不写
volume_clear = zero[lvm-ssd]
volume_backend_name = lvm-ssd
volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver
volume_group = cinder-volumes-ssd
iscsi_protocol = iscsi
#iscsi_helper = lioadm
volume_clear = zero[root@rg2-test-control003 ~]#

② 修改glance-api配置

参考官方文档：https://docs.openstack.org/glance/pike/install/install-rdo.html

# 三个节点均需修改(或者说glance-api节点所在节点都修改)
[root@rg2-test-control003 ~]# cat /etc/kolla/glance-api/glance-api.conf| grep '\[glance_store\]' -A 10
[glance_store]
#default_store = rbd
#filesystem_store_datadir = /var/lib/glance/images/
#stores = rbd
#rbd_store_user = glance
#rbd_store_pool = images
#rbd_store_ceph_conf = /etc/ceph/ceph.conf
#rbd_volume_pool = sata_pool
default_store = file
filesystem_store_datadir = /var/lib/glance/images/
stores = file

② 修改nova-compute配置

[root@rg2-test-control003 ~]# cat /etc/kolla/nova-compute/nova.conf  | grep '\[libvirt\]' -A 10
[libvirt]
connection_uri = qemu+tcp://192.168.162.125/system
#images_rbd_pool = vms
#images_type = rbd
#images_rbd_ceph_conf = /etc/ceph/ceph.conf
#rbd_user = nova
#rbd_secret_uuid = df5fbfad-02c7-43b2-9938-900c80d6bc99
images_type = lvm
images_volume_group=cinder-volumes1

3、修改相关配置文件 (解决方案二)

参考官方文档：
https://docs.openstack.org/cinder/pike/scheduler-filters.html
https://github.com/openstack/cinder/blob/master/cinder/scheduler/filters/instance_locality_filter.py
https://opendev.org/openstack/cinder/src/commit/40a6fff6d408f2ef0317c08fb165cc75a92c380a/setup.cfg
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/14/html/
openstack_data_processing/instancelocality
https://specs.openstack.org/openstack/sahara-specs/specs/kilo/volume-instance-locality.html

# 在方案一的基础上新增如下配置
# 该配置可以尝试使我们的虚机实例和系统盘在同一个节点上创建
[root@rg2-test-control003 ~]# cat /etc/kolla/cinder-scheduler/cinder.conf  | grep scheduler_default_filters -A 3
scheduler_default_filters=AvailabilityZoneFilter,CapacityFilter,CapabilitiesFilter,InstanceLocalityFilter
volume_local_to_instance = True # value for true or false 

四、运行相关容器

# iscsi-initiator-utils  scsi-target-utils.x86_64 
# cinder-volume容器挂载一下-v iscsi_info:/etc/iscsi  nova-compute也挂在一下cat /etc/kolla/iscsid/config.json 
{"command": "iscsid -d 8 -f --pid=/run/iscsid.pid","config_files": []
}
cat /etc/kolla/tgtd/config.json {"command": "tgtd -d 1 -f --iscsi portal=192.168.162.123:3260","config_files": []
}# nova_compute
docker run \
-d --tty=true --net=host --restart=always \
--privileged \
--name=nova_compute \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=nova-compute \
-v /etc/kolla/nova-compute/:/var/lib/kolla/config_files/:ro \
-v /etc/localtime:/etc/localtime:ro \
-v /lib/modules:/lib/modules:ro \
-v /run:/run:shared \
-v /dev:/dev \
-v kolla_logs:/var/log/kolla/ \
-v libvirtd:/var/lib/libvirt \
-v nova_compute:/var/lib/nova/ \
-v iscsi_info:/etc/iscsi \
mmwei.com/lokolla/centos-source-nova-compute:15.0.6-ccr-1.1.3-priv.7-2#cinder_volume
docker run \
-d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=cinder_volume \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS  \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /dev/:/dev/ \
-v /lib/modules:/lib/modules:ro \
-v /run/:/run/:shared \
-v cinder:/var/lib/cinder \
-v iscsi_info:/etc/iscsi \
-v /etc/kolla/cinder-volume/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-volume:10.0.3-ccr-1.0.6docker run \
-d --tty=true --net=host --restart=always \
--name=cinder_api \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS  \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /etc/kolla/cinder-api/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-api:10.0.3-ccr-1.0.6docker run \
-d --tty=true --net=host --restart=always \
--name=cinder_scheduler \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS  \
-v /etc/localtime:/etc/localtime:ro \
-v kolla_logs:/var/log/kolla/:rw \
-v /etc/kolla/cinder-scheduler/:/var/lib/kolla/config_files/:ro \
mmwei.com/lokolla/centos-source-cinder-scheduler:10.0.3-ccr-1.0.6mmwei.com/lokolla/centos-source-cinder-volume:10.0.3-ccr-1.0.8# iscsi-tgtd
docker run -d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=tgtd \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=tgtd \
-v /etc/kolla/tgtd/:/var/lib/kolla/config_files/:ro \
-v kolla_logs:/var/log/kolla/ 
-v /etc/localtime:/etc/localtime:ro \
-v /dev:/dev \
-v /run:/run:shared \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \-v /lib/modules:/lib/modules:ro \-v /sys/kernel/config:/configfs \-v iscsi_info:/etc/iscsi \mmwei.com/lokolla/centos-source-tgtd:train-1# iscsid
docker run -d --tty=true --net=host --restart=always \
--privileged \
--ipc=host \
--name=iscsid \
-e KOLLA_CONFIG_STRATEGY=COPY_ALWAYS \
-e KOLLA_SERVICE_NAME=iscsid \
-v /etc/kolla/iscsid/:/var/lib/kolla/config_files/:ro \
-v kolla_logs:/var/log/kolla/ \
-v /etc/localtime:/etc/localtime:ro \
-v /dev:/dev \
-v /run:/run:shared \
-v /sys/fs/cgroup:/sys/fs/cgroup:ro \
-v /lib/modules:/lib/modules:ro \
-v /sys/kernel/config:/configfs \
-v iscsi_info:/etc/iscsi \
mmwei.com/lokolla/centos-source-iscsid:train-v1

五、验证方案

1、虚机(卷启动)+系统盘+数据盘 三者在同一计算节点。

① 上传镜像

[root@rg2-test-control001 ~]# glance image-create    
--disk-format raw \
--container-format bare \
--file /root/mmwei3/Centos-7.4.raw  \
--property  hw_qemu_guest_agent=yes \
--name Centos-7.4-lvm \
--visibility public \
--progress

② 新建主机聚合

# 先查看服务状态 是否正常，zone是否生效
[root@rg2-test-control001 ~]# openstack volume service list  -c Binary -c  Host -c Zone
+-----------------+----------------------------------------------------+---------------------------------------------+
| Binary          | Host                                               | Zone                                        |
+-----------------+----------------------------------------------------+---------------------------------------------+
| cinder-scheduler| rg2-test-control002.ostack.hfb3.iflytek.net        | nova                                        |
| cinder-scheduler| rg2-test-control003.ostack.hfb3.iflytek.net        | nova                                        |
| cinder-scheduler| rg2-test-control001.ostack.hfb3.iflytek.net        | nova                                        |
| cinder-volume   | rg2-test-control001.ostack.hfb3.iflytek.net@lvm    | nova                                        |
| cinder-volume   | rg2-test-control003.ostack.hfb3.iflytek.net@lvm    | rg2-test-control003.ostack.hfb3.iflytek.net |
| cinder-volume   | rg2-test-control002.ostack.hfb3.iflytek.net@lvm    | rg2-test-control002.ostack.hfb3.iflytek.net |
| cinder-volume   | rbd:sata@rbd-sata                                  | nova                                        |
| cinder-volume   | rg2-test-control003.ostack.hfb3.iflytek.net@lvm-ssd| rg2-test-control003.ostack.hfb3.iflytek.net |
+-----------------+----------------------------------------------------+---------------------------------------------+[root@rg2-test-control001 ~]# nova  aggregate-create
usage: nova aggregate-create <name> [<availability-zone>][root@rg2-test-control001 ~]# nova  aggregate-create normal
[root@rg2-test-control001 ~]# nova  aggregate-create lvm
[root@rg2-test-control001 ~]# nova  aggregate-create lvm2# 加入聚合
[root@rg2-test-control001 ~]# nova   aggregate-add-host
usage: nova aggregate-add-host <aggregate> <host>
[root@rg2-test-control001 ~]# nova   aggregate-add-host normal rg2-test-control001.ostack.hfb3.iflytek.net
[root@rg2-test-control001 ~]# nova   aggregate-add-host lvm rg2-test-control002.ostack.hfb3.iflytek.net
[root@rg2-test-control001 ~]# nova   aggregate-add-host lvm2 rg2-test-control003.ostack.hfb3.iflytek.net[root@rg2-test-control001 ~]# nova  aggregate-list
+----+--------+---------------------------------------------+--------------------------------------+
| Id | Name   | Availability Zone                           | UUID                                 |
+----+--------+---------------------------------------------+--------------------------------------+
| 1  | normal | nova                                        | e1b4ddac-957f-49a8-ac08-0054a33edc3d |
| 9  | lvm    | rg2-test-control002.ostack.hfb3.iflytek.net | 1beb7c26-a49e-4bdc-8469-c844c8dac756 |
| 11 | lvm2   | rg2-test-control003.ostack.hfb3.iflytek.net | 1ba48e46-6443-4557-9098-1bff0e0edb0f |
+----+--------+---------------------------------------------+--------------------------------------+

③ 新建卷类型

[root@rg2-test-control001 ~]# openstack volume type create --public --property volume_backend_name='lvm-ssd' lvm-ssd
[root@rg2-test-control001 ~]# openstack volume type create --public --property volume_backend_name='lvm' lvm
[root@rg2-test-control001 ~]# openstack volume type list
+--------------------------------------+---------+-----------+
| ID                                   | Name    | Is Public |
+--------------------------------------+---------+-----------+
| a2a42521-ab9c-42aa-b15a-2af147ce03d2 | lvm-ssd | True      |
| 96a28f6a-c3b6-43fc-bd99-887daa579663 | lvm     | True      |
+--------------------------------------+---------+-----------+
[root@rg2-test-control001 ~]# openstack volume type show  a2a42521-ab9c-42aa-b15a-2af147ce03d2
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| access_project_ids | None                                 |
| description        | None                                 |
| id                 | a2a42521-ab9c-42aa-b15a-2af147ce03d2 |
| is_public          | True                                 |
| name               | lvm-ssd                              |
| properties         | volume_backend_name='lvm-ssd'        |
| qos_specs_id       | None                                 |
+--------------------+--------------------------------------+[root@rg2-test-control001 ~]# openstack volume type show   96a28f6a-c3b6-43fc-bd99-887daa579663
+--------------------+--------------------------------------+
| Field              | Value                                |
+--------------------+--------------------------------------+
| access_project_ids | None                                 |
| description        | None                                 |
| id                 | 96a28f6a-c3b6-43fc-bd99-887daa579663 |
| is_public          | True                                 |
| name               | lvm                                  |
| properties         | volume_backend_name='lvm'            |
| qos_specs_id       | None                                 |
+--------------------+--------------------------------------+

④ 新建卷及云盘

# 1、从镜像创建卷(作为系统盘)，指定availability-zone
[root@rg2-test-control001 ~]# openstack volume create    \ 
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100   \
--type lvm --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \vol-zone2-os1+---------------------+---------------------------------------------+
| Field               | Value                                       |
+---------------------+---------------------------------------------+
| attachments         | []                                          |
| availability_zone   | rg2-test-control002.ostack.hfb3.iflytek.net |
| bootable            | false                                       |
| consistencygroup_id | None                                        |
| created_at          | 2021-12-23T03:49:10.027995                  |
| description         | None                                        |
| encrypted           | False                                       |
| id                  | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b        |
| migration_status    | None                                        |
| multiattach         | False                                       |
| name                | vol-zone2-os1                               |
| properties          |                                             |
| replication_status  | None                                        |
| size                | 100                                         |
| snapshot_id         | None                                        |
| source_volid        | None                                        |
| status              | creating                                    |
| type                | lvm                                         |
| updated_at          | None                                        |
| user_id             | 96f12fa88f0e4bb29123dd33021307ad            |
+---------------------+---------------------------------------------+### 时间问题 耗时约1分钟
相关日志
[root@rg2-test-control00 ~]# tail -f /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log2021-12-23 10:25:29.012 31 INFO cinder.volume.flows.manager.create_volume\[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \dd94c9e5d29048b68b56131a0d6ea2ba - default default] \Volume 1661215d-1302-4065-8e80-7f3e29591742: being created as image with specification:\{'status': u'creating', 'image_location': (u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690',\[{u'url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690', u'metadata': {}}]),\'volume_size': 100, 'volume_name': 'volume-1661215d-1302-4065-8e80-7f3e29591742', \'image_id': '2e3c5301-e2ed-4357-884e-f28ad0428690', 'image_service': \<cinder.image.glance.GlanceImageService object at 0x7fb5f0d09a50>, \'image_meta': {u'status': u'active', u'file': u'/v2/images/2e3c5301-e2ed-4357-884e-f28ad0428690/file',\u'virtual_size': None, u'name': u'Centos-7.4-lvm', u'tags': [], \u'container_format': u'bare', u'created_at': datetime.datetime(2021, 12, 20, 14, 21, 13, tzinfo=<iso8601.Utc>),\u'disk_format': u'raw',\u'locations': [{u'url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690', u'metadata': {}}], \u'visibility': u'public', u'updated_at': datetime.datetime(2021, 12, 20, 14, 21, 37, tzinfo=<iso8601.Utc>), \u'owner': u'dd94c9e5d29048b68b56131a0d6ea2ba', u'protected': False, u'id': u'2e3c5301-e2ed-4357-884e-f28ad0428690', \u'min_ram': 0, u'checksum': u'afcb6cdfb41dd41343c122915c741026',\u'min_disk': 0, u'direct_url': u'file:///var/lib/glance/images/2e3c5301-e2ed-4357-884e-f28ad0428690',\
'properties': {u'hw_qemu_guest_agent': u'yes'}, u'size': 5368709120}}
2021-12-23 10:25:52.487 31 INFO cinder.image.image_utils \
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad\dd94c9e5d29048b68b56131a0d6ea2ba - default default] Image download 5120.00 MB at 218.14 MB/s2021-12-23 10:26:31.357 31 INFO cinder.image.image_utils\[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \dd94c9e5d29048b68b56131a0d6ea2ba - default default] Converted 5120.00 MB image at 172.83 MB/s \
2021-12-23 10:26:33.556 31 INFO cinder.volume.flows.manager.create_volume \[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad \ dd94c9e5d29048b68b56131a0d6ea2ba - default default]  \Volume volume-1661215d-1302-4065-8e80-7f3e29591742 (1661215d-1302-4065-8e80-7f3e29591742): \created successfully \
2021-12-23 10:26:33.561 31 INFO cinder.volume.manager  \
[req-5b949e95-bf05-4065-8423-b9e5496d9e09 96f12fa88f0e4bb29123dd33021307ad  \
dd94c9e5d29048b68b56131a0d6ea2ba - default default] Created volume successfully. # 2、 新建空白卷 耗时秒级 5s以内
[root@rg2-test-control001 ~]#  openstack volume create \
--size 8  --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \test-volume-vdb
+---------------------+---------------------------------------------+
| Field               | Value                                       |
+---------------------+---------------------------------------------+
| attachments         | []                                          |
| availability_zone   | rg2-test-control002.ostack.hfb3.iflytek.net |
| bootable            | false                                       |
| consistencygroup_id | None                                        |
| created_at          | 2021-12-23T03:50:53.717870                  |
| description         | None                                        |
| encrypted           | False                                       |
| id                  | e56a2896-5f59-4789-8ff6-f272683ffbfa        |
| migration_status    | None                                        |
| multiattach         | False                                       |
| name                | test-volume-vdb                             |
| properties          |                                             |
| replication_status  | None                                        |
| size                | 8                                           |
| snapshot_id         | None                                        |
| source_volid        | None                                        |
| status              | creating                                    |
| type                | lvm                                         |
| updated_at          | None                                        |
| user_id             | 96f12fa88f0e4bb29123dd33021307ad            |
+---------------------+---------------------------------------------+# 3、 新建虚机
[root@rg2-test-control001 ~]# openstack flavor list
+--------------------------------------+------------+------+------+-----------+-------+-----------+
| ID                                   | Name       |  RAM | Disk | Ephemeral | VCPUs | Is Public |
+--------------------------------------+------------+------+------+-----------+-------+-----------+
| 4bea2583-f630-430a-95cf-4665414f8a78 | t4.4medium | 4096 |   50 |         0 |     4 | True      |
+--------------------------------------+------------+------+------+-----------+-------+-----------+
[root@rg2-test-control001 ~]# openstack network list
+--------------------------------------+-----------------------+--------------------------------------+
| ID                                   | Name                  | Subnets                              |
+--------------------------------------+-----------------------+--------------------------------------+
| bd23d0fd-e513-4f4e-a901-ef01b0476f43 | Classical_Network_163 | 1a5aefa9-aa0d-40e7-af4e-78d34f146346 |
+--------------------------------------+-----------------------+--------------------------------------+# 这种启动方式 可以保证虚机qemu进程及系统盘、数据盘三者都在同一台机器上
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--block-device source=volume,id=e56a2896-5f59-4789-8ff6-f272683ffbfa,dest=volume,shutdown=preserve \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43   \
--availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \test-zone2-1# 如果不需要数据盘则：
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43   \
--availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \test-zone2-1#######------------------------------如果采用方案二 加入 InstanceLocalityFilter 过滤器--------------
# 此方案适用于新建虚机不建数据盘或者数据盘在其他节点或者其他存储池
# 且volume_local_to_instance = True 则新建虚机时不需要指定zone,新建系统盘时 也不需要指定zone
# 但是新建数据盘需要指定zone 
# 如果新建虚机不加数据盘则直接新建便可
[root@rg2-test-control001 ~]# openstack volume create    \ 
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100   \
--type lvm  vol-zone2-os1   [root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78 \
--boot-volume 76a5ef99-2888-4fb9-99f0-8b3a23e4854b \
--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43   \test-zone2-1
# 如果新建虚机同时需要加入数据盘，则 新建系统盘时 需要指定zone,这样新建数据盘便可以更好的识别zone.
[root@rg2-test-control001 ~]# openstack volume create    \ 
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100   \
--type lvm --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \vol-zone2-os1[root@rg2-test-control001 ~]#  openstack volume create \
--size 8  --availability-zone rg2-test-control002.ostack.hfb3.iflytek.net \test-volume-vdb

2、虚机可以挂本计算节点的数据盘也可以挂载其他计算节点的数据盘。

[root@rg2-test-control001 ~]# nova  volume-attachments 368c291b-981e-4fef-90ce-643397b00870
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | DEVICE   | SERVER ID                            | VOLUME ID                            |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 76a5ef99-2888-4fb9-99f0-8b3a23e4854b | /dev/vda | 368c291b-981e-4fef-90ce-643397b00870 | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b |
| e56a2896-5f59-4789-8ff6-f272683ffbfa | /dev/vdb | 368c291b-981e-4fef-90ce-643397b00870 | e56a2896-5f59-4789-8ff6-f272683ffbfa |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
[root@rg2-test-control001 ~]#
[root@rg2-test-control001 ~]# cinder show  e56a2896-5f59-4789-8ff6-f272683ffbfa
+--------------------------------+-----------------------------------------------------+
| Property                       | Value                                               |
+--------------------------------+-----------------------------------------------------+
| attached_servers               | ['368c291b-981e-4fef-90ce-643397b00870']            |
| attachment_ids                 | ['3bd98c3a-3501-428e-8044-293f028297b1']            |
| availability_zone              | rg2-test-control002.ostack.hfb3.iflytek.net         |
| bootable                       | false                                               |
| consistencygroup_id            | None                                                |
| created_at                     | 2021-12-23T03:50:54.000000                          |
| description                    | None                                                |
| encrypted                      | False                                               |
| id                             | e56a2896-5f59-4789-8ff6-f272683ffbfa                |
| metadata                       | attached_mode : rw                                  |
| migration_status               | None                                                |
| multiattach                    | False                                               |
| name                           | test-volume-vdb                                     |
| os-vol-host-attr:host          | rg2-test-control002.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | None                                                |
| os-vol-mig-status-attr:name_id | None                                                |
| os-vol-tenant-attr:tenant_id   | dd94c9e5d29048b68b56131a0d6ea2ba                    |
| replication_status             | None                                                |
| size                           | 8                                                   |
| snapshot_id                    | None                                                |
| source_volid                   | None                                                |
| status                         | in-use                                              |
| updated_at                     | 2021-12-23T03:53:48.000000                          |
| user_id                        | 96f12fa88f0e4bb29123dd33021307ad                    |
| volume_type                    | lvm                                                 |
+--------------------------------+-----------------------------------------------------+[root@rg2-test-control001 ~]#  openstack volume create --size 8 \--availability-zone rg2-test-control003.ostack.hfb3.iflytek.net  \test-volume-vdb
+---------------------+---------------------------------------------+
| Field               | Value                                       |
+---------------------+---------------------------------------------+
| attachments         | []                                          |
| availability_zone   | rg2-test-control003.ostack.hfb3.iflytek.net |
| bootable            | false                                       |
| consistencygroup_id | None                                        |
| created_at          | 2021-12-23T04:08:01.380129                  |
| description         | None                                        |
| encrypted           | False                                       |
| id                  | 8a6d7913-6253-4f44-b654-186ae87ed886        |
| migration_status    | None                                        |
| multiattach         | False                                       |
| name                | test-volume-vdb                             |
| properties          |                                             |
| replication_status  | None                                        |
| size                | 8                                           |
| snapshot_id         | None                                        |
| source_volid        | None                                        |
| status              | creating                                    |
| type                | lvm                                         |
| updated_at          | None                                        |
| user_id             | 96f12fa88f0e4bb29123dd33021307ad            |
+---------------------+---------------------------------------------+[root@rg2-test-control001 ~]# nova  volume-attach  \368c291b-981e-4fef-90ce-643397b00870 \8a6d7913-6253-4f44-b654-186ae87ed886
+----------+--------------------------------------+
| Property | Value                                |
+----------+--------------------------------------+
| device   | /dev/vdc                             |
| id       | 8a6d7913-6253-4f44-b654-186ae87ed886 |
| serverId | 368c291b-981e-4fef-90ce-643397b00870 |
| volumeId | 8a6d7913-6253-4f44-b654-186ae87ed886 |
+----------+--------------------------------------+
[root@rg2-test-control001 ~]# nova  volume-attachments 368c291b-981e-4fef-90ce-643397b00870
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | DEVICE   | SERVER ID                            | VOLUME ID                            |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| 76a5ef99-2888-4fb9-99f0-8b3a23e4854b | /dev/vda | 368c291b-981e-4fef-90ce-643397b00870 | 76a5ef99-2888-4fb9-99f0-8b3a23e4854b |
| e56a2896-5f59-4789-8ff6-f272683ffbfa | /dev/vdb | 368c291b-981e-4fef-90ce-643397b00870 | e56a2896-5f59-4789-8ff6-f272683ffbfa |
| 8a6d7913-6253-4f44-b654-186ae87ed886 | /dev/vdc | 368c291b-981e-4fef-90ce-643397b00870 | 8a6d7913-6253-4f44-b654-186ae87ed886 |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+

3、虚机可以使用本节点上的HDD做系统盘及云盘，也可以使用本节点上的SSD做系统盘及云盘

# 仅需要将--type lvm 修改为 --type lvm-ssd便可
[root@rg2-test-control003 ~]#  openstack volume create     \
--image 2e3c5301-e2ed-4357-884e-f28ad0428690 --size 100  \--type lvm-ssd vol-zone2-vdb
# 如[root@rg2-test-control003 ~]# cinder show 6e426442-967c-4c32-bb90-e82ffb6428c7
+--------------------------------+-------------------------------------------------------------+
| Property                       | Value                                                       |
+--------------------------------+-------------------------------------------------------------+
| attached_servers               | ['ec164258-db3f-43b2-a7cd-2e62a0c0af41']                    |
| attachment_ids                 | ['59b8212f-87c5-45e3-b587-b76261f4fb25']                    |
| availability_zone              | rg2-test-control003.ostack.hfb3.iflytek.net                 |
| bootable                       | false                                                       |
| consistencygroup_id            | None                                                        |
| created_at                     | 2021-12-22T08:05:16.000000                                  |
| description                    | None                                                        |
| encrypted                      | False                                                       |
| id                             | 6e426442-967c-4c32-bb90-e82ffb6428c7                        |
| metadata                       | attached_mode : rw                                          |
| migration_status               | None                                                        |
| multiattach                    | False                                                       |
| name                           | lvm-ssd-vdb                                                 |
| os-vol-host-attr:host          | rg2-test-control003.ostack.hfb3.iflytek.net@lvm-ssd#lvm-ssd |
| os-vol-mig-status-attr:migstat | None                                                        |
| os-vol-mig-status-attr:name_id | None                                                        |
| os-vol-tenant-attr:tenant_id   | dd94c9e5d29048b68b56131a0d6ea2ba                            |
| replication_status             | None                                                        |
| size                           | 10                                                          |
| snapshot_id                    | None                                                        |
| source_volid                   | None                                                        |
| status                         | in-use                                                      |
| updated_at                     | 2021-12-22T08:15:33.000000                                  |
| user_id                        | 96f12fa88f0e4bb29123dd33021307ad                            |
| volume_type                    | lvm-ssd                                                     |
+--------------------------------+-------------------------------------------------------------+

4、虚机(本地盘)+数据盘 三者在同一计算节点。(在方案1的基础上调整)

① 修改nova-compute配置

[root@rg2-test-control003 ~]# cat /etc/kolla/nova-compute/nova.conf  | grep '\[libvirt\]' -A 10
[libvirt]
connection_uri = qemu+tcp://192.168.162.125/system
#images_rbd_pool = vms
#images_type = rbd
#images_rbd_ceph_conf = /etc/ceph/ceph.conf
#rbd_user = nova
#rbd_secret_uuid = df5fbfad-02c7-43b2-9938-900c80d6bc99
#images_type = lvm
#images_volume_group=cinder-volumes1

# 新建虚机
[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78\--block-device source=image,id=2e3c5301-e2ed-4357-884e-f28ad0428690,size=50,bootindex=0,dest=local \--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 instance_lvm001[root@rg2-test-control001 ~]# nova boot --flavor 4bea2583-f630-430a-95cf-4665414f8a78\--image 2e3c5301-e2ed-4357-884e-f28ad0428690 \--nic net-id=bd23d0fd-e513-4f4e-a901-ef01b0476f43 instance_lvm002# 新建空白卷，根据新建的虚机分配的zone[root@rg2-test-control001 ~]#  openstack volume migrate  \--host rg2-test-control001.ostack.hfb3.iflytek.net  d3ef7ec0-0f43-43ba-b4cc-e1136e182a30[root@rg2-test-control001 ~]# nova volume-attach \f72d7b2e-12f4-4857-89d5-177bc845e15f d3ef7ec0-0f43-43ba-b4cc-e1136e182a30[root@rg2-test-control001 ~]# nova   volume-attachments f72d7b2e-12f4-4857-89d5-177bc845e15f
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| ID                                   | DEVICE   | SERVER ID                            | VOLUME ID                            |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+
| d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 | /dev/vdb | f72d7b2e-12f4-4857-89d5-177bc845e15f | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 |
+--------------------------------------+----------+--------------------------------------+--------------------------------------+

---

5、虚机迁移方案

#冷迁移(cold migration)，也叫静态迁移。关闭电源的虚拟机进行迁移。通过冷迁移，\
#可以选择将关联的磁盘从一个数据存储移动到另一个数据存储。#好处：虚拟机不需要位于共享存储器上，数据丢失率小。#缺点：需要关闭电源，业务中断。#虚拟机的迁移，就是数据的转移，如果计算节点之间没有共享存储，所以要转移的数据包括两部分：#1、静态数据：存储在本地的虚拟机的镜像文件，包括后端镜像(libvirt Base)和虚拟机单独的增量镜像文件(libvirt instance disks)。#2、动态数据：内存里虚拟机的运行时数据，内存里的数据是动态变化的数据，虚拟机里运行的负载的大小直接影响迁移的时间长短。

① 共享存储与非共享存储

# 虚拟机的数据存在共享磁盘上(Shared storage-based live migration)，迁移只需要完成内存数据的迁移。
# 虚拟机的数据存在本地磁盘(block migration)，需要对镜像文件和内存数据同时迁移。

② 冷迁移

# 冷迁移实现方法有多种，例如有快照来迁移实例、实例文件迁移。以文件迁移为例，完成冷迁移。
# 虚机冷迁移步骤：#1、关闭虚拟机#2、找到虚拟机位于/var/lib/nova/instances下文件#3、将虚拟机的文件全部copy到目标主机的相同位置下#4、修改用户组#5、更新数据库中host,node字段为目标主机的名字#6、重启目标主机的nova-compute服务# 测试虚机
[root@rg2-test-control001 ~]# nova   live-migration --block-migrate f72d7b2e-12f4-4857-89d5-177bc845e15f
# 迁移卷 
[root@rg2-test-control001 ~]# cinder  migrate  \
d3ef7ec0-0f43-43ba-b4cc-e1136e182a30 rg2-test-control001.ostack.hfb3.iflytek.net@lvm#lvm[root@rg2-test-control001 ~]# cinder show   d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
+--------------------------------+-----------------------------------------------------+
| Property                       | Value                                               |
+--------------------------------+-----------------------------------------------------+
| attached_servers               | ['f72d7b2e-12f4-4857-89d5-177bc845e15f']            |
| attachment_ids                 | ['e59dbb3c-0f5b-443d-84c4-c81519c03794']            |
| availability_zone              | rg2-test-control003.ostack.hfb3.iflytek.net         |
| bootable                       | false                                               |
| consistencygroup_id            | None                                                |
| created_at                     | 2022-01-04T03:36:46.000000                          |
| description                    | None                                                |
| encrypted                      | False                                               |
| id                             | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30                |
| metadata                       | attached_mode : rw                                  |
| migration_status               | migrating                                           | #状态
| multiattach                    | False                                               |
| name                           | test-volume-vdb                                     |
| os-vol-host-attr:host          | rg2-test-control003.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | migrating                                           |
| os-vol-mig-status-attr:name_id | None                                                |
| os-vol-tenant-attr:tenant_id   | dd94c9e5d29048b68b56131a0d6ea2ba                    |
| replication_status             | None                                                |
| size                           | 8                                                   |
| snapshot_id                    | None                                                |
| source_volid                   | None                                                |
| status                         | in-use                                              |
| updated_at                     | 2022-01-04T03:45:37.000000                          |
| user_id                        | 96f12fa88f0e4bb29123dd33021307ad                    |
| volume_type                    | lvm                                                 |
+--------------------------------+-----------------------------------------------------++--------------------------------+-----------------------------------------------------+
[root@rg2-test-control001 ~]# cinder show   d3ef7ec0-0f43-43ba-b4cc-e1136e182a30
+--------------------------------+-----------------------------------------------------+
| Property                       | Value                                               |
+--------------------------------+-----------------------------------------------------+
| attached_servers               | ['f72d7b2e-12f4-4857-89d5-177bc845e15f']            |
| attachment_ids                 | ['0903c19d-c1e9-4a8e-a376-46eeb8eabc15']            |
| availability_zone              | rg2-test-control003.ostack.hfb3.iflytek.net         |
| bootable                       | false                                               |
| consistencygroup_id            | None                                                |
| created_at                     | 2022-01-04T03:36:46.000000                          |
| description                    | None                                                |
| encrypted                      | False                                               |
| id                             | d3ef7ec0-0f43-43ba-b4cc-e1136e182a30                |
| metadata                       | attached_mode : rw                                  |
| migration_status               | success                                             | #状态
| multiattach                    | False                                               |
| name                           | test-volume-vdb                                     |
| os-vol-host-attr:host          | rg2-test-control001.ostack.hfb3.iflytek.net@lvm#lvm |
| os-vol-mig-status-attr:migstat | success                                             |
| os-vol-mig-status-attr:name_id | 44d67e42-f17d-4114-8b64-569f07b2d9e6                | #新盘
| os-vol-tenant-attr:tenant_id   | dd94c9e5d29048b68b56131a0d6ea2ba                    |
| replication_status             | None                                                |
| size                           | 8                                                   |
| snapshot_id                    | None                                                |
| source_volid                   | None                                                |
| status                         | in-use                                              |
| updated_at                     | 2022-01-04T03:47:44.000000                          |
| user_id                        | 96f12fa88f0e4bb29123dd33021307ad                    |
| volume_type                    | lvm                                                 |
+--------------------------------+-----------------------------------------------------+

---

采用的最终方案

经过验证及当期场景，采用如下方案：
4、虚机(本地盘)+数据盘 三者在同一计算节点。(在方案1的基础上调整)#环境准备：
# ① 需要单独为nova准备一个lv，用来存放虚机系统盘及数据,可以单独VG，也可以和数据盘混用一个VG
# ② 挂载到指定节点 /var/lib/docker/volumes/nova_compute/_data
# ③ 关于数据盘问题: 单独新建一个vg，用来存储数据盘，每一个数据盘都是一个独立的LV，迁移后会生成新的LV。
#   参考该字段value: os-vol-mig-status-attr:name_id[root@rg2-test-control003 ~]#  lvcreate -L 1024G -n lv_nova_instances cinder-volumes1Logical volume "lv_nova_instances" created.
[root@rg2-test-control003 ~]#  mkfs.xfs /dev/cinder-volumes1/lv_nova_instances
meta-data=/dev/cinder-volumes1/lv_nova_instances isize=512    agcount=4, agsize=67108864 blks=                       sectsz=512   attr=2, projid32bit=1=                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=268435456, imaxpct=5=                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=131072, version=2=                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0[root@rg2-test-control003 nova_compute]# mv _data _data1
[root@rg2-test-control003 nova_compute]# mount /dev/cinder-volumes1/lv_nova_instances \
/var/lib/docker/volumes/nova_compute/_data
[root@rg2-test-control003 nova_compute]# mv _data1/*  _data/
[root@rg2-test-control003 nova_compute]# docker restart nova_libvirt nova_compute(nova-compute)[root@rg2-test-control003 instances]# ll
total 4
drwxr-xr-x  2 nova nova 73 Jan  4 15:22 1acd4814-314c-4482-803a-8d74fbd3d5f2
drwxr-xr-x. 2 nova nova  6 Dec 21 00:49 a01ffde2-4131-450d-a83b-09ced05035c0
drwxr-xr-x. 2 nova nova 54 Jan  4 10:55 _base
drwxr-xr-x  2 nova nova 44 Dec 23 10:14 cc932408-2806-4798-85f4-b0946a207c68
-rw-r--r--. 1 nova nova 65 Jan  4 14:58 compute_nodes
drwxr-xr-x  2 nova nova 61 Dec 23 18:01 f29a8284-f4ad-473d-90b6-6b3f0c73869f
drwxr-xr-x  2 nova nova 73 Jan  4 15:22 f34b4532-d07b-40fa-83fe-91972bcf9f5c
drwxr-xr-x  2 nova nova 73 Jan  4 11:40 f72d7b2e-12f4-4857-89d5-177bc845e15f
drwxr-xr-x. 2 nova nova 93 Jan  4 10:54 locks
(nova-compute)[root@rg2-test-control003 instances]# cd 1acd4814-314c-4482-803a-8d74fbd3d5f2
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# ls
console.log  disk  disk.config  disk.info
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# du -sh *
0       console.log
26M     disk
460K    disk.config
4.0K    disk.info(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]# df -Th
Filesystem                                     Type      Size  Used Avail Use% Mounted on
overlay                                        overlay   223G  207G   17G  93% /
tmpfs                                          tmpfs      63G     0   63G   0% /sys/fs/cgroup
devtmpfs                                       devtmpfs   63G     0   63G   0% /dev
tmpfs                                          tmpfs      63G     0   63G   0% /dev/shm
tmpfs                                          tmpfs      63G  486M   63G   1% /run
tmpfs                                          tmpfs      13G     0   13G   0% /run/user/0
/dev/sda2                                      xfs       223G  207G   17G  93% /etc/iscsi
/dev/mapper/cinder--volumes1-lv_nova_instances xfs       1.0T   11G 1014G   1% /var/lib/nova #存储侧
(nova-compute)[root@rg2-test-control003 1acd4814-314c-4482-803a-8d74fbd3d5f2]#

[root@rg2-test-control001 ~]# docker ps -a | grep nova
276b84b58fd1        mmwei.com/lokolla/centos-source-nova-compute:stable-15.0.6-ccr-1.2.1                    "kolla_start"            2 days ago          Up 2 days                               nova_compute
ad16ad4c615d        mmwei.com/lokolla/centos-source-nova-libvirt:15.0.6-ccr-1.1.4                           "kolla_start"            3 days ago          Up 3 days                               nova_libvirt
6dcdb7423867        mmwei.com/lokolla/centos-source-nova-novncproxy:stable-15.0.6-ccr-1.1.4                 "kolla_start"            4 days ago          Up 4 days                               nova_novncproxy
1b1c7451ff36        mmwei.com/lokolla/centos-source-nova-consoleauth:15.0.6-ccr-1.1.2                       "kolla_start"            4 days ago          Up 4 days                               nova_consoleauth
eb19b7296467        mmwei.com/lokolla/centos-source-nova-placement-api:15.0.6-ccr-1.1.2                     "kolla_start"            4 days ago          Up 4 days                               placement_api
b43c7de31f2e        mmwei.com/lokolla/centos-source-nova-scheduler:15.0.6-ccr-1.1.2                         "kolla_start"            4 days ago          Up 4 days                               nova_scheduler
8fdd76b9e13a        mmwei.com/lokolla/centos-source-nova-conductor:stable-15.0.6-ccr-1.1.10                 "kolla_start"            4 days ago          Up 4 days                               nova_conductor
6609b35806ae        mmwei.com/lokolla/centos-source-nova-api:stable-15.0.6-ccr-1.2.1                        "kolla_start"            4 days ago          Up 4 days                               nova_api
abad2cdb9be2        mmwei.com/lokolla/centos-source-nova-ssh:15.0.6-ccr-1.1.4                               "kolla_start"            4 days ago          Up 4 days                               nova_ssh
[root@rg2-test-control001 ~]# 

集群环境

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 223.1G  0 disk 
├─sda1   8:1    0   500M  0 part /boot
└─sda2   8:2    0 222.6G  0 part /
sdb      8:16   0   1.8T  0 disk 
sdc      8:32   0   1.8T  0 disk 
sdd      8:48   0   2.2T  0 disk 
sde      8:64   0   2.2T  0 disk 
sdf      8:80   0   2.2T  0 disk 
sdg      8:96   0   2.2T  0 disk 
sdh      8:112  0   2.2T  0 disk 
sdi      8:128  0   2.2T  0 disk 
sdj      8:144  0   2.2T  0 disk 
sdk      8:160  0   2.2T  0 disk 两块ssd做raid18块sata 做raid10 其中 raid10 做1个VG，划分出一个1T的lvm 用来存放虚机系统盘数据，其余做数据盘