图文精华

About云-梭伦科技»专题 › 交流区 › OpenStack系列 › [解决后端使用ceph qem性能问题]（升级kvm后使用virt-ty ...
分享

[解决后端使用ceph qem性能问题]（升级kvm后使用virt-type=kvm（qemu-system-x86_6...

s060403072 发表于 2015-4-7 18:47:51 [显示全部楼层] 只看大图 回帖奖励 阅读模式 关闭右栏 0 30557
About云VIP帮助找工作和提升
[解决后端使用ceph qem性能问题]（升级kvm后使用virt-type=kvm（qemu-system-x86_64、qemu-kvm、qemu-img）

由于升级libvirt、qemu后没有qemu-kvm模块。nova配置文件配置为 virt-type=kvm将会报错、配置为qemu则性能低。
关于kvm、和qemu类型选择请参考博文：qemu中KVM硬件虚拟化的初始化分析 (tcg、xen、kvm、qtest)

本次测试环境配置： 
  1. horizon 的ip 192.168.10.71

  2. 其他组件ip 192.168.10.191
复制代码


1、使用virt-type=qemu启动的虚拟机：

1.png 
  1. root 24502 17.6 0.4 2530112 307192 ? Sl 21:51 1:23 /usr/bin/qemu-system-x86_64 -name instance-00000001 -S -machine pc-i440fx-1.6,accel=tcg,usb=off -m 512 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 8ee337e0-4808-49e7-9e66-45dc4a7b1f40 -smbios type=1,manufacturer=Red Hat Inc.,product=OpenStack Nova,version=2014.1.1-3.el6,serial=44454c4c-3600-1058-8033-b3c04f463232,uuid=8ee337e0-4808-49e7-9e66-45dc4a7b1f40 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/usr/local/libvirt/var/lib/libvirt/qemu/instance-00000001.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=rbd:liunova/8ee337e0-4808-49e7-9e66-45dc4a7b1f40_disk:id=admin:key=AQBlwSxU0HsWBxAAMfhhWANPIIDihk7JTFunKQ==:auth_supported=cephx\;none:mon_host=192.168.10.186\:6789\;192.168.10.187\:6789\;192.168.10.188\:6789\;192.168.10.189\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=18,id=hostnet0 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=fa:16:3e:99:7c:a0,bus=pci.0,addr=0x3 -chardev file,id=charserial0,path=/var/lib/nova/instances/8ee337e0-4808-49e7-9e66-45dc4a7b1f40/console.log -device isa-serial,chardev=charserial0,id=serial0 -chardev pty,id=charserial1 -device isa-serial,chardev=charserial1,id=serial1 -device usb-tablet,id=input0 -spice port=5900,addr=0.0.0.0,disable-ticketing,seamless-migration=on -k en-us -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5
复制代码




很明显看到上面使用的是/usr/bin/qemu-system-x86_64。

看看：/usr/bin/qemu-system-x86_64 --help

2.png

2、virt-type=kvm用openstack生成虚拟机，报错：
  1. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8] Traceback (most recent call last):

  2. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1305, in _build_instance

  3. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     set_access_ip=set_access_ip)

  4. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 393, in decorated_function

  5. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     return function(self, context, *args, **kwargs)

  6. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1717, in _spawn

  7. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     LOG.exception(_('Instance failed to spawn'), instance=instance)

  8. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__

  9. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     six.reraise(self.type_, self.value, self.tb)

  10. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 1714, in _spawn

  11. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     block_device_info)

  12. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 2265, in spawn

  13. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     block_device_info)

  14. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3656, in _create_domain_and_network

  15. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     power_on=power_on)

  16. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3559, in _create_domain

  17. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     domain.XMLDesc(0))

  18. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/openstack/common/excutils.py", line 68, in __exit__

  19. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     six.reraise(self.type_, self.value, self.tb)

  20. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/nova/virt/libvirt/driver.py", line 3554, in _create_domain

  21. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     domain.createWithFlags(launch_flags)

  22. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 183, in doit

  23. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     result = proxy_call(self._autowrap, f, *args, **kwargs)

  24. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 141, in proxy_call

  25. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     rv = execute(f, *args, **kwargs)

  26. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 122, in execute

  27. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     six.reraise(c, e, tb)

  28. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib/python2.6/site-packages/eventlet/tpool.py", line 80, in tworker

  29. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     rv = meth(*args, **kwargs)

  30. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]   File "/usr/lib64/python2.6/site-packages/libvirt.py", line 727, in createWithFlags

  31. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8]     if ret == -1: raise libvirtError ('virDomainCreateWithFlags() failed', dom=self)

  32. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee1tafd7f8] libvirtError: internal error: process exited while connecting to monitor: char device redirected to /dev/pts/4

  33. 2014-11-18 21:53:55.498 25214 TRACE nova.compute.manager [instance: 5b50ed5b-c3d9-40cd-889d-0aee14afd7f8] q%mu-kvm: -drive file=rbd:liunova/5b50ed5b-c3d9-40cd-889d-0aee14afd7f8_disk:id=admin:key=AQBlwSxU0HsWBxAAMfhhWANPIIDihk7JTFunKQ==:auth_supported=cephx\;none:mon_host=192.168.10.186\:6789\;192.168.10.187\:6789\;192.168.10.188\:6789\;192.168.10.189\:6789,if=none,id=drive-virtio-disk0,format=raw,cache=none%sA could not open disk image rbd:liunova/5b50ed5b-c3d9-40cd-889d-0aee14afd7f8_disk:id=admin:key=AQBlwSxU0HsWBxAAMfhhWANPIIDihk7JTFunKQ==:auth_supported=cephx\;none:mon_host=192.168.10.186\:6789\;192.168.10.187\:6789\;192.168.10.188\:6789\;192.168.10.189\:6789: No such file or directory
复制代码



3.png

3、关于qemu-system-x86_64、qemu-kvm、qemu-img
图3.1


4.png

查看上面两个qemu-img对应的版本及支持格式：
  1. /usr/bin/qemu-img | grep -E "version|Supported"
复制代码


图3.2

5.png 

  1. /usr/local/qemu/bin/qemu-img | grep -E  "version|Supported"
复制代码


图3.3

6.png



在回到上述错误：
图3.4

7.png

此处启动用的qemu-kvm：

查找qemu-kvm：
图3.5

8.png

查看qemu-kvm版本：
图3.6

9.png

再找到对应的0.12.1版本的qemu-img（图3.2），发现不支持rbd（ceph后端使用需要rbd支持）。（注：图3.3qemu-img支持rbd。）

通过上面分析总结一下：
/usr/bin/qemu-system-x86_64与/usr/local/qemu/bin/qemu-img是配套的。版本为：version 1.6.1（手动编译1.6.1 qemu的源码包之后版本）

/usr/libexec/qemu-kvm与/usr/bin/qemu-img是配套的。版本为：version 0.12.1（为升级前版本）



这样就可以解释图3.4的错误了，因为图3.4使用的是V0.12.1，而V0.12.1的qemu-img没有编译支持rbd的模块，因此出现图3.4错误。

解决办法：
将图3.4中启动的qemu-kvm替换为新版version 1.6.1的/usr/bin/qemu-system-x86_64来启动虚拟机。

操作步骤：
  1. mv /usr/libexec/qemu-kvm /usr/libexec/qemu-kvm.bak
复制代码



再次生成虚拟机：
10.png

发现虚拟机生成成功。

再来看看底册启动参数：
  1. ps aux | grep qemu
复制代码



11.png

发现后面生成的虚拟机，将nova.conf配置为virt-type=kvm之后。再将原生自带的qemu-kvm移除掉则能正常生成虚拟机。且生成的虚拟机使用了kvm虚拟机（accel=kvm），提高了性能。

结论：
  1. 1、当nova.conf配置为virt-type=qemu时候，会使用qemu软件特性模拟生成虚拟机，一次性能降低。

  2. 

  3. 2、当nova.conf配置为virt-type=kvm时候，首先回去查找qemu-kvm模块。如果存在qemu-kvm模块则直接使用此模块启动虚拟机。如果qemu-kvm模块找不到，则会去寻找qemu-system-x86_64，找到则用qemu-system-x86_64启动虚拟机，并使用kvm虚拟机特性。及开启--enable-kvm特性参数
复制代码



注：kvm虚拟化模块：

  1. 在老版本中有单独的qemu-kvm模块存在，结合qemu一起做虚拟机工作。在后续新版本中，已经将qemu-kvm模块完全合并到qemu中去。因此当需要使用kvm特性时候，只需要qemu-system-x86_64 启动命令中增加参数 --enable-kvm参数使能即可。
复制代码








回复

使用道具 举报

自定义广告语

没找到任何评论，期待你打破沉寂

高级模式
B Color Image Link Quote Code Smilies
您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

发表新帖
s060403072

高级会员

关注

  • 53

    主题

  • 592

    帖子

  • 18

    粉丝

TA的主题
关闭

推荐上一条 /2 下一条

关于我们· 联系我们· 加入我们· 小黑屋· 合作伙伴   

Copyright © 2001-2024 About云-梭伦科技 Powered by Discuz! X3.4 Licensed Discuz Team.

简书 / Smrz 京ICP备2020039040号 Wxb 简书网举报电话：021-34700000