openstack虚拟机占用CPU高定位排查

简介: openstack虚拟机占用CPU高定位排查

起因

使用top的时候发现有一个进程占用很高,不难看出这是一台虚拟机  

[root@node10 ~]# top -c 
top - 09:56:38 up 21 days, 16:40,  1 user,  load average: 4.60, 4.56, 4.59
Tasks: 1229 total,   1 running, 599 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.1 us,  1.3 sy,  0.0 ni, 95.6 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52795177+total, 38830100 free, 46541856+used, 23703120 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 53799924 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                   
3631246 qemu      20   0 54.513g 564816  23032 S 432.8  0.1  40708:18 /usr/libexec/qemu-kvm -name guest=instance-000010e1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-51-instance-000010e1/ma+
3626859 qemu      20   0 54.463g 829640  23272 S  19.2  0.2  12322:27 /usr/libexec/qemu-kvm -name guest=instance-000010de,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-49-instance-000010de/ma+
3627164 qemu      20   0 54.456g 923188  22340 S  18.9  0.2  11907:12 /usr/libexec/qemu-kvm -name guest=instance-000010d5,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-50-instance-000010d5/ma+
3626589 qemu      20   0 54.433g 869120  22984 S  18.5  0.2  11899:47 /usr/libexec/qemu-kvm -name guest=instance-000010d8,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-48-instance-000010d8/ma+
   2395 root      10 -10 7707908 265212  14004 S   3.0  0.1 864:38.19 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/+
   3833 neutron   20   0  345824  84400  10012 S   2.0  0.0 630:14.43 /usr/bin/python2 /usr/bin/neutron-lbaasv2-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /usr/share/neutron/neutron-lbaas-dist.conf --config-file+
   3846 neutron   20   0  328252  66856   9996 S   2.0  0.0 719:21.04 /usr/bin/python2 /usr/bin/neutron-openvswitch-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /etc/neutron+
   3835 neutron   20   0  319756  58260   9968 S   1.7  0.0 632:00.95 /usr/bin/python2 /usr/bin/neutron-l3-agent --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/l3_agent --config-file /etc/neutron/neutron+
   4375 nova      20   0 3381644 238220  31192 S   1.7  0.0 913:57.44 /usr/bin/python2 /usr/bin/nova-compute                                                                                                                                    
2146459 qemu      20   0 67.497g 551796  22052 S   1.3  0.1 166:53.77 /usr/libexec/qemu-kvm -name guest=instance-000010ba,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-34-instance-000010ba/ma+
   3893 ceph      20   0 3113128 1.358g  24804 S   1.0  0.3 581:34.38 /usr/bin/ceph-osd -f --cluster ceph --id 40 --setuser ceph --setgroup ceph                                                                                                
1481773 qemu      20   0 5731192 421528  21148 S   1.0  0.1 119:18.05 /usr/libexec/qemu-kvm -name guest=instance-0000109c,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-32-instance-0000109c/ma+
   3882 root      20   0  362240  16064   3620 S   0.7  0.0  60:42.74 /usr/sbin/zabbix_agentd: forward listener #1 [waiting for connection]                                                                                                     
   3895 ceph      20   0 2520352 1.182g  23984 S   0.7  0.2 800:07.02 /usr/bin/ceph-osd -f --cluster ceph --id 37 --setuser ceph --setgroup ceph                                                                                                
3489135 root      20   0  149772   5292   3280 R   0.3  0.0   0:00.80 top -c                                                                      
## 小提示
top之后按 P 是以CPU占用排序,按 M 是以内存占用排序 

经过

找到了这个进程之后就得去查看虚拟机状态了

[root@node10 ~]# virsh dominfo instance-000010e1
Id:             51
Name:           instance-000010e1
UUID:           d8194be5-b464-4a31-9bc1-7d2e04bc0841
OS Type:        hvm
State:          running
CPU(s):         12
CPU time:       2443906.6s
Max memory:     54525952 KiB
Used memory:    54525952 KiB
Persistent:     yes
Autostart:      disable
Managed save:   no
Security model: none
Security DOI:   0
## 通过这个命令可以查询到 UUID ,nova中uuid是唯一的
## 然后就可以通过nova命令来查询确认状态了
[root@node10 ~]# nova show d8194be5-b464-4a31-9bc1-7d2e04bc0841
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Property                             | Value                                                                                                                                                                                                                                  |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| OS-DCF:diskConfig                    | MANUAL                                                                                                                                                                                                                                 |
| OS-EXT-AZ:availability_zone          | zs                                                                                                                                                                                                                               |
| OS-EXT-SRV-ATTR:host                 | node10                                                                                                                                                                                                                                 |
| OS-EXT-SRV-ATTR:hostname             | dy-0                                                                                                                                                                                                                                   |
| OS-EXT-SRV-ATTR:hypervisor_hostname  | node10                                                                                                                                                                                                                                 |
| OS-EXT-SRV-ATTR:instance_name        | instance-000010e1                                                                                                                                                                                                                      |
| OS-EXT-SRV-ATTR:kernel_id            |                                                                                                                                                                                                                                        |
| OS-EXT-SRV-ATTR:launch_index         | 0                                                                                                                                                                                                                                      |
| OS-EXT-SRV-ATTR:ramdisk_id           |                                                                                                                                                                                                                                        |
| OS-EXT-SRV-ATTR:reservation_id       | r-976ib014                                                                                                                                                                                                                             |
| OS-EXT-SRV-ATTR:root_device_name     | /dev/sda                                                                                                                                                                                                                               |
| OS-EXT-SRV-ATTR:user_data            | -                                                                                                                                                                                                                                      |
| OS-EXT-STS:power_state               | 1                                                                                                                                                                                                                                      |
| OS-EXT-STS:task_state                | -                                                                                                                                                                                                                                      |
| OS-EXT-STS:vm_state                  | active                                                                                                                                                                                                                                 |
| OS-SRV-USG:launched_at               | 2023-04-28T01:55:41.000000                                                                                                                                                                                                             |
| OS-SRV-USG:terminated_at             | -                                                                                                                                                                                                                                      |
| accessIPv4                           |                                                                                                                                                                                                                                        |
| accessIPv6                           |                                                                                                                                                                                                                                        |
| attached_iso                         | -                                                                                                                                                                                                                                      |
| attached_usb                         | []                                                                                                                                                                                                                                     |
| config_drive                         | True                                                                                                                                                                                                                                   |
| created                              | 2023-04-28T01:55:26Z                                                                                                                                                                                                                   |
| description                          | -                                                                                                                                                                                                                                      |
| distribution_user_id                 | -                                                                                                                                                                                                                                      |
| flavor                               | image01 (65fccbf5-423c-44ad-826c-4ac09e059eb0)                                                                                                                                                                                      |
| floating_ips                         | []                                                                                                                                                                                                                                     |
| hostId                               | 005e2fb7c75ec9942d79802a9ed8f63b73544a2babee0cd2bb41d08a                                                                                                                                                                               |
| host_status                          | UP                                                                                                                                                                                                                                     |
| hot_add_info                         | {"use_vcpus": 0, "use_memory": 0, "numa_memory_info": [], "max_memory": 1024, "max_memory_slot": 255, "max_vcpus": 128, "use_memory_slot": 0}                                                                                          |
| id                                   | d8194be5-b464-4a31-9bc1-7d2e04bc0841                                                                                                                                                                                                   |
| image                                | 发明家 (2a927fb1-3606-43cf-ba8a-669d16a10228)                                                                                                                                                                                          |
| key_name                             | -                                                                                                                                                                                                                                      |
| kms_server                           | -                                                                                                                                                                                                                                      |
| ldap                                 | False                                                                                                                                                                                                                                  |
| ldap_server                          | -                                                                                                                                                                                                                                      |
| life_cycle                           | -                                                                                                                                                                                                                                      |
| locked                               | False                                                                                                                                                                                                                                  |
| logout_time                          | 0                                                                                                                                                                                                                                      |
| metadata                             | {"reserve_pci_resources": "true"}                                                                                                                                                                                                      |
| name                                 | dy-0                                                                                                                                                                                                                                   |
| numa_topology                        | {"numa_info": [{"node": 0, "virtual_node": 0}, {"node": 1, "virtual_node": 1}], "use_numa": 2}                                                                                                                                         |
| os-extended-volumes:volumes_attached | [{"device": "/dev/sda", "delete_on_termination": true, "id": "e7678c71-ee57-48ae-923f-8b09f0b04e3b", "size": 200}, {"device": "/dev/sdc", "delete_on_termination": false, "id": "67ffbb38-40c8-4cbd-b26d-5ab3ac9c544c", "size": 1024}] |
| os-pci:pci_devices                   | [{"vendor_id": "10de", "vendor_name": "NVIDIA Corporation", "product_id": "2204", "id": 370, "product_name": "GA102 [GeForce RTX 3090]"}]                                                                                              |
| os-pci:request_pci_devices           | [{"count": 1, "vendor_name": "NVIDIA Corporation", "vendor_id": "10de", "product_name": "GA102 [GeForce RTX 3090]", "product_id": "2204"}]                                                                                             |
| pool                                 | False                                                                                                                                                                                                                                  |
| progress                             | 0                                                                                                                                                                                                                                      |
| security_groups                      | default                                                                                                                                                                                                                                |
| serial_number                        | -                                                                                                                                                                                                                                      |
| status                               | ACTIVE                                                                                                                                                                                                                                 |
| tags                                 | []                                                                                                                                                                                                                                     |
| tenant_id                            | e377f6bbc0e04fa0862e1a98bc62fe69                                                                                                                                                                                                       |
| updated                              | 2023-04-28T02:43:16Z                                                                                                                                                                                                                   |
| user_id                              | 0e9ca51a76df4b1b8c4944c7a11a2d75                                                                                                                                                                                                       |
| vlan1021 network                     | 10.10.21.216                                                                                                                                                                                                                           |
+--------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+  

当然还有一种定位虚拟机的方法

[root@node10 ~]# virsh  edit instance-000010e1 
Domain instance-000010e1 XML configuration not changed.
## 在edit页面找到 uuid
<uuid>d8194be5-b464-4a31-9bc1-7d2e04bc0841</uuid>
## 在edit页面找到 nova:name
<nova:name>dy-0</nova:name>
## uuid可以直接 nova show , 名字可以通过 dashboard 去查询 

处理

找到虚拟机之后直接进入

发现有一个已经运行完成的进程消耗CPU资源很高,关掉这个进程,过了一会再看发现CPU占用已经降下去了

[root@node10 ~]# top -c 
top - 10:16:59 up 21 days, 17:00,  1 user,  load average: 0.80, 2.20, 3.63
Tasks: 1219 total,   1 running, 596 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.4 us,  0.4 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem : 52795177+total, 38820768 free, 46538003+used, 23750980 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 53799416 avail Mem 
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                   
3626859 qemu      20   0 54.463g 829640  23272 S  20.9  0.2  12326:28 /usr/libexec/qemu-kvm -name guest=instance-000010de,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-49-instance-000010de/ma+
3631246 qemu      20   0 54.513g 536928  23032 S  18.5  0.1  40775:08 /usr/libexec/qemu-kvm -name guest=instance-000010e1,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-51-instance-000010e1/ma+
3627164 qemu      20   0 54.456g 923188  22340 S  17.5  0.2  11911:04 /usr/libexec/qemu-kvm -name guest=instance-000010d5,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-50-instance-000010d5/ma+
3626589 qemu      20   0 54.433g 869120  22984 S  16.2  0.2  11903:38 /usr/libexec/qemu-kvm -name guest=instance-000010d8,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-48-instance-000010d8/ma+
   2395 root      10 -10 7707908 265212  14004 S   3.0  0.1 865:13.35 ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --no-chdir --log-file=/var/log/openvswitch/ovs-vswitchd.log --pidfile=/+


目录
相关文章
|
2月前
CPU过高问题定位
CPU过高问题定位
31 0
|
2月前
|
监控 Java 索引
cpu使用率过高和jvm old占用过高排查过程
cpu使用率过高和jvm old占用过高排查过程
86 2
|
2月前
|
设计模式 监控 安全
如何定位当生产环境CPU飙升的时候的问题
在当今的信息化时代,计算机系统在各行各业都发挥着重要的作用。然而,当生产环境中的CPU飙升时,系统性能会受到影响,甚至导致整个系统瘫痪。这不仅会对企业造成经济损失,还会对用户体验造成严重影响。因此,如何定位并解决生产环境中CPU飙升的问题,已成为众多企业和开发人员亟待解决的问题之一。本文旨在探讨如何定位生产环境中CPU飙升的问题,并提供相应的解决方案。通过了解CPU飙升的原因、定位方法以及解决方案,企业和开发人员可以更好地应对生产环境中出现的CPU飙升问题,提高系统性能和用户体验。
99 1
|
22天前
|
缓存 弹性计算 监控
云服务器 CPU 使用率高的问题排查与优化
云服务器 CPU 使用率高的问题排查与优化
51 0
|
27天前
|
虚拟化 iOS开发 MacOS
客户机操作系统已禁用 CPU。请关闭或重置虚拟机。解决方法
客户机操作系统已禁用 CPU。请关闭或重置虚拟机。解决方法
196 0
|
1月前
|
SQL Java Linux
Linux系统cpu飙升到100%排查方案
Linux系统cpu飙升到100%排查方案
22 0
|
2月前
|
Java Linux
Linux下如何定位最耗CPU的JAVA代码
Linux下如何定位最耗CPU的JAVA代码
53 0
|
2月前
|
存储 弹性计算 运维
|
2月前
|
Java Shell
java中jvm使用jststak定位线程cpu占用内存高的线程
java中jvm使用jststak定位线程cpu占用内存高的线程
28 5
|
2月前
|
存储 弹性计算 运维