先說一下環(huán)境：

一、硬件

８臺(tái)服務(wù)器做的超融合架構(gòu)，軟件存儲(chǔ)池，
每臺(tái)服務(wù)器是９６Ｇ內(nèi)存，兩顆Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz，３２線程。
每臺(tái)服務(wù)器是１Ｔ?。樱樱募尤龎K４Ｔ　ＳＡＴＡ，無ｒａｉｄ卡，做的兩副本。
服務(wù)器集成２個(gè)１Ｇ的網(wǎng)口，做端口聚合，對(duì)外提供服務(wù)。
另加一塊四端口千兆網(wǎng)卡，做端口聚合，做內(nèi)部數(shù)據(jù)傳輸。

二、軟件

基礎(chǔ)系統(tǒng)是Ubuntu 16.04 LTS，基于ＫＶＭ，libvirt的虛擬化管理平臺(tái)。
虛擬機(jī)：ｗｉｎｄｏｗｓ７中文專業(yè)版，３２位，
每臺(tái)虛擬機(jī)分配３０７１Ｍ內(nèi)存，兩顆ＣＰＵ(2 sockets) 。
libvirt：1.3.1

三、現(xiàn)象

服務(wù)器上的windows，不定時(shí)的會(huì)卡死，多數(shù)每天發(fā)生一次到兩次，少數(shù)三四天卡死一次，極少數(shù)會(huì)間隔時(shí)間更久，
多數(shù)發(fā)生在電腦空閑時(shí)間，
能上互聯(lián)網(wǎng)的虛擬機(jī)多發(fā)，
此外無其他明顯規(guī)律。

四、問題排查

１、先是排查了服務(wù)器配置，終端的配置，網(wǎng)絡(luò)

因?yàn)檫@套架構(gòu)已經(jīng)在多地部署，并穩(wěn)定運(yùn)行多年，排查完客戶的現(xiàn)場(chǎng)網(wǎng)絡(luò)后，此原因基本排除。

２、后來懷疑存儲(chǔ)的問題

使用iostat持續(xù)檢測(cè)兩臺(tái)服務(wù)器的三個(gè)小時(shí)。

iostat -x -m 1

從記錄下的日志看服務(wù)器存儲(chǔ)的平均響應(yīng)時(shí)間一直在正常范圍，讀寫負(fù)載也不高。

３、檢查服務(wù)器的日志

默認(rèn)libvirt的日志是關(guān)閉的，編輯/etc/libvirt/libvirtd.conf，打開warnning級(jí)別的日志記錄。

log_level = 3
log_outputs="3:file:/var/log/libvirt/libvirtd.log"

從/var/log/libvirt/libvirtd.log看libvirt日志：

2017-03-15 06:03:08.894+0000: 13919: info : libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200)
2017-03-15 06:03:08.894+0000: 13919: info : hostname: sc7
2017-03-15 06:03:08.894+0000: 13919: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'
2017-03-15 06:03:48.180+0000: 13914: error : qemuMonitorIO:689 : internal error: End of file from monitor
2017-03-15 06:03:48.180+0000: 13914: error : virNetClientProgramDispatchError:177 : Cannot open log file: '/var/log/libvirt/qemu/vm105.log': Device or resource busy
2017-03-15 06:04:22.540+0000: 13918: error : qemuMigrationCheckJobStatus:2641 : operation failed: migration job: unexpectedly failed
2017-03-15 06:04:57.546+0000: 13918: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'
2017-03-15 06:06:12.507+0000: 13916: error : qemuMigrationCheckJobStatus:2641 : operation failed: migration job: unexpectedly failed
2017-03-15 06:11:21.182+0000: 13918: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'
2017-03-15 06:13:08.112+0000: 13920: error : qemuMigrationCheckJobStatus:2641 : operation failed: migration job: unexpectedly failed
2017-03-15 06:15:09.467+0000: 13919: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'
2017-03-15 06:35:17.680+0000: 13914: error : qemuMonitorIO:689 : internal error: End of file from monitor
2017-03-15 06:35:17.680+0000: 13914: error : virNetClientProgramDispatchError:177 : Cannot open log file: '/var/log/libvirt/qemu/vm102.log': Device or resource busy

2017-03-16 00:33:08.324+0000: 13914: error : virNetClientProgramDispatchError:177 : Cannot open log file: '/var/log/libvirt/qemu/vm105.log': Device or resource busy
2017-03-16 02:47:26.712+0000: 13919: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'
2017-03-16 02:52:03.184+0000: 13914: error : qemuMonitorIO:689 : internal error: End of file from monitor
2017-03-16 02:52:03.184+0000: 13914: error : virNetClientProgramDispatchError:177 : Cannot open log file: '/var/log/libvirt/qemu/vm113.log': Device or resource busy
2017-03-16 02:52:26.683+0000: 13919: warning : virStorageBackendVolOpen:1535 : ignoring permission error for '/media/vmpool/win7_test.img'

看到日志中有一些virStorageBackendVolOpen，virNetClientProgramDispatchError，qemuMonitorIO的報(bào)錯(cuò)（重復(fù)部分略去），但都不是關(guān)鍵問題，不會(huì)導(dǎo)致虛擬機(jī)死機(jī)。

從/var/log/libvirt/qemu/里看虛擬機(jī)的日志：

2017-03-16 02:55:46.227+0000: starting up libvirt version: 1.3.1, package: 1ubuntu10.5 (Stefan Bader <stefan.bader@canonical.com> Thu, 06 Oct 2016 13:07:20 +0200), qemu version: 2.5.0 (Debian 1:2.5+dfsg-5ubuntu10.8), hostname: sc7
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin QEMU_AUDIO_DRV=spice /usr/bin/kvm-spice -name vm113 -S -machine pc-i440fx-xenial,accel=kvm,usb=off -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 3072 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid 3fec386e-23ee-4853-b46a-37feb9ee3cf9 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-vm113/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -boot order=cd,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/media/vmpool/vm113.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/media/vmdpool/vmd113.qcow2,format=qcow2,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-1-1,readonly=on -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=34,id=hostnet0,vhost=on,vhostfd=42 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:a6:3a:68,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -spice port=5908,addr=0.0.0.0,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on
char device redirected to /dev/pts/1 (label charserial0)
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support requested feature: CPUID.01H:EDX.acpi [bit 22]
warning: host doesn't support requested feature: CPUID.01H:EDX.ht [bit 28]
warning: host doesn't support requested feature: CPUID.01H:EDX.tm [bit 29]
warning: host doesn't support requested feature: CPUID.01H:EDX.pbe [bit 31]
warning: host doesn't support requested feature: CPUID.01H:ECX.dtes64 [bit 2]
warning: host doesn't support requested feature: CPUID.01H:ECX.monitor [bit 3]
warning: host doesn't support requested feature: CPUID.01H:ECX.ds_cpl [bit 4]
warning: host doesn't support requested feature: CPUID.01H:ECX.smx [bit 6]
warning: host doesn't support requested feature: CPUID.01H:ECX.est [bit 7]
warning: host doesn't support requested feature: CPUID.01H:ECX.tm2 [bit 8]
warning: host doesn't support requested feature: CPUID.01H:ECX.xtpr [bit 14]
warning: host doesn't support requested feature: CPUID.01H:ECX.pdcm [bit 15]
warning: host doesn't support requested feature: CPUID.01H:ECX.dca [bit 18]
warning: host doesn't support requested feature: CPUID.01H:ECX.osxsave [bit 27]
warning: host doesn't support requested feature: CPUID.01H:EDX.ds [bit 21]
warning: host doesn't support requested feature: CPUID.01H:EDX.acpi [bit 22]
warning: host doesn't support requested feature: CPUID.01H:EDX.ht [bit 28]
warning: host doesn't support requested feature: CPUID.01H:EDX.tm [bit 29]
warning: host doesn't support requested feature: CPUID.01H:EDX.pbe [bit 31]
warning: host doesn't support requested feature: CPUID.01H:ECX.dtes64 [bit 2]
warning: host doesn't support requested feature: CPUID.01H:ECX.monitor [bit 3]
warning: host doesn't support requested feature: CPUID.01H:ECX.ds_cpl [bit 4]
warning: host doesn't support requested feature: CPUID.01H:ECX.smx [bit 6]
warning: host doesn't support requested feature: CPUID.01H:ECX.est [bit 7]
warning: host doesn't support requested feature: CPUID.01H:ECX.tm2 [bit 8]
warning: host doesn't support requested feature: CPUID.01H:ECX.xtpr [bit 14]
warning: host doesn't support requested feature: CPUID.01H:ECX.pdcm [bit 15]
warning: host doesn't support requested feature: CPUID.01H:ECX.dca [bit 18]
warning: host doesn't support requested feature: CPUID.01H:ECX.osxsave [bit 27]
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 15.188000 ms, bitrate 120548590 bps (114.964094 Mbps)
inputs_connect: inputs channel client create
red_dispatcher_set_cursor_peer: 
red_channel_client_disconnect: rcc=0x55f0084c69c0 (channel=0x55f007faa6f0 type=3 id=0)
red_channel_client_disconnect: rcc=0x7f9120236180 (channel=0x7f912020b990 type=2 id=0)
red_channel_client_disconnect: rcc=0x7f912022ff60 (channel=0x7f912020bf60 type=4 id=0)
red_peer_receive: Connection reset by peer
red_channel_client_disconnect: rcc=0x55f009402710 (channel=0x55f007f9f160 type=1 id=0)
main_channel_client_on_disconnect: rcc=0x55f009402710
snd_receive: Connection reset by peer
red_channel_client_disconnect_dummy: rcc=0x55f00919e910 (channel=0x55f00925ab90 type=6 id=0)
snd_channel_put: SndChannel=0x55f0087d8c50 freed
snd_receive: Connection reset by peer
red_channel_client_disconnect_dummy: rcc=0x55f0092349b0 (channel=0x55f00925a8a0 type=5 id=0)
snd_channel_put: SndChannel=0x55f008af3550 freed
red_client_destroy: destroy client 0x55f009496b80 with #channels=6
red_dispatcher_disconnect_cursor_peer: 
red_dispatcher_disconnect_display_peer: 
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 16.994000 ms, bitrate 115386782 bps (110.041410 Mbps)
inputs_connect: inputs channel client create

中間的warnning信息是libvirt遷移虛擬機(jī)時(shí)添加的cpu feature不支持。也沒有會(huì)導(dǎo)致死機(jī)的關(guān)鍵的錯(cuò)誤。

4、分析虛擬機(jī)本身的問題

該虛擬機(jī)經(jīng)過用戶的修改之后，使用了多款優(yōu)化工具進(jìn)行優(yōu)化，出于管理的目的，手動(dòng)調(diào)整了一些系統(tǒng)的設(shè)置，做了一些權(quán)限上的修改。

從ｗｉｎｄｏｗｓ里拷出崩潰時(shí)的內(nèi)存轉(zhuǎn)儲(chǔ)文件。memory.dmp。
運(yùn)行windbg，按ctrl+s配置symboal地址：

SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols

打開dmp文件：

Microsoft (R) Windows Debugger Version 6.11.0001.404 X86
Copyright (c) Microsoft Corporation. All rights reserved.


Loading Dump File [C:\dmp\(98)MEMORY.DMP]
Kernel Summary Dump File: Only kernel address space is available

Symbol search path is: SRV*C:\Symbols*http://msdl.microsoft.com/download/symbols;SRV*c:\mysymbol* http://msdl.microsoft.com/download/symbols
Executable search path is: 
Windows 7 Kernel Version 7601 (Service Pack 1) MP (2 procs) Free x86 compatible
Product: WinNt, suite: TerminalServer SingleUserTS
Built by: 7601.18205.x86fre.win7sp1_gdr.130708-1532
Machine Name:
Kernel base = 0x83c13000 PsLoadedModuleList = 0x83d5c4d0
Debug session time: Wed Mar 15 13:11:54.777 2017 (GMT+8)
System Uptime: 0 days 5:00:52.740
Loading Kernel Symbols
...............................................................
................................................................
.........................
Loading User Symbols

Loading unloaded module list
.......

可以看出系統(tǒng)是運(yùn)行了５個(gè)小時(shí)之后死機(jī)的。
在windbg里執(zhí)行：

0: kd> !analyze -v
*******************************************************************************
*                                                                             *
*                        Bugcheck Analysis                                    *
*                                                                             *
*******************************************************************************

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 00000061, Clock interrupt time out interval in nominal clock ticks.
Arg2: 00000000, 0.
Arg3: 807a3120, The PRCB address of the hung processor.
Arg4: 00000001, 0.

Debugging Details:
------------------

*** ERROR: Module load completed but symbols could not be loaded for Ntfs.sys
*** ERROR: Module load completed but symbols could not be loaded for qutmdrv.sys

BUGCHECK_STR:  CLOCK_WATCHDOG_TIMEOUT_2_PROC

DEFAULT_BUCKET_ID:  VISTA_DRIVER_FAULT

PROCESS_NAME:  System

CURRENT_IRQL:  1c

STACK_TEXT:  
807fc5e4 83c8e117 00000101 00000061 00000000 nt!KeBugCheckEx+0x1e
807fc620 83c8d763 00026160 00000000 0011a800 nt!KeAccumulateTicks+0x242
807fc660 83c8d610 83c4a2be 2e50d92a 00000000 nt!KeUpdateRunTime+0x145
807fc6b8 83c8ce13 8736e71b 8736e71b 000000d1 nt!KeUpdateSystemTime+0x613
807fc6b8 83c4a2be 8736e71b 8736e71b 000000d1 nt!KeUpdateSystemTimeAssist+0x13
807fc75c 83c93268 94484000 00000000 807fc7b4 nt!KeFlushMultipleRangeTb+0x2d5
807fc770 83c70ca0 00001000 876424e8 00000000 nt!MiFlushPteList+0x22
807fc838 83c9acdb 94485000 00001000 94272250 nt!MmSetAddressRangeModified+0x205
807fc8cc 8c0ebd3f 8763f998 00000000 00000000 nt!CcFlushCache+0x294
WARNING: Stack unwind information not available. Following frames may be wrong.
807fc91c 8c0f2dfc 87604da8 942670f8 00000000 Ntfs+0x7ed3f
807fc990 8c0ecd92 87604da8 87baa428 0c757330 Ntfs+0x85dfc
807fc9f8 83c49c1e 87361020 87baa428 87baa428 Ntfs+0x7fd92
807fca10 843ce20c 8735a300 87baa428 00000000 nt!IofCallDriver+0x63
807fca34 843ce3cb 807fca54 8735a300 00000000 FLTMGR!FltpLegacyProcessingAfterPreCallbacksCompleted+0x2aa
807fca6c 83c49c1e 8735a300 87baa428 874e9db0 FLTMGR!FltpDispatch+0xc5
807fca84 937684db 807fcaa4 83c49c1e 874e9db0 nt!IofCallDriver+0x63
807fca8c 83c49c1e 874e9db0 87baa428 87baa428 qutmdrv+0x4db
807fcaa4 83e3dbe9 87baa428 87646c98 00000000 nt!IofCallDriver+0x63
807fcac4 83e1c4bb 874e9db0 87646c98 00000000 nt!IopSynchronousServiceTail+0x1f8
807fcb30 83c508c6 87f0a590 807fcbc8 807fcbd0 nt!NtFlushBuffersFile+0x1d7
807fcb30 83c4e7ad 87f0a590 807fcbc8 807fcbd0 nt!KiSystemServicePostCall
807fcbb0 83e22b5d 80000564 807fcbc8 863e7798 nt!ZwFlushBuffersFile+0x11
807fcbd0 83eab328 8ce1a740 00000000 00000000 nt!CmpFileFlush+0x5a
807fcc00 83c9014b 00000000 00000000 863e7798 nt!CmpLazyFlushWorker+0x35
807fcc50 83e1c12b 00000001 b2d8e202 00000000 nt!ExpWorkerThread+0x10d
807fcc90 83cc3559 83c9003e 00000001 00000000 nt!PspSystemThreadStartup+0x9e
00000000 00000000 00000000 00000000 00000000 nt!KiThreadStartup+0x19


STACK_COMMAND:  kb

SYMBOL_NAME:  ANALYSIS_INCONCLUSIVE

FOLLOWUP_NAME:  MachineOwner

MODULE_NAME: Unknown_Module

IMAGE_NAME:  Unknown_Image

DEBUG_FLR_IMAGE_TIMESTAMP:  0

FAILURE_BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_2_PROC_ANALYSIS_INCONCLUSIVE

BUCKET_ID:  CLOCK_WATCHDOG_TIMEOUT_2_PROC_ANALYSIS_INCONCLUSIVE

Followup: MachineOwner

看到關(guān)鍵的錯(cuò)誤信息了：

CLOCK_WATCHDOG_TIMEOUT (101)
An expected clock interrupt was not received on a secondary processor in an
MP system within the allocated interval. This indicates that the specified
processor is hung and not processing interrupts.
Arguments:
Arg1: 00000061, Clock interrupt time out interval in nominal clock ticks.
Arg2: 00000000, 0.
Arg3: 807a3120, The PRCB address of the hung processor.
Arg4: 00000001, 0.

CLOCK_WATCHDOG_TIMEOUT
就是這個(gè)了。
看ｍｓｄｎ上解釋：
https://msdn.microsoft.com/en-us/library/windows/hardware/ff557211(v=vs.85).aspx
就是在多處理器的情況中，第二顆處理器沒有及時(shí)響應(yīng)系統(tǒng)的中斷請(qǐng)求。
此外就沒有更多的信息了。

五、解決問題

問題找到了。
嘗試的解決辦法：

１、將虛擬機(jī)改成單ＣＰＵ

觀察了一段時(shí)間，故障消失，但這是治標(biāo)不治本的辦法。

2、將虛擬機(jī)改成單 socket雙核心的模式

（windows７默認(rèn)最多支持雙路ｃｐｕ，kvm默認(rèn)分配兩顆vcpu時(shí)是雙路）

#libvirt xml <CPU>
 <topology sockets=‘1‘ cores=‘2‘ threads=‘1‘/>

觀察無效。

3、bugzilla.redhat.com里發(fā)現(xiàn)：

Bug 784836 - new cpu's flags, to control hyper-v related features

• "hv_relaxed" to depress some ugly Windows watchdog timer, and enable "relaxed" timing

直覺感覺和這個(gè)ＢＳＯＤ相關(guān)。

libvirt xml里增加：

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>


......
.....


  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv_vendor_id=xeon'/>
  </qemu:commandline>




</domain>

觀察一晚上，問題解決。

這時(shí)從命令行看qemu的執(zhí)行參數(shù)如下：

libvirt+  4573  3.1  3.3 6576684 3363956 ?     Sl   06:53  26:10 qemu-system-x86_64 -enable-kvm -name vm84 -S -machine pc-i440fx-xenial,accel=kvm,usb=off -cpu SandyBridge,+pdpe1gb,+osxsave,+dca,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme -m 3072 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 1bae86e4-4e9e-44cb-a8ae-2a8a87142f75 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-vmtest/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime -no-shutdown -boot order=cd,menu=on,strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/media/vmpool/vmtest.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0 -drive file=/media/vmdpool/vmdtest.qcow2,format=qcow2,if=none,id=drive-virtio-disk1 -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x7,drive=drive-virtio-disk1,id=virtio-disk1 -drive if=none,id=drive-ide0-1-1,readonly=on -device ide-cd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=24,id=hostnet0,vhost=on,vhostfd=40 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:6c:0f:62,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -spice port=5906,addr=0.0.0.0,disable-ticketing,seamless-migration=on -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2 -device intel-hda,id=sound0,bus=pci.0,addr=0x6 -device hda-duplex,id=sound0-codec0,bus=sfaxianound0.0,cad=0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -cpu host,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,kvm=off,hv_vendor_id=xeon -msg timestamp=on

可以看到啟用了：hv_relaxed 。

六、補(bǔ)充

后來發(fā)現(xiàn)libvirt-1.1.1之后開始支持hyper-v的特性，HV開頭。

在libvirt xml <feature>里增加：

    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>

也可以起到同樣的效果。

七、參考資料

https://bugzilla.redhat.com/show_bug.cgi?id=784836
http://www.windowswally.com/how-to-fix-the-clock_watchdog_timeout-error/
https://msdn.microsoft.com/en-us/library/windows/hardware/ff557211(v=vs.85).aspx
http://lists.nongnu.org/archive/html/qemu-devel/2014-02/msg00335.html