记一次vmware虚拟机无法添加网卡的处理过程
Vcenter版本 VMware VirtualCenter, version: 6.0.0, build: build-2559268
最近遇到一个问题,在给新开的机器添加网卡时发现,无法正常添加报错
A general system error occurred: vDS host error: see faultCause
错误如下图:
报错详细信息
Cannot create DVPort 14930 of VDS POD21-dvSwitch on the host pod21-clu03-h014.cdscloud.local
A general system error occurred:
大致问题是给虚拟机分配端口ID没有成功,手工测试了不是所有的portgroup不行
经过简单的排查,没有看到具体有用的错误信息,发现VDS有几台宿主机没有同步
VDS out of sync
针对不同步问题,强制主机同步,和重启大法都不行,我们开始怀疑可能数据库有脏数据导致,
官方和网上都没有相关的原因说明,正常的话这个应该是正常同步的。重启主机总有那么几台是不同步的
为了确认影响范围,我们写了一个powercli脚本来确认那些portgroup不可以
!=======代码片段
#测试每个portgroup,虚拟机尝试添加每个portgroup确认是否有问题
#失败的日志保存位置
[string]$Out_Errlog_File = "d:\alex\errorlog.txt"
#连接到vmware vcenter服务器
Connect-VIServer -Server vcenter地址 -Protocol https -User "用户名" -Password "密码"
#获取所有的portgroup对象
$myVDPortGroup = Get-VDPortgroup
#遍历所有portgroup尝试添加到虚拟机上
foreach($portgroup in $myVDPortGroup){
#获得虚拟机的网卡对象
$myNetworkAdapters = Get-VM test-2018-10-30 | Get-NetworkAdapter -Name "Network adapter 1"
#尝试修改虚拟机网卡的portgroup
Set-NetworkAdapter -NetworkAdapter $myNetworkAdapters -Portgroup $portgroup -Confirm:$false
if(!$?)
{
#失败记录到文件中
Write-output "portgroup_name:$portgroup" | Out-File -Append $Out_Errlog_File
}
}
!===============
通过上面的脚本跑完,我们发现一共有六个portgroup有相同的问题
针对怀疑的数据库脏数据问题,对数据库又是一顿的折腾,
我们使用的MSSQL数据库,使用SQL的profiler工具抓取trace把VC的操作过程给截取了下来,
经过分析,在给虚拟机添加网卡,数据库执行了两个操作
第一:读取表,返回空的portid
整理后的相关SQL语句
select TOP 1 t1.DVPORT_KEY
from VPX_DVPORT_MEMBERSHIP t1 left outer join VPX_DVPORT t2 on t1.DVS_ID = t2.DVS_ID and t1.DVPORT_KEY = t2.DVPORT_KEY
where t1.DVPORTGROUP_ID = 674 and t1.HOST_ID = 0 and t2.CONNECTEE is NULL
第二步:相关信息写入到表中
整理后的相关SQL语句
UPDATE VPX_DVPORT SET PROXY_HOST = NULL , CONNECTION_COOKIE = 49790401 , PERSISTENCE_LOCATION = 'ds:///vmfs/volumes/5a568853-7af66eed-2d80-6c92bf82278c//.dvsData/a7 7f 3f 50 6c e1 71 e0-2f ad 3c d3 16 86 b7 b4/14930' ,
CONNECTEE = 'vm-6091' , CONNECTEE_TYPE = 'vmVnic' , CONNECTEE_NIC_KEY = '4000' , CONNECTEE_ADDRESS = NULL ,
RESERVED_TIME = 1541069175987649 , POWERING_ON_FLG = 0 , LAST_CONNECTEE = NULL WHERE DVS_ID = 35 AND DVPORT_KEY = 14930
把数据库复制一份出来,测试插入语句执行没有异常,通过观察trace文件发现,VC最后把修改的数据又修改回去了,对于怀疑数据库脏数据问题步成立。
我们重新建portgroup设置相同VLAN,可以正常使用,
计划下一步把六个有问题的portgroup全部替换掉
操作完成后神奇的发现VDS的同步也正常了,各种测试都正常了
这个问题最终也没有确定到问题,但最终通过替换大法,恢复了正常,特此分享出来,如有入到相同问题可以进行尝试下。
怀疑是触发了BUG照成,我们截取到了部分相关的日志如下:
2018-10-30T23:54:32.176+08:00 info vpxd[06104] [Originator@6876 sub=pbm opID=7284884A-00000085-e8] PBMCallback: PbmFunctionTracer::PbmFunctionTracer: Entering PbmCallBack::PreProcessReconfigureSpec
2018-10-30T23:54:32.177+08:00 info vpxd[06104] [Originator@6876 sub=pbm opID=7284884A-00000085-e8] PBMCallback: PbmFunctionTracer::~PbmFunctionTracer: Leaving PbmCallBack::PreProcessReconfigureSpec
2018-10-30T23:54:32.266+08:00 error vpxd[06660] [Originator@6876 sub=hostMethod] [HostMethodDispatcher::ProcessTaskResult] Host call [applyDVPort]for host [host-3991] failed with exception [vmodl.fault.SystemError]
2018-10-30T23:54:32.266+08:00 error vpxd[06104] [Originator@6876 sub=DvsCore opID=7284884A-00000085-e8] [DvsKeeper::GetTaskFault] API [applyDVPort] on host [pod21-clu03-h013.cdscloud.local] failed with exception [vmodl.fault.SystemError]
2018-10-30T23:54:32.275+08:00 error vpxd[06104] [Originator@6876 sub=VmProv opID=7284884A-00000085-e8 opName=create vmName=test-2018-10-30] Unexpec
2018-10-30T23:54:32.176+08:00 info vpxd[06104] [Originator@6876 sub=pbm opID=7284884A-00000085-e8] PBMCallback: PbmFunctionTracer::PbmFunctionTracer: Entering PbmCallBack::PreProcessReconfigureSpec
2018-10-30T23:54:32.177+08:00 info vpxd[06104] [Originator@6876 sub=pbm opID=7284884A-00000085-e8] PBMCallback: PbmFunctionTracer::~PbmFunctionTracer: Leaving PbmCallBack::PreProcessReconfigureSpec
2018-10-30T23:54:32.266+08:00 error vpxd[06660] [Originator@6876 sub=hostMethod] [HostMethodDispatcher::ProcessTaskResult] Host call [applyDVPort]for host [host-3991] failed with exception [vmodl.fault.SystemError]
2018-10-30T23:54:32.266+08:00 error vpxd[06104] [Originator@6876 sub=DvsCore opID=7284884A-00000085-e8] [DvsKeeper::GetTaskFault] API [applyDVPort] on host [pod21-clu03-h013.cdscloud.local] failed with exception [vmodl.fault.SystemError]
2018-10-30T23:54:32.275+08:00 error vpxd[06104] [Originator@6876 sub=VmProv opID=7284884A-00000085-e8 opName=create vmName=test-2018-10-30] Unexpected exception (vmodl.fault.SystemError) while reconfiguring VM. Aborting.
2018-10-30T23:54:32.279+08:00 error vpxd[06104] [Originator@6876 sub=VmProv opID=7284884A-00000085-e8 opName=create vmName=test-2018-10-30] Backtrace:
--> [backtrace begin] product: VMware VirtualCenter, version: 6.0.0, build: build-2559268, tag: vpxd
--> backtrace[00] vmacore.dll[0x001BC4DA]
--> backtrace[01] vmacore.dll[0x0005B99F]
--> backtrace[02] vmacore.dll[0x0005CACE]
--> backtrace[03] vmacore.dll[0x00012C3B]
--> backtrace[04] vmomi.dll[0x0011CDD2]
--> backtrace[05] vpxd.exe[0x00DABAD9]
--> backtrace[06] vpxd.exe[0x00DB751A]
--> backtrace[07] vpxd.exe[0x00CCC8B1]
--> backtrace[08] vpxd.exe[0x00C4CBD0]
--> backtrace[09] vpxd.exe[0x0091BAEF]
--> backtrace[10] vpxd.exe[0x0091CE61]
--> backtrace[11] types.dll[0x0008E0F6]
--> backtrace[12] vpxd.exe[0x0007580C]
--> backtrace[13] vmomi.dll[0x0004B41C]
--> backtrace[14] vpxd.exe[0x000AF412]
--> backtrace[15] vpxd.exe[0x0008614B]
--> backtrace[16] vpxd.exe[0x00089A74]
--> backtrace[17] vpxd.exe[0x0009A4A8]
--> backtrace[18] vpxd.exe[0x000A1243]
--> backtrace[19] vmacore.dll[0x0015081E]
--> backtrace[20] vmacore.dll[0x0015405F]
--> backtrace[21] vmacore.dll[0x001555B1]
--> backtrace[22] vmacore.dll[0x001572E5]
--> backtrace[23] vmacore.dll[0x00064F4B]
--> backtrace[24] vmacore.dll[0x001527F0]
--> backtrace[25] vmacore.dll[0x001D059B]
--> backtrace[26] MSVCR90.dll[0x00002FDF]
--> backtrace[27] MSVCR90.dll[0x00003080]
--> backtrace[28] kernel32.dll[0x0001652D]
--> backtrace[29] ntdll.dll[0x0002C521]
--> [backtrace end]
-->
- 本文标签: 杂谈
- 本文链接: http://www.iamlk.cn/article/103
- 版权声明: 本文由Leonidax原创发布,转载请遵循《署名-非商业性使用-相同方式共享 4.0 国际 (CC BY-NC-SA 4.0)》许可协议授权