The errors seen in the dsmerror.log were:
07/22/2014 13:27:59 ANS9365E VMware vStorage API error for virtual machine '<vm hostname>'.
TSM function name : visdkCreateVmSnapshotMoRef
TSM file : vmvisdk.cpp (5416)
API return code : 67
API error message : Another task is already in progress.
07/22/2014 13:27:59 ANS5250E An unexpected error was encountered.
TSM function name : vmVddkFullVMPrePareToOpenVMDKs
TSM function : snapshot targetMoRefP is null
TSM return code : 115
TSM file : ..\..\common\vm\vmbackvddk.cpp (12439)
07/22/2014 13:27:59 ANS1228E Sending of object '<vm hostname>' failed.
07/22/2014 13:27:59 ANS4015E Error processing '<vm hostname>': unexpected TSM error (115)
The problem was at the vm level. We found a hung snap vmtools install preventing the snapshot execution.
Resolution: On the vCenter console we manaully stopped the VMtools install and were ableto manually execute a TSM4VE back-up
Wednesday, July 23, 2014
Friday, July 11, 2014
ANS1311E Server out of Storage space -- TSM4VE back-ups fail
The Tivoli Storage Manager for Virtual Environments (TSM4VE) is a complex infrastructure to support and trouble-shoot.
Recently, this problem re-occured and so I thought that I'd give back to the TSM community online that has helped me so many times resolve problems.
TSM4VE back-ups can fail for many reasons. If you manuallly run the back-up, then you can retrieve helpful errors OR you cna look at the dsmerror.log. Many times the errors are generic enough to confused admins who have little experience.
Our TSM4VE back-ups were failing with the error message:
TSM4VE back-ups fail ANS1311E Server out of Storage space
Now, TSM4VE back-ups use 2 storagepools: one for data and one for vmware control files.
We use a Falconstor VTL to store data and a diskpool.
So, first check the VTL for scratch volumes by logging into dsadmc:
q libv vlib1
VLIB1 C1128OL4 Scratch 3,929 LTO
VLIB1 C1128PL4 Scratch 3,930 LTO
VLIB1 C1128QL4 Scratch 3,931 LTO
VLIB1 C1128RL4 Scratch 3,932 LTO
VLIB1 C1128SL4 Scratch 3,933 LTO
VLIB1 C1128TL4 Scratch 3,934 LTO
q libv vlib2
VLIB2 C2120BL4 Scratch 3,628 LTO
VLIB2 C2120CL4 Scratch 3,629 LTO
VLIB2 C2120DL4 Scratch 3,630 LTO
VLIB2 C2120EL4 Scratch 3,631 LTO
VLIB2 C2120FL4 Scratch 3,632 LTO
VLIB2 C2120GL4 Scratch 3,633 LTO
Well, have scratches so there is plenty of space for the VMs' data.
Next, we checked to see if we have space for the VMs' control files.
q stgpool VMCTLPOOL
Storage Device Estimated Pct Pct High Low Next Stora-
Pool Name Class Name Capacity Util Migr Mig Mig ge Pool
Pct Pct
----------- ---------- ---------- ----- ----- ---- --- -----------
VMCTLPOOL DISK 1,418 G 100.00 100.00 99 94
So, there is no space to store new VM control files.
So we created another volume in the VMCTLPOOL.
def volume VMCTLPOOL /vmctl1/stg33.dsm formatsize=42709
After, this completed back-ups stalled started sessions with the TSM server and transferred their back-up.
Recently, this problem re-occured and so I thought that I'd give back to the TSM community online that has helped me so many times resolve problems.
TSM4VE back-ups can fail for many reasons. If you manuallly run the back-up, then you can retrieve helpful errors OR you cna look at the dsmerror.log. Many times the errors are generic enough to confused admins who have little experience.
Our TSM4VE back-ups were failing with the error message:
TSM4VE back-ups fail ANS1311E Server out of Storage space
Now, TSM4VE back-ups use 2 storagepools: one for data and one for vmware control files.
We use a Falconstor VTL to store data and a diskpool.
So, first check the VTL for scratch volumes by logging into dsadmc:
q libv vlib1
VLIB1 C1128OL4 Scratch 3,929 LTO
VLIB1 C1128PL4 Scratch 3,930 LTO
VLIB1 C1128QL4 Scratch 3,931 LTO
VLIB1 C1128RL4 Scratch 3,932 LTO
VLIB1 C1128SL4 Scratch 3,933 LTO
VLIB1 C1128TL4 Scratch 3,934 LTO
q libv vlib2
VLIB2 C2120BL4 Scratch 3,628 LTO
VLIB2 C2120CL4 Scratch 3,629 LTO
VLIB2 C2120DL4 Scratch 3,630 LTO
VLIB2 C2120EL4 Scratch 3,631 LTO
VLIB2 C2120FL4 Scratch 3,632 LTO
VLIB2 C2120GL4 Scratch 3,633 LTO
Well, have scratches so there is plenty of space for the VMs' data.
Next, we checked to see if we have space for the VMs' control files.
q stgpool VMCTLPOOL
Storage Device Estimated Pct Pct High Low Next Stora-
Pool Name Class Name Capacity Util Migr Mig Mig ge Pool
Pct Pct
----------- ---------- ---------- ----- ----- ---- --- -----------
VMCTLPOOL DISK 1,418 G 100.00 100.00 99 94
So, there is no space to store new VM control files.
So we created another volume in the VMCTLPOOL.
def volume VMCTLPOOL /vmctl1/stg33.dsm formatsize=42709
After, this completed back-ups stalled started sessions with the TSM server and transferred their back-up.
Subscribe to:
Posts (Atom)