Wednesday, July 23, 2014

ANS9365E TSM4VE back-up failed visdkCreateVmSnapshotMoRef API return code 67

The errors seen in the dsmerror.log were:
07/22/2014 13:27:59 ANS9365E VMware vStorage API error for virtual machine '<vm hostname>'.
TSM function name : visdkCreateVmSnapshotMoRef
TSM file : vmvisdk.cpp (5416)
API return code : 67
API error message : Another task is already in progress.
07/22/2014 13:27:59 ANS5250E An unexpected error was encountered.
TSM function name : vmVddkFullVMPrePareToOpenVMDKs
TSM function : snapshot targetMoRefP is null
TSM return code : 115
TSM file : ..\..\common\vm\vmbackvddk.cpp (12439)
07/22/2014 13:27:59 ANS1228E Sending of object '<vm hostname>' failed.
07/22/2014 13:27:59 ANS4015E Error processing '<vm hostname>': unexpected TSM error (115)

The problem was at the vm level. We found a hung snap vmtools install preventing the snapshot execution.
Resolution: On the vCenter console we manaully stopped the VMtools install and were ableto manually execute a TSM4VE back-up

Friday, July 11, 2014

ANS1311E Server out of Storage space -- TSM4VE back-ups fail

The Tivoli Storage Manager for Virtual Environments (TSM4VE) is a complex infrastructure to support and trouble-shoot.
Recently, this problem re-occured and so I thought that I'd give back to the TSM community online that has helped me so many times resolve problems.

TSM4VE back-ups can fail for many reasons. If you manuallly run the back-up, then you can retrieve helpful errors OR you cna look at the dsmerror.log.  Many times the errors are generic enough to confused admins who have little experience.

Our TSM4VE back-ups were failing with the error message:
TSM4VE back-ups fail  ANS1311E Server out of Storage space

Now, TSM4VE back-ups use 2 storagepools: one for data and one for vmware control files.
We use a Falconstor VTL to store data and a diskpool.
So, first check the VTL for scratch volumes by logging into dsadmc:
q libv vlib1
VLIB1        C1128OL4    Scratch                               3,929   LTO
VLIB1        C1128PL4    Scratch                               3,930   LTO
VLIB1        C1128QL4    Scratch                               3,931   LTO
VLIB1        C1128RL4    Scratch                               3,932   LTO
VLIB1        C1128SL4    Scratch                               3,933   LTO
VLIB1        C1128TL4    Scratch                               3,934   LTO
q libv vlib2
VLIB2        C2120BL4    Scratch                               3,628   LTO
VLIB2        C2120CL4    Scratch                               3,629   LTO
VLIB2        C2120DL4    Scratch                               3,630   LTO
VLIB2        C2120EL4    Scratch                               3,631   LTO
VLIB2        C2120FL4    Scratch                               3,632   LTO
VLIB2        C2120GL4    Scratch                               3,633   LTO
Well, have scratches so there is plenty of  space for the VMs' data.
Next, we checked to see if we have space for the VMs'  control files.
q stgpool VMCTLPOOL
Storage       Device        Estimated     Pct     Pct   High   Low   Next Stora-
Pool Name     Class Name     Capacity    Util    Migr    Mig   Mig   ge Pool
                                                         Pct   Pct
-----------   ----------   ----------   -----   -----   ----   ---   -----------
VMCTLPOOL     DISK            1,418 G    100.00   100.00     99    94

So, there is no space to store new VM control files.
So we created another volume in the VMCTLPOOL.
def volume VMCTLPOOL /vmctl1/stg33.dsm formatsize=42709

After, this completed back-ups stalled started sessions with the TSM server and transferred their back-up.