Friday, July 11, 2014

ANS1311E Server out of Storage space -- TSM4VE back-ups fail

The Tivoli Storage Manager for Virtual Environments (TSM4VE) is a complex infrastructure to support and trouble-shoot.
Recently, this problem re-occured and so I thought that I'd give back to the TSM community online that has helped me so many times resolve problems.

TSM4VE back-ups can fail for many reasons. If you manuallly run the back-up, then you can retrieve helpful errors OR you cna look at the dsmerror.log.  Many times the errors are generic enough to confused admins who have little experience.

Our TSM4VE back-ups were failing with the error message:
TSM4VE back-ups fail  ANS1311E Server out of Storage space

Now, TSM4VE back-ups use 2 storagepools: one for data and one for vmware control files.
We use a Falconstor VTL to store data and a diskpool.
So, first check the VTL for scratch volumes by logging into dsadmc:
q libv vlib1
VLIB1        C1128OL4    Scratch                               3,929   LTO
VLIB1        C1128PL4    Scratch                               3,930   LTO
VLIB1        C1128QL4    Scratch                               3,931   LTO
VLIB1        C1128RL4    Scratch                               3,932   LTO
VLIB1        C1128SL4    Scratch                               3,933   LTO
VLIB1        C1128TL4    Scratch                               3,934   LTO
q libv vlib2
VLIB2        C2120BL4    Scratch                               3,628   LTO
VLIB2        C2120CL4    Scratch                               3,629   LTO
VLIB2        C2120DL4    Scratch                               3,630   LTO
VLIB2        C2120EL4    Scratch                               3,631   LTO
VLIB2        C2120FL4    Scratch                               3,632   LTO
VLIB2        C2120GL4    Scratch                               3,633   LTO
Well, have scratches so there is plenty of  space for the VMs' data.
Next, we checked to see if we have space for the VMs'  control files.
q stgpool VMCTLPOOL
Storage       Device        Estimated     Pct     Pct   High   Low   Next Stora-
Pool Name     Class Name     Capacity    Util    Migr    Mig   Mig   ge Pool
                                                         Pct   Pct
-----------   ----------   ----------   -----   -----   ----   ---   -----------
VMCTLPOOL     DISK            1,418 G    100.00   100.00     99    94

So, there is no space to store new VM control files.
So we created another volume in the VMCTLPOOL.
def volume VMCTLPOOL /vmctl1/stg33.dsm formatsize=42709

After, this completed back-ups stalled started sessions with the TSM server and transferred their back-up.