What to do when "vxdisk list" shows status of 'online dgdisabled'.

Details:

HPSRV01:# vxdisk -o alldgs list
DEVICE           TYPE            DISK             GROUP        STATUS
EMC_CLARiiON0_0  auto:cdsdisk    EMC_CLARiiON0_0  dygy2502     online
EMC_CLARiiON0_1  auto:cdsdisk    -               (dvgy2500)    online
EMC_CLARiiON0_2  auto:cdsdisk    EMC_CLARiiON0_4  dvgyappl     online
EMC_CLARiiON0_3  auto:cdsdisk    EMC_CLARiiON0_3  dvgy2503     online
EMC_CLARiiON0_4  auto:cdsdisk    EMC_CLARiiON0_4  dvgy2504     online
EMC_CLARiiON0_5  auto:cdsdisk    EMC_CLARiiON0_5  dvgy25       online
EMC_CLARiiON0_6  auto:cdsdisk    EMC_CLARiiON0_9  dvgy26       online dgdisabled
EMC_CLARiiON0_7  auto:cdsdisk    EMC_CLARiiON0_8  dygy2501     online
EMC_CLARiiON0_8  auto:cdsdisk    -               (dvgy2506)    online
EMC_CLARiiON0_9  auto:cdsdisk    -               (dvgy2505)    online
EMC_CLARiiON0_10 auto:cdsdisk    -               (dvgy2507)    online
EMC_CLARiiON0_11 auto:cdsdisk    EMC_CLARiiON0_11 dvgy25db2    online

This situation can happen when every disk in a disk group is lost from a bad power supply, power turned off to the disk array, cable disconnected, zoning problems, etc.

The disk group will not show in the output from vxprint -ht.

HPSRV01:# vxprint -htg dvgy26
VxVM vxprint ERROR V-5-1-582 Disk group dvgy26: No such disk group

The disk group will show as disabled in vxdg list:

HPSRV01:# vxdg list
NAME         STATE                ID
dygy2501     enabled,cds          1189621899.78.HPSRV01
dvgyappl     enabled,cds          1190904062.52.HPSRV01
dvgy25       enabled,cds          1189622068.88.HPSRV01
dvgy25db2    enabled,cds          1189622043.86.HPSRV01
dvgy26       disabled             1189538508.74.HPSRV01
dvgy2503     enabled,cds          1189621988.82.HPSRV01
dvgy2504     enabled,cds          1189622014.84.HPSRV01
dygy2502     enabled,cds          1189621955.80.HPSRV01

This is the output of vxdg list dvgy26:

HPSRV01:# vxdg list dvgy26
Group:            dvgy26
dgid:             1189538508.74.HPSRV01
import-id:        1024.22
flags:            disabled
version:          0
alignment:        0 (bytes)
local-activation: read-write
ssb:              off
detach-policy:    invalid
copies:           nconfig=default nlog=default
config:           seqno=0.1103 permlen=1280 free=1259 templen=11 loglen=192
config disk EMC_CLARiiON0_6 copy 1 len=1280 state=clean online
log disk EMC_CLARiiON0_6 copy 1 len=192

Your filesystems will of course fail and the operating system will report it as corrupted.

HPSRV01:# df -k > /dev/null
df: /db2/dwins26q: I/O error
df: /backup: I/O error
df: /db/dwdb26q/dwins25q/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/syscatspace/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/tempspace01/NODE0000: I/O error
df: /dba/dwins26q: I/O error
df: /db2/dwmysld: I/O error
df: /backup/wiminst: I/O error

Once you have confirmed that the disk storage is powered-up, running, and operational and if the LUNs are in a SAN, zoning is configured right, this problem can be remedied by deporting, and then importing the disk group:

# vxdg deport dvgy26

# vxdg import dvgy26
VxVM vxdg ERROR V-5-1-587 Disk group dvgy26: import failed: No valid disk found containing disk group

If volume manager can't see the disks, and your SAN or storage administrator has confirmed that the LUNs were fine and presented to your server, then rescan the disks.

HPSRV01:# vxdisk scandisks
HPSRV01:# vxdctl enable
HPSRV01:# vxdg import dvgy26

Otherwise, your diskgroup should be showing up as enabled.

HPSRV01:# vxdg list
NAME         STATE           ID
dygy2501     enabled,cds          1189621899.78.HPSRV01
dvgyappl     enabled,cds          1190904062.52.HPSRV01
dvgy25       enabled,cds          1189622068.88.HPSRV01
dvgy25db2    enabled,cds          1189622043.86.HPSRV01
dvgy26       enabled,cds          1189538508.74.HPSRV01
dvgy2503     enabled,cds          1189621988.82.HPSRV01
dvgy2504     enabled,cds          1189622014.84.HPSRV01
dygy2502     enabled,cds          1189621955.80.HPSRV01

The disk group now shows in vxprint -ht with the volumes and plexes disabled:

HPSRV01:# vxprint -htg dvgy26
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg dvgy26       default      default  9000     1189538508.74.HPSRV01

dm EMC_CLARiiON0_9 EMC_CLARiiON0_6 auto 2048   67102464 -

v  backup       -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl backup-01    backup       DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-02 backup-01 EMC_CLARiiON0_9 8388608 4194304 0 EMC_CLARiiON0_6 ENA

v  db           -            DISABLED ACTIVE   1048576  SELECT    -        fsgen
pl db-01        db           DISABLED ACTIVE   1048576  CONCAT    -        RW
sd EMC_CLARiiON0_9-04 db-01  EMC_CLARiiON0_9 16777216 1048576 0   EMC_CLARiiON0_6 ENA

v  dba          -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl dba-01       dba          DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-03 dba-01 EMC_CLARiiON0_9 12582912 4194304 0   EMC_CLARiiON0_6 ENA

v  db2          -            DISABLED ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          DISABLED ACTIVE   8388608  CONCAT    -        RW
sd EMC_CLARiiON0_9-01 db2-01 EMC_CLARiiON0_9 0 8388608  0         EMC_CLARiiON0_6 ENA

v  dwmysld      -            DISABLED ACTIVE   2097152  SELECT    -        fsgen
pl dwmysld-01   dwmysld      DISABLED ACTIVE   2097152  CONCAT    -        RW
sd EMC_CLARiiON0_9-09 dwmysld-01 EMC_CLARiiON0_9 55574528 2097152 0 EMC_CLARiiON0_6 ENA

v  lg1          -            DISABLED ACTIVE   10485760 SELECT    -        fsgen
pl lg1-01       lg1          DISABLED ACTIVE   10485760 CONCAT    -        RW
sd EMC_CLARiiON0_9-08 lg1-01 EMC_CLARiiON0_9 45088768 10485760 0  EMC_CLARiiON0_6 ENA

v  syscat       -            DISABLED ACTIVE   2097152  SELECT    -        fsgen
pl syscat-01    syscat       DISABLED ACTIVE   2097152  CONCAT    -        RW
sd EMC_CLARiiON0_9-05 syscat-01 EMC_CLARiiON0_9 17825792 2097152 0 EMC_CLARiiON0_6 ENA

v  tp01         -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl tp01-01      tp01         DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-07 tp01-01 EMC_CLARiiON0_9 40894464 4194304 0  EMC_CLARiiON0_6 ENA

v  ts01         -            DISABLED ACTIVE   20971520 SELECT    -        fsgen
pl ts01-01      ts01         DISABLED ACTIVE   20971520 CONCAT    -        RW
sd EMC_CLARiiON0_9-06 ts01-01 EMC_CLARiiON0_9 19922944 20971520 0 EMC_CLARiiON0_6 ENA

Verify that the disks on the diskgroup are all online.

HPSRV01:# vxdisk -o alldgs list
DEVICE           TYPE            DISK             GROUP        STATUS
EMC_CLARiiON0_0  auto:cdsdisk    EMC_CLARiiON0_0  dygy2502     online
EMC_CLARiiON0_1  auto:cdsdisk    -               (dvgy2500)    online
EMC_CLARiiON0_2  auto:cdsdisk    EMC_CLARiiON0_4  dvgyappl     online
EMC_CLARiiON0_3  auto:cdsdisk    EMC_CLARiiON0_3  dvgy2503     online
EMC_CLARiiON0_4  auto:cdsdisk    EMC_CLARiiON0_4  dvgy2504     online
EMC_CLARiiON0_5  auto:cdsdisk    EMC_CLARiiON0_5  dvgy25       online
EMC_CLARiiON0_6  auto:cdsdisk    EMC_CLARiiON0_9  dvgy26       online
EMC_CLARiiON0_7  auto:cdsdisk    EMC_CLARiiON0_8  dygy2501     online
EMC_CLARiiON0_8  auto:cdsdisk    -               (dvgy2506)    online
EMC_CLARiiON0_9  auto:cdsdisk    -               (dvgy2505)    online
EMC_CLARiiON0_10 auto:cdsdisk    -               (dvgy2507)    online
EMC_CLARiiON0_11 auto:cdsdisk    EMC_CLARiiON0_11 dvgy25db2    online

Now the volumes can be started:

HPSRV01:# vxvol -g dvgy26 startall

HPSRV01:# vxprint -htg dvgy26 | egrep '^v|^pl'
v  backup       -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl backup-01    backup       ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  db           -            ENABLED  ACTIVE   1048576  SELECT    -        fsgen
pl db-01        db           ENABLED  ACTIVE   1048576  CONCAT    -        RW
v  dba          -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl dba-01       dba          ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  db2          -            ENABLED  ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          ENABLED  ACTIVE   8388608  CONCAT    -        RW
v  dwmysld      -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl dwmysld-01   dwmysld      ENABLED  ACTIVE   2097152  CONCAT    -        RW
v  lg1          -            ENABLED  ACTIVE   10485760 SELECT    -        fsgen
pl lg1-01       lg1          ENABLED  ACTIVE   10485760 CONCAT    -        RW
v  syscat       -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl syscat-01    syscat       ENABLED  ACTIVE   2097152  CONCAT    -        RW
v  tp01         -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl tp01-01      tp01         ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  ts01         -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
pl ts01-01      ts01         ENABLED  ACTIVE   20971520 CONCAT    -        RW

The filesystems on these volumes may not be in consistent state. So, run a filesystem check before mounting them.

HPSRV01:# for i in `grep dvgy26 /etc/fstab | awk '{ print $1 }'`
> do
>   fsck -y $i
>   mount $i
> done