소소한 일상에서 책읽기 중

11gR2 Exadata V2 - DB Instance 장애처리 (asm disk group 장애처리) 본문

DB까다롭다

11gR2 Exadata V2 - DB Instance 장애처리 (asm disk group 장애처리)

다솜여우 2012. 6. 19. 19:11

1. 문제 발생 : alert.log에 아래와 같은 에러메시지가 기록되면서 DB Instance가 내려가서 올라가지 않음

Tue Jun 19 18:10:47 2012
Exadata error: 201 (Generic I/O error)
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/incident/incdir_1154831/idba1_p020_8884_i1154831.trc:
ORA-27603: Cell storage I/O error, I/O failed on disk o/10.10.30.39/DATA_CD_05_idbadsk21 at offset 3434086400 for data length 1048576
ORA-00600: internal error code, arguments: [1236], [kxfx.c], [7568], [0x3EE509108], [40], [0], [1148147089], [], [], [], [], []
WARNING: IO Failed. group:1 disk(number.incarnation):136.0xe96edf4e disk_path:o/10.10.30.39/DATA_CD_05_idbadsk21
         AU:818 disk_offset(bytes):3434086400 io_size:1048576 operation:Read type:asynchronous
         result:I/O error process_id:8884
         Exadata error:201 (Generic I/O error)
WARNING: failed to read mirror side 1 of virtual extent 2098 logical extent 0 of file 273 in group [1.3183356272] from disk DATA_CD_05_IDBADSK21  allocation unit 818 reason error; if possible,will try another mirror side
NOTE: successfully read mirror side 2 of virtual extent 2098 logical extent 1 of file 273 in group [1.3183356272] from disk DATA_CD_08_IDBADSK22 allocation unit 624
ERROR: cannot read disk header of disk DATA_CD_05_IDBADSK21 (136:3916357454)
NOTE: process 8884 initiating offline of disk 136.3916357454 (DATA_CD_05_IDBADSK21) with mask 0x7e in group 1
WARNING: group 1 dismounted: failed to submit a read to virtual extent 2181 of file 273
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 97081 of file 312
WARNING: group 1 dismounted: failed to read virtual extent 97081 of file 312
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 99282 of file 546
WARNING: group 1 dismounted: failed to read virtual extent 99282 of file 546
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to read virtual extent 201574 of file 312
WARNING: group 1 dismounted: failed to submit a read to virtual extent 201573 of file 312
WARNING: group 1 dismounted: failed to read virtual extent 201573 of file 312
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/incident/incdir_1154831/idba1_p020_8884_i1154831.trc:
ORA-00333: redo log read error block 17872896 count 2048
ORA-00334: archived log: '+DATA/idba/onlinelog/group_21.273.720297751'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
ORA-15066: offlining disk "DATA_CD_05_IDBADSK21" may result in a data loss
ORA-00600: internal error code, arguments: [1236], [kxfx.c], [7568], [0x3EE509108], [40], [0], [1148147089], [], [], [], [], []
WARNING: group 1 dismounted: failed to submit a read to virtual extent 2181 of file 273
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/incident/incdir_1154831/idba1_p020_8884_i1154831.trc:
ORA-00333: redo log read error block 17872896 count 2048
ORA-00334: archived log: '+DATA/idba/onlinelog/group_21.273.720297751'
ORA-15081: failed to submit an I/O operation to a disk
ORA-00600: internal error code, arguments: [1236], [kxfx.c], [7568], [0x3EE509108], [40], [0], [1148147089], [], [], [], [], []
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 62959 of file 546
WARNING: group 1 dismounted: failed to read virtual extent 62959 of file 546
WARNING: group 1 dismounted: failed to submit a read to virtual extent 2166 of file 273
WARNING: group 1 dismounted: failed to read virtual extent 2166 of file 273
WARNING: group 1 dismounted: failed to submit a read to virtual extent 2166 of file 273
WARNING: group 1 dismounted: failed to read virtual extent 2166 of file 273
WARNING: group 1 dismounted: failed to submit a read to virtual extent 2166 of file 273
WARNING: group 1 dismounted: failed to read virtual extent 2166 of file 273
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 75369 of file 546
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a write to virtual extent 73 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 263
WARNING: group 1 dismounted: failed to submit a write to virtual extent 42310 of file 304
WARNING: group 1 dismounted: failed to submit a write to virtual extent 230284 of file 305
WARNING: group 1 dismounted: failed to submit a write to virtual extent 30345 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 40935 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 96547 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 103062 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 114097 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 115237 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 123272 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 130250 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 130356 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 268410 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 270078 of file 312
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 75 of file 316
WARNING: group 1 dismounted: failed to submit a write to virtual extent 73 of file 357
WARNING: group 1 dismounted: failed to submit a write to virtual extent 73 of file 357
WARNING: group 1 dismounted: failed to submit a write to virtual extent 87 of file 358
WARNING: group 1 dismounted: failed to submit a write to virtual extent 87 of file 358
WARNING: group 1 dismounted: failed to submit a write to virtual extent 87 of file 358
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 76 of file 362
WARNING: group 1 dismounted: failed to submit a write to virtual extent 78 of file 382
WARNING: group 1 dismounted: failed to submit a write to virtual extent 78 of file 382
WARNING: group 1 dismounted: failed to submit a write to virtual extent 137051 of file 390
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 80 of file 470
WARNING: group 1 dismounted: failed to submit a write to virtual extent 79 of file 489
Tue Jun 19 18:10:50 2012
NOTE:Waiting for all pending writes to complete before de-registering: grpnum 1
KCF: read, write or open error, block=0x9e6c online=1
        file=121 '+DATA/idba/datafile/undotbs1.489.737928837'
        error=15081 txt: ''
WARNING: group 1 dismounted: failed to submit a read to virtual extent 0 of file 256
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/trace/idba1_dbw0_4653.trc:
ORA-00202: control file: '+DATA/idba/controlfile/current.256.720296933'
ORA-15081: failed to submit an I/O operation to a disk
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to write virtual extent 230585 of file 305
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to write virtual extent 42305 of file 304
WARNING: group 1 dismounted: failed to write virtual extent 42305 of file 304
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to write virtual extent 5015 of file 258
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/trace/idba1_lgwr_4736.trc:
ORA-00345: redo log write error block 41087133 count 1950
ORA-00312: online log 2 thread 1: '+DATA/idba/onlinelog/group_2.258.720297023'
ORA-15081: failed to submit an I/O operation to a disk
ORA-15081: failed to submit an I/O operation to a disk
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 0 of file 256
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/trace/idba1_lmon_4117.trc:
ORA-00202: control file: '+DATA/idba/controlfile/current.256.720296933'
ORA-15081: failed to submit an I/O operation to a disk
Tue Jun 19 18:10:50 2012
WARNING: group 1 dismounted: failed to submit a read to virtual extent 121922 of file 312
WARNING: group 1 dismounted: failed to read virtual extent 121922 of file 312
Errors in file /u01/app/oracle/diag/rdbms/idba/idba1/trace/idba1_dbw0_4653.trc:
ORA-00204: error in reading (block 1, # blocks 1) of control file
ORA-00202: control file: '+DATA/idba/controlfile/current.256.720296933'
ORA-15081: failed to submit an I/O operation to a disk
DBW0 (ospid: 4653): terminating the instance due to error 204


2. 복구 방법 : 수동으로 instance를 재기동 하려했으나 에러메시지만 뿌리고 DB가 올라오지 않는데, 해당 노드의 DATA Group이 dismount여서 mount 로 변경하고 나니 DB Instance가 자동으로 올라와서 복구 됨

[oracle@idbadb01.sktelecom.com:/home/oracle/DBA/31.APP/99.PT_EXCH/93.HourExch_Loading/LOG] sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Jun 19 18:42:27 2012

Copyright (c) 1982, 2009, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup
ORA-01078: failure in processing system parameters
ORA-01565: error in identifying file '+DATA/idba/spfileidba.ora'
ORA-17503: ksfdopn:2 Failed to open file +DATA/idba/spfileidba.ora
ORA-15056: additional error message
ORA-17503: ksfdopn:DGOpenFile05 Failed to open file +DATA/idba/spfileidba.ora
ORA-17503: ksfdopn:2 Failed to open file +DATA/idba/spfileidba.ora
ORA-15001: diskgroup "DATA" does not exist or is not mounted
ORA-06512: at line 4
SQL> ----------------------------------
SQL>
SQL> exit
[oracle@idbadb01.sktelecom.com:/home/oracle/DBA] . oraenv
ORACLE_SID = [idba1] ? +ASM1
The Oracle base for ORACLE_HOME=/u01/app/11.2.0/grid is /u01/app/oracle
[oracle@idbadb01.sktelecom.com:/home/oracle/DBA] sqlplus / as sysasm
SQL> set lines 200
SQL> set pages 200
SQL> col GROUP_NUMBER for 99
SQL> col NAME for a16
SQL> col STATE for a12

SQL> select INST_ID,GROUP_NUMBER,NAME,STATE,TYPE,TOTAL_MB,FREE_MB,HOT_USED_MB,REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB,OFFLINE_DISKS from gv$asm_diskgroup order by INST_ID, GROUP_NUMBER  ;

   INST_ID GROUP_NUMBER NAME             STATE        TYPE           TOTAL_MB    FREE_MB HOT_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
---------- ------------ ---------------- ------------ ------------ ---------- ---------- ----------- ----------------------- -------------- -------------
         1            0 DATA             DISMOUNTED                         0          0           0                       0              0         0
         1            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         1            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         1            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         2            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         2            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         2            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         2            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         3            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         3            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         3            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         3            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         4            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         4            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         4            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         4            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         5            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         5            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         5            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         5            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         6            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         6            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         6            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         6            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         7            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         7            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         7            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         7            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         8            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         8            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         8            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         8            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         9            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         9            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         9            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         9            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        10            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        10            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        10            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        10            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        11            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        11            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        11            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        11            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        12            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        12            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        12            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        12            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        13            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        13            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        13            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        13            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        14            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        14            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        14            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        14            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        15            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        15            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        15            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        15            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        16            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        16            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        16            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        16            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0

64 rows selected.

SQL> rem 1호기 DATA Group이 dismount 상태이므로, mount로 변경해야 함

SQL> ALTER DISKGROUP data MOUNT;

SQL> select INST_ID,GROUP_NUMBER,NAME,STATE,TYPE,TOTAL_MB,FREE_MB,HOT_USED_MB,REQUIRED_MIRROR_FREE_MB,USABLE_FILE_MB,OFFLINE_DISKS from gv$asm_diskgroup order by INST_ID, GROUP_NUMBER  ;

   INST_ID GROUP_NUMBER NAME             STATE        TYPE           TOTAL_MB    FREE_MB HOT_USED_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB OFFLINE_DISKS
---------- ------------ ---------------- ------------ ------------ ---------- ---------- ----------- ----------------------- -------------- -------------
         1            1 DATA             MOUNTED      NORMAL        522547200   48638260           0                47504290         566985            16
         1            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         1            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         1            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         2            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         2            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         2            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         2            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         3            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         3            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         3            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         3            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         4            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         4            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         4            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         4            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         5            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         5            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         5            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         5            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         6            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         6            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         6            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         6            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         7            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         7            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         7            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         7            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         8            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         8            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         8            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         8            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
         9            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
         9            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
         9            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
         9            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        10            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        10            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        10            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        10            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        11            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        11            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        11            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        11            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        12            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        12            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        12            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        12            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        13            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        13            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        13            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        13            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        14            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        14            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        14            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        14            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        15            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        15            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        15            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        15            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0
        16            1 DATA             MOUNTED      NORMAL        522547200   48638132           0                47504290         566921            16
        16            2 RECO             MOUNTED      NORMAL         96192624   42864880           0                 8744784       17060048         0
        16            3 SYSTEMDG         MOUNTED      NORMAL          8316432    8311216           0                  756039        3777588         0
        16            4 TEMP             MOUNTED      NORMAL          5035008    1917320           0                  457728         729796         0

64 rows selected.

[oracle@idbadb01.sktelecom.com:/home/oracle/DBA] . oraenv
ORACLE_SID = [+ASM1] ? idba1

[oracle@idbadb01.sktelecom.com:/home/oracle/DBA] sqlplus / as sysdba

SQL> set pages 200
SQL> col HOST_NAME for a30
SQL> col INSTANCE_NAME for a12
SQL> set lines 200
SQL> select INST_ID,INSTANCE_NAME,HOST_NAME,STARTUP_TIME,STATUS from gv$instance order by 1;
   INST_ID INSTANCE_NAM HOST_NAME                      STARTUP_TIME STATUS
---------- ------------ ------------------------------ ------------ ------------------------
         1 idba1        idbadb01.sktelecom.com         19-JUN-12    OPEN
         2 idba2        idbadb02.sktelecom.com         16-JUN-12    OPEN
         3 idba3        idbadb03.sktelecom.com         16-JUN-12    OPEN
         4 idba4        idbadb04.sktelecom.com         16-JUN-12    OPEN
         5 idba5        idbadb05.sktelecom.com         16-JUN-12    OPEN
         6 idba6        idbadb06.sktelecom.com         16-JUN-12    OPEN
         7 idba7        idbadb07.sktelecom.com         16-JUN-12    OPEN
         8 idba8        idbadb08.sktelecom.com         16-JUN-12    OPEN
         9 idba9        idbadb09.sktelecom.com         16-JUN-12    OPEN
        10 idba10       idbadb10.sktelecom.com         16-JUN-12    OPEN
        11 idba11       idbadb11.sktelecom.com         16-JUN-12    OPEN
        12 idba12       idbadb12.sktelecom.com         16-JUN-12    OPEN
        13 idba13       idbadb13.sktelecom.com         16-JUN-12    OPEN
        14 idba14       idbadb14.sktelecom.com         16-JUN-12    OPEN
        15 idba15       idbadb15.sktelecom.com         16-JUN-12    OPEN
        16 idba16       idbadb16.sktelecom.com         16-JUN-12    OPEN

16 rows selected.