Introduction

This is my third article in the series. In previous two articles we have seen how we can install/configure ASM Instance inside a non global zone and we have also seen how we can add any additional devices to an existing ASM instance running in a non global zone.  In this article we will see how we can remove the  provisioned ASM devices from Solaris-11 Non global zone.

We have to be very careful with the device management  specifically if we are using PRODUCTION Environments in a non-global zones. 

Please refer below links for the previous articles:

http://www.toadworld.com/platforms/oracle/w/wiki/11365.installing-and-configuring-oracle-12c-grid-infrasturucture-and-asm-in-solaris-11-non-global-zones

http://www.toadworld.com/platforms/oracle/w/wiki/11379.managing-asm-devices-in-solaris-11-non-global-zone-part1

If you have not read my previous articles then I would highly recommend to read it to have better understanding about this article.

Environment details

I will be using the same environment used in my previous articles.

-  soltest is the hostname for global zone.

- dbnode1 is the hostname for the non global zone

Global Zone - GZ

soltest

Non Global Zone - NGZ

dbnode1

 

 Demonstration:

In this demonstration we will execute TEST cases for destroying the ASM devices provisioned in non-global zone.  Actually the TEST cases are situations that can be executed accidentally on any system running in PROD/DEV/TEST and eventually we will loose data and it may cost huge time for restoring/ recovering the lost data from the Disk Group.  

In this current demonstration the text code displayed in blue color is from the global zone and text color displayed in red  is from non global zone.

The below mentioned are the existing ASM devices and Disk Groups exists in the non global zone:

SQL> column name format a15
SQL> column path format a45
SQL> select name, path from v$asm_disk;

NAME                      PATH
--------------- ---------------------------------------------
                 /dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
                 /dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
GRID_0000        /dev/rdsk/c2t2d0s5
GRID_0001        /dev/rdsk/c2t3d0s5
DATA_0000        /dev/rdsk/c2t4d0s5
DATA_0001        /dev/rdsk/c2t5d0s5
DATA3_0000       /dev/zvol/rdsk/dbzone/dbnode1/asmdisk01
DATA4_0000       /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02

8 rows selected.

SQL>

----------------------------------------------------------------

SQL> select name from v$asm_diskgroup;

NAME
---------------
DATA4
DATA3
DATA
GRID

SQL>

In this demonstration we will work on Disk groups "DATA" and "DATA3"

- DATA3 Disk Group is configured on ZFS volume "/dev/zvol/rdsk/dbzone/dbnode1/asmdisk01"  and DATA Disk Group is configured on "/dev/rdsk/c2t4d0s5, /dev/rdsk/c2t4d0s5"

TEST CASES


In this demonstration we will see the different situations that can happen in real life while working on ASM Instances in global and non-global zones. 

TEST CASE -1:

We will try to destroy the ZFS volume from the global zone and then we will see the impact on the underlying ASM Disk Groups.

 

- Check the exists ZFS volumes:

root@soltest:~# zfs list -t volume
NAME                      USED AVAIL REFER MOUNTPOINT
dbzone/dbnode1/asmdisk01 1.03G 39.9G 53.4M   -
dbzone/dbnode1/asmdisk02 1.03G 39.9G 53.4M   -
dbzone/dbnode1/asmdisk03 1.03G 40.0G 16K     -
dbzone/dbnode1/asmdisk04 1.03G 40.0G 16K     -
rpool/dump               2.06G 16.2G 2.00G   -
rpool/swap               5.16G 16.3G 5.00G   -
root@soltest:~#

 

- Lets try to destroy the ZFS volume "asmdiks01" belonging to DG - DATA3:

root@soltest:~# zfs destroy dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~#

-Try with the force option:


root@soltest:~# zfs destroy -f dbzone/dbnode1/asmdisk01 
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~#

Great - Still its not able to destroy the zfs volume as the volume is by ASM Instance in a non-global zone. So the ASM Devices are safer as we are not able to destroy the volumes from the global zone as well.

TEST CASE -2 :

In this test case we will try to dismount the ASM Diskgroup "DATA3" and then we will try to destroy the ZFS volume  belonging to the ASM Disk group "DATA3" from the global zone.

- Check the existing mounted Disk Groups:

- Dismount the Disk Group using asmca or CLI

- Disk Group "DATA3" is dismounted

 - Disk Group "DATA3" is already dismounted now we will try to destroy the ZFS volume (asmdiks01) belonging to DG - DATA3 from the global zone.

- Try to destroy ZFS volume

root@soltest:~# zfs destroy dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy

- Now try using force option

root@soltest:~# zfs destroy -f dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~#

-  Good still we are not able to destroy the ZFS volume from the non global zone.

TEST CASE -3:

 In this scenario we will try to drop the ASM Disk Group and then we will try to destroy the ZFS volume from the non-global zone. The DG - "DATA3" is already dismounted simply we will drop the required Disk Group.

- After Dropping the disk group now try to destroy the required ZFS volume.

root@soltest:~# zfs destroy dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~# zfs destroy -f dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~#

- We are not able to destroy ZFS volume even after dropping the Disk Group associated with the ZFS volume.

TEST CASE -4:

In this test case we will remove the zfs device from the non-global zone and then will try to destroy the ZFS volume.

root@soltest:~# zonecfg -z dbnode1
zonecfg:dbnode1> info
zonename: dbnode1
zonepath: /dbzone/dbnode1
brand: solaris
autoboot: true
autoshutdown: shutdown
bootargs: -m verbose
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
tenant:
fs-allowed:
anet:
linkname: net0
lower-link: auto
allowed-address not specified
configure-allowed-address: true
defrouter not specified
allowed-dhcp-cids not specified
link-protection: mac-nospoof
mac-address: auto
auto-mac-address: 2:8:20:1c:41:61
mac-prefix not specified
mac-slot not specified
vlan-id not specified
priority not specified
rxrings not specified
txrings not specified
mtu not specified
maxbw not specified
rxfanout not specified
vsi-typeid not specified
vsi-vers not specified
vsi-mgrid not specified
etsbw-lcl not specified
cos not specified
pkey not specified
linkmode not specified
evs not specified
vport not specified
device:
match: /dev/rdsk/c2t2d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t3d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t3d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t2d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t2d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t3d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t3d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t5d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t5d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t4d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t4d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/dsk/dbzone/dbnode1/asmvol1
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk01
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
storage not specified
allow-partition not specified
allow-raw-io not specified
zonecfg:dbnode1>

- Once you are connected to zone using zonecfg command then we can see all devices configuration of non-global zone using "info" command. Now we will remove "/dev/zvol/rdsk/dbzone/dbnode1/asmdisk01" ZFS volume from the non-global zone.

zonecfg:dbnode1> remove device match=/dev/zvol/rdsk/dbzone/dbnode1/asmdisk01
zonecfg:dbnode1> commit;
zonecfg:dbnode1>

- After removing the devices the removed device will not be listed in zonecfg info output command.

device:
match: /dev/zvol/dsk/dbzone/dbnode1/asmvol1
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
storage not specified
allow-partition not specified
allow-raw-io not specified
zonecfg:dbnode1>

- After removing the device try to destroy the ZFS volume.

Before proceeding to destroy the ZFS volume from Global zone, lets see whether this device is still visible  under ASM Instance and at OS level:

SQL> column name format a15
SQL> column path format a45
SQL> select name, path from v$asm_disk;

NAME                      PATH
--------------- ---------------------------------------------
                      /dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
                      /dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
                      /dev/zvol/rdsk/dbzone/dbnode1/asmdisk01
GRID_0000             /dev/rdsk/c2t2d0s5
GRID_0001             /dev/rdsk/c2t3d0s5
DATA_0000             /dev/rdsk/c2t4d0s5
DATA_0001             /dev/rdsk/c2t5d0s5
DATA4_0000            /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02

8 rows selected.

SQL>

grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$ ls -lrt
total 0
crw-rw---- 1 grid12c dba 303, 6 Oct 14 16:28 asmdisk04
crw-rw---- 1 grid12c dba 303, 5 Oct 14 16:28 asmdisk03
crw-rw---- 1 grid12c dba 303, 4 Oct 14 16:57 asmdisk01
crw-rw---- 1 grid12c dba 303, 3 Oct 14 17:11 asmdisk02
grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$

- Still asmdisk01 volume is visible in V$asm_disk

- Now dynamically apply the configuration changes to a non-global zone for relecting recently removed asm volume.

root@soltest:~# zoneadm -z dbnode1 apply
zone 'dbnode1': Checking: Removing device match=/dev/zvol/rdsk/dbzone/dbnode1/asmdisk01
zone 'dbnode1': Applying the changes
root@soltest:~#

grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$ ls -lrt
total 0
crw-rw---- 1 grid12c dba 303, 6 Oct 14 16:28 asmdisk04
crw-rw---- 1 grid12c dba 303, 5 Oct 14 16:28 asmdisk03
crw-rw---- 1 grid12c dba 303, 4 Oct 14 16:57 asmdisk01
crw-rw---- 1 grid12c dba 303, 3 Oct 14 17:14 asmdisk02
grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$

- Now lets try to destory the ZFS volume:

root@soltest:~# zfs destroy -f dbzone/dbnode1/asmdisk01
cannot destroy 'dbzone/dbnode1/asmdisk01': volume is busy
root@soltest:~#

After reconfiguration also still we are not able to destroy the zfs volume. This I beleive is an unexpected behaviour. Once we remove the zfs volume then it should not be visible
in the non-global zone.

To remove the devices from the non-global zone requires restarting of Grid Infrastructure Services.

- Stop Grid Infrastructure Services running in the non-global zone:

root@dbnode1:/u01/grid12c/product/12.1.0/grid/bin# pwd
/u01/grid12c/product/12.1.0/grid/bin
root@dbnode1:/u01/grid12c/product/12.1.0/grid/bin# ./crsctl stop has
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'dbnode1'
CRS-2673: Attempting to stop 'ora.DATA4.dg' on 'dbnode1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'dbnode1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'dbnode1' succeeded
CRS-2677: Stop of 'ora.DATA4.dg' on 'dbnode1' succeeded
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'dbnode1'
CRS-2673: Attempting to stop 'ora.GRID.dg' on 'dbnode1'
CRS-2677: Stop of 'ora.DATA.dg' on 'dbnode1' succeeded
CRS-2677: Stop of 'ora.GRID.dg' on 'dbnode1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'dbnode1'
CRS-2677: Stop of 'ora.asm' on 'dbnode1' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'dbnode1'
CRS-2677: Stop of 'ora.evmd' on 'dbnode1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'dbnode1'
CRS-2677: Stop of 'ora.cssd' on 'dbnode1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'dbnode1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
root@dbnode1:/u01/grid12c/product/12.1.0/grid/bin#

- Now lets try to destroy the ZFS volume:

root@soltest:~# zfs destroy dbzone/dbnode1/asmdisk01
root@soltest:~#

This time ZFS volume has been destroyed successfully. So its very clear that if we are using  ZFS volumes as ASM Devices then we cannot destroy it before stopping the Grid Infrastructure services.

After starting up of GI services check the destroyed ZFS volume from ASM Instance and OS:

SQL> column name format a15
SQL> column path format a45
SQL> select name, path from v$asm_disk;

NAME PATH
--------------- ---------------------------------------------
/dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
/dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
GRID_0000 /dev/rdsk/c2t2d0s5
GRID_0001 /dev/rdsk/c2t3d0s5
DATA_0000 /dev/rdsk/c2t4d0s5
DATA_0001 /dev/rdsk/c2t5d0s5
DATA4_0000 /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02

7 rows selected.

SQL>

grid12c@dbnode1:/u01/grid12c/product/12.1.0/grid$ cd /dev/zvol/rdsk/dbzone/dbnode1/
grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$ ls -lrt
total 0
crw-rw---- 1 grid12c dba 303, 6 Oct 14 16:28 asmdisk04
crw-rw---- 1 grid12c dba 303, 5 Oct 14 16:28 asmdisk03
crw-rw---- 1 grid12c dba 303, 3 Oct 14 17:50 asmdisk02
grid12c@dbnode1:/dev/zvol/rdsk/dbzone/dbnode1$

Now "asmdisk01" is not visible  in the list.

TEST CASE -5:

All executed test cases before was on zfs volumes. In this test case we will try to format the disk belongs to non-global zone and under ASM from the global zone. We will see the impact of these changes on ASM Disk group. The disk group DATA is confugured with devices "/dev/rdsk/c2t4d0s5" & "dev/rdsk/c2t5d0s5". In this scenario we will format device "dev/rdsk/c2t5d0s5" from the non global zone.

 

root@soltest:~# format
Searching for disks...done


AVAILABLE DISK SELECTIONS:
0. c2t0d0 <VBOX-HARDDISK-1.0-100.00GB>
/pci@0,0/pci1000,8000@14/sd@0,0
1. c2t1d0 <VBOX-HARDDISK-1.0-50.00GB>
/pci@0,0/pci1000,8000@14/sd@1,0
2. c2t2d0 <VBOX-HARDDISK-1.0 cyl 2557 alt 2 hd 128 sec 32>
/pci@0,0/pci1000,8000@14/sd@2,0
3. c2t3d0 <VBOX-HARDDISK-1.0 cyl 2557 alt 2 hd 128 sec 32>
/pci@0,0/pci1000,8000@14/sd@3,0
4. c2t4d0 <VBOX-HARDDISK-1.0 cyl 1021 alt 2 hd 64 sec 32>
/pci@0,0/pci1000,8000@14/sd@4,0
5. c2t5d0 <VBOX-HARDDISK-1.0 cyl 1021 alt 2 hd 64 sec 32>
/pci@0,0/pci1000,8000@14/sd@5,0
6. c2t6d0 <VBOX-HARDDISK-1.0 cyl 1021 alt 2 hd 64 sec 32>
/pci@0,0/pci1000,8000@14/sd@6,0
Specify disk (enter its number): 5
selecting c2t5d0
[disk formatted]


FORMAT MENU:
disk - select a disk
type - select (define) a disk type
partition - select (define) a partition table
current - describe the current disk
format - format and analyze the disk
fdisk - run the fdisk program
repair - repair a defective sector
label - write label to the disk
analyze - surface analysis
defect - defect list management
backup - search for backup labels
verify - read and display labels
save - save new disk/partition definitions
inquiry - show disk ID
volname - set 8-character volume name
!<cmd> - execute <cmd>, then return
quit
format> p


PARTITION MENU:
0 - change `0' partition
1 - change `1' partition
2 - change `2' partition
3 - change `3' partition
4 - change `4' partition
5 - change `5' partition
6 - change `6' partition
7 - change `7' partition
select - select a predefined table
modify - modify a predefined partition table
name - name the current table
print - display the current table
label - write partition map and label to the disk
!<cmd> - execute <cmd>, then return
quit
partition> p
Current partition table (original):
Total disk cylinders available: 1021 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 1020 1021.00MB (1021/0/0) 2091008
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 1 - 1000 1000.00MB (1000/0/0) 2048000
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 - 0 1.00MB (1/0/0) 2048
9 unassigned wm 0 0 (0/0/0) 0

partition>

partition> 5
Part Tag Flag Cylinders Size Blocks
5 unassigned wm 1 - 1000 1000.00MB (1000/0/0) 2048000

Enter partition id tag[unassigned]:
Enter partition permission flags[wm]:
Enter new starting cyl[1]: 0
Enter partition size[2048000b, 1000c, 999e, 1000.00mb, 0.98gb]: 0
partition> l
Ready to label disk, continue? y

partition> p
Current partition table (unnamed):
Total disk cylinders available: 1021 + 2 (reserved cylinders)

Part Tag Flag Cylinders Size Blocks
0 unassigned wm 0 0 (0/0/0) 0
1 unassigned wm 0 0 (0/0/0) 0
2 backup wu 0 - 1020 1021.00MB (1021/0/0) 2091008
3 unassigned wm 0 0 (0/0/0) 0
4 unassigned wm 0 0 (0/0/0) 0
5 unassigned wm 0 0 (0/0/0) 0
6 unassigned wm 0 0 (0/0/0) 0
7 unassigned wm 0 0 (0/0/0) 0
8 boot wu 0 - 0 1.00MB (1/0/0) 2048
9 unassigned wm 0 0 (0/0/0) 0



- Once we formatted the disk slice associated with "DATA"  from the global zone then it must not allow any I/O operation on that Disk group.

Lets try to create some file on DG - DATA

grid12c@dbnode1:~$ asmcmd
ASMCMD> ls
DATA/
DATA4/
GRID/
ASMCMD> cd DATA
ASMCMD> ls
ASMCMD> mkdir test1
ORA-15032: not all alterations performed
ORA-15130: diskgroup "DATA" is being dismounted (DBD ERROR: OCIStmtExecute)
ASMCMD> exit
grid12c@dbnode1:~$


Content - ASM alert logfile:

Wed Oct 14 17:39:46 2015
NOTE: diskgroup resource ora.DATA4.dg is online
NOTE: diskgroup resource ora.DATA.dg is online
NOTE: diskgroup resource ora.GRID.dg is online
Wed Oct 14 18:35:01 2015
SQL> /* ASMCMD */alter diskgroup /*ASMCMD*/ "DATA" add directory '+DATA/test1'
Wed Oct 14 18:35:01 2015
WARNING: Write Failed. group:3 disk:1 AU:10 offset:57344 size:4096
path:/dev/rdsk/c2t5d0s5
incarnation:0xf0d0e8e9 synchronous result:'I/O error'
subsys:System krq:0xffff80ffbf733688 bufp:0x9e590000 osderr1:0x0 osderr2:0x0
IO elapsed time: 0 usec Time waited on I/O: 0 usec
NOTE: unable to write any mirror side for diskgroup DATA
Wed Oct 14 18:35:01 2015
Errors in file /u01/grid12c/diag/asm/+asm/+ASM/trace/+ASM_lgwr_14504.trc:
ORA-15080: synchronous I/O operation failed to write block 0 of disk 1 in disk group DATA
ORA-27072: File I/O error
Solaris-AMD64 Error: 5: I/O error
Additional information: 4
Additional information: 20592
Additional information: 4294967295
NOTE: cache initiating offline of disk 1 group DATA
NOTE: process _lgwr_+asm (14504) initiating offline of disk 1.4040222953 (DATA_0001) with mask 0x7e in group 3 with client assisting
NOTE: initiating PST update: grp 3 (DATA), dsk = 1/0xf0d0e8e9, mask = 0x6a, op = clear
Wed Oct 14 18:35:01 2015
GMON updating disk modes for group 3 at 75 for pid 10, osid 14504
ERROR: disk 1(DATA_0001) in group 3(DATA) cannot be offlined because the disk group has external redundancy.
Wed Oct 14 18:35:01 2015
ERROR: too many offline disks in PST (grp 3)
Wed Oct 14 18:35:01 2015
NOTE: cache dismounting (not clean) group 3/0x8DC01812 (DATA)
Wed Oct 14 18:35:01 2015
NOTE: sending clear offline flag message (3717243651) to 1 disk(s) in group 3
Wed Oct 14 18:35:01 2015
WARNING: Disk 1 (DATA_0001) in group 3 mode 0x7f offline is being aborted
Wed Oct 14 18:35:01 2015
NOTE: halting all I/Os to diskgroup 3 (DATA)
Wed Oct 14 18:35:01 2015
NOTE: unable to offline disks after getting write error for diskgroup DATA
Wed Oct 14 18:35:01 2015
Errors in file /u01/grid12c/diag/asm/+asm/+ASM/trace/+ASM_lgwr_14504.trc:
ORA-15066: offlining disk "DATA_0001" in group "DATA" may result in a data loss
ORA-27072: File I/O error
Solaris-AMD64 Error: 5: I/O error
Additional information: 4
Additional information: 20592
Additional information: 4294967295
NOTE: disk 1 had IO error
Wed Oct 14 18:35:01 2015
ORA-15032: not all alterations performed
ORA-15130: diskgroup "DATA" is being dismounted

Wed Oct 14 18:35:01 2015
ERROR: /* ASMCMD */alter diskgroup /*ASMCMD*/ "DATA" add directory '+DATA/test1'
Wed Oct 14 18:35:01 2015
NOTE: messaging CKPT to quiesce pins Unix process pid: 21743, image: oracle@dbnode1 (B000)
Wed Oct 14 18:35:02 2015
ERROR: ORA-15130 in COD recovery for diskgroup 3/0x8dc01812 (DATA)
ERROR: ORA-15130 thrown in RBAL for group number 3
Wed Oct 14 18:35:02 2015
Errors in file /u01/grid12c/diag/asm/+asm/+ASM/trace/+ASM_rbal_14514.trc:
ORA-15130: diskgroup "DATA" is being dismounted
Wed Oct 14 18:35:02 2015
NOTE: LGWR doing non-clean dismount of group 3 (DATA) thread 1
NOTE: LGWR sync ABA=7.12 last written ABA 7.13
Wed Oct 14 18:35:02 2015
NOTE: cache dismounted group 3/0x8DC01812 (DATA)
NOTE: cache deleting context for group DATA 3/0x8dc01812
Wed Oct 14 18:35:02 2015
SQL> alter diskgroup DATA dismount force /* ASM SERVER:2378176530 */
Wed Oct 14 18:35:02 2015
GMON dismounting group 3 at 76 for pid 23, osid 21743
Wed Oct 14 18:35:02 2015
NOTE: Disk DATA_0000 in mode 0x7f marked for de-assignment
NOTE: Disk DATA_0001 in mode 0x7f marked for de-assignment
SUCCESS: diskgroup DATA was dismounted
Wed Oct 14 18:35:02 2015
SUCCESS: alter diskgroup DATA dismount force /* ASM SERVER:2378176530 */
SUCCESS: ASM-initiated MANDATORY DISMOUNT of group DATA
Wed Oct 14 18:35:02 2015
NOTE: diskgroup resource ora.DATA.dg is offline
Wed Oct 14 18:35:02 2015
ASM Health Checker found 1 new failures

Once we format the configured ASM device from the global zone. Its allowing us to format it without any warning and it dismounted the ASM diskgroup immediately when we tried to perform the I/O on that specific Disk Group.

Correct Sequence for removing ASM devices from NGZ

  1. stop database or backup any data residing on the Disk group
  2. Drop the required disk group
  3. Stop Grid Infrastructure services (If using ZFS volumes)
  4. Remove devices from the global zone
  5. Apply the chnanges dynamically for reflecting the changes (applicable from solaris 11.2)
  6. Unprovisioned or destroy the devices from the global zone
  7. startup the Grid Infrastructure services

dropping the required diks group:

We must drop the ASM disk group before removing the devices from the non-global zone. 

grid12c@dbnode1:~$ sqlplus / as sysasm

SQL*Plus: Release 12.1.0.1.0 Production on Wed Oct 14 20:22:48 2015

Copyright (c) 1982, 2013, Oracle. All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Automatic Storage Management option

SQL> drop diskgroup data including contents;
drop diskgroup data including contents
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup "DATA" does not exist or is not mounted


SQL> drop diskgroup data force including contents;

Diskgroup dropped.

SQL>

As we formatted one of the devices from the global zone hence its not allowing us  to drop the DG normally, we must use force option for dropping the ASM  Disk Group.

content from ASM Alert log:

SQL> drop diskgroup data force including contents
Wed Oct 14 20:23:08 2015
NOTE: Assigning number (1,0) to disk (/dev/rdsk/c2t4d0s5)
NOTE: erasing header on grp 1 disk DATA_0000
NOTE: Disk DATA_0000 in mode 0x18 marked for de-assignment
SUCCESS: diskgroup DATA was force dropped
Wed Oct 14 20:23:14 2015
SUCCESS: drop diskgroup data force including contents

Remove devices from the non-global zone:

After dropping the Disk group we must remove the required devices from the non-global zone.

root@soltest:~# zonecfg -z dbnode1
zonecfg:dbnode1> info
zonename: dbnode1
zonepath: /dbzone/dbnode1
brand: solaris
autoboot: true
autoshutdown: shutdown
bootargs: -m verbose
file-mac-profile:
pool:
limitpriv:
scheduling-class:
ip-type: exclusive
hostid:
tenant:
fs-allowed:
anet:
linkname: net0
lower-link: auto
allowed-address not specified
configure-allowed-address: true
defrouter not specified
allowed-dhcp-cids not specified
link-protection: mac-nospoof
mac-address: auto
auto-mac-address: 2:8:20:1c:41:61
mac-prefix not specified
mac-slot not specified
vlan-id not specified
priority not specified
rxrings not specified
txrings not specified
mtu not specified
maxbw not specified
rxfanout not specified
vsi-typeid not specified
vsi-vers not specified
vsi-mgrid not specified
etsbw-lcl not specified
cos not specified
pkey not specified
linkmode not specified
evs not specified
vport not specified
device:
match: /dev/rdsk/c2t2d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t3d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t3d0s0
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t2d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t2d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t3d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t3d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t5d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t5d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/dsk/c2t4d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/rdsk/c2t4d0s5
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/dsk/dbzone/dbnode1/asmvol1
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk02
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk03
storage not specified
allow-partition not specified
allow-raw-io not specified
device:
match: /dev/zvol/rdsk/dbzone/dbnode1/asmdisk04
storage not specified
allow-partition not specified
allow-raw-io not specified
zonecfg:dbnode1>

We will remove the highlighted devices from the non-global zone. These are devices that was belonging to the Disk group "DATA3"

zonecfg:dbnode1> remove device match=/dev/rdsk/c2t4d0s5
zonecfg:dbnode1> remove device match=/dev/dsk/c2t4d0s5
zonecfg:dbnode1> remove device match=/dev/rdsk/c2t5d0s5
zonecfg:dbnode1> remove device match=/dev/dsk/c2t5d0s5
zonecfg:dbnode1> commit;
zonecfg:dbnode1>

After removing the devices from the non global zone the devices will not be visible at OS level as well as under the ASM Instance. But the devices will still be visible under global zone and we need to follow the regular OS utilities to remove these devices from the global zone.

Conclusion:

In this part of article we have seen different test cases for removing the ASM disk group from the non-global zone in Solaris 11.2. We have seen demonstration for removing two type of ASM devices one is ZFS volumes and another is raw devices. ZFS volumes are seems to much safer than regular raw devices under ASM instance. As ZFS is not allowing us to destroy the ZFS volume (even with force -f option) once the ASM instance is active. Its only allowing us to destroy the volume once the GI services are stopped.  Whereas we can format/modify any raw device from the global zone which is directly effecting the ASM Disk group. So we must be very careful if are using raw devices under ASM instance. Its very difficult to keep track which raw device is provisioned to which global zone. So its highly recommenced to maintain a catalog where we can keep all information related to raw devices that belong to non-global zones. It may not be difficult in environments where one or two non global zones are running. But Imagine a system with 100 non global zones and each NGZ has 10 - 20 raw devices, In such situation the device management will be a big challenge for DBA/System administrators.

On other hand ZFS volumes are safer to use but its not recommended  by oracle as ZFS itself is a volume manager and it has its own file system. Oracle do not recommend to use any other volume manager with ASM. It may create performance issues related to I/O.