This is completely understandable. Anything that changes the underlying architecture of a database system is sure to cause undue stress and apprehension.
Recently, I had the opportunity to go through this exercise and while it did require a good amount of planning, the actual steps to perform this activity were relatively simple and straightforward. The system we were resizing was an Oracle SuperCluster that uses Oracle ASM for memory storage. The SuperCluster was split into multiple zones with the end goal of moving free space from certain disk groups in one zone to other disk groups in other zones to avoid running out of free space.
One prerequisite that I would stress before starting this procedure is to run the exachk utility on your system and fix any issues that appear on the report. Also, you will want to be sure recent backups are valid and available, just in case.
I used both MOS Notes 1684112.1 and 2176737.1 as a reference. I strongly recommend reviewing them before going through this procedure on a live system. They feature links to other MOS notes featuring potential issues which could arise, as well as a section on known issues that could apply to your system. With this review, you will be mentally prepared for potential issues ahead of time with a solid game plan for resolution, just in case. Now, let’s get started.
Resizing ASM Disk Groups
1. Determine the amount of disk space that will be transferred between disk groups:
The easiest way to do this is to first run the below two queries on the ASM database as sysasm.
select dg.name, d.total_mb total_mb_per_disk, d.os_mb, count(1) num_disks
from v$asm_diskgroup dg, v$asm_disk d
where dg.group_number = d.group_number
and dg.name like '%DB01G%'
group by dg.name, d.total_mb, d.os_mb;
select name, total_mb, free_mb, total_mb - free_mb used_mb, round(100*free_mb/total_mb,2)
pct_free
from v$asm_diskgroup dg
where dg.name like '%DB01G%'
order by 1;
The results of these queries will show you what the current space allocation is. For our example, we will be moving space from DB01G_RECO to DB03P_DATA.
NAME TOTAL_MB_PER_DISK OS_MB NUM_DISKS
----------------------------- ----------------- ---------- ---------RECO_DB01G 582656 582656 36
NAME TOTAL_MB FREE_MB USED_MB PCT_FREE
------------------------- ---------- ---------- ---------- ---------RECO_DB01G 20975616 17155315 3820301 81.79
NAME TOTAL_MB_PER_DISK OS_MB NUM_DISKS
---------------------------- ----------------- ---------- ----------DATA_DB03P 171008 171008 36
NAME TOTAL_MB FREE_MB USED_MB PCT_FREE
-------------------------- ---------- ---------- ---------- --------DATA_DB03P 6156288 1098424 5057864 17.84
Looking at the results, we will move a total of 8108928MB from DB01G to DB03P. A final query to run before we get started is run on the cell nodes through cellcli:
CellCLI> list griddisk attributes name, size order by name
DB01G_RECO_GD00_stg01celadm01 569G
DB01G_RECO_GD01_stg01celadm01 569G
DB01G_RECO_GD02_stg01celadm01 569G
DB01G_RECO_GD03_stg01celadm01 569G
DB01G_RECO_GD04_stg01celadm01 569G
DB01G_RECO_GD05_stg01celadm01 569G
DB01G_RECO_GD06_stg01celadm01 569G
DB01G_RECO_GD07_stg01celadm01 569G
DB01G_RECO_GD08_stg01celadm01 569G
DB01G_RECO_GD09_stg01celadm01 569G
DB01G_RECO_GD10_stg01celadm01 569G
DB01G_RECO_GD11_stg01celadm01 569G
DB03P_DATA_GD00_stg01celadm01 167G
DB03P_DATA_GD01_stg01celadm01 167G
DB03P_DATA_GD02_stg01celadm01 167G
DB03P_DATA_GD03_stg01celadm01 167G
DB03P_DATA_GD04_stg01celadm01 167G
DB03P_DATA_GD05_stg01celadm01 167G
DB03P_DATA_GD06_stg01celadm01 167G
DB03P_DATA_GD07_stg01celadm01 167G
DB03P_DATA_GD08_stg01celadm01 167G
DB03P_DATA_GD09_stg01celadm01 167G
DB03P_DATA_GD10_stg01celadm01 167G
DB03P_DATA_GD11_stg01celadm01 167G
2. Verify the amount of space to be moved fits the 16MB boundary:
In this scenario, we want to move a total of 8108928 MB (approximately 8.1 TB) from one disk group to another disk group. Some of this operation is done at the grid disk level so we will want to divide this overall number by the number of grid disks we have. In our case, that is 36 grid disks (8108928/36 = 225248).
Finally, we want to make sure this value fits within a 16 MB boundary so that there is no wasted/lost space (225248/16= 14078). The resulting value is a whole number, so we are good. If this number had been a decimal you would want to modify the amount of disk space being moved until it can be divided by 16 evenly.
3. Shrink disk group through ASM:
You will resize the disk group to the size you want it to be. That involves just subtracting the current total with the amount you want to reduce. In this scenario, that number is 357408M. Finally, we can start making updates. On the database node as sysasm, run the below command:
$ sqlplus / as sysasm
alter diskgroup RECO_DB01G resize all size 357408M rebalance power 32;
select * from gv$asm_operation;
Re-run the second query until it displays “no rows selected”, then move on to the next step.
4. Shrink grid disks through cellcli:
The grid disks now need to be shrunk to the correct size. This needs to be done on each cell node.
alter griddisk DB01G_RECO_GD00_stg01celadm01, DB01G_RECO_GD01_stg01celadm01, DB01G_RECO_GD02_stg01celadm01, DB01G_RECO_GD03_stg01celadm01, DB01G_RECO_GD04_stg01celadm01, DB01G_RECO_GD05_stg01celadm01, DB01G_RECO_GD06_stg01celadm01, DB01G_RECO_GD07_stg01celadm01, DB01G_RECO_GD08_stg01celadm01, DB01G_RECO_GD09_stg01celadm01, DB01G_RECO_GD10_stg01celadm01, DB01G_RECO_GD11_stg01celadm01 size=357408M
alter griddisk DB01G_RECO_GD00_stg01celadm02, DB01G_RECO_GD01_stg01celadm02, DB01G_RECO_GD02_stg01celadm02, DB01G_RECO_GD03_stg01celadm02, DB01G_RECO_GD04_stg01celadm02, DB01G_RECO_GD05_stg01celadm02, DB01G_RECO_GD06_stg01celadm02, DB01G_RECO_GD07_stg01celadm02, DB01G_RECO_GD08_stg01celadm02, DB01G_RECO_GD09_stg01celadm02, DB01G_RECO_GD10_stg01celadm02, DB01G_RECO_GD11_stg01celadm02 size=357408M
alter griddisk DB01G_RECO_GD00_stg01celadm03, DB01G_RECO_GD01_stg01celadm03, DB01G_RECO_GD02_stg01celadm03, DB01G_RECO_GD03_stg01celadm03, DB01G_RECO_GD04_stg01celadm03, DB01G_RECO_GD05_stg01celadm03, DB01G_RECO_GD06_stg01celadm03, DB01G_RECO_GD07_stg01celadm03, DB01G_RECO_GD08_stg01celadm03, DB01G_RECO_GD09_stg01celadm03, DB01G_RECO_GD10_stg01celadm03, DB01G_RECO_GD11_stg01celadm03 size=357408M
Once these steps complete, you can run the first query from step 1 again. The numbers should match what the disk group and grid disks were altered to.
select dg.name, d.total_mb total_mb_per_disk, d.os_mb, count(1) num_disks from v$asm_diskgroup dg, v$asm_disk d
where dg.group_number = d.group_number
and dg.name like '%DB01G%'
group by dg.name, d.total_mb, d.os_mb;
NAME TOTAL_MB_PER_DISK OS_MB NUM_DISKS
---------------------------- ----------------- ---------- ----------RECO_DB01G 357408 357408 36
5. Grow grid disks through cellcli:
Now it’s time to grow the grid disks of disk group DB03P_DATA. These steps are done in the reverse order of how we shrunk space. We will grow the grid disks of the data group we want to add space to. In this case, that number will be (171008 + 225248 = 396256). On each cell node, run the below cellcli command:
alter griddisk DB03P_DATA_GD00_stg01celadm01, DB03P_DATA_GD01_stg01celadm01, DB03P_DATA_GD02_stg01celadm01, DB03P_DATA_GD03_stg01celadm01, DB03P_DATA_GD04_stg01celadm01, DB03P_DATA_GD05_stg01celadm01, DB03P_DATA_GD06_stg01celadm01, DB03P_DATA_GD07_stg01celadm01, DB03P_DATA_GD08_stg01celadm01, DB03P_DATA_GD09_stg01celadm01, DB03P_DATA_GD10_stg01celadm01, DB03P_DATA_GD11_stg01celadm01 size=396256M
alter griddisk DB03P_DATA_GD00_stg01celadm02, DB03P_DATA_GD01_stg01celadm02, DB03P_DATA_GD02_stg01celadm02, DB03P_DATA_GD03_stg01celadm02, DB03P_DATA_GD04_stg01celadm02, DB03P_DATA_GD05_stg01celadm02, DB03P_DATA_GD06_stg01celadm02, DB03P_DATA_GD07_stg01celadm02, DB03P_DATA_GD08_stg01celadm02, DB03P_DATA_GD09_stg01celadm02, DB03P_DATA_GD10_stg01celadm02, DB03P_DATA_GD11_stg01celadm02 size=396256M
alter griddisk DB03P_DATA_GD00_stg01celadm03, DB03P_DATA_GD01_stg01celadm03, DB03P_DATA_GD02_stg01celadm03, DB03P_DATA_GD03_stg01celadm03, DB03P_DATA_GD04_stg01celadm03, DB03P_DATA_GD05_stg01celadm03, DB03P_DATA_GD06_stg01celadm03, DB03P_DATA_GD07_stg01celadm03, DB03P_DATA_GD08_stg01celadm03, DB03P_DATA_GD09_stg01celadm03, DB03P_DATA_GD10_stg01celadm03, DB03P_DATA_GD11_stg01celadm03 size=396256M
6. Grow disk group through ASM:
The final step in this process is to grow the disk group size through ASM. You will notice that for this command, we don’t even need to put the number that we want to grow the disk group to. That’s because the grid disks have already been updated, so the disk group will grow to the size that was previously defined. This command is run from the database node as sysasm:
alter diskgroup DATA_DB03P resize all rebalance power 32;
select * from gv$asm_operation;
Once this completes, you are done! You can verify that everything has moved over by running the same queries from Step 1.
Prep work makes this easy
Overall, this process was much faster than originally anticipated. While age/power of the system, network speed, and network traffic all play a part in the duration of this process; with the above procedure, we were able to move around 8TB of space and have everything configured in under ten minutes.
The main takeaway from this is that as long as you put in the time to correctly calculate and verify the amount of space you want to move, the process itself is not very time-consuming and relatively stress-free.
