Upgrading Oracle 12.2 Clusterware... ASM Notes of Interest...

If you are new to Oracle 12.2 RAC, you may have heard that Oracle is requiring you to use ASM. While you don't have to store the entire database in ASM, you do have to store the voting disks and the OCR registry into ASM. But, that's not all... as you will see shortly.

Why Move to ASM Before You Upgrade?

If you are not running ASM now, I strongly suggest you make the move to ASM before you upgrade to the 12.2 Grid Infrastructure. Why is this? Here are a few thoughts:

1. You are going to face a learning curve with ASM. Tackle it before you tackle the learning curve related to Oracle 12.2.
2. Even if you are an ASM master, just the process of upgrading between Oracle Clusterware and RAC database versions is a big bowl of risk anyway. Divide and conquer the upgrade, and the complexity, by upgrading to ASM first.
3. Making sure that ASM is up, running and stable before you do your 12.2 upgrade will help to ensure that the 12.2 upgrade goes more smoothly. ASM is a critical component of the GI, so having it already stable goes a long way towards success.
4. While upgrading to ASM pre-12.2, if something goes wrong backing out is easy. Once you are in the middle of a 12.2 Clusterware upgrade, having an stability problem with ASM will only make things worse at upgrade time - and backing out is much harder.  

Some ASM Upgrade Tips...

Use the ASMCA...

To install ASM you can use the ASM Configuration assistant (asmca). It's located in the bin directory in your Grid software home. It will setup the ASM instance for you, and offer you the option to create ASM disk groups - of course, if you have not made any ASM disks available first, then you won't be creating any ASM diskgroups!

Creating Your ASM Disks

There are a few different ways of creating ASM disks. Here, we use Netapp on Linux. The result is the presentation of shared, cooked file systems. Once the file systems are mounted, we pre-create the ASM disks using dd as seen here:

dd if=/dev/zero of=/asm01/grid/grid1 bs=1024k count=700 oflag=direct
dd if=/dev/zero of=/asm01/grid/grid2 bs=1024k count=700 oflag=direct
dd if=/dev/zero of=/asm01/grid/grid3 bs=1024k count=700 oflag=direct
dd if=/dev/zero of=/asm01/grid/mgmtdb1 bs=1024k count=35000 oflag=direct

Make sure the new disk has the ownership and permissions required by your operating system. In our case, oracle:dba are the owner and group memberships. The ASM disks have a 660 permissions as well.

Once you get your ASM instance up, you simply connect to the ASM instance. Then, use the alter system command to set the ASM_DISKSTRING parameter to point to the directory where you created your ASM disks, and then use the ASM create diskgroup command to create a diskgroup.

Here is an example of these commands in use. You will notice that I am creating two disk groups. There is a reason for that, which I'll discuss next.

alter system set asm_diskstring='/asm01/grid/*';

create diskgroup GRID normal redundancy
failgroup falgroup1 disk '/asm01/grid/grid1' name griddg1 size 700m
failgroup falgroup2 disk '/asm01/grid/grid2' name griddg2 size 700m
failgroup falgroup3 disk '/asm01/grid/grid3' name griddg3 size 700m
ATTRIBUTE 'compatible.asm'='11.2.0.0.0','compatible.rdbms'='11.2.0.0.0', 'au_size'='1M';

create diskgroup MGMTDB external redundancy
disk '/asm01/grid/mgmtdb1' name mgmtdb1 size 35000m
ATTRIBUTE 'compatible.asm'='11.2.0.2.0','compatible.rdbms'='11.2.0.0.2', 'au_size'='1M'; 

Balloons - MGMTDB

If you upgrade to ASM prior to your Oracle 12.2 upgrade, you need to be aware of the space requirements that Oracle 12.2 Grid Infrastructure will demand of you, and plan for those requirements. If you are upgrading to ASM while on Oracle 11.2 or 12.1, then all you need to do is allocate enough space in your disk groups for the voting disks and the OCR. Together, quite a slim storage requirement. Typically, these get put in their own diskgroup. I often will call that diskgroup something like GRID or CRS.

However, you will be surprised when you upgrade to Oracle GI 12.2 that the space requirements for the GRID diskgroup skyrocket. This is because of a new requirement in the form of a database called MGMTDB.

MGMTDB is a database that Oracle uses to manage Clusterware related data. MGMTDB was introduced in Oracle 12.1, and then it was an optional component. In Oracle 12.2, MGMTDB is no longer an option - and it's inclusion adds greatly to the disk space required to install Oracle 12.2 Clusterware.

The minimum disk space requirements for Clusterware in 12.2 have ballooned to around 32GB of space (see the documentation for the exact number as it varies based on a few different factors). That is just for a disk group setup to use external redundancy. Now, consider that it's a really good idea to have your voting disk files and OCR files on more than one disk (or in more than one failgroup in ASM) and the size requirement skyrockets (3 failgroups would required 3 times the space for example).

Add to this the fact that when you upgrade Clusterware, you have to install MGMTDB in the same ASM diskgroup that contains the voting disk and the OCR. The OUI does not give you an option to create the MGMTDB database in a different disk group - which I think is simply insane.

If you are a seasoned DBA, the inclusion of database files with the OCR and the voting disks should make your skin crawl. Fear not - there is a solution - all be it a round-a-bout one.

Deflating the Balloons - Kind of....

So, what should you do? First, you have no choice, you have to install MGMTDB, and it has to initially be included in the ASM disk group where the Clusterware related files are located. After that, however, Oracle gives you the option to drop and re-create the MGMTDB database, which effectively provides the ability to move that database to another diskgroup. I'd avoid the thought of just not re-creating it all-together, because you never know what problems that might cause now, or in the future.

However, there is another approach that I think is less time consuming than dropping and recreating the MGMTDB database. Here is what I've done for the last few upgrades, and it seems to work really well. 

1. Stand up your ASM instance and convert your RAC cluster to use it before the 12.2 upgrade. We will call this initial diskgroup GRID. Initially, just make the GRID diskgroup big enough for the OCR, voting disks and the ASM SPFILE.  I'd make sure this disk group provides enough protection for your Clusterware files with at least 3 failgroups, with the ASM disks as isolated from each other as possible.
   
2. Before you upgrade to 12.2, add a new disk group called MGMTDB. This disk group must be big enough to create the MGMTDB database in it, as well as store the other Clusterware related files.
   
   This diskgroup will need to be about 35GB in size. If space is an issue, you can make this disk group one that uses external redundancy. We won't need it that long, just for the time we are doing the upgrade.
3. Before you start your 12.2 upgrade, move the voting disk, the OCR and the ASM SPFILE to the new MGMTDB disk group.
   
   This new group will provide enough space for the new MGMTDB database, as well as the storage required by the Clusterware files. We will move the Clusterware related files back later. The basic commands that  you will use to move these files include:
   
   
   • ocrconfig -add +MGMTDB
   • ocrconfig -delete +GRID
   • crsctl -replace votedisk +MGMTDB
   • ocrcheck
   
   Also, from ASM, you need to move the SPFILE from the GRID diskgroup to the new MGMTDB diskgroup. Do keep in mind, when we have moved these files to the MGMTDB disk group, we have a somewhat lower level of file protection. This should be OK for the short term, but you need to determine if this will meet the recovery SLA's that you must meet.
4. Having moved all of the Clusterware files to MGMTDB, you can perform your Clusterware Upgrade. The Oracle Installer will create the MGMTDB database in the GRID_TEMP disk group as a part of the upgrade process.
5. Once the upgrade has completed, and you feel certain that you have a stable system, reverse the process in step 3 and move the voting files, the OCR and the ASM SPFILE back to the original GRID diskgroup. Leave the MGMTDB database in the MGMTDB diskgroup.

Oracle also offers the ability to drop and recreate the MGMTDB database. You can review that process by bringing up document 1589394.1 when connected to Oracle Support.

The one thing I've had problems with is moving the SPFILE in ASM. It seems to sometimes get bent out of sort and I've had to shutdown and restart the ASM instance once or twice to get everything to work right. I've seen a couple of bugs on Metalink, but I've not really dug into it that deeply.

Also, keep in mind that you can use the create spfile from memory command to create the SPFILE in the diskgroup that you want it in. In some cases, this can prevent you from having to create a pfile and altering it, before re-creating the SPFILE in the new diskgroup.

Finally, the nice thing in Oracle Clusterware in 12c, is that the local database instance does not depend on the local ASM instance in order to remain open. Thus far, I've not had to take an outage when I needed to shutdown ASM to deal with parameter file issues. No outage is ... well, no outage.