Restarting Oracle Clusterware 12.2.0.1 OUI (via gridSetup.sh) after a rootupgrade.sh failure

Gads!

I was doing a 12.2.0.1 Clusterware upgrade.... I'd run rootupgrade.sh node 1 (of 2 nodes). Everything was looking good, then on the next to last step it crashed.

The reasons for the crash are still unclear. Looking at the logs, there was a port that apparently wasn't listening when rootupgrade.sh tried to create a checkpoint via cluutil. Further, we suspect that something on the CRS shutdown/restart didn't go right and that there was a port that was either not available for connections or that was blocked.

Either way, this post isn't about the problem in general, after some discussion with Oracle, we simply stopped and restarted the CRS stack on node 1 (under the new 12.2 Grid Home) and then reran rootupgrade.sh - and it completed just fine. We then moved onto node 2, ran rootupgrade.sh there - which ran fine again.

However, after node 2's upgrade had completed, we had an interesting problem. The OUI session that had been started by running gridSetup.sh (and had prompted me to run rootupgrade.sh on each node) was no longer running. This was because I had started the OUI using a X session directly on my laptop, rather than running it out of a VNC connection. So, when it was time to go home, I had to let go of the OUI session. So sad....


This is a problem because after you run rootupgrade.sh on each node, there are some post clusterware install processes that need to be run. Sure, I could do them manually, but I prefer to let the OUI do them, to be honest. I'm just lazy that way.

So, the question is, how to restart the OUI via gridSetup.sh in order to finish the final grid setup steps (which include things like the creation of the MGMTDB database and the final run of the cluster verify utility) without it going through the entire upgrade cycle.

The answer is contained in Oracle documents 1360798.1, 1363785.1 and also the Oracle GI Install and Upgrade Guide. Yes, I know - the documentation.

The long and short of it is that in Oracle Clusterware 12.2, you can restart the OUI, and instruct it to start it's work by using the -executeCOnfigTools parameter. Also, Oracle Database 12.2 allows you to use the response file that was already created by the OUI during the initial Clusterware install. This is a new feature in 12.2, and it is covered in the documents above.

To re-start the OUI, we need to first find the response file used during the install. The install response file can be found in the NEW GRID_HOME directory using the path $GRID_HOME/install/response as seen in listing 2018-04-03-01:

Listing 2018-04-03-01 Listing of the Response Files
 [root@sbxrac1db1 ~]# ls -alt /app/grid/product/12.2.0.1_180116/gi_1/install/response
total 156
drwxr-xr-x. 11 oracle oinstall  4096 Apr  2 16:06 ..
-rw-r--r--.  1 oracle oinstall 35326 Mar 28 14:41 grid_2018-03-28_02-23-44PM.rsp
drwxr-xr-x.  2 oracle oinstall  4096 Mar 28 14:41 .
-rw-r--r--.  1 oracle oinstall 35266 Mar 20 15:57 grid_2018-03-20_03-52-23PM.rsp
-rw-r--r--.  1 oracle oinstall 34357 Jan 26  2017 grid_2017-01-26_04-10-28PM.rsp
-rw-r-----.  1 oracle oinstall 35087 Jan 26  2017 gridsetup.rsp
-rw-r-----.  1 oracle oinstall  1541 May 20  2016 sample.ccf
 There may be more than one response file, I'd suggest using the most current one.

Once you know which response file you are going to use, you are ready to run the gridSetup.sh command again, using the -executeCOnfigTools option as seen in listing 2018-04-03-02.

Listing 2018-04-03-02 Restarting gridSetup.sh After Failed, and Corrected rootupgrade.sh Execution
cd $GRID_HOME
./gridSetup.sh -executeCOnfigTools -responseFile /app/grid/product/12.2.0.1_180116/gi_1/install/response/grid_2018-03-28_02-23-44PM.rsp

Side Note
The ability to restart gridSetup.sh is available for earlier versions of gridSetup.sh too. However, you will have to create the response file yourself. The Oracle Support documents above cover this process in detail for you too.