I am getting more and more experience with patching clusters with the local-mode automaton. The whole process would be very complex, but the local-mode automaton makes it really easy.<\/p>\n
I have had nevertheless a couple of clusters where the process did not work:<\/strong><\/p>\n #1: The very first cluster that I installed in 18c<\/strong><\/p>\n This cluster has “kind of failed” patching the first node. Actually, the rhpctl command exited with an error:<\/p>\n But actually, the helper kept running and configured everything properly:<\/p>\n The cluster was OK on the first node, with the correct patch level. The second node, however, was failing with:<\/p>\n I am not sure about the cause, but let’s assume it is irrelevant for the moment.<\/p>\n #2: A cluster with new GI home not properly linked with RAC<\/strong><\/p>\n This was another funny case, where the first node patched successfully, but the second one failed upgrading in the middle of the process with a java NullPointer exception. We did a few bad tries of prePatch and postPatch to solve, but after that the second node of the cluster was in an inconsistent state: in ROLLING_UPGRADE mode and not possible to patch anymore.<\/p>\n Common solution: removing the node from the cluster and adding it back<\/strong><\/p>\n In both cases we were in the following situation:<\/p>\n So, for me, the easiest solution has been removing the failing node and adding it back with the new patched version.<\/p>\n Steps to remove the node<\/strong><\/p>\n Although the steps are described here: https:\/\/docs.oracle.com\/en\/database\/oracle\/oracle-database\/18\/cwadd\/adding-and-deleting-cluster-nodes.html#GUID-8ADA9667-EC27-4EF9-9F34-C8F65A757F2A, <\/a>there are a few differences that I will highlight:<\/p>\n Stop of the cluster:<\/p>\n The actual procedure to remove a node asks to deconfigure the databases and managed homes from the active cluster version. But as we manage our homes with golden images, we do not need this; we rather want to keep all the entries in the OCR so that when we add it back, everything is in place.<\/p>\n Once stopped the CRS, we have deinstalled the CRS home on the failing node:<\/p>\n This\u00a0 complained about the CRS that was down, but it continued and ask for this script to be executed:<\/p>\n We’ve got errors also for this script, but the remove process was OK afterall.<\/p>\n Then, from the surviving node:<\/p>\n Adding the node back<\/strong><\/p>\n From the surviving node, we ran gridSetup.sh<\/strong> and followed the steps to ad the node.<\/p>\n Wait before running root.sh<\/strong>.<\/p>\n In our case, we have originally installed the cluster starting with a SW_ONLY install. This type of installation keeps some leftovers in the configuration files that prevent the root.sh from configuring the cluster…we have had to modify rootconfig.sh:<\/p>\n then, after running root.sh and the config tools, everything was back as before removing the node form the cluster.<\/p>\n For one of the clusters , both nodes were at the same patch level, but the cluster was still in ROLLING_PATCH mode. So we have had to do a<\/p>\n —<\/p>\n Ludo<\/p>\n","protected":false},"excerpt":{"rendered":" I am getting more and more experience with patching clusters with the local-mode automaton. The whole process would be very complex, but the local-mode automaton makes it really easy. I have had nevertheless a couple of clusters where the process … Continue reading $ rhpctl move gihome -sourcehome \/u01\/crs\/crs1830 -desthome \/u01\/crs\/crs1860 -node server1\r\nserver1.cern.ch: Audit ID: 2\r\nserver1.cern.ch: verifying versions of Oracle homes ...\r\nserver1.cern.ch: verifying owners of Oracle homes ...\r\nserver1.cern.ch: verifying groups of Oracle homes ...\r\nserver1.cern.ch: starting to move the Oracle Grid Infrastructure home from \"\/u01\/crs\/crs1830\" to \"\/u01\/crs\/crs1860\" on server cluster \"AISTEST-RAC16\"\r\n[...]\r\n2019\/07\/08 09:45:06 CLSRSC-329: Replacing Clusterware entries in file 'oracle-ohasd.service'\r\nPRCG-1239 : failed to close a proxy connection\r\nConnection refused to host: server1.cern.ch; nested exception is:\r\n java.net.ConnectException: Connection refused (Connection refused)\r\nPRCG-1079 : Internal error: ClientFactoryImpl-submitAction-error1\r\nPROC-32: Cluster Ready Services on the local node is not running Messaging error [gipcretConnectionRefused] [29]<\/pre>\n
$ tail -f \/ORA\/dbs01\/oracle\/crsdata\/server1\/crsconfig\/crs_postpatch_server1_2019-07-08_09-41-36AM.log\r\n2019-07-08 09:55:25:\r\n2019-07-08 09:55:25: Succeeded in writing the checkpoint:'ROOTCRS_POSTPATCH' with status:SUCCESS\r\n2019-07-08 09:55:25: Executing cmd: \/u01\/crs\/crs1860\/bin\/clsecho -p has -f clsrsc -m 672\r\n2019-07-08 09:55:25: Executing cmd: \/u01\/crs\/crs1860\/bin\/clsecho -p has -f clsrsc -m 672\r\n2019-07-08 09:55:25: Command output:\r\n> CLSRSC-672: Post-patch steps for patching GI home successfully completed.\r\n>End Command output\r\n2019-07-08 09:55:25: CLSRSC-672: Post-patch steps for patching GI home successfully completed.\r\n<\/pre>\n
$ rhpctl move gihome -sourcehome \/u01\/crs\/crs1830 -desthome \/u01\/crs\/crs1860 -node server2\r\nserver1.cern.ch: retrieving status of databases ...\r\nserver1.cern.ch: retrieving status of services of databases ...\r\nPRCT-1011 : Failed to run \"rhphelper\". Detailed error: <HLP_EMSG>,RHPHELP_procCmdLine-05,<\/HLP_EMSG>,<HLP_VRES>3<\/HLP_VRES>,<HLP_IEEMSG>,PRCG-1079 : Internal error: RHPHELP122_main-01,<\/HLP_IEEMSG>,<HLP_ERES>1<\/HLP_ERES><\/pre>\n
\n
(root)# crsctl stop crs<\/pre>\n
(oracle)$ $OH\/deinstall\/deinstall -local<\/pre>\n
\/u01\/crs\/crs1830\/crs\/install\/rootcrs.sh -force -deconfig -paramfile \"\/tmp\/deinstall2019-07-08_11-37-20AM\/response\/deinstall_1830.rsp\"<\/pre>\n
root # crsctl delete node -n server2\r\noracle $ srvctl stop vip -vip server2\r\nroot $ srvctl remove vip -vip server2<\/pre>\n
check\/modify \/u01\/crs\/crs1860\/crs\/config\/rootconfig.sh and change this:\r\n# before:\r\n# SW_ONLY=true\r\n# after:\r\nSW_ONLY=false<\/pre>\n
(root) # crsctl stop rollingpatch<\/pre>\n