{"id":1162,"date":"2015-09-27T21:32:35","date_gmt":"2015-09-27T19:32:35","guid":{"rendered":"http:\/\/www.ludovicocaldara.net\/dba\/?p=1162"},"modified":"2020-08-18T16:30:04","modified_gmt":"2020-08-18T14:30:04","slug":"recover-grid-gimr-12c","status":"publish","type":"post","link":"https:\/\/www.ludovicocaldara.net\/dba\/recover-grid-gimr-12c\/","title":{"rendered":"Grid Infrastructure 12c: Recovering the GRID Disk Group and recreating the GIMR"},"content":{"rendered":"

Losing the Disk Group that contains OCR and voting files has always been a challenge. It requires you to take regular backups of OCR<\/strong>, spfile<\/strong> and diskgroup metadata<\/strong>.<\/p>\n

Since Oracle 12cR1, there are a few additional components you must take care of:<\/p>\n

– The ASM password file<\/strong> (if you have Flex ASM it can be quite critical)<\/p>\n

– The Grid Infrastructure Management Repository<\/strong><\/p>\n

Why ASM password file is important? Well, you can read this good blog post form my colleague Robert Bialek:\u00a0http:\/\/blog.trivadis.com\/b\/robertbialek\/archive\/2014\/10\/26\/are-you-using-oracle-12c-flex-asm-if-yes-do-you-have-asm-password-file-backup.aspx<\/a><\/p>\n

So the problem here, is not whether you should back them up or not, but how you can restore them quickly.<\/p>\n

Assumptions: you back up regularly:<\/p>\n

ASM parameter\u00a0 file:<\/strong><\/p>\n

SQL> create pfile='\/backup\/spfileASM.ora' from spfile;\r\n\r\nFile created.<\/pre>\n
Oracle Cluster Registry:<\/strong><\/p>\n
grid@tvdrach01:~\/ [+ASM1] sudo $ORACLE_HOME\/bin\/ocrconfig -manualbackup\r\ntvdrach03 2015\/09\/21 14:30:39 \/u01\/app\/grid\/12.1.0.2\/cdata\/tvdrac-cluster\/backup_20150921_143039.ocr 0<\/pre>\n
ASM Diskgroup Metadata:<\/strong><\/p>\n
ASMCMD [+] > md_backup GRID.dg -G GRID\r\nDisk group metadata to be backed up: GRID\r\nCurrent alias directory path: _MGMTDB\/DATAFILE\r\nCurrent alias directory path: _MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\r\nCurrent alias directory path: tvdrac-cluster\r\nCurrent alias directory path: _MGMTDB\/FD9AC0F7C36E4438E043B6A9E80A24D5\/DATAFILE\r\nCurrent alias directory path: _MGMTDB\/FD9AC0F7C36E4438E043B6A9E80A24D5\r\nCurrent alias directory path: ASM\/PASSWORD\r\nCurrent alias directory path: _MGMTDB\/TEMPFILE\r\nCurrent alias directory path: tvdrac-cluster\/ASMPARAMETERFILE\r\nCurrent alias directory path: _MGMTDB\/20BC39F0F36C18F4E0533358A8C058F7\/TEMPFILE\r\nCurrent alias directory path: _MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\r\nCurrent alias directory path: _MGMTDB\/20BC2691871B0B14E0533358A8C01AC6\r\nCurrent alias directory path: _MGMTDB\/ONLINELOG\r\nCurrent alias directory path: _MGMTDB\r\nCurrent alias directory path: ASM\r\nCurrent alias directory path: tvdrac-cluster\/OCRFILE\r\nCurrent alias directory path: _MGMTDB\/20BC39F0F36C18F4E0533358A8C058F7\r\nCurrent alias directory path: _MGMTDB\/20BC2691871B0B14E0533358A8C01AC6\/TEMPFILE\r\nCurrent alias directory path: _MGMTDB\/CONTROLFILE\r\nCurrent alias directory path: _MGMTDB\/PARAMETERFILE\r\n<\/pre>\n
ASM password file:<\/strong><\/p>\n
ASMCMD [+GRID] > pwcopy +GRID\/orapwASM \/backup\/\r\ncopying +GRID\/orapwASM -> \/backup\/orapwASM<\/pre>\n
What about the GIMR<\/strong>?<\/p>\n
According to the MOS Note: FAQ: 12c Grid Infrastructure Management Repository (GIMR) (Doc ID 1568402.1),\u00a0<\/strong>there is no such need for the moment.<\/p>\n
Weird, huh? The -MGMTDB itself contains\u00a0for the moment<\/em> just the Cluster Health Monitor\u00a0repository, but expect to see its important increasing with the next versions of Oracle Grid Infrastructure.<\/p>\n
If you REALLY want to back it up (even if not fundamental, it is not a bad idea, after all), you can do it.<\/p>\n
The -MGMTDB is in noarchivelog<\/strong> by default. You need to either put it in archivelog mode (and set a recovery area, etc etc) or back it up while it is mounted.<\/p>\n
Because the Cluster Health Monitor (ora.crf) \u00a0depends on it, you have to stop it beforehand:<\/p>\n
grid@tvdrach01:~\/ [-MGMTDB] crsctl stop resource ora.crf -init\r\nCRS-2673: Attempting to stop 'ora.crf' on 'tvdrach01'\r\nCRS-2677: Stop of 'ora.crf' on 'tvdrach01' succeeded\r\n<\/pre>\n
Then you can operate with -MGMTDB:<\/p>\n
grid@tvdrach01:~\/ [-MGMTDB] srvctl stop mgmtdb -stopoption IMMEDIATE\r\ngrid@tvdrach01:~\/ [-MGMTDB] srvctl start mgmtdb -startoption MOUNT\r\n\r\ngrid@tvdrach01:~\/ [-MGMTDB]\r\n\r\ngrid@tvdrach02:~\/ [-MGMTDB] rman\r\n\r\nRecovery Manager: Release 12.1.0.2.0 - Production on Sun Sep 27 17:59:55 2015\r\n\r\nCopyright (c) 1982, 2014, Oracle and\/or its affiliates.  All rights reserved.\r\n\r\nRMAN> connect target \/\r\n\r\nconnected to target database: _MGMTDB (DBID=1095800268, not open)\r\n\r\nRMAN> backup as compressed backupset database format '+DATA';\r\n\r\nStarting backup at 27-SEP-15\r\nusing target database control file instead of recovery catalog\r\nallocated channel: ORA_DISK_1\r\nchannel ORA_DISK_1: SID=24 device type=DISK\r\nchannel ORA_DISK_1: starting compressed full datafile backup set\r\nchannel ORA_DISK_1: specifying datafile(s) in backup set\r\ninput datafile file number=00011 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/sysmgmtdata.269.891526555\r\ninput datafile file number=00007 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/system.270.891526555\r\ninput datafile file number=00008 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/sysaux.271.891526555\r\ninput datafile file number=00010 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/sysgridhomedata.272.891526555\r\ninput datafile file number=00012 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/sysmgmtdatadb.273.891526555\r\ninput datafile file number=00009 name=+GRID\/_MGMTDB\/FD9B43BF6A646F8CE043B6A9E80A2815\/DATAFILE\/users.274.891526555\r\nchannel ORA_DISK_1: starting piece 1 at 27-SEP-15\r\nchannel ORA_DISK_1: finished piece 1 at 27-SEP-15\r\npiece handle=+DATA\/_MGMTDB\/20BC39F0F36C18F4E0533358A8C058F7\/BACKUPSET\/2015_09_27\/nnndf0_tag20150927t180016_0.256.891540019 tag=TAG20150927T180016 comment=NONE\r\nchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:25\r\nchannel ORA_DISK_1: starting compressed full datafile backup set\r\nchannel ORA_DISK_1: specifying datafile(s) in backup set\r\ninput datafile file number=00001 name=+GRID\/_MGMTDB\/DATAFILE\/system.258.891526155\r\ninput datafile file number=00003 name=+GRID\/_MGMTDB\/DATAFILE\/sysaux.257.891526135\r\ninput datafile file number=00004 name=+GRID\/_MGMTDB\/DATAFILE\/undotbs1.259.891526181\r\nchannel ORA_DISK_1: starting piece 1 at 27-SEP-15\r\nchannel ORA_DISK_1: finished piece 1 at 27-SEP-15\r\npiece handle=+DATA\/_MGMTDB\/BACKUPSET\/2015_09_27\/nnndf0_tag20150927t180016_0.257.891540043 tag=TAG20150927T180016 comment=NONE\r\nchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:25\r\nchannel ORA_DISK_1: starting compressed full datafile backup set\r\nchannel ORA_DISK_1: specifying datafile(s) in backup set\r\ninput datafile file number=00005 name=+GRID\/_MGMTDB\/FD9AC0F7C36E4438E043B6A9E80A24D5\/DATAFILE\/system.265.891526233\r\ninput datafile file number=00006 name=+GRID\/_MGMTDB\/FD9AC0F7C36E4438E043B6A9E80A24D5\/DATAFILE\/sysaux.266.891526233\r\nchannel ORA_DISK_1: starting piece 1 at 27-SEP-15\r\nchannel ORA_DISK_1: finished piece 1 at 27-SEP-15\r\npiece handle=+DATA\/_MGMTDB\/20BC2691871B0B14E0533358A8C01AC6\/BACKUPSET\/2015_09_27\/nnndf0_tag20150927t180016_0.258.891540069 tag=TAG20150927T180016 comment=NONE\r\nchannel ORA_DISK_1: backup set complete, elapsed time: 00:00:15\r\nFinished backup at 27-SEP-15\r\n\r\nStarting Control File and SPFILE Autobackup at 27-SEP-15\r\npiece handle=\/u01\/app\/grid\/12.1.0.2\/dbs\/c-1095800268-20150927-00 comment=NONE\r\nFinished Control File and SPFILE Autobackup at 27-SEP-15\r\n\r\nRMAN> alter database open;\r\n\r\nStatement processed\r\n\r\nRMAN>\r\n<\/pre>\n
Now, imagine that you loose the GRID diskgroup (nowadays, with the ASM Filter Driver, it’s more complex to corrupt a device by mistake, but let’s assume that you do it):<\/p>\n
root@tvdrach01:~\/ [-MGMTDB] dd if=\/dev\/zero of=\/dev\/asm-disk1 bs=1M count=128\r\n128+0 records in\r\n128+0 records out\r\n134217728 bytes (134 MB) copied, 0.360653 s, 372 MB\/s\r\n<\/pre>\n
The cluster will not start anymore, you need to disable the crs, reboot and start it in exclusive mode:<\/p>\n
root@tvdrach01:~\/ [-MGMTDB] crsctl start crs -excl -nocrs\r\nCRS-4123: Oracle High Availability Services has been started.\r\nCRS-2672: Attempting to start 'ora.evmd' on 'tvdrach01'\r\nCRS-2672: Attempting to start 'ora.mdnsd' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.mdnsd' on 'tvdrach01' succeeded\r\nCRS-2676: Start of 'ora.evmd' on 'tvdrach01' succeeded\r\nCRS-2672: Attempting to start 'ora.gpnpd' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.gpnpd' on 'tvdrach01' succeeded\r\nCRS-2672: Attempting to start 'ora.cssdmonitor' on 'tvdrach01'\r\nCRS-2672: Attempting to start 'ora.gipcd' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.cssdmonitor' on 'tvdrach01' succeeded\r\nCRS-2676: Start of 'ora.gipcd' on 'tvdrach01' succeeded\r\nCRS-2672: Attempting to start 'ora.cssd' on 'tvdrach01'\r\nCRS-2672: Attempting to start 'ora.diskmon' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.diskmon' on 'tvdrach01' succeeded\r\nCRS-2676: Start of 'ora.cssd' on 'tvdrach01' succeeded\r\nCRS-2672: Attempting to start 'ora.drivers.acfs' on 'tvdrach01'\r\nCRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'tvdrach01'\r\nCRS-2672: Attempting to start 'ora.ctssd' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.ctssd' on 'tvdrach01' succeeded\r\nCRS-2676: Start of 'ora.drivers.acfs' on 'tvdrach01' succeeded\r\nCRS-2676: Start of 'ora.cluster_interconnect.haip' on 'tvdrach01' succeeded\r\nCRS-2672: Attempting to start 'ora.asm' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.asm' on 'tvdrach01' succeeded\r\nroot@tvdrach01:~\/ [-MGMTDB]<\/pre>\n
 <\/p>\n
Then you can recreate\u00a0the GRID disk group and restore everything inside it:<\/p>\n
SQL> alter system set asm_diskstring='\/dev\/asm*';\r\n\r\nSystem altered.\r\n\r\nSQL> create diskgroup GRID  external redundancy disk '\/dev\/asm-disk1' attribute 'COMPATIBLE.ADVM'='12.1.0.0.0', 'COMPATIBLE.ASM'='12.1.0.0.0';\r\n\r\nDiskgroup created.\r\n\r\nSQL> create spfile='+GRID' from pfile='\/backup\/spfileASM.ora';\r\n\r\nFile created.\r\n\r\nSQL> \r\n\r\nroot@tvdrach01:~\/ [+ASM1] ocrconfig -restore \/u01\/app\/grid\/12.1.0.2\/cdata\/tvdrac-cluster\/backup_20150927_174702.ocr\r\nroot@tvdrach01:~\/ [+ASM1]\r\n\r\ngrid@tvdrach01:~\/ [+ASM1] crsctl replace votedisk '+GRID'\r\nSuccessful addition of voting disk a375f4bdb7854f8fbf7a92cd880fba60.\r\nSuccessfully replaced voting disk group with +GRID.\r\nCRS-4266: Voting file(s) successfully replaced\r\n\r\n\r\nroot@tvdrach01:~\/ [+ASM1]  crsctl stop crs -f\r\n...\r\nroot@tvdrach01:~\/ [+ASM1]  crsctl start crs\r\n...\r\n\r\n\r\nASMCMD [+] >  pwcopy --asm \/backup\/orapwASM +GRID\/orapwASM\r\ncopying \/backup\/orapwASM -> +GRID\/orapwASM<\/pre>\n
Finally, the last missing component: the GIMR.<\/strong><\/p>\n
You can recreate it or restore it (if you backed it up at some point in time).<\/p>\n
Let’s see how to recreate it:<\/strong><\/p>\n
grid@tvdrach03:~\/ [-MGMTDB] srvctl disable mgmtdb\r\ngrid@tvdrach03:~\/ [-MGMTDB] srvctl remove mgmtdb\r\nRemove the database _mgmtdb? (y\/[n]) y\r\ngrid@tvdrach01:~\/ [+ASM1] dbca -silent -createDatabase -sid -MGMTDB \\\r\n> -createAsContainerDatabase true -templateName MGMTSeed_Database.dbc \\\r\n> -gdbName _mgmtdb -storageType ASM -diskGroupName +GRID \\\r\n> -datafileJarLocation $ORACLE_HOME\/assistants\/dbca\/templates -characterset AL32UTF8 \\\r\n> -autoGeneratePasswords -skipUserTemplateCheck\r\nCleaning up failed steps\r\n5% complete\r\nRegistering database with Oracle Grid Infrastructure\r\n11% complete\r\nCopying database files\r\n12% complete\r\n14% complete\r\n21% complete\r\n27% complete\r\n34% complete\r\n41% complete\r\n44% complete\r\nCreating and starting Oracle instance\r\n46% complete\r\n51% complete\r\n52% complete\r\n53% complete\r\n58% complete\r\n62% complete\r\n63% complete\r\n66% complete\r\nCompleting Database Creation\r\n70% complete\r\n80% complete\r\n90% complete\r\n100% complete\r\nLook at the log file \"\/u01\/app\/oracle\/cfgtoollogs\/dbca\/_mgmtdb\/_mgmtdb0.log\" for further details.\r\ngrid@tvdrach01:~\/ [+ASM1] dbca -silent -createPluggableDatabase -sourceDB -MGMTDB \\\r\n>  -pdbName tvdrac_cluster -createPDBFrom RMANBACKUP \\\r\n>  -PDBBackUpfile $ORACLE_HOME\/assistants\/dbca\/templates\/mgmtseed_pdb.dfb \\\r\n>  -PDBMetadataFile $ORACLE_HOME\/assistants\/dbca\/templates\/mgmtseed_pdb.xml \\\r\n>  -createAsClone true -internalSkipGIHomeCheck\r\nCreating Pluggable Database\r\nCreating Pluggable Database\r\n4% complete\r\n12% complete\r\n21% complete\r\n38% complete\r\n55% complete\r\nO-GRINF Grid Infrastructure Disaster Recovery\r\nPage 21\r\n85% complete\r\nCompleting Pluggable Database Creation\r\n100% complete\r\nLook at the log file \"\/u01\/app\/oracle\/cfgtoollogs\/dbca\/_mgmtdb\/tvdrac_cluster\/_mgmtdb.log\" for further details.\r\ngrid@tvdrach01:~\/ [+ASM1] srvctl status mgmtdb\r\nDatabase is enabled\r\nInstance -MGMTDB is running on node tvdrach01\r\n\r\ngrid@tvdrach01:~\/ [+ASM1] sudo $ORACLE_HOME\/bin\/crsctl modify res ora.crf -attr ENABLED=1 -init\r\ngrid@tvdrach01:~\/ [+ASM1] crsctl start res ora.crf -init\r\nCRS-2672: Attempting to start 'ora.crf' on 'tvdrach01'\r\nCRS-2676: Start of 'ora.crf' on 'tvdrach01' succeeded\r\ngrid@tvdrach01:~\/ [+ASM1]<\/pre>\n
Conclusion<\/strong><\/p>\n
Recovering from a lost Disk Group \/ Cluster is not rocket science. Just practice it every now and then. If you do not have a test RAC, you can build your lab on your laptop using the RAC Attack instructions<\/a>. If you want to test all the scenarios, the RAC SIG webcast: Oracle 11g Clusterware failure scenarios with practical demonstrations by Kamran Agayev<\/a> is the best starting point, IMHO. Just keep in mind that Flex ASM and the GIMR add more complexity.<\/p>\n
HTH<\/p>\n
—<\/p>\n
Ludovico<\/p>\n","protected":false},"excerpt":{"rendered":"
Losing the Disk Group that contains OCR and voting files has always been a challenge. It requires you to take regular backups of OCR, spfile and diskgroup metadata. Since Oracle 12cR1, there are a few additional components you must take … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[327,326,3,52,330,149,132],"tags":[],"class_list":["post-1162","post","type-post","status-publish","format-standard","hentry","category-oracle-maa","category-oracle","category-oracledb","category-12c","category-oracle-inst-upg","category-oracle-rac","category-triblog"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/1162","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/comments?post=1162"}],"version-history":[{"count":7,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/1162\/revisions"}],"predecessor-version":[{"id":1198,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/1162\/revisions\/1198"}],"wp:attachment":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/media?parent=1162"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/categories?post=1162"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/tags?post=1162"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}