{"id":744,"date":"2014-08-29T00:27:41","date_gmt":"2014-08-28T22:27:41","guid":{"rendered":"http:\/\/www.ludovicocaldara.net\/dba\/?p=744"},"modified":"2020-08-18T16:39:42","modified_gmt":"2020-08-18T14:39:42","slug":"a-pdb-is-cloned-while-in-read-write-data-guard-loose-its-marbles-12-1-0-2-ora-19729","status":"publish","type":"post","link":"https:\/\/www.ludovicocaldara.net\/dba\/a-pdb-is-cloned-while-in-read-write-data-guard-loose-its-marbles-12-1-0-2-ora-19729\/","title":{"rendered":"A PDB is cloned while in read-write, Data Guard loose its marbles (12.1.0.2, ORA-19729)"},"content":{"rendered":"<p style=\"padding-left: 30px;\"><em><strong>UPDATE<\/strong>: please check my more recent post about this problem and the information I&#8217;ve got at the Oracle Demo Grounds during OOW14: <a href=\"https:\/\/www.ludovicocaldara.net\/dba\/demo-grounds-clone-pdb-rw\/\">https:\/\/www.ludovicocaldara.net\/dba\/demo-grounds-clone-pdb-rw\/<\/a><\/em><\/p>\n<p>I feel the strong need to blog abut this very recent problem because I&#8217;ve spent a lot of time debugging it&#8230; especially because there&#8217;s no information about this error on the MOS.<\/p>\n<p><strong>Introduction<\/strong><br \/>\nFor a lab, I have prepared two RAC Container databases in physical stand-by.<br \/>\nReal-time query is configured (real-time apply, standby in read-only mode).<\/p>\n<p>Following the doc, <a href=\"http:\/\/docs.oracle.com\/database\/121\/SQLRF\/statements_6010.htm#CCHDFDDG\">http:\/\/docs.oracle.com\/database\/121\/SQLRF\/statements_6010.htm#CCHDFDDG<\/a>, I&#8217;ve cloned one local pluggable database to a new PDB and, because Active Data Guard is active, I was expecting the PDB to be created on the standby and its files copied without problems.<\/p>\n<p>BUT! I&#8217;ve forgot to put my source PDB in read-only mode on the primary and, strangely:<\/p>\n<ul>\n<li>The pluggable database has been created on the primary WITHOUT PROBLEMS (despite the documentation explicitly states that it needs to be read-only)<\/li>\n<li>The recovery process on the standby stopped with error.<\/li>\n<\/ul>\n<pre class=\"lang:plsql decode:true\">Recovery copied files for tablespace SYSTEM\r\nRecovery successfully copied file +DATA\/CDBGVA\/01B838F74693443FE053334EA8C03527\/DATAFILE\/system.437.856805523 from +DATA\/CDBGVA\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/system.435.856802413\r\nMRP0: Background Media Recovery terminated with error 1274\r\nThu Aug 28 17:32:05 2014\r\nErrors in file \/u01\/app\/oracle\/diag\/rdbms\/cdbgva\/CDBGVA_1\/trace\/CDBGVA_1_mrp0_13949.trc:\r\nORA-01274: cannot add data file that was originally created as '+DATA\/CDBATL\/01B838F74693443FE053334EA8C03527\/DATAFILE\/system.477.856805517'\r\nORA-19729: File 22 is not the initial version of the plugged in datafile\r\nThu Aug 28 17:32:05 2014<\/pre>\n<p>&nbsp;<\/p>\n<p>Now, the primary had all its datafiles (the new PDB has con_id 4):<\/p>\n<pre class=\"lang:plsql decode:true\">CON_ID NAME\r\n---------- ----------------------------------------------------------------------------------------------------\r\n1 +DATA\/CDB\/DATAFILE\/system.283.854626623\r\n1 +DATA\/CDB\/DATAFILE\/undotbs1.290.854627639\r\n1 +DATA\/CDB\/DATAFILE\/users.291.854627695\r\n1 +DATA\/CDB\/DATAFILE\/undotbs2.287.854627063\r\n1 +DATA\/CDB\/DATAFILE\/sysaux.285.854626879\r\n2 +DATA\/CDB\/FFBCECBB503D606BE043334EA8C019B7\/DATAFILE\/sysaux.286.854627011\r\n2 +DATA\/CDB\/FFBCECBB503D606BE043334EA8C019B7\/DATAFILE\/system.284.854626785\r\n3 +DATA\/CDBATL\/00B29F47A2D71CC2E053334EA8C03B13\/DATAFILE\/sysaux.390.855681795\r\n3 +DATA\/CDBATL\/00B29F47A2D71CC2E053334EA8C03B13\/DATAFILE\/system.389.855681795\r\n4 +DATA\/CDBATL\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/sysaux.459.856788061\r\n4 +DATA\/CDBATL\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/system.458.856788061<\/pre>\n<p>&nbsp;<\/p>\n<p>and the standby was missing the datafiles of the new PDB:<\/p>\n<pre class=\"lang:plsql decode:true\">1* select con_id, name from v$datafile order by 1\r\n\r\nCON_ID NAME\r\n---------- ----------------------------------------------------------------------------------------------------\r\n1 +DATA\/CDBGVA\/DATAFILE\/system.319.855054997\r\n1 +DATA\/CDBGVA\/DATAFILE\/undotbs2.283.855055141\r\n1 +DATA\/CDBGVA\/DATAFILE\/users.285.855055149\r\n1 +DATA\/CDBGVA\/DATAFILE\/undotbs1.284.855055145\r\n1 +DATA\/CDBGVA\/DATAFILE\/sysaux.281.855055061\r\n2 +DATA\/CDBGVA\/FFBCECBB503D606BE043334EA8C019B7\/DATAFILE\/sysaux.282.855055127\r\n2 +DATA\/CDBGVA\/FFBCECBB503D606BE043334EA8C019B7\/DATAFILE\/system.280.855055053\r\n3 +DATA\/CDBGVA\/00B29F47A2D71CC2E053334EA8C03B13\/DATAFILE\/sysaux.363.855681865\r\n3 +DATA\/CDBGVA\/00B29F47A2D71CC2E053334EA8C03B13\/DATAFILE\/system.362.855681863<\/pre>\n<p>&nbsp;<\/p>\n<p>But, on the standby database, the PDB somehow was existing.<\/p>\n<pre class=\"lang:plsql decode:true\">16:20:58 SYS@CDBGVA_1&gt; select name from v$pdbs;\r\n\r\nNAME\r\n------------------------------\r\nPDB$SEED\r\nMAAZ\r\nLUDO<\/pre>\n<p>&nbsp;<\/p>\n<p>I&#8217;ve tried to play a little, and finally decided to disable the recovery for the PDB (new in 12.1.0.2).<br \/>\nBut to disable the recovery I was needing to connect to the PDB, but the PDB was somehow &#8220;inexistent&#8221;:<\/p>\n<pre class=\"lang:plsql decode:true\">16:21:35 SYS@CDBGVA_1&gt; alter session set container=LUDO;\r\nERROR:\r\nORA-65011: Pluggable database LUDO does not exist.\r\n\r\n16:21:39 SYS@CDBGVA_1&gt; select name, open_mode from v$pdbs;\r\n\r\nNAME OPEN_MODE\r\n------------------------------ ----------\r\nPDB$SEED READ ONLY\r\nMAAZ MOUNTED\r\nLUDO MOUNTED<\/pre>\n<p>&nbsp;<\/p>\n<p>So I&#8217;ve tried to drop it, but off course, the standby was read-only and I could not drop the PDB:<\/p>\n<pre class=\"lang:plsql decode:true\">16:22:01 SYS@CDBGVA_1&gt; drop pluggable database ludo;\r\ndrop pluggable database ludo\r\n*\r\nERROR at line 1:\r\nORA-16000: database or pluggable database open for read-only access<\/pre>\n<p>&nbsp;<\/p>\n<p>Then I&#8217;ve shutted down the standby, but one instance hung and I&#8217;ve needed to do a shutdown abort (I don&#8217;t know if it was related with my original problem..)<\/p>\n<pre class=\"lang:plsql decode:true\"># [ oracle@racb02:\/u01\/app\/oracle\/diag\/rdbms\/cdbgva\/CDBGVA_1\/trace [16:22:45] [12.1.0.2.0 EE SID=CDBGVA_1] 1 ] #\r\n# srvctl stop database -d CDBGVA -o immediate\r\n[HANGS]<\/pre>\n<p>&nbsp;<\/p>\n<p>After mounting again the standby, the PDB was also accessible:<\/p>\n<pre class=\"lang:plsql decode:true\">SQL*Plus: Release 12.1.0.2.0 Production on Thu Aug 28 16:30:19 2014\r\n\r\nCopyright (c) 1982, 2014, Oracle. All rights reserved.\r\n\r\nConnected to:\r\nOracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production\r\nWith the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,\r\nAdvanced Analytics and Real Application Testing options\r\n\r\n16:30:19 SYS@CDBGVA_1&gt; alter session set container=LUDO;\r\n\r\nSession altered.<\/pre>\n<p>&nbsp;<\/p>\n<p>So I&#8217;ve been able to disable the recovery:<\/p>\n<pre class=\"lang:plsql decode:true\">16:31:19 SYS@CDBGVA_1&gt; alter pluggable database ludo disable recovery;\r\n\r\nPluggable database altered.<\/pre>\n<p>&nbsp;<\/p>\n<p>Then, on the primary, I&#8217;ve took a fresh backup of the involved datafiles:<\/p>\n<pre class=\"lang:plsql decode:true\">RMAN&gt; backup as copy datafile 16,17 format '\/tmp\/%f.dbf';\r\n\r\nStarting backup at 28-AUG-14\r\nusing target database control file instead of recovery catalog\r\nallocated channel: ORA_DISK_1\r\nchannel ORA_DISK_1: SID=88 instance=CDBATL_2 device type=DISK\r\nchannel ORA_DISK_1: starting datafile copy\r\ninput datafile file number=00017 name=+DATA\/CDBATL\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/sysaux.459.856788061\r\noutput file name=\/tmp\/17.dbf tag=TAG20140828T163251 RECID=4 STAMP=856801976\r\nchannel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07\r\nchannel ORA_DISK_1: starting datafile copy\r\ninput datafile file number=00016 name=+DATA\/CDBATL\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/system.458.856788061\r\noutput file name=\/tmp\/16.dbf tag=TAG20140828T163251 RECID=5 STAMP=856801981\r\nchannel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03\r\nFinished backup at 28-AUG-14\r\n\r\nStarting Control File and SPFILE Autobackup at 28-AUG-14\r\npiece handle=+DATA\/CDBATL\/AUTOBACKUP\/2014_08_28\/s_856801982.471.856801983 comment=NONE\r\nFinished Control File and SPFILE Autobackup at 28-AUG-14<\/pre>\n<p>&nbsp;<\/p>\n<p>and I&#8217;ve copied and cataloged the copies to the controlfile:<\/p>\n<pre class=\"lang:plsql decode:true\">RMAN&gt; catalog start with '\/tmp\/1';\r\n\r\nsearching for all files that match the pattern \/tmp\/1\r\n\r\nList of Files Unknown to the Database\r\n=====================================\r\nFile Name: \/tmp\/17.dbf\r\nFile Name: \/tmp\/16.dbf\r\n\r\nDo you really want to catalog the above files (enter YES or NO)? yes\r\ncataloging files...\r\ncataloging done\r\n\r\nList of Cataloged Files\r\n=======================\r\nFile Name: \/tmp\/17.dbf\r\nFile Name: \/tmp\/16.dbf<\/pre>\n<p>&nbsp;<\/p>\n<p>but the restore was impossible, because the controlfile was not knowing these datafiles!!<\/p>\n<pre class=\"lang:plsql decode:true\">16:38:48 SYS@CDBGVA_1&gt; select file# from v$datafile;\r\n\r\nFILE#\r\n----------\r\n1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n10\r\n11\r\n\r\nRMAN&gt; run {\r\n2&gt; set newname for datafile 16 to new;\r\n3&gt; set newname for datafile 17 to new;\r\n4&gt; restore datafile 16,17;\r\n5&gt; }\r\n\r\nexecuting command: SET NEWNAME\r\n\r\nexecuting command: SET NEWNAME\r\n\r\nStarting restore at 28-AUG-14\r\nallocated channel: ORA_DISK_1\r\nchannel ORA_DISK_1: SID=64 instance=CDBGVA_1 device type=DISK\r\nRMAN-00571: ===========================================================\r\nRMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============\r\nRMAN-00571: ===========================================================\r\nRMAN-03002: failure of restore command at 08\/28\/2014 16:37:02\r\nRMAN-20201: datafile not found in the recovery catalog\r\nRMAN-06010: error while looking up datafile: 17\r\n\r\nRMAN&gt; exit<\/pre>\n<p>&nbsp;<\/p>\n<p>So I&#8217;ve <strong>RESTARTED the recovery for a few seconds<\/strong>, and because the PDB had the recovery disabled,<strong> the recovery process has added the datafiles and set them offline<\/strong>.<\/p>\n<pre class=\"lang:plsql decode:true\">16:38:08 SYS@CDBGVA_1&gt; alter database recover managed standby database ;\r\nalter database recover managed standby database\r\n*\r\nERROR at line 1:\r\nORA-16043: Redo apply has been canceled.\r\nORA-01013: user requested cancel of current operation\r\n\r\n16:38:48 SYS@CDBGVA_1&gt; select file# from v$datafile;\r\n\r\nFILE#\r\n----------\r\n1\r\n2\r\n3\r\n4\r\n5\r\n6\r\n7\r\n10\r\n11\r\n16\r\n17<\/pre>\n<p>&nbsp;<\/p>\n<p>Then I&#8217;ve been able to restore the datafiles \ud83d\ude42<\/p>\n<pre class=\"lang:plsql decode:true\">RMAN&gt; run {\r\n2&gt; set newname for datafile 16 to new;\r\n3&gt; set newname for datafile 17 to new;\r\n4&gt; restore datafile 16,17;\r\n5&gt; }\r\n\r\nexecuting command: SET NEWNAME\r\n\r\nexecuting command: SET NEWNAME\r\n\r\nStarting restore at 28-AUG-14\r\nusing target database control file instead of recovery catalog\r\nallocated channel: ORA_DISK_1\r\nchannel ORA_DISK_1: SID=33 instance=CDBGVA_1 device type=DISK\r\n\r\nchannel ORA_DISK_1: restoring datafile 00016\r\ninput datafile copy RECID=21 STAMP=856802136 file name=\/tmp\/16.dbf\r\ndestination for restore of datafile 00016: +DATA\r\nchannel ORA_DISK_1: copied datafile copy of datafile 00016\r\noutput file name=+DATA\/CDBGVA\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/system.435.856802413 RECID=22 STAMP=856802416\r\nchannel ORA_DISK_1: restoring datafile 00017\r\ninput datafile copy RECID=20 STAMP=856802136 file name=\/tmp\/17.dbf\r\ndestination for restore of datafile 00017: +DATA\r\nchannel ORA_DISK_1: copied datafile copy of datafile 00017\r\noutput file name=+DATA\/CDBGVA\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/sysaux.355.856802417 RECID=23 STAMP=856802421\r\nFinished restore at 28-AUG-14\r\n\r\nRMAN&gt;\r\n\r\nRMAN&gt; switch datafile 16, 17 to copy;\r\n\r\ndatafile 16 switched to datafile copy \"+DATA\/CDBGVA\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/system.435.856802413\"\r\ndatafile 17 switched to datafile copy \"+DATA\/CDBGVA\/01B431F9BDF51AB7E053334EA8C06877\/DATAFILE\/sysaux.355.856802417\"\r\n\r\nRMAN&gt;<\/pre>\n<p>&nbsp;<\/p>\n<p>Finally, I&#8217;ve enabled again the recovery for the PDB and restarted the apply process.<\/p>\n<pre class=\"lang:plsql decode:true\">16:41:14 SYS@CDBGVA_1&gt; alter session set container=LUDO;\r\n\r\nSession altered.\r\n\r\n16:41:19 SYS@CDBGVA_1&gt; alter pluggable database ludo enable recovery;\r\n\r\nPluggable database altered.<\/pre>\n<p>&nbsp;<\/p>\n<p>Lesson learned: if you want to clone a PDB never, ever, forget to put your source PDB in read-only mode or you&#8217;ll have to deal with it!! \ud83d\ude42<\/p>\n","protected":false},"excerpt":{"rendered":"<p>UPDATE: please check my more recent post about this problem and the information I&#8217;ve got at the Oracle Demo Grounds during OOW14: https:\/\/www.ludovicocaldara.net\/dba\/demo-grounds-clone-pdb-rw\/ I feel the strong need to blog abut this very recent problem because I&#8217;ve spent a lot &hellip; <a href=\"https:\/\/www.ludovicocaldara.net\/dba\/a-pdb-is-cloned-while-in-read-write-data-guard-loose-its-marbles-12-1-0-2-ora-19729\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"om_disable_all_campaigns":false,"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"footnotes":""},"categories":[327,326,329,3,52,328,149,132],"tags":[292,289,189,188,190,161],"class_list":["post-744","post","type-post","status-publish","format-standard","hentry","category-oracle-maa","category-oracle","category-oracle-dg","category-oracledb","category-12c","category-oracle-mt","category-oracle-rac","category-triblog","tag-data-guard","tag-multitenant","tag-ora-01274","tag-ora-19729","tag-pluggable-database","tag-recovery"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/744","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/comments?post=744"}],"version-history":[{"count":8,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/744\/revisions"}],"predecessor-version":[{"id":861,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/744\/revisions\/861"}],"wp:attachment":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/media?parent=744"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/categories?post=744"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/tags?post=744"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}