My Collaborate 14 articles about Active Data Guard 12c and Policy Managed Databases

After almost 1 year, I’ve decided to publish these articles on my Slideshare account. You may have already seen them in the IOUG Collaborate 14 conference content or in the SOUG Newsletter 2014/4. Nothing really new, but I hope you’ll still enjoy them.


Cheers

Ludo

A 2014 of technology, sharing, friends, fun. And ideas for the next year.

2014 has been a great year, and it’s time to summarize it quickly.

A few numbers

  • 24 blog posts
  • ~37000 page views, >29000 visits
  • Speaker at 3 major conferences (#C14LV, #OOW14, #UKOUG_TECH14)
  • Speaker at 2  Trivadis internal conferences
  • Speaker at 2 local user group events
  • Speaker at an Oracle Business Breakfast
  • A total of 14 public speeches
  • I’ve been included in the ACE Program
  • 3 RAC Attack workshops (one co-organized and another self-organized)
  • 1 roundtable as co-organizer
  • 2 T-shirt designed as gifts for 2 RAC Attack workshops
  • 3 articles published
  • 3 new websites for the community (The new @IT_OUG website and two still in progress)
  • Joined 3 user group boards (SOUG-R, ITOUG and RAC SIG (well, RAC SIG and ITOUG were actually in 2013…))
  • Countless new friends and/or contacts

 

UKOUG Tech 14 wrap-up

I’ve attended UKOUG Tech for the first time this year and have not found the time to blog about it. The conference is very good. There is a lot of good content, the agenda is great. But I’ve heard that the conference was better and bigger in the past, that’s why the UKOUG is going to move it back to Birmingham in 2015. I sincerely hope that they will get back the attendance this conference deserves.

Sadly, the RAC Attack has not been organized and advertised properly, so it has been a failure comparing to te previous ones I’ve attended/organized.

I have learnt many new technical things and I also have learned some lessons:

  • Never under-estimate the time it takes to say goodbye to everyone before running to the train station if you’re in a rush for the train to the airport
  • Even if the temperature is higher than in Switzerland, the wind-chill in Liverpool may get it seems much colder
  • Be careful when English people invite you for a quick beer, it’s not one beer and it’s not quick :-) (especially when you go back home at 2AM and pretend to be awake at 6AM, check-out at the hotel and prepare the demo for the first presentation in the morning)

DSC_0391 DSC_0393 DSC_0407 DSC_0367 DSC_0374 DSC_0382 DSC_0386

DSC_0363[1]

 

B4fAw_iCYAAJ8wB

 

Once again, I’ve met many new friends :-)

 

6 months in the ACE program

I’ve spent my first 6 months in the ACE Program, it would take too much to talk about it. The best I’ve taken is not being in the program by itself. The best is the effort, the feedback and the new connections I’ve got as an active member of the community. Is a path that I recommend to all my Oracle-involved friends and colleagues because it’s very rewarding. I hope to be able to blog more about it in 2015 ;-)

 

A couple of pictures from 2014…

10532989_10204805457406497_9118297674875147610_n By-X9wECEAAYiFb DSC02619 1781065_612698348817395_3758740240585039433_o BlJFNzQCAAA4Ikp BkpvaPBCAAAULIB

 


B1vurMxIEAAXgRg

Projects for 2015

  • My fligths are already booked for the next Collaborate 15 #C15LV. See you in Vegas :-)
  • About the ITOUG, the die is cast, I’ll certainly contribute more in Italian with articles, webcasts and general duties related to the OUG administration. Keep an eye on www.itoug.it
  • Other secret plans that I cannot unveil now ;-)

Happy new year to everyone!

Ludo

Cloning a PDB with ASM and Data Guard (no ADG) without network transfer

Ok, if you’re reading this post, you may want to read also the previous one that explains something more about the problem.

Briefly said, if you have a CDB running on ASM in a MAA architecture and you do not have Active Data Guard, when you clone a PDB you have to “copy” the datafiles somehow on the standby. The only solution offered by Oracle (in a MOS Note, not in the documentation) is to restore the PDB from the primary to the standby site, thus transferring it over the network. But if you have a huge PDB this is a bad solution because it impacts your network connectivity. (Note: ending up with a huge PDB IMHO can only be caused by bad consolidation. I do not recommend to consolidate huge databases on Multitenant).

So I’ve worked out another solution, that still has many defects and is almost not viable, but it’s technically interesting because it permits to discover a little more about Multitenant and Data Guard.

The three options

At the primary site, the process is always the same: Oracle copies the datafiles of the source, and it modifies the headers so that they can be used by the new PDB (so it changes CON_ID, DBID, FILE#, and so on).

On the standby site, by opposite, it changes depending on the option you choose:

Option 1: Active Data Guard

If you have ADG, the ADG itself will take care of copying the datafile on the standby site, from the source standby pdb to the destination standby pdb. Once the copy is done, the MRP0 will continue the recovery. The modification of the header block of the destination PDB is done by the MRP0 immediately after the copy (at least this is what I understand).

ADG_PDB_copy

Option 2: No Active Data Guard, but STANDBYS=none

In this case, the copy on the standby site doesn’t happen, and the recovery process just add the entry of the new datafiles in the controlfile, with status OFFLINE and name UNKNOWNxxx.  However, the source file cannot be copied anymore, because the MRP0 process will expect to have a copy of the destination datafile, not the source datafile. Also, any tentative of restore of the datafile 28 (in this example) will give an error because it does not belong to the destination PDB. So the only chance is to restore the destination PDB from the primary.
NOADG_PDB_STANDBYS_NONE_copy

Option 3: No Active Data Guard, no STANDBYS=none

This is the case that I want to explain actually. Without the flag STANDBYS=none, the MRP0 process will expect to change the header of the new datafile, but because the file does not exist yet, the recovery process dies.
We can then copy it manually from the source standby pdb, and restart the recovery process, that will change the header. This process needs to be repeated for each datafile. (that’s why it’s not a viable solution, right now).

NOADG_PDB_copy

Let’s try it together:

The Environment

Primary

Standby

The current user PDB (any resemblance to real people is purely coincidental ;-) #haveUSeenMaaz):

Cloning the PDB on the primary

First, make sure that the source PDB is open read-only

Then, clone the PDB on the primary without the clause STANDBYS=NONE:

Review the clone on the Standby

At this point, on the standby the alert log show that the SYSTEM datafile is missing, and the recovery process stops.

One remarkable thing, is that in the standby controlfile, ONLY THE SYSTEM DATAFILE exists:

We need to fix the datafiles one by one, but most of the steps can be done once for all the datafiles.

Copy the source PDB from the standby

What do we need to do? Well, the recovery process is stopped, so we can safely copy the datafiles of  the source PDB from the standby site because they have not moved yet. (meanwhile, we can put the primary source PDB back in read-write mode).

Copy the datafiles:

Do the magic

Now there’s the interesting part: we need to assign the datafile copies of the maaz PDB to LUDO.

Sadly, the OMF will create the copies on the bad location (it’s a copy, to they are created on the same location as the source PDB).

We cannot try to uncatalog and recatalog the copies, because they will ALWAYS be affected to the source PDB. Neither we can use RMAN because it will never associate the datafile copies to the new PDB. We need to rename the files manually.

It’s better to uncatalog the datafile copies before, so we keep the catalog clean:

Then, because we cannot rename files on a standby database with standby file management set to AUTO, we need to put it temporarily to MANUAL.

standby_file_management is not PDB modifiable, so we need to do it for the whole CDB.

then we need to set back the standby_file_management=auto or the recover will not start:

We can now restart the recovery.

The recovery process will:
– change the new datafile by modifying the header for the new PDB
– create the entry for the second datafile in the controlfile
– crash again because the datafile is missing

We already have the SYSAUX datafile, right? So we can alter the name again:

This time all the datafiles have been copied (no user datafile for this example) and the recovery process will continue!! :-) so we can hit ^C and start it in background.

The Data Guard configuration reflects the success of this operation.

Do we miss anything?

Of course, we do!! The datafile names of the new PDB reside in the wrong ASM path. We need to fix them!

 

I know there’s no practical use of this procedure, but it helps a lot in understanding how Multitenant has been implemented.

I expect some improvements in 12.2!!

Cheers

Ludo

 

Tales from the Demo Grounds part 2: cloning a PDB with ASM and Data Guard (no ADG)

In my #OOW14 presentation about MAA and Multitenant, more precisely at slide #59, “PDB Creation from other PDB without ADG*”, I list a few commands that you can use to achieve a “correct” Pluggable Database clone in case you’re not using Active Data Guard.

What’s the problem with cloning a PDB in a MAA environment without ADG? If you’ve attended my session you should know the answer…

If you read the book “Data Guard Concepts and Administration 12c Release 1 (12.1)“, paragraph 3.5 Creating a PDB in a Primary Database, you’ll see that:

If you plan to create a PDB as a clone from a different PDB, then copy the data files that belong to the source PDB over to the standby database. (This step is not necessary in an Active Data Guard environment because the data files are copied automatically when the PDB is created on the standby database.)

But because there are good possibilities (99%?) that in a MAA environment you’re using ASM, this step is not so simple: you cannot copy the datafiles exactly where you want, it’s OMF, and the recovery process expects the files to be where the controlfile says they should be.

So, if you clone the PDB, the recovery process on the standby doesn’t find the datafiles at the correct location, thus the recovery process will stop and will not start until you fix manually. That’s why Oracle has implemented the new syntax “STANDBYS=NONE” that disables the recovery on the standby for a specific PDB: it lets you disable the recovery temporarily while the recovery process continues to apply logs on the remaining PDBs. (Note, however, that this feature is not intended as a generic solution for having PDBs not replicated. The recommended solution in this case is having two distinct CDBs, one protected by DG, the other not).

With ADG, when you clone the PDB on the primary, on the standby the ADG takes care of the following steps, no matter if on ASM or FS:

  1. recover up to the point where the file# is registered in the controlfile
  2. copy the datafiles from the source DB ON THE STANDBY DATABASE (so no copy over the network)
  3. rename the datafile in the controlfile
  4. continue with the recovery

If you don’t have ADG, and you’re on ASM, Oracle documentation says nothing with enough detail to let you solve the problem. So in August I’ve worked out the “easy” solution that I’ve also included in my slides (#59 and #60):

  1. SQL> create pluggable database DEST from SRC standbys=none;
  2. RMAN> backup as copy pluggable database DEST format ‘/tmp/dest%f.dbf';
  3. $ scp  /tmp/dest*.dbf remote:/tmp
  4. RMAN> catalog start with ‘/tmp/dest’
  5. RMAN> set newnamefor pluggable database DEST to new;
  6. RMAN> restore pluggable database DEST;
  7. RMAN> switch pluggable database DEST to copy;
  8. DGMGRL> edit database ‘STBY’ set state=’APPLY-OFF';
  9. SQL> Alter pluggable database DEST enable recovery;
  10. DGMGRL> edit database ‘STBY’ set state=’APPLY-ON';

Once at #OOW14, after endless conversations at the Demo Grounds, I’ve discovered that Oracle has worked out the very same solution requiring network transfer and that it has been documented in a new note.

Making Use of the STANDBYS=NONE Feature with Oracle Multitenant (Doc ID 1916648.1)

This note is very informative and I recommend to read it carefully!

What changes (better) in comparison with my first solution, is that Oracle suggests to use the new feature “restore from service”:

I’ve questioned the developers at the Demo Grounds about the necessity to use network transfer (I had the chance to speak directly with the developer who has written this piece of code!! :-)) and they said that they had worked out only this solution. So, if you have a huge PDB to clone, the network transfer from the primary to standby may impact severely your Data  Guard environment and/or your whole infrastructure, for the time of the transfer.

Of course, I have a complex, undocumented solution, I hope I will find the time to document it, so stay tuned if you’re curious! :-)

Tales from Demo Grounds part 1: Clone PDBs while open READ-WRITE

DISCLAIMER: I’ve got this information by chatting with Oracle developers at the Demo Grounds. The functionality is not documented yet and Oracle may change it at its sole discretion. Please refer to the documentation if/when it will be updated ;-)

In one of my previous posts named “A PDB is cloned while in read-write, Data Guard loose its marbles (12.1.0.2, ORA-19729)” I’ve blogged about a weird behaviour:

The documentation states that you can create a pluggable database from another one only if the source PDB is open read-only.

Indeed, If I try to clone it when the source PDB is MOUNTED, I get error ORA-65036:

The weird behavior is that if you do it when the source is in read-write mode, it works from release 12.1.0.2 (onward?)

I’ve questioned the developers at the DEMO Grounds and they have confirmed that:

  • With the 12.1.0.2, they have initially planned to disclose this functionality (clone PDBS in READ-WRITE).
  • That they had problems in making it work with an Active Data Guard environment (a-ah! so my post was not completely wrong)
  • Finally they have released it as undocumented feature
  • In the next release “they will fix it, maybe” and document it
  • The process of cloning the PDB anyway freeze the transactions on the source

I hope that this update helps clarifying both the behavior and my previous post about this problem! :-)

Cheers

Ludo

OOW14… I know I’m late… but it’s worth to blog about it, at least I won’t forget!

One month after the Open World, and still I’ve got no time to blog about it… The first reason is… well… I’ve been working on new stuff (a couple of new websites are coming), then I’ve had to adapt old slides with the great content got at the conference (ZDLRA and other topics).  Finally, I was also tired about working off-hours, I needed to spend good time with my family after this summer’s rush (moved to the new house and meanwhile worked on one gazillion different topics).

Open World 2014, a different personal feeling

The Open World has been again THE big conference that no Oracle professional can miss. But this time it has been very different to me comparing to 2013.

It wasn’t my first time as attendee, so it has been less surprising. I’ve attended as a Speaker (that made me feel part of the conference, and not just a spectator). I’ve attended also as an Oracle ACE (I’ll blog about thisin another post, soon or later). I was already friend with many people that I was eager to see again (and I feel the same now, waiting for the next conference).

My sessions

MAA+MT session at OOW14

Surprisingly, from the very beginning (at the time of the session acceptance from Oracle) I felt very relaxed about my presentations. One presentation after the other, I’m getting aware of my limits (especially my bad English! ;-)) but I’m also getting experienced. So I went to Open World without stress, even knowing that the first one, on Sunday, was already full booked a couple of weeks before the beginning of the conference.

Oracle has assigned to my sessions the VERY FIRST and the VERY LAST slots of the conference, so I’ve had less attendees than expected (around 170 for both sessions out of 230 and 270 registered people respectively). At least, I’ve had the chance to open and close the conference by saying “Welcome to Open World” and “Thank you, see you next year!” :-)

By-X9wECEAAYiFb

I’ve got very few feedback from the official OOW website, just 3 people for each session (anyway with a good, statistically irrelevant score: 4.33 and 4.67 out of 5), but I’ve been much more excited by the direct feedback of people that joined me for a chat right after the sessions and by the feedback on Twitter. I have to say that the interactions on this social media made my week, again.

 

 

RAC Attack

This year we organized (or… Bobby Curtis and Laura Ramsey did) the RAC Attack at the OTN Lounge. It’s always a great meeting point for all the bloggers and Oracle ACEs, but due to the day (Sunday) and the location (the mezzanine of the Moscone South), we’ve got less people than expected. Anyway, it’s always my favorite project and workshop. We’ve got T-shirts, bandannas, famous ninjas and the most famous attendee ever. Who? Try to figure it out yourself:  

Networking events

Saturday I’ve missed a great bike ride, and Sunday I had to miss the “Official” Golden Gate Run organized by Jeff Smith because it was overlapping with my session. So the first big event has been the ACE Dinner on Sunday, kindly sponsored by the Oracle ACE Program. A dream come true!

DSC02629_2

DSC02631_2

 

Monday morning, despite a very bad weather, I’ve been part of the great Swim in the Bay organized by Oraclenerd. The air was very cold (15°C) and the water was even colder!

10532989_10204805457406497_9118297674875147610_n

 

 

On Monday evening there was the Tech Fest organized at the Oracle Plaza (once Howard St.), I’ve been there with friends and colleagues and then we headed together to the Mikkeller Bar for a beer.DSC02664_2
DSC02675_2

 

I’ve planned my Tuesday longtime ago, when I’ve realized that there was a concert of the Royal Blood and the Pixies at the Masonic Center (Nob Hill) I’ve bought a couple of tickets for my and my ex-colleague and friend Franck Pachot.

DSC02682_2
DSC02687_2pixies_SF_2014

Wednesday morning I’ve organized the replica of the Golden Gate Run. Franck and Sten Vesterli joined me. Doing it early in the morning is beautiful!

 

DSC_0088_2Wednesday evening was the time of the great Blogger Meetup organized by Pythian and the OTN. Nowhere else you can find the same concentration of Oracle bloggers! Sadly this year the gadget was electronic (an app on the smartphone requiring network connectivity, so almost unusable and much less interactive, IMHO). I’ve finally met my good tweep Kevin Closson that wasn’t there in 2013.

DSC02741_2After the meetup, me and a team of very good friends have organized a dinner at the Fisherman’s Wharf, far from the crowd going to the traditional Oracle Appreciation Event. Try to recognize them in the picture below, I couldn’t ask for a better companionship!! :-)

By6SG9kCYAAo5KR By686V2CEAArLzy DSC02749_2Thursday is always sad, everybody leaves and after the conference the Oracle Plaza gets immediately so empty! The afternoon we’ve organized a meeting of the RAC SIG, and the evening I’ve managed to meet one more time many friends. Thank you guys :-)

 

My ride on Friday

I’ve ended my week with a long bike ride. I’ve under estimated the size of San Francisco, after more than four hours, 50km and almost 800m of cumulated ascent, I was completely out of order! But I’ve got the chance to visit for the first time most attractions… The City Hall, the Painted Ladies, the Golden Gate Park, the beach on the ocean side, the Coastal Trail, the Legion of Honor, the Presidio and the Hawk Hill on the other side of the bridge. DSC02765_2

DSC_0121_2DSC02781_2 gg_ride

DSC_0148_2


DSC_0137_2

 

To resume

Listing all the friends I would like to thank for the OOW week is long… Heli, Ovind, Kyle, Martin, Yury, Björn, Franck, Tim, Kellyn, Rene, Leighton, Seth, Laura, Vikki, Jennifer, Mina, Marc, Nelson, Markus, Larry, Edelweiss, Emiliano, Roman, Daniele, Konrad, Chris, Christian, Sten, Hans, Rooq, Andrejs, Paul, Michelle, Kai, Ian, Michael, Alex, Vanessa, Mark, Osama, Bobby, Rick, Gurkan, Laurent, Kevin, Jason, Chet, Carlos, Mauro, Danny, Don, Jan, Vit, Biju, Stacey,  Mike, Henning, Jeff, Christophe, Dominique, Hervé…  I know I’m forgetting too many, you know who you are, thank you!

 

Oracle RAC, Oracle Data Guard, and Pluggable Databases: When MAA Meets Oracle Multitenant (OOW14)

Here you can find the material related to my session at Oracle Open World 2014. I’m sorry I’m late in publishing them, but I challenge you to find spare time during Oracle Open World! It’s the busiest week of the year! (Hard Work, Hard Play)

 Slides

 Demo 1 video

Demo 2 video

Demo 1 script

 

Demo 2 script

 

There’s one slide describing the procedure for cloning one PDB using the standbys clause. Oracle has released a Note while I was preparing my slides (one month ago) and I wasn’t aware of it, so you may also checkout this note on MOS:

Making Use of the STANDBYS=NONE Feature with Oracle Multitenant (Doc ID 1916648.1)

UPDATE: I’ve blogged about it in a more recent post: Tales from the Demo Grounds part 2: cloning a PDB with ASM and Data Guard (no ADG)

Cheers!

Ludovico

Oracle Active Data Guard 12c: Far Sync Instance, Real-Time Cascade Standby, and Other Goodies

Here you can find the content related to my second presentation at Oracle Open World 2014.

 Slides

Demo video1: Real-Time Cascade

Demo video2: Far Sync Instance

Demo 1 Script

 

Demo 2 script

For the demo I’ve used 5 machines running 3 database instances and 2 Far Sync instances. I cannot provide the documentation for creating the demo environment, but the scripts may be useful to understand how the demo works.

Cheers

Ludo

My agenda at Oracle Open World 2014

It’s time to prepare my luggage for the OOW, it will be my second time in San Francisco and the first as ACE and speaker. I still need to figure out if I’ll manage to get my badge Sunday morning, because Saturday I won’t be in the city before the registration desks close.

If you want to reach me during the conference, this is my “expected” plan:

OOW14_Schedule

I’m looking forward particularly to meet my many community friends at the ACE dinner (the first as non-infiltrated ;-)), the blogger meetup and the crazy swim in the bay.

See you on Sunday! :-)

RAC Attack 12c in Switzerland, it’s a wrap!

Last Wednesday, September 17th, we’ve done the first RAC Attack in Switzerland (as far as I know!). I have to say that it has been a complete success like all other RAC Attacks I’ve been involved in.

DSC_0019

This time I’ve been particularly happy and proud because I’ve organized it almost all alone. Trivadis, my employer, has kindly sponsored everything: the venue (the new, cool Trivadis offices in Geneva), the T-shirts (I’ve done the design, very similar to the one I’ve designed for Collaborate 14),  beers and pizza!

For beer lovers,we’ve got the good “Blanche des Neiges” from Belgium, “La Helles” and “La Rossa” from San Martino Brewery, Ticino (Italian speaking region of Switzerland). People have appreciated :-)

DSC_0027

We’ve had 4 top-class Ninjas and 10 people actively installing Oracle RAC (plus a famous blogger that joined for networking), sadly two people have renounced at the last minute. For the very first time, all the participants have downloaded the Oracle Software in advance. When they’ve registered I’ve reminded twice that the software was necessary because we cannot provide it due to legal constraints.

DSC_0023

 

People running the lab on Windows laptops have reported problems with VirtualBox 4.3.16 (4.3.14 has been skipped directly because of known problems). So every one had to fallback to version 4.3.12 (the last stable release, IMO).

The best praise I’ve got has been the presence of a Senior DBA coming from Nanterre! 550Km (> 5h00 by public transport door-to-door) and an overnight stay just for this event, can you believe it? :-)

This makes me think seriously about the real necessity of organizing this kind of events around the world.

DSC02614 DSC02600 DSC02581

 

Off course, we’ve got a photo session with a lot of jumps ;-) We could not miss this RAC Attack tradition!

We’ve wrapped everything around 10:30pm, after a bit more than 5 hours of event. We’ve enjoyed a lot and had good time together chatting about Oracle RAC and about our work in general.

DSC02619

Thank you again to all participants!! :-)