A winning week in Vegas

Collaborate 14 is over. After a small week of vacation, it’s time for me to think about it and blog a short review about what I’ve learnt and done.


RAC Attack 12c (Monday)



RAC Attack is definitely a great project, and the live event at Collaborate has been greater than ever before. The “official pre-conference workshop” has attracted 40 active participants (where active really means 40 people with 40 laptops installing 40 RAC 12c stacks!). Big user base means also new problems. We’ve faced a strange issue with VirtualBox showing only 32bit versions for the guest OSes (hint, hardware acceleration MUST be enabled in the laptop’s BIOS) and a few copy & paste problems from the electronic PDF (copying & pasting from the online website solves the problem). We’re working in order to fix them.

As usual, most of the problems faced by the participants were related to some steps skipped by mistake.

The OTN (our great friend @OracleDBDev :-) ) has sponsored pizzas, drinks and beers!!!


Moreover, always thanks to the OTN, we’ve got a few prizes to give to the best participants (the most active in installing and/or helping others) and cool T-shirts. We’ve got a great feedback from the attendees and the entire Oracle community, we’re looking forward to see another success like this one.


It has been possible thanks to the many local and remote contributors!

The Ninjas: Yury Velikanov, Seth Miller, Erik Benner, Andrejs Karpovs, (me)
The additional new Ninja: Ryan Weber
The OTN staff: Laura Ramsey, Bob Rhubart
The IOUG staff: Alexis Bauer Kolak, Apryl Alexander-Savino, Tricia Chiamas
The RAC Attack founder and legend: Jeremy Schneider
Other volunteers: Leighton Nelson, Osama Mustafa, Kamran Agayev, Alvaro Miranda, Maaz Anjum, Björn Rost, Bobby Curtis, Marcin Przepiorowski, Marc Fielding, <missing names here>

Most of these people are very active on Twitter, make sure to follow them! :-)

Check out this small video by Erik Benner, it’s made by many different pictures taken during the workshop.


The RAC SIG meeting (Tuesday)

Tuesday evening I’ve organized with my friend Yury a SIG meeting about the best ways to learn Oracle RAC 12c. Despite the late time and free beer served in the exhibition hall, we’ve got a good attendance. Many new faces but also many faces from the RAC Attack. We’ve been honored by the presence of Oracle (the RAC Product Manager Markus Michalewicz in person!).  It has been really interesting.



My first sessions about Data Guard (Wednesday)

I was expecting many people, actually I’ve got between 40 and 50 attendees, not sure if it’s OK, but I’ve been very excited about the presence of Larry Carpenter (Oracle Data Guard Product Manager) in the room. My speech was not good enough to fulfill my personal expectation, but I’ve got a quite good feedback from many people and a very good feedback about my live demos! :-)


My second session about Policy-managed Databases (Friday)

My second presentation has been slight better than the first one (read: my English has been more fluent) and I’ve also got a great feedback. Again, I’ve executed the demos without problems (I’ve prepared my lab on the plane :-)) Sadly the audience has been much smaller, I think something like 15 attendees. I can blame the time (last day, early morning) and the topic: how many people know about PMDB? The next time I’ll find a more attractive title.


Networking and great content

Collaborate is big enough to network with a huge amount of experts and famous technologists, but not big enough to have difficulties in finding the people you’re looking for.

In the networking area I’ve always been able to find many great peers for great discussions.




The sessions were great, I’m always excited to hear from Oracle Employees, ACEs, ACE Directors and members of the OAK Table. And this Collaborate was no exception.
The most exciting news has been the introduction of the “lightning talks” at the Oak Table World. A+++ !


Friends, more friends and a lot of friends

What I’m most excited about, is the great number of old and new friends that I’ve met. Thank you my friends, you know who you are, I hope to see you very soon! :-)



1277296_612698062150757_8173502953307905657_o 1781065_612698348817395_3758740240585039433_o


One spare day in New York (Saturday)

I’ve scheduled my flights with the intent of spending one day in New York, so I’ve landed Saturday very early and taken off the same day, in the evening. First time in NYC, I’ve loved it! :-)

IMG_0409 IMG_0293


And remember, Collaborate 14 is over, but collaboration is not! Start your involvement with the community today! 

Oracle Multitenant and custom DBCA templates

Today I’ve encountered an annoying issue while adapting a few scripts for automatic database creation. I track it here so hopefully it may save a few hours of troubleshooting to someone…

I’ve used the DBCA to prepare a new template:


Then continued by customizing other options, including init parameters and datafile/logfile paths. Finally, I’ve saved it as a new template instead of creating the database.

I’ve checked the resulting .dbc and seen that there was, as expected, the parameter “enable_pluggable_database”=”true”.

Then I’ve moved the template file to my $HOME directory and tested the silent database creation with the option “-createAsContainerDatabase true”:

The database configuration has completed successfully, without errors. I’ve accessed my new container, and I’ve been surprised by seing:

In fact, there were no pdb$seed datafiles:

After little investigation, I’ve found these lines in the dbca trace.log:


Then I’ve struggled with dbca and templates a few times before finding that, actually, the magic “enable pluggable database” is done by dbca only if the template file name is not customized.

Running the same exact command with the very same template file but renamed to $ORACLE_HOME/assistants/dbca/templates/General_Purpose.dbc actually works (notice the diff at the first line):



I’ve also tried to cheat and use a symlink to my previous custom template, and surprisingly, it still works:

In the dbca trace log the message saying that the DB will be NON-CDB disappears:

So the problem is really caused by the different filename/location of the template.

IMHO it’s a kind of bug, the decision between a CDB and NON-CDB should not be taken by DBCA.  Moreover, it’s not based on the content of the template, which would be logic. But today I’m late and lazy, I will not open a SR for this.


RAC Attack 12c at Collaborate 14 and advanced labs

I’ve just published an advanced lab on SlideShare that RAC Attack attendees may do at Collaborate this year, instead of just doing the basic 2-node RAC installation on their laptop.

We’ll offer also an advanced lab about Flex Clusters and Flex ASM (written by Maaz Anjum). Moreover, I’m working on an additional lab that allows to implement a multi-node RAC by using Virtual Box linked clones and GI Home clones like I’ve shown in my previous post.

RAC Attack at #C14LV will be like fun again. We’ll have a few t-shirts for the attendants, designed by me and Chet “Oraclenerd” Justice, kindly sponsored by OTN.

The workshop will end up with beers and snaks (again, thank you OTN for sponsoring this :-)).



If you’re planning to attend Collaborate, join us and start your conference week in a good mood :-)


Multinode RAC 12c cluster on VirtualBox using linked clones

Recently I’ve had to install a four-node RAC cluster on my laptop in order to do some tests. I have found an “easy” (well, easy, it depends), fast and space-efficient way to do it so I would like to track it down.

The quick step list

  • Install the OS on the first node
  • Add the shared disks
  • Install the clusterware in RAC mode on on the first node only
  • Remove temporarily the shared disks
  • Clone the server as linked clone as many times as you want
  • Reconfigure the new nodes with the new ip and naming
  • Add back the shared disks on the first node and on all other nodes
  • Clone the GI + database homes in order to add them to the cluster

Using this method the Oracle binaries (the most space consuming portion of the RAC installation) are installed and allocated on the first node only.

The long step list

Actually you can follow many instruction steps from the RAC Attack 12c book.

  • Review the HW requirements  but let at least 3Gb RAM for each guest + 2Gb more for your host (you may try with less RAM but everything will slow down).
  • Download all the SW components , additionally you may download the latest PSU ( from MOS.
  • Prepare the host and install linux on the first node. When configuring the OS, make sure you enter all the required IP addresses for the additional nodes. RAC Attack has two nodes collabn1, collabn2. Add as many nodes as you want to configure. As example, the DNS config may have four nodes

At this point, the procedure starts differing from the RAC Attack book.

  •  Go to the VirtualBox VM settings and delete all the shared disks2014_03_16_22_05_26_Oracle_VM_VirtualBox_Manager
  •  Clone the first server as linked clone (right-click, clone, choose the name, flag “Linked Clone” as many times as the number of additional servers you want.

2014-03-16 22_09_42-Clone Virtual Machine


  • By using this method  the new servers will use the same virtual disk file of the first server and a second file will be used to track the differences. This will save a lot of space on the disk.
  • Add back the shared disks to all the servers.
  • Start the other nodes and configure them following the RAC Attack instructions again.
  • Once all the nodes are configured, the GI installation has to be cleaned out on all the cloned servers using these guidelines:

  • Then, on each cloned server, run the perl clone.pl as follows to clone the GI home, but change the LOCAL_NODE accordingly (note: the GI Home name must be identical to the one specified in the original installation!):

  •  Then, on the first node (that you have started and you have reactivated the clusterware stack on it with crsctl enable crs / crsctl start crs ;-)), run this command to add the new nodes in the definition of the cluster:


  • from the first server copy these files on all the other nodes:

  •  Then clone also the DB Home (again, run it on each new server and specify the same DB home name that you have used in the original installation):

  •  On each new node run also the updatenodelist and the DB root.sh command to update the node list for the DB home:

  •  and finally, run the GI root.sh on each new node to finalize their inclusion in the cluster!! :-)


  • As result, you should be able to seen all the cluster resources started correctly on all the nodes.


I know it seems a little complex, but if you have several nodes this is dramatically faster than the standard installation and also the space used is reduced. This is good if you have invested in a high-performance but low-capacity SSD disk like I did :-(.

Hope it helps, I paste here the official documentation links that I’ve used to clone the installations. The other steps are my own work.



Some notes about Grid Infrastructure PSU

I’m creating a new 12c RAC environment from scratch, Ii just want to track a few notes (primarily for my personal use ;-)) about the _PSU.

The opatch utility bundled with the GI does not contain the emocmrsp, so it is necessary to install the latest opatch (6880880).

The patching process can patch both GI and RAC homes at once, but if you don’t have a valid database registered, an error is raised:

So you need to patch the Oracle Homes individually if it’s a new installation.

Remind that:

  • The patch must be unzipped by the oracle/grid user in a directory readable to oracle and root (or it will fail with Argument(s) Error… Patch Location not valid) or other funny errors (permission denied errors in the middle of the patch process)
  • Must be applied by the root user
  • Must be applied individually and on every node, one node at time.
  • The opatchauto executable must belong to one of the OH you’re patching (so if you patch GI and RAC separately,  you have to use the correspondent opatch.





Speaker and Ninja at Collaborate14 – #C14LV


This year I will have the honor to present at Collaborate14, from April 7th to 11th. First of all, many thanks to Trivadis that has kindly agreed to send me to the conference.

My session (#603):
Oracle Data Guard 12c: Real-Time Cascade, Far Sync Instances and other goodies
has been accepted, so if you plan to attend Collaborate, I will be glad to see you there!
My paper and presentation are ready, but I’ll wait the post-conference before publishing them. Meanwhile, you can get a little sneak peak of my live demo (I’ll cut something, somewhere, but my new SSD disk should reduce the time elapsed, I have to do it again with the new hardware to get correct timings :-) ). There’s no audio, since it’s supposed to be my failover demo if I’ll have problems during my session.

Part I

Part II

I’ve submitted another abstract about Policy Managed Databases, but it has been put in the waiting list, assuming that Data Guard has a lot more users and the interest in new Data Guard 12c features will be higher than PMDBs that are rarely used in production environments (and I’m sad about it, keep in touch if you want to know more about this great technology).


RAC Attack 12c!

I’ll be organizing the RAC Attack again, along with Seth Miller, Yury Velikanov and Kamran Agayev. Sharing this exciting role with an Oracle ACE and two ACE Directors makes me  proud of what I’m doing, but more than this, I’m happy to repeat another exciting experience like I had at OOW13.

This Year RAC Attack will be an official pre-conference workshop. We have been contacted directly by the IOUG, and we’re making improvements. We’ll install RAC 12c and discuss about advanced topics, have a lot of fun, drink a beer together and jump a lot! :-)

Other mentors at the workshop will be Leighton Nelson, Maaz Anjum, Biju Thomas. You should know them already, so join us!

And don’t forget, register before February 12th, so you take benefit of the early bird discount!


Removing passwords from Oracle scripts: Wallets and Proxy Users

Very often I encounter customers that include Oracle account passwords in their scripts in order to connect to the database.
For DBAs, sometimes things are
easier when they run scripts locally using the oracle account, since the “connect / as sysdba” usually do the job, even with some security concerns. But what if we need other users or we need to connect remotely?
Since longtime Oracle supplies secure wallets and the proxy authentication. Let’s see what they are and how to use them.

Secure Wallet
Secure wallets are managed through the tool mkstore. They allow to store a username and password in a secure wallet accessible only by its owner. The wallet is then accessed by the Oracle Client to connect to a remote database, meaning that you DON’T HAVE to specify any username and password!

Let’s see how to implement this the quick way:

Create a directory that will contain your wallet:

Create the wallet, use an arbitrary complex password to protect it:

Immagine that you’ve a user created with a very complex password:

Then you need to insert these credentials, including the connect string, into the wallet.

Keep in mind that you can have multiple credentials in the wallet for different remote descriptors (connect strings), but if you want many credentials for the very same connect string you need to create different wallets in different directories.

Now you need to tell your Oracle Client to use that wallet by using the wallet_location parameter in your sqlnet.ora, so you need to have a private TNS_ADMIN:

If everything’s alright, you should be able to connect to the database PROD as the batch user, without specifying any username or password.

Attention: when mkstore modifies the wallet, only the clients with the same or above versions will be able to access the wallet, so if you access your wallet with a 11g client you shouldn’t modify the wallet with a 12c version of mkstore. This is not documented by Oracle, but you can infer it from different “not a bug” entries on MOS :-)

Proxy Users
You can push things a little farther, and hook your wallet with a proxy user, in order to connect to arbitrary users. That’s it, a proxy user is entitled to connect to the database on behalf of other users. In this example, we’ll see how, through the batch account, we can connect as OE, SH or HR:

Now I can specify with which user I want to work on the DB, connect to it through the batch account, without specifying the password thanks to the wallet:


See how it’s easy? But don’t forget to keep your wallet secure using unix/windows permissions!

Oracle Instances and real memory consumption on Linux and Solaris

There’s a way to know the REAL memory usage by Oracle Instance, including all connecting processes and using the shell rather than a connection to oracle?

The short answer is “I think so” :-)

Summing up RSS column from ps output, is not reliable because Linux uses a copy-on-write on process forks and also doesn’t take into account correctly the shared memory and other shared allocations.

I’ve come across this post on Pythian’s Blog from Marc Billette.

While it seems good I’ve had discording results depending on platform and release.

Instead, I’ve tried to create a shell snippet that always uses pmap but works differently and SEEMS to work correctly on Linux ans Solaris.

Basically, using the pmap script I get a lot of information about the different memory areas allocated to the process:


Initially I’ve tried to decode correctly the different kinds of memory the same way other scripts I’ve found online do:

but finally the ADDRESS is the same from different processes when the memory area is shared, so my script now just get a unique line for each address and sums up the memory size (not the rss one!):

This should give the total virtual memory allocated by the different Oracle instances.

The results I get are plausible both on Linux and Solaris.


If you find any error let me know and I’ll fix the script!


Regular TNS-12508 critical alerts in EM12c

Yesterday I’ve come across a small request from a customer.
They were receiving REGULARLY critical alerts in EM12c from some listeners due to error TNS-12508.

The facts:

  • only 10g listeners were affected
  • every day, only one occurrence of the error and always at the same time on a named host
  • no apparent correlations between times on different hosts

I’ve analyzed the log to see the error.


Notice that after the error there are two requests: show log_directory and show trc_directory.
So I’ve supposed that it’s an additional request failing on 10g listeners but not on 11g listeners.
The “help” command of 10g and 11g releases shows that the two releases have some different commands. One of them is “show oracle_home” that has been introduced in 11g.

  • First I’ve searched for scheduled scripts (the customer literally have a huge library of scripts run against the databases to automate maintenance tasks)
  • Then I’ve asked to the team that manages an automatic discovery tool that feeds the CMDB
  • Finally, I’ve come across this note on Metalink that explain the error:

Repetitive TNS-12508 Errors logged for a listener target after upgrade to DB plugin or higher (Doc ID 1596633.1)

I’ve applied the change to the metrics in EM12c to ignore the error for 10g listeners.