<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>DBA survival BLOG &#187; Performance</title>
	<atom:link href="http://www.ludovicocaldara.net/dba/category/perf/feed/" rel="self" type="application/rss+xml" />
	<link>http://www.ludovicocaldara.net/dba</link>
	<description>Keep DBA job simple. Thanks.</description>
	<lastBuildDate>Thu, 27 Aug 2009 13:58:01 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.2</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>A great talk</title>
		<link>http://www.ludovicocaldara.net/dba/a-great-talk/</link>
		<comments>http://www.ludovicocaldara.net/dba/a-great-talk/#comments</comments>
		<pubDate>Fri, 10 Jul 2009 08:16:02 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Performance]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=79</guid>
		<description><![CDATA[John Allspaw and Paul Hammond (Flickr) talk about Dev and Ops Cooperation at Flickr (very good).
 
Have a look at Kitchen Soap blog.
]]></description>
			<content:encoded><![CDATA[<p>John Allspaw and Paul Hammond (Flickr) talk about Dev and Ops Cooperation at Flickr (very good).</p>
<p><embed src="http://blip.tv/play/AYGMoH+LqzQ" type="application/x-shockwave-flash" width="460" height="390" allowscriptaccess="always" allowfullscreen="true"></embed> </p>
<p>Have a look at <a href="http://www.kitchensoap.com/2009/06/23/slides-for-velocity-talk-2009/">Kitchen Soap</a> blog.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/a-great-talk/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Oracle capacity planning with RRDTOOL</title>
		<link>http://www.ludovicocaldara.net/dba/oracle-capacity-planning-with-rrdtool/</link>
		<comments>http://www.ludovicocaldara.net/dba/oracle-capacity-planning-with-rrdtool/#comments</comments>
		<pubDate>Mon, 25 May 2009 16:18:50 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Oracle Database]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[awr]]></category>
		<category><![CDATA[Capacity Planning]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[rrdtool]]></category>
		<category><![CDATA[workload]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=68</guid>
		<description><![CDATA[RRDize everything, chapter 2
Oracle Database Server has the most powerful system catalog that allows to query almost any aspect inside an oracle instance.
You can query many v$ fixed views at regular intervals and populate many RRD files through rrdtool: space usage, wait events. system statistics and so on&#8230;
Since release 10.1 Oracle has introduced Automatic Workload [...]]]></description>
			<content:encoded><![CDATA[<p><strong>RRDize everything, chapter 2</strong></p>
<p>Oracle Database Server has the most powerful system catalog that allows to query almost any aspect inside an oracle instance.<br />
You can query many <em>v$</em> fixed views at regular intervals and populate many RRD files through <a href="http://oss.oetiker.ch/rrdtool/">rrdtool</a>: space usage, wait events. system statistics and so on&#8230;</p>
<p>Since release 10.1 Oracle has introduced <a href="www.oracle.com/technology/pub/articles/10gdba/week6_10gdba.html">Automatic Workload Repository</a>, a finer version of old good Statspack.<br />
No matter if you are using AWR or statspack, you can rely on their views to collect data for your RRDs.</p>
<p>If you are administering a new instance and you haven&#8217;t collected its statistics so far, you can query (as example) the DBA_HIST_BG_EVENT_SUMMARY view to gather all AWR data about wait events. Historical views could be useful also to collect historical data once a week rather than query the fixed views every few minutes doing the hard work twice (you and AWR).</p>
<p>The whole process of gathering performance data and update rrd files can be resumed into the following steps:</p>
<p>- connect to the database<br />
- query the AWR&#8217;s views<br />
- build and execute an rrdtool update command<br />
- check if rrd file exists or create it<br />
- update the rrd file</p>
<p>The less rrdtool update commands you will execute, the better the whole process will perform.<br />
Do it in a language you are comfortable with and that supports easily connection descriptors.</p>
<p>Since I&#8217;m very comfortable with php, I did it this way.</p>
<p>This is a very basilar script that works greatly for me with good performances:</p>

<div class="wp_syntax"><div class="code"><pre class="php" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/usr/bin/php -f
</span><span style="color: #339933;">&lt;</span> ?php                                         
&nbsp;
<span style="color: #990000;">define</span><span style="color: #009900;">&#40;</span><span style="color: #0000ff;">'WD'</span><span style="color: #339933;">,</span><span style="color: #0000ff;">'/opt/oracle/awr'</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$cs</span>         <span style="color: #339933;">=</span> <span style="color: #000088;">$_SERVER</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'argv'</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#91;</span><span style="color: #cc66cc;">1</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$user</span>       <span style="color: #339933;">=</span> <span style="color: #0000ff;">'mymonitoruser'</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$pass</span>       <span style="color: #339933;">=</span> <span style="color: #0000ff;">'mystrongpassword'</span><span style="color: #339933;">;</span> 
&nbsp;
<span style="color: #666666; font-style: italic;">/* open a new connection */</span>
<span style="color: #000088;">$ds</span> <span style="color: #339933;">=</span> oci_connect<span style="color: #009900;">&#40;</span><span style="color: #000088;">$user</span><span style="color: #339933;">,</span> <span style="color: #000088;">$pass</span><span style="color: #339933;">,</span> <span style="color: #000088;">$cs</span><span style="color: #009900;">&#41;</span>
        or <span style="color: #990000;">die</span> <span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;Cannot connect to Oracle Database &quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;<span style="color: #000099; font-weight: bold;">\n</span>&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
<span style="color: #666666; font-style: italic;">/* setting client nls environment */</span>
<span style="color: #000088;">$sql</span> <span style="color: #339933;">=</span> <span style="color: #0000ff;">&quot;alter session set nls_timestamp_format='MM/DD/YY HH24:MI'&quot;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$stmt</span> <span style="color: #339933;">=</span> oci_parse<span style="color: #009900;">&#40;</span><span style="color: #000088;">$ds</span><span style="color: #339933;">,</span> <span style="color: #000088;">$sql</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_execute<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_free_statement<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>                                         
&nbsp;
<span style="color: #666666; font-style: italic;">/* create directory that will contain rrds (if not exists) */</span>
<span style="color: #b1b100;">if</span><span style="color: #009900;">&#40;</span><span style="color: #339933;">!</span><span style="color: #990000;">file_exists</span><span style="color: #009900;">&#40;</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">'/'</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #990000;">mkdir</span><span style="color: #009900;">&#40;</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">'/'</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #b1b100;">if</span><span style="color: #009900;">&#40;</span><span style="color: #339933;">!</span><span style="color: #990000;">file_exists</span><span style="color: #009900;">&#40;</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">'/'</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">'/wait'</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #990000;">mkdir</span><span style="color: #009900;">&#40;</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">'/'</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">'/wait'</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>                   
&nbsp;
<span style="color: #666666; font-style: italic;">/* function to create new RRDs */</span>
<span style="color: #000000; font-weight: bold;">function</span> createRRD<span style="color: #009900;">&#40;</span><span style="color: #000088;">$name</span><span style="color: #339933;">,</span> <span style="color: #000088;">$interval</span><span style="color: #339933;">,</span> <span style="color: #000088;">$cs</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #000088;">$hb</span> <span style="color: #339933;">=</span> <span style="color: #000088;">$interval</span><span style="color: #339933;">*</span><span style="color: #cc66cc;">5</span><span style="color: #339933;">;</span> <span style="color: #666666; font-style: italic;">//heartbeat</span>
        <span style="color: #000088;">$cmd</span><span style="color: #339933;">=</span><span style="color: #0000ff;">&quot;rrdtool create &quot;</span><span style="color: #339933;">.</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/wait/<span style="color: #006699; font-weight: bold;">${name}</span>.rrd -s &quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$interval</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot; <span style="color: #000099; font-weight: bold;">\
</span>                -b <span style="color: #000099; font-weight: bold;">\&quot;</span>now -3month<span style="color: #000099; font-weight: bold;">\&quot;</span> DS:waits:DERIVE:<span style="color: #006699; font-weight: bold;">$hb</span>:0:U <span style="color: #000099; font-weight: bold;">\
</span>                DS:mswaited:DERIVE:<span style="color: #006699; font-weight: bold;">$hb</span>:0:U <span style="color: #000099; font-weight: bold;">\
</span>                RRA:AVERAGE:0.5:1:1440 RRA:AVERAGE:0.5:30:336 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:AVERAGE:0.5:120:372 RRA:AVERAGE:0.5:720:730 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:MIN:0.5:1:1440 RRA:MIN:0.5:30:336 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:MIN:0.5:120:372 RRA:MIN:0.5:720:730 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:MAX:0.5:1:1440 RRA:MAX:0.5:30:336 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:MAX:0.5:120:372 RRA:MAX:0.5:720:730 <span style="color: #000099; font-weight: bold;">\
</span>                RRA:LAST:0.5:1:1440&quot;</span><span style="color: #339933;">;</span>
        <span style="color: #666666; font-style: italic;">//print $cmd.&quot;\n&quot;;</span>
        <span style="color: #b1b100;">return</span> <span style="color: #990000;">passthru</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$cmd</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>                                                                              
&nbsp;
<span style="color: #666666; font-style: italic;">/* take the snapshot frequency from dba_hist_wr_control
 to create the RDD with correct heartbeat value */</span>
<span style="color: #000088;">$sql</span> <span style="color: #339933;">=</span> <span style="color: #0000ff;">'select extract(hour from snap_interval)*3600 +
extract(minute from snap_interval)*60 as SEED from DBA_HIST_WR_CONTROL'</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$stmt</span> <span style="color: #339933;">=</span> oci_parse<span style="color: #009900;">&#40;</span><span style="color: #000088;">$ds</span><span style="color: #339933;">,</span> <span style="color: #000088;">$sql</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_execute<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$row</span> <span style="color: #339933;">=</span> oci_fetch_assoc<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$interval</span> <span style="color: #339933;">=</span> <span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'SEED'</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
<span style="color: #990000;">unset</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_free_statement<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>                                              
&nbsp;
<span style="color: #666666; font-style: italic;">/* statement definition that will collect
 all snapshots for a certain wait event with more than
 a certain amonut of time waited.
 Gathering ALL EVENTS could be time consuming and useless.
 I fetch rows ordered by event_name rather
 then by date because I can update many values
 into the same rrd with very few rrdupdate commands
*/</span>
<span style="color: #000088;">$sql</span> <span style="color: #339933;">=</span> <span style="color: #0000ff;">'select s.END_INTERVAL_TIME END_INTERVAL_TIME,
    g.EVENT_NAME, g.WAIT_CLASS, g.TOTAL_WAITS,
    round(g.TIME_WAITED_MICRO/1000) MS
  from DBA_HIST_SNAPSHOT s,
   dba_hist_bg_event_summary g,
   v$instance i
 where s.SNAP_ID=g.SNAP_ID and g.wait_class!=\'Idle\'
  and g.TIME_WAITED_MICRO&amp;gt;100000
  and s.instance_number=i.instance_number
  and s.instance_number=g.instance_number
 order by 2,1'</span><span style="color: #339933;">;</span>                                      
&nbsp;
<span style="color: #666666; font-style: italic;">/* default prefetch size (148) matches default snapshot retention (24hx7dd) */</span>
<span style="color: #000088;">$stmt</span> <span style="color: #339933;">=</span> oci_parse<span style="color: #009900;">&#40;</span><span style="color: #000088;">$ds</span><span style="color: #339933;">,</span> <span style="color: #000088;">$sql</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_set_prefetch<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #339933;">,</span> <span style="color: #cc66cc;">148</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_execute<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
<span style="color: #000088;">$i</span><span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
<span style="color: #000088;">$oldevent</span><span style="color: #339933;">=</span><span style="color: #0000ff;">&quot;&quot;</span><span style="color: #339933;">;</span>
<span style="color: #b1b100;">while</span> <span style="color: #009900;">&#40;</span><span style="color: #000088;">$row</span> <span style="color: #339933;">=</span> oci_fetch_assoc<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000088;">$oldevent</span> <span style="color: #339933;">!=</span> <span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'EVENT_NAME'</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #666666; font-style: italic;">//NEW EVENT DETECTED: WILL START A NEW UPDATE CMD</span>
                <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000088;">$i</span> <span style="color: #339933;">!=</span> <span style="color: #cc66cc;">0</span> <span style="color: #339933;">&amp;</span>amp<span style="color: #339933;">;&amp;</span>amp<span style="color: #339933;">;</span> <span style="color: #339933;">!</span><span style="color: #990000;">empty</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$cmd</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                        <span style="color: #666666; font-style: italic;">/* not the first occurrence,
                         I bet there's something in my buffer */</span>
                        <span style="color: #990000;">passthru</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$cmd</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                <span style="color: #000088;">$cleanName</span> <span style="color: #339933;">=</span> <span style="color: #990000;">preg_replace</span> <span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;([^[:alnum:]_-])&quot;</span><span style="color: #339933;">,</span><span style="color: #0000ff;">&quot;_&quot;</span><span style="color: #339933;">,</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'EVENT_NAME'</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #666666; font-style: italic;">// if there is no rrd for this event, I create a new one</span>
                <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #339933;">!</span><span style="color: #990000;">file_exists</span><span style="color: #009900;">&#40;</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/wait/<span style="color: #006699; font-weight: bold;">${cleanName}</span>.rrd&quot;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                        createRRD<span style="color: #009900;">&#40;</span><span style="color: #000088;">$cleanName</span><span style="color: #339933;">,</span> <span style="color: #000088;">$interval</span><span style="color: #339933;">,</span> <span style="color: #000088;">$cs</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
                <span style="color: #666666; font-style: italic;">/*
                * I initialize a new update command. This string act as a buffer: I append many
                * values to be updated so I'll update many values in a single command line:
                * less forks of rrdtool and less file opens: the whole update process has an
                * enormous improvement.
                */</span>
                <span style="color: #000088;">$precmd</span><span style="color: #339933;">=</span><span style="color: #0000ff;">&quot;rrdtool update &quot;</span><span style="color: #339933;">.</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/wait/<span style="color: #006699; font-weight: bold;">${cleanName}</span>.rrd &quot;</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$lastcmd</span><span style="color: #339933;">=</span><span style="color: #0000ff;">&quot;rrdtool info &quot;</span><span style="color: #339933;">.</span>WD<span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$cs</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;/wait/<span style="color: #006699; font-weight: bold;">${cleanName}</span>.rrd&quot;</span><span style="color: #339933;">.</span>
                        <span style="color: #0000ff;">&quot;| grep last_update | awk '{print <span style="color: #000099; font-weight: bold;">\$</span>NF}'&quot;</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$last</span><span style="color: #339933;">=</span><span style="color: #990000;">trim</span><span style="color: #009900;">&#40;</span>`<span style="color: #000088;">$lastcmd</span>`<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #990000;">printf</span> <span style="color: #009900;">&#40;</span><span style="color: #0000ff;">&quot;<span style="color: #009933; font-weight: bold;">%s</span> - <span style="color: #009933; font-weight: bold;">%s</span> - last: <span style="color: #009933; font-weight: bold;">%d</span><span style="color: #000099; font-weight: bold;">\n</span>&quot;</span><span style="color: #339933;">,</span> <span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'EVENT_NAME'</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> <span style="color: #000088;">$cleanName</span><span style="color: #339933;">,</span> <span style="color: #000088;">$last</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$i</span><span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$cmd</span><span style="color: #339933;">=</span><span style="color: #000088;">$precmd</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$oldevent</span><span style="color: #339933;">=</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'EVENT_NAME'</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #000088;">$time</span><span style="color: #339933;">=</span><span style="color: #990000;">strtotime</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'END_INTERVAL_TIME'</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #666666; font-style: italic;">//print &quot;time: &quot;.$time.&quot;  last: &quot;.$last.&quot;\n&quot;;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #000088;">$time</span> <span style="color: #339933;">&amp;</span>gt<span style="color: #339933;">;</span> <span style="color: #000088;">$last</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #000088;">$cmd</span><span style="color: #339933;">.=</span><span style="color: #0000ff;">&quot; &quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$time</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;:&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'TOTAL_WAITS'</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">.</span><span style="color: #0000ff;">&quot;:&quot;</span><span style="color: #339933;">.</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#91;</span><span style="color: #0000ff;">'MS'</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$i</span><span style="color: #339933;">++;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000088;">$i</span> <span style="color: #339933;">&amp;</span>gt<span style="color: #339933;">;=</span> <span style="color: #cc66cc;">40</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #666666; font-style: italic;">// when I reach 40 values per commandline I force</span>
                <span style="color: #666666; font-style: italic;">// the update: next loop will reinitialize a new commandline.</span>
                <span style="color: #990000;">passthru</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$cmd</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$cmd</span><span style="color: #339933;">=</span><span style="color: #000088;">$precmd</span><span style="color: #339933;">;</span>
                <span style="color: #000088;">$i</span><span style="color: #339933;">=</span><span style="color: #cc66cc;">0</span><span style="color: #339933;">;</span>
        <span style="color: #009900;">&#125;</span>
        <span style="color: #990000;">unset</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$row</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
<span style="color: #009900;">&#125;</span>
<span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span><span style="color: #000088;">$i</span> <span style="color: #339933;">!=</span> <span style="color: #cc66cc;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #666666; font-style: italic;">/* one more update pending in my buffer */</span>
        <span style="color: #990000;">passthru</span><span style="color: #009900;">&#40;</span><span style="color: #000088;">$cmd</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
oci_free_statement<span style="color: #009900;">&#40;</span><span style="color: #000088;">$stmt</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
oci_close<span style="color: #009900;">&#40;</span><span style="color: #000088;">$ds</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #000000; font-weight: bold;">?&gt;</span></pre></div></div>

<p>Depending on how many different wait events you have, you&#8217;ll have a certain number of rrd files:</p>
<pre># ls -l
total 3864
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 Streams_AQ__enqueue_blocked_on_low_memory.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 20 08:18 buffer_busy_waits.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 control_file_parallel_write.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 control_file_sequential_read.rrd
-rw-r--r-- 1 ludovico ludovico 165304 Apr 30 10:12 cursor__pin_S_wait_on_X.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 db_file_scattered_read.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 db_file_sequential_read.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 events_in_waitclass_Other.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 latch__cache_buffers_chains.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 latch__library_cache.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 11 13:22 latch__library_cache_lock.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 20 08:18 latch__redo_writing.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 latch__row_cache_objects.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 latch__shared_pool.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 library_cache_load_lock.rrd
-rw-r--r-- 1 ludovico ludovico 165304 Apr 15 13:17 library_cache_lock.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 log_buffer_space.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 log_file_parallel_write.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 log_file_sequential_read.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 log_file_single_write.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 log_file_switch_completion.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 11 13:22 log_file_sync.rrd
-rw-r--r-- 1 ludovico ludovico 165304 May 25 15:00 os_thread_startup.rrd</pre>
<p>As you can see, they are not so big&#8230;</p>
<p>Once you have your data in rrd files, it&#8217;s quite simple to script even complex plots with several datasources. Everything depends on the results you want.<br />
This script stack all my wait events for a certain instance: it takes the directory containing all the rrds as first argument and the number of hours we want to be plotted as second argument:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #007800;">cs</span>=$<span style="color: #000000;">1</span>
<span style="color: #007800;">hours</span>=<span style="color: #800000;">${2:-148}</span>
&nbsp;
<span style="color: #007800;">eventlist</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">ls</span> <span style="color: #007800;">$cs</span><span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/*</span>rrd<span style="color: #000000; font-weight: bold;">`</span>
&nbsp;
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">1</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#000000&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">2</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#000055&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">3</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#0000aa&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">4</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#0000ff&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">5</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#550055&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">6</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#aa00aa&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">7</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#ff00ff&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">8</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#550000&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">9</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#aa0000&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">10</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#ff0000&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">11</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#555500&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">12</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#aaaa00&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">13</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#ffff00&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">14</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#005500&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">15</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#00aa00&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">16</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#00ff00&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">17</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#005555&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">18</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#00aaaa&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">19</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#00ffff&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">20</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#555555&quot;</span>
colors<span style="color: #7a0874; font-weight: bold;">&#91;</span><span style="color: #000000;">21</span><span style="color: #7a0874; font-weight: bold;">&#93;</span>=<span style="color: #ff0000;">&quot;#aaaaaa&quot;</span>
&nbsp;
<span style="color: #007800;">i</span>=<span style="color: #000000;">0</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">for</span> event <span style="color: #000000; font-weight: bold;">in</span> <span style="color: #007800;">$eventlist</span> ; <span style="color: #000000; font-weight: bold;">do</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #007800;">$i</span> <span style="color: #660033;">-eq</span> <span style="color: #000000;">0</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span> ; <span style="color: #000000; font-weight: bold;">then</span>
                <span style="color: #007800;">end</span>=<span style="color: #000000; font-weight: bold;">`</span>rrdtool info <span style="color: #007800;">$event</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> last_update <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">awk</span> <span style="color: #ff0000;">'{print $NF}'</span><span style="color: #000000; font-weight: bold;">`</span>
                <span style="color: #007800;">end</span>=<span style="color: #000000; font-weight: bold;">`</span>rrdtool info <span style="color: #007800;">$cs</span><span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>control_file_parallel_write.rrd <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">grep</span> last_update <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">awk</span> <span style="color: #ff0000;">'{print $NF}'</span><span style="color: #000000; font-weight: bold;">`</span>
                <span style="color: #007800;">cmd</span>=<span style="color: #ff0000;">&quot;rrdtool graph - -s end-<span style="color: #007800;">${hours}</span>hours -e <span style="color: #007800;">$end</span>  -v <span style="color: #000099; font-weight: bold;">\&quot;</span>milliseconds waited<span style="color: #000099; font-weight: bold;">\&quot;</span> -l 0 -w 640 -h 240 -t <span style="color: #000099; font-weight: bold;">\&quot;</span><span style="color: #007800;">$cs</span> WAIT PROFILE<span style="color: #000099; font-weight: bold;">\&quot;</span>&quot;</span>
                <span style="color: #007800;">i</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #007800;">$i</span>+<span style="color: #000000;">1</span><span style="color: #7a0874; font-weight: bold;">&#41;</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
        <span style="color: #000000; font-weight: bold;">fi</span>
        <span style="color: #007800;">color</span>=<span style="color: #800000;">${colors[$i]}</span>
        <span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #007800;">$color</span>
        <span style="color: #007800;">evname</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">basename</span> <span style="color: #007800;">$event</span> <span style="color: #000000; font-weight: bold;">|</span> <span style="color: #c20cb9; font-weight: bold;">sed</span> <span style="color: #660033;">-e</span> s<span style="color: #000000; font-weight: bold;">/</span>\.rrd\$<span style="color: #000000; font-weight: bold;">//`</span>
        <span style="color: #007800;">cmd</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$cmd</span>  DEF:<span style="color: #007800;">$evname</span>=<span style="color: #007800;">$event</span>:mswaited:AVERAGE&quot;</span>
        <span style="color: #007800;">cmd</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$cmd</span>  AREA:<span style="color: #007800;">${evname}</span><span style="color: #007800;">${color}</span>:&quot;</span><span style="color: #007800;">$evname</span><span style="color: #ff0000;">&quot;:STACK&quot;</span>
        <span style="color: #007800;">i</span>=$<span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #7a0874; font-weight: bold;">&#40;</span><span style="color: #007800;">$i</span>+<span style="color: #000000;">1</span><span style="color: #7a0874; font-weight: bold;">&#41;</span><span style="color: #7a0874; font-weight: bold;">&#41;</span>
        <span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #007800;">$i</span> <span style="color: #660033;">-eq</span> <span style="color: #000000;">20</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span> ; <span style="color: #000000; font-weight: bold;">then</span>
                <span style="color: #007800;">i</span>=<span style="color: #000000;">1</span>
        <span style="color: #000000; font-weight: bold;">fi</span>
<span style="color: #000000; font-weight: bold;">done</span>
        <span style="color: #007800;">cmd</span>=<span style="color: #ff0000;">&quot;<span style="color: #007800;">$cmd</span>  |display /dev/input&quot;</span>
        <span style="color: #7a0874; font-weight: bold;">echo</span> <span style="color: #007800;">$cmd</span>
        <span style="color: #7a0874; font-weight: bold;">eval</span> <span style="color: #007800;">$cmd</span>
<span style="color: #7a0874; font-weight: bold;">exit</span></pre></div></div>

<p>The resulting command is very long:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;">rrdtool graph - <span style="color: #660033;">-s</span> end-148hours <span style="color: #660033;">-e</span> <span style="color: #000000;">1243252800</span> \
 <span style="color: #660033;">-v</span> <span style="color: #ff0000;">&quot;milliseconds waited&quot;</span> <span style="color: #660033;">-l</span> <span style="color: #000000;">0</span> <span style="color: #660033;">-w</span> <span style="color: #000000;">640</span> <span style="color: #660033;">-h</span> <span style="color: #000000;">240</span> <span style="color: #660033;">-t</span> <span style="color: #ff0000;">&quot;mydb WAIT PROFILE&quot;</span>\
 DEF:<span style="color: #007800;">Streams_AQ__enqueue_blocked_on_low_memory</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>Streams_AQ__enqueue_blocked_on_low_memory.rrd:mswaited:AVERAGE \
 AREA:Streams_AQ__enqueue_blocked_on_low_memory<span style="color: #666666; font-style: italic;">#000000:Streams_AQ__enqueue_blocked_on_low_memory:STACK\</span>
 DEF:<span style="color: #007800;">buffer_busy_waits</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>buffer_busy_waits.rrd:mswaited:AVERAGE \
 AREA:buffer_busy_waits<span style="color: #666666; font-style: italic;">#000055:buffer_busy_waits:STACK\</span>
 DEF:<span style="color: #007800;">control_file_parallel_write</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>control_file_parallel_write.rrd:mswaited:AVERAGE \
 AREA:control_file_parallel_write<span style="color: #666666; font-style: italic;">#0000aa:control_file_parallel_write:STACK\</span>
 DEF:<span style="color: #007800;">control_file_sequential_read</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>control_file_sequential_read.rrd:mswaited:AVERAGE \
 AREA:control_file_sequential_read<span style="color: #666666; font-style: italic;">#0000ff:control_file_sequential_read:STACK\</span>
 DEF:<span style="color: #007800;">cursor__pin_S_wait_on_X</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>cursor__pin_S_wait_on_X.rrd:mswaited:AVERAGE \
 AREA:cursor__pin_S_wait_on_X<span style="color: #666666; font-style: italic;">#550055:cursor__pin_S_wait_on_X:STACK\</span>
 DEF:<span style="color: #007800;">db_file_scattered_read</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>db_file_scattered_read.rrd:mswaited:AVERAGE \
 AREA:db_file_scattered_read<span style="color: #666666; font-style: italic;">#aa00aa:db_file_scattered_read:STACK\</span>
 DEF:<span style="color: #007800;">db_file_sequential_read</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>db_file_sequential_read.rrd:mswaited:AVERAGE \
 AREA:db_file_sequential_read<span style="color: #666666; font-style: italic;">#ff00ff:db_file_sequential_read:STACK\</span>
 DEF:<span style="color: #007800;">events_in_waitclass_Other</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>events_in_waitclass_Other.rrd:mswaited:AVERAGE \
 AREA:events_in_waitclass_Other<span style="color: #666666; font-style: italic;">#550000:events_in_waitclass_Other:STACK\</span>
 DEF:<span style="color: #007800;">latch__cache_buffers_chains</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__cache_buffers_chains.rrd:mswaited:AVERAGE \
 AREA:latch__cache_buffers_chains<span style="color: #666666; font-style: italic;">#aa0000:latch__cache_buffers_chains:STACK\</span>
 DEF:<span style="color: #007800;">latch__library_cache</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__library_cache.rrd:mswaited:AVERAGE \
 AREA:latch__library_cache<span style="color: #666666; font-style: italic;">#ff0000:latch__library_cache:STACK\</span>
 DEF:<span style="color: #007800;">latch__library_cache_lock</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__library_cache_lock.rrd:mswaited:AVERAGE \
 AREA:latch__library_cache_lock<span style="color: #666666; font-style: italic;">#555500:latch__library_cache_lock:STACK\</span>
 DEF:<span style="color: #007800;">latch__redo_writing</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__redo_writing.rrd:mswaited:AVERAGE \
 AREA:latch__redo_writing<span style="color: #666666; font-style: italic;">#aaaa00:latch__redo_writing:STACK\</span>
 DEF:<span style="color: #007800;">latch__row_cache_objects</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__row_cache_objects.rrd:mswaited:AVERAGE \
 AREA:latch__row_cache_objects<span style="color: #666666; font-style: italic;">#ffff00:latch__row_cache_objects:STACK\</span>
 DEF:<span style="color: #007800;">latch__shared_pool</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>latch__shared_pool.rrd:mswaited:AVERAGE \
 AREA:latch__shared_pool<span style="color: #666666; font-style: italic;">#005500:latch__shared_pool:STACK\</span>
 DEF:<span style="color: #007800;">library_cache_load_lock</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>library_cache_load_lock.rrd:mswaited:AVERAGE \
 AREA:library_cache_load_lock<span style="color: #666666; font-style: italic;">#00aa00:library_cache_load_lock:STACK\</span>
 DEF:<span style="color: #007800;">library_cache_lock</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>library_cache_lock.rrd:mswaited:AVERAGE \
 AREA:library_cache_lock<span style="color: #666666; font-style: italic;">#00ff00:library_cache_lock:STACK\</span>
 DEF:<span style="color: #007800;">log_buffer_space</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_buffer_space.rrd:mswaited:AVERAGE \
 AREA:log_buffer_space<span style="color: #666666; font-style: italic;">#005555:log_buffer_space:STACK\</span>
 DEF:<span style="color: #007800;">log_file_parallel_write</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_file_parallel_write.rrd:mswaited:AVERAGE \
 AREA:log_file_parallel_write<span style="color: #666666; font-style: italic;">#00aaaa:log_file_parallel_write:STACK\</span>
 DEF:<span style="color: #007800;">log_file_sequential_read</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_file_sequential_read.rrd:mswaited:AVERAGE \
 AREA:log_file_sequential_read<span style="color: #666666; font-style: italic;">#00ffff:log_file_sequential_read:STACK\</span>
 DEF:<span style="color: #007800;">log_file_single_write</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_file_single_write.rrd:mswaited:AVERAGE \
 AREA:log_file_single_write<span style="color: #666666; font-style: italic;">#000000:log_file_single_write:STACK\</span>
 DEF:<span style="color: #007800;">log_file_switch_completion</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_file_switch_completion.rrd:mswaited:AVERAGE \
 AREA:log_file_switch_completion<span style="color: #666666; font-style: italic;">#000055:log_file_switch_completion:STACK\</span>
 DEF:<span style="color: #007800;">log_file_sync</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>log_file_sync.rrd:mswaited:AVERAGE \
 AREA:log_file_sync<span style="color: #666666; font-style: italic;">#0000aa:log_file_sync:STACK\</span>
 DEF:<span style="color: #007800;">os_thread_startup</span>=mydb<span style="color: #000000; font-weight: bold;">/</span>wait<span style="color: #000000; font-weight: bold;">/</span>os_thread_startup.rrd:mswaited:AVERAGE \
 AREA:os_thread_startup<span style="color: #666666; font-style: italic;">#0000ff:os_thread_startup:STACK |display /dev/input</span></pre></div></div>

<p>This is the resulting graph:<br />
<a href="http://www.ludovicocaldara.net/images/mydb_events.png"><img alt="Graph plotted with rrdtool displaying Oracle instance Wait Events" src="http://www.ludovicocaldara.net/images/mydb_events.png" title="Oracle Instance Wait Events" width="737" height="398" /></a></p>
<p><strong>OHHHHHHHHHHHH COOOOL!!!</strong><br />
 <img src='http://www.ludovicocaldara.net/dba/wp-includes/images/smilies/icon_wink.gif' alt=';-)' class='wp-smiley' /> </p>
<p>Any comment is appreciated! thanks</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/oracle-capacity-planning-with-rrdtool/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>How to collect Oracle Application Server performance data with DMS and RRDtool</title>
		<link>http://www.ludovicocaldara.net/dba/how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool/</link>
		<comments>http://www.ludovicocaldara.net/dba/how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool/#comments</comments>
		<pubDate>Mon, 02 Mar 2009 17:08:31 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[Oracle Application Server]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[Capacity Planning]]></category>
		<category><![CDATA[rrdtool]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=56</guid>
		<description><![CDATA[RRDize everything, chapter 1
If you are managing some Application Server deployments you should have wondered how to check and collect performance data.
As stated in documentation, you can gather performance metrics with the dmstool utility.
AFAIK, this can be done from 9.0.2 release upwards, but i&#8217;m concerned DMS will not work on Weblogic.
Mainly, you should have an [...]]]></description>
			<content:encoded><![CDATA[<p><strong>RRDize everything, chapter 1</strong></p>
<p>If you are managing some Application Server deployments you should have wondered how to check and collect performance data.<br />
As stated in documentation, you can gather performance metrics with the dmstool utility.<br />
AFAIK, this can be done from 9.0.2 release upwards, but i&#8217;m concerned DMS will not work on Weblogic.</p>
<p>Mainly, you should have an external server that acts as collector (it could be a server in the Oracle AS farm as well): copy the dms.jar library from an Oracle AS installation to your collector and use it as you would use dmstool:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;">java -jar dms.jar [dmstool options]</pre></div></div>

<p>There are three basilar methods to get data:</p>
<p>Get all metrics at once:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;">java -jar dms.jar -dump -a &quot;youraddress://...&quot; [format=xml]</pre></div></div>

<p>Get only the interesting metrics:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;">java -jar dms.jar -a &quot;youraddress://...&quot; metric metric ...</pre></div></div>

<p>Get metrics included into specific DMS tables:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;">java -jar dms.jar -a &quot;youraddress://...&quot; -table table table ...</pre></div></div>

<p>What youraddress:// is, it depends on the component you are trying to connect:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;">opmn://asserver:6003
http://asserver:7200/dms0/Spy
ajp13://asserver:3301/dmsoc4j/Spy</pre></div></div>

<p>If you are trying to connect to the OHS (Apache), be careful to allow remote access from the collector by editing the dms.conf file.</p>
<p>Now that you can query dms data, you should store it somewhere.<br />
Personally, I did a first attempt with dmstool -dump format=xml. I wrote a parser in PHP with SimpleXML extension and I did a lot of inserts into a MySQL database. After a few months the whole data collected from tens of servers was too much to be mantained&#8230;<br />
To avoid the maintenance of a DWH-grade database I investigated and found RRDTool. Now I&#8217;m asking how could I live without it!</p>
<p>I then wrote a parser in awk that parse the output of the dms.jar call and invoke an rrdtool update command.<br />
I always use dms.jar -table command. The output has always the same format:</p>
<pre>###SOF

Mon Mar 02 17:01:19 CET 2009

---------------
TABLE1_Name
---------------

record1_metric1.name:     value       units
record1_metric2.name:     value       units
....

record2_metric1.name:     value       units
record2_metric2.name:     value       units
....

---
TABLE2_Name
---

record1_metric1.name:     value       units
record1_metric2.name:     value       units
....

record2_metric1.name:     value       units
record2_metric2.name:     value       units
....

##EOF</pre>
<p>So I written an awk file that works for me.<br />
use it this way:</p>

<div class="wp_syntax"><div class="code"><pre class="sh" style="font-family:monospace;"> java -jar dms.jar ... | awk -f parse_output.awk</pre></div></div>


<div class="wp_syntax"><div class="code"><pre class="c" style="font-family:monospace;"><span style="color: #339933;">####################</span>
<span style="color: #339933;"># parse_output.awk #</span>
<span style="color: #339933;">####################</span>
&nbsp;
<span style="color: #339933;">#function pl() replaces all non alphanumeric occurrences with an underscore</span>
<span style="color: #000000; font-weight: bold;">function</span> pl<span style="color: #009900;">&#40;</span>input<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #b1b100;">return</span> gensub<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;[^[:alnum:]_-]&quot;</span><span style="color: #339933;">,</span><span style="color: #ff0000;">&quot;_&quot;</span><span style="color: #339933;">,</span><span style="color: #ff0000;">&quot;G&quot;</span><span style="color: #339933;">,</span>input<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #339933;"># function get_rrd_path() returns a path where the rrd files should be placed</span>
<span style="color: #339933;"># I should rewrite a new path for each dms table... I'll skip many of them</span>
<span style="color: #000000; font-weight: bold;">function</span> get_rrd_path<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>table <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;mod_oc4j_destination_metrics&quot;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span>
                    pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Name.value&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>table <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;mod_oc4j_mount_pt_metrics&quot;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span>
                    pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Destination.value&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Name.value&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>table <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;ohs_server&quot;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>table <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;JVM&quot;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span>
                    pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Process&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>table <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;opmn_process&quot;</span><span style="color: #009900;">&#41;</span>
                <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s/%s/%s/%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span>
                  pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;iasInstance.value&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;opmn_ias_component&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span>
                  pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;opmn_process_type&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span>pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;opmn_process_set&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span>
                  pl<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Name&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
&nbsp;
        <span style="color: #b1b100;">return</span> sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;%s/%s/%s.rrd&quot;</span><span style="color: #339933;">,</span> record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;Host&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>table<span style="color: #009900;">&#41;</span><span style="color: #339933;">,</span> pl<span style="color: #009900;">&#40;</span>var<span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
<span style="color: #339933;"># function process_record actually does the dirty work of invoking the update script</span>
<span style="color: #000000; font-weight: bold;">function</span> process_record<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #339933;">#every record has a timeStamp.ts metric that I should use to update my rrd</span>
        ts<span style="color: #339933;">=</span>substr<span style="color: #009900;">&#40;</span>record<span style="color: #009900;">&#91;</span><span style="color: #ff0000;">&quot;timeStamp.ts&quot;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">,</span><span style="color: #0000dd;">0</span><span style="color: #339933;">,</span><span style="color: #0000dd;">10</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
        <span style="color: #b1b100;">for</span> <span style="color: #009900;">&#40;</span> var in record <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> var <span style="color: #339933;">!=</span> <span style="color: #ff0000;">&quot;timeStamp.ts&quot;</span> <span style="color: #339933;">&amp;</span>amp<span style="color: #339933;">;&amp;</span>amp<span style="color: #339933;">;</span> record<span style="color: #009900;">&#91;</span>var<span style="color: #009900;">&#93;</span> ~ <span style="color: #339933;">/^</span><span style="color: #009900;">&#91;</span><span style="color: #009900;">&#91;</span><span style="color: #339933;">:</span>digit<span style="color: #339933;">:</span><span style="color: #009900;">&#93;</span><span style="color: #009900;">&#93;</span><span style="color: #339933;">+</span>$<span style="color: #339933;">/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
            <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> var ~ <span style="color: #339933;">/</span>\.<span style="color: #009900;">&#40;</span>count<span style="color: #339933;">|</span>completed<span style="color: #339933;">|</span>time<span style="color: #009900;">&#41;</span>$<span style="color: #339933;">/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                dstype<span style="color: #339933;">=</span><span style="color: #ff0000;">&quot;DERIVE&quot;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span> <span style="color: #b1b100;">else</span> <span style="color: #009900;">&#123;</span>
                <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> var <span style="color: #339933;">==</span> <span style="color: #ff0000;">&quot;responseSize.value&quot;</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
                    dstype<span style="color: #339933;">=</span><span style="color: #ff0000;">&quot;DERIVE&quot;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span> <span style="color: #b1b100;">else</span> <span style="color: #009900;">&#123;</span>
                    dstype<span style="color: #339933;">=</span><span style="color: #ff0000;">&quot;GAUGE&quot;</span><span style="color: #339933;">;</span>
                <span style="color: #009900;">&#125;</span>
            <span style="color: #009900;">&#125;</span>
            rrdFile<span style="color: #339933;">=</span>sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;/path_to_data/%s&quot;</span><span style="color: #339933;">,</span>get_rrd_path<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #339933;">#### update_metric_rrd is a shell script listed below!!!!!</span>
            cmd<span style="color: #339933;">=</span>sprintf<span style="color: #009900;">&#40;</span><span style="color: #ff0000;">&quot;/path_to_scripts/update_metric_rrd %s %s %d %d&quot;</span><span style="color: #339933;">,</span>
                rrdFile<span style="color: #339933;">,</span>dstype<span style="color: #339933;">,</span>ts<span style="color: #339933;">,</span>record<span style="color: #009900;">&#91;</span>var<span style="color: #009900;">&#93;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            system<span style="color: #009900;">&#40;</span>cmd<span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
            <span style="color: #009900;">&#125;</span>
        <span style="color: #009900;">&#125;</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #339933;"># parse_record() populates an hash array</span>
<span style="color: #339933;"># with all metrics belonging to the table record</span>
<span style="color: #000000; font-weight: bold;">function</span> parse_record<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
    <span style="color: #339933;">#print &quot;RRRR -  START OF RECORD (table &quot; table &quot;)&quot;</span>
    delete record
    <span style="color: #b1b100;">while</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">!</span> <span style="color: #339933;">/^</span>$<span style="color: #339933;">/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #339933;"># I'm parsing the record as far I'm in this while statement</span>
        <span style="color: #339933;"># the array hash is the name of the dms metric basename.</span>
        <span style="color: #339933;"># $1 is the metric name but I have to trim the final &quot;:&quot;</span>
        key<span style="color: #339933;">=</span>substr<span style="color: #009900;">&#40;</span>$<span style="color: #0000dd;">1</span><span style="color: #339933;">,</span><span style="color: #0000dd;">0</span><span style="color: #339933;">,</span>length<span style="color: #009900;">&#40;</span>$<span style="color: #0000dd;">1</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">-</span><span style="color: #0000dd;">1</span><span style="color: #009900;">&#41;</span>
        record<span style="color: #009900;">&#91;</span>key<span style="color: #009900;">&#93;</span><span style="color: #339933;">=</span>$<span style="color: #0000dd;">2</span>
        getline
    <span style="color: #009900;">&#125;</span>
    <span style="color: #339933;"># this function is included in funcions.awk:</span>
    <span style="color: #339933;"># I invoke it to process the record I've just parsed</span>
    process_record<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span><span style="color: #339933;">;</span>
<span style="color: #009900;">&#125;</span>
BEGIN <span style="color: #009900;">&#123;</span>
    <span style="color: #339933;"># as far as started is 0, I've never reached the first table</span>
    started<span style="color: #339933;">=</span><span style="color: #0000dd;">0</span>
<span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #339933;">#MAIN</span>
<span style="color: #009900;">&#123;</span>
    <span style="color: #339933;"># I jump over the first lines until I reach the first table</span>
    <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span>started<span style="color: #339933;">==</span><span style="color: #0000dd;">0</span><span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #b1b100;">while</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">!</span> <span style="color: #339933;">/^---/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
           getline
        <span style="color: #009900;">&#125;</span>
        started<span style="color: #339933;">=</span><span style="color: #0000dd;">1</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #339933;"># looking for the next occurrence of a table</span>
    <span style="color: #339933;"># all tables start with:</span>
    <span style="color: #339933;"># ----------</span>
    <span style="color: #339933;"># table_name</span>
    <span style="color: #339933;"># ----------</span>
    <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">/^---/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #339933;"># first table reached: the next row is my table name,</span>
        <span style="color: #339933;"># then I reach again a dashed line -----</span>
        getline table
        getline trash
        <span style="color: #339933;">#print &quot;&quot;</span>
        <span style="color: #339933;">#print &quot;##########################&quot;</span>
        print <span style="color: #ff0000;">&quot;  TABELLA &quot;</span> table
        <span style="color: #339933;">#print &quot;##########################&quot;</span>
        next
    <span style="color: #009900;">&#125;</span>
&nbsp;
    <span style="color: #b1b100;">if</span> <span style="color: #009900;">&#40;</span> <span style="color: #339933;">!</span> <span style="color: #339933;">/^</span>$<span style="color: #339933;">/</span> <span style="color: #009900;">&#41;</span> <span style="color: #009900;">&#123;</span>
        <span style="color: #339933;"># reached an empty line: could be the end of a record or the and of a table</span>
        <span style="color: #339933;"># since a new table is threated in previous &quot;if&quot; statement, I'm starting a new record.</span>
        parse_record<span style="color: #009900;">&#40;</span><span style="color: #009900;">&#41;</span>
    <span style="color: #009900;">&#125;</span>
&nbsp;
<span style="color: #009900;">&#125;</span>
&nbsp;
END <span style="color: #009900;">&#123;</span>
<span style="color: #009900;">&#125;</span></pre></div></div>

<p>And this is the code for update_metric_rrd:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;"><span style="color: #666666; font-style: italic;">#!/bin/bash</span>
<span style="color: #007800;">RRDFILE</span>=$<span style="color: #000000;">1</span>
<span style="color: #007800;">DSTYPE</span>=$<span style="color: #000000;">2</span>
<span style="color: #007800;">TS</span>=$<span style="color: #000000;">3</span>
<span style="color: #007800;">VALUE</span>=$<span style="color: #000000;">4</span>
&nbsp;
rrdtool update <span style="color: #007800;">$RRDFILE</span> <span style="color: #800000;">${TS}</span>:<span style="color: #800000;">${VALUE}</span>
&nbsp;
<span style="color: #000000; font-weight: bold;">if</span> <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #007800;">$?</span> <span style="color: #660033;">-ne</span> <span style="color: #000000;">0</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span> ; <span style="color: #000000; font-weight: bold;">then</span>
        <span style="color: #007800;">DIR</span>=<span style="color: #000000; font-weight: bold;">`</span><span style="color: #c20cb9; font-weight: bold;">dirname</span> <span style="color: #007800;">$RRDFILE</span><span style="color: #000000; font-weight: bold;">`</span>
&nbsp;
        <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #660033;">-d</span> <span style="color: #007800;">$DIR</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span> <span style="color: #000000; font-weight: bold;">||</span> <span style="color: #c20cb9; font-weight: bold;">mkdir</span> <span style="color: #660033;">-p</span> <span style="color: #007800;">$DIR</span>
        <span style="color: #7a0874; font-weight: bold;">&#91;</span> <span style="color: #660033;">-f</span> <span style="color: #007800;">$RRDFILE</span> <span style="color: #7a0874; font-weight: bold;">&#93;</span> <span style="color: #000000; font-weight: bold;">||</span> rrdtool create <span style="color: #007800;">$RRDFILE</span> <span style="color: #660033;">-b</span> <span style="color: #ff0000;">&quot;now-1month&quot;</span> <span style="color: #660033;">-s</span> <span style="color: #000000;">1800</span> \
                DS:metric:<span style="color: #800000;">${DSTYPE}</span>:<span style="color: #000000;">7200</span>:<span style="color: #000000;">0</span>:U \
                RRA:AVERAGE:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">1</span>:<span style="color: #000000;">672</span> \
                RRA:AVERAGE:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">4</span>:<span style="color: #000000;">1080</span> \
                RRA:AVERAGE:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">12</span>:<span style="color: #000000;">1460</span> \
                RRA:AVERAGE:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">48</span>:<span style="color: #000000;">1095</span> \
                RRA:MAX:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">4</span>:<span style="color: #000000;">1080</span> \
                RRA:MAX:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">12</span>:<span style="color: #000000;">1460</span> \
                RRA:MAX:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">48</span>:<span style="color: #000000;">1095</span> \
                RRA:LAST:<span style="color: #000000;">0.5</span>:<span style="color: #000000;">1</span>:<span style="color: #000000;">672</span>
        rrdtool update <span style="color: #007800;">$RRDFILE</span> <span style="color: #800000;">${TS}</span>:<span style="color: #800000;">${VALUE}</span>
<span style="color: #000000; font-weight: bold;">fi</span></pre></div></div>

<p>Once you have all your rrd files populated, it&#8217;s easy to script automatic reporting. You would probably want a graph with the request count served by your Apache cluster, along with its linear regression:</p>

<div class="wp_syntax"><div class="code"><pre class="bash" style="font-family:monospace;">rrdtool graph - <span style="color: #660033;">-s</span> <span style="color: #ff0000;">&quot;end-<span style="color: #007800;">${hours}</span>hours&quot;</span> <span style="color: #660033;">-e</span> <span style="color: #007800;">$end</span> \
                <span style="color: #660033;">-v</span> <span style="color: #ff0000;">&quot;Requests Completed/sec&quot;</span> \
        <span style="color: #660033;">-w</span> <span style="color: #000000;">640</span> <span style="color: #660033;">-h</span> <span style="color: #000000;">240</span> <span style="color: #660033;">--slope-mode</span> \
                <span style="color: #660033;">-t</span> <span style="color: #ff0000;">&quot;HTTP Requests for www.ludovicocaldara.net&quot;</span> \
                DEF:<span style="color: #000000;">1</span><span style="color: #007800;">request_completed</span>=<span style="color: #000000; font-weight: bold;">/</span>data<span style="color: #000000; font-weight: bold;">/</span>wwwserver1<span style="color: #000000; font-weight: bold;">/</span>ohs_server<span style="color: #000000; font-weight: bold;">/</span>request_completed.rrd:metric:AVERAGE \
                DEF:<span style="color: #000000;">2</span><span style="color: #007800;">request_completed</span>=<span style="color: #000000; font-weight: bold;">/</span>data<span style="color: #000000; font-weight: bold;">/</span>wwwserver2<span style="color: #000000; font-weight: bold;">/</span>ohs_server<span style="color: #000000; font-weight: bold;">/</span>request_completed.rrd:metric:AVERAGE \
                CDEF:<span style="color: #007800;">request_completed</span>=1request_completed,2request_completed,+ \
                VDEF:<span style="color: #007800;">slope</span>=request_completed,LSLSLOPE \
                VDEF:<span style="color: #007800;">lslint</span>=request_completed,LSLINT \
                CDEF:<span style="color: #007800;">reg</span>=request_completed,POP,slope,COUNT,<span style="color: #000000; font-weight: bold;">*</span>,lslint,+ \
                LINE1:reg<span style="color: #666666; font-style: italic;">#666666:&quot;Regression&quot; \</span>
                AREA:1request_completed<span style="color: #666666; font-style: italic;">#4040AA:&quot;wwwserver1&quot;  \</span>
                AREA:2request_completed<span style="color: #666666; font-style: italic;">#6666FF:&quot;wwwserver1&quot;:STACK  \</span>
        <span style="color: #000000; font-weight: bold;">&amp;</span>gt; mygraph.png</pre></div></div>

<p>This is the result:<br />
<img src="http://www.ludovicocaldara.net/images/dba/dms_ohs_requests.png" alt="OHS request completed" /><br />
<strong>OHHHHHHHHHHHH!!!! COOL!!!!</strong></p>
<p>That&#8217;s all for DMS capacity planning. Stay tuned, more about rrdtool is coming!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Interesting Slides about Capacity Planning</title>
		<link>http://www.ludovicocaldara.net/dba/interesting-slides-about-capacity-planning/</link>
		<comments>http://www.ludovicocaldara.net/dba/interesting-slides-about-capacity-planning/#comments</comments>
		<pubDate>Thu, 29 Jan 2009 21:43:17 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[Capacity Planning]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=46</guid>
		<description><![CDATA[Browsing the web I found this interesting presentation by John Allspaw:
Slides from ‘Capacity Planning for LAMP’ talk at MySQL Conf 2007.
(Damn, I had to clean out the embedded flash because of invalid markup&#8230;)

I&#8217;m preparing some stuff related to RRDTool, I&#8217;ll post soon here some code snippets.
]]></description>
			<content:encoded><![CDATA[<p>Browsing the web I found this interesting presentation by <b>John Allspaw</b>:</p>
<p><a title="View Slides from ‘Capacity Planning for LAMP’ talk at MySQL Conf 2007 on Scribd" href="http://www.scribd.com/doc/43281/Slides-from-Capacity-Planning-for-LAMP-talk-at-MySQL-Conf-2007" style="margin: 12px auto 6px auto; font-family: Helvetica,Arial,Sans-serif; font-style: normal; font-variant: normal; font-weight: normal; font-size: 14px; line-height: normal; font-size-adjust: none; font-stretch: normal; -x-system-font: none; display: block; text-decoration: underline;">Slides from ‘Capacity Planning for LAMP’ talk at MySQL Conf 2007</a>.<br />
(Damn, I had to clean out the embedded flash because of invalid markup&#8230;)<br />
<br />
I&#8217;m preparing some stuff related to <a href="http://oss.oetiker.ch/rrdtool/">RRDTool</a>, I&#8217;ll post soon here some code snippets.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/interesting-slides-about-capacity-planning/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>JBoss Portal and MySQL scalability: What The&#8230;???</title>
		<link>http://www.ludovicocaldara.net/dba/jboss-portal-and-mysql-scalability/</link>
		<comments>http://www.ludovicocaldara.net/dba/jboss-portal-and-mysql-scalability/#comments</comments>
		<pubDate>Tue, 09 Dec 2008 22:06:33 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[JBoss]]></category>
		<category><![CDATA[MySQL]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[Portal]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=33</guid>
		<description><![CDATA[I found several queries running on a MySQL 5.0 database like this one:

SELECT PATH, NAME FROM JBP_OBJECT_NODE  WHERE PK IN &#40;
SELECT NODE_KEY FROM JBP_OBJECT_NODE_SEC WHERE ROLE IN &#40;
SELECT jr.jbp_name FROM jbp_users ju, jbp_role_membership jrm,
jbp_roles jr
WHERE jrm.jbp_uid = ju.jbp_uid
AND jr.jbp_rid = jrm.jbp_rid
AND ju.jbp_uname = 'LUDOVICO'
AND ju.jbp_enabled = 1&#41;&#41;;

This query is related to JBoss Portal and does [...]]]></description>
			<content:encoded><![CDATA[<p>I found several queries running on a MySQL 5.0 database like this one:</p>

<div class="wp_syntax"><div class="code"><pre class="sql" style="font-family:monospace;"><span style="color: #993333; font-weight: bold;">SELECT</span> PATH<span style="color: #66cc66;">,</span> NAME <span style="color: #993333; font-weight: bold;">FROM</span> JBP_OBJECT_NODE  <span style="color: #993333; font-weight: bold;">WHERE</span> PK <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #66cc66;">&#40;</span>
<span style="color: #993333; font-weight: bold;">SELECT</span> NODE_KEY <span style="color: #993333; font-weight: bold;">FROM</span> JBP_OBJECT_NODE_SEC <span style="color: #993333; font-weight: bold;">WHERE</span> ROLE <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #66cc66;">&#40;</span>
<span style="color: #993333; font-weight: bold;">SELECT</span> jr<span style="color: #66cc66;">.</span>jbp_name <span style="color: #993333; font-weight: bold;">FROM</span> jbp_users ju<span style="color: #66cc66;">,</span> jbp_role_membership jrm<span style="color: #66cc66;">,</span>
jbp_roles jr
<span style="color: #993333; font-weight: bold;">WHERE</span> jrm<span style="color: #66cc66;">.</span>jbp_uid <span style="color: #66cc66;">=</span> ju<span style="color: #66cc66;">.</span>jbp_uid
<span style="color: #993333; font-weight: bold;">AND</span> jr<span style="color: #66cc66;">.</span>jbp_rid <span style="color: #66cc66;">=</span> jrm<span style="color: #66cc66;">.</span>jbp_rid
<span style="color: #993333; font-weight: bold;">AND</span> ju<span style="color: #66cc66;">.</span>jbp_uname <span style="color: #66cc66;">=</span> <span style="color: #ff0000;">'LUDOVICO'</span>
<span style="color: #993333; font-weight: bold;">AND</span> ju<span style="color: #66cc66;">.</span>jbp_enabled <span style="color: #66cc66;">=</span> <span style="color: #cc66cc;">1</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span>;</pre></div></div>

<p>This query is related to JBoss Portal and does a full scan on table JBP_OBJECT_NODE.</p>
<p>It has bad performances (&gt;0.8 sec) with just a few records:</p>
<p>mysql&gt; select count(*) from JBP_OBJECT_NODE;<br />
+&#8212;&#8212;&#8212;-+<br />
| count(*) |<br />
+&#8212;&#8212;&#8212;-+<br />
|    33461 |<br />
+&#8212;&#8212;&#8212;-+</p>
<p>If I rewrite the query using an inner join (à la Oracle, please forgive me) instead of a subquery I get an index scan:</p>

<div class="wp_syntax"><div class="code"><pre class="sql" style="font-family:monospace;"><span style="color: #993333; font-weight: bold;">SELECT</span> <span style="color: #993333; font-weight: bold;">DISTINCT</span> a<span style="color: #66cc66;">.</span>PATH<span style="color: #66cc66;">,</span> a<span style="color: #66cc66;">.</span>NAME <span style="color: #808080; font-style: italic;">/* , b.NODE_KEY */</span> <span style="color: #993333; font-weight: bold;">FROM</span>  JBP_OBJECT_NODE
a<span style="color: #66cc66;">,</span> JBP_OBJECT_NODE_SEC b
<span style="color: #993333; font-weight: bold;">WHERE</span> a<span style="color: #66cc66;">.</span>pk<span style="color: #66cc66;">=</span>b<span style="color: #66cc66;">.</span>NODE_KEY
<span style="color: #993333; font-weight: bold;">AND</span> b<span style="color: #66cc66;">.</span>ROLE <span style="color: #993333; font-weight: bold;">IN</span> <span style="color: #66cc66;">&#40;</span>
<span style="color: #993333; font-weight: bold;">SELECT</span> jr<span style="color: #66cc66;">.</span>jbp_name <span style="color: #993333; font-weight: bold;">FROM</span> jbp_users ju<span style="color: #66cc66;">,</span> jbp_role_membership jrm<span style="color: #66cc66;">,</span>
jbp_roles jr
<span style="color: #993333; font-weight: bold;">WHERE</span> jrm<span style="color: #66cc66;">.</span>jbp_uid <span style="color: #66cc66;">=</span> ju<span style="color: #66cc66;">.</span>jbp_uid
<span style="color: #993333; font-weight: bold;">AND</span> jr<span style="color: #66cc66;">.</span>jbp_rid <span style="color: #66cc66;">=</span> jrm<span style="color: #66cc66;">.</span>jbp_rid
<span style="color: #993333; font-weight: bold;">AND</span> ju<span style="color: #66cc66;">.</span>jbp_uname <span style="color: #66cc66;">=</span> <span style="color: #ff0000;">'UTDEMO'</span>
<span style="color: #993333; font-weight: bold;">AND</span> ju<span style="color: #66cc66;">.</span>jbp_enabled <span style="color: #66cc66;">=</span> <span style="color: #cc66cc;">1</span><span style="color: #66cc66;">&#41;</span>;</pre></div></div>

<p>With 30k records the execution time falls down from 0.8 secs to 0.01 secs&#8230;<br />
That&#8217;s NOT all! I found this open bug:</p>
<p>https://jira.jboss.org/jira/browse/JBPORTAL-2040</p>
<p>With many users registered in, the JBoss Portal Admin console tooks over a minute to show a single page&#8230;</p>
<p>I don&#8217;t like portals&#8230;</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/jboss-portal-and-mysql-scalability/feed/</wfw:commentRss>
		<slash:comments>2</slash:comments>
		</item>
		<item>
		<title>Oracle RAC Standard Edition to achieve low cost and high performance</title>
		<link>http://www.ludovicocaldara.net/dba/oracle-rac-standard-edition-to-achieve-low-cost-and-high-performance/</link>
		<comments>http://www.ludovicocaldara.net/dba/oracle-rac-standard-edition-to-achieve-low-cost-and-high-performance/#comments</comments>
		<pubDate>Thu, 27 Nov 2008 23:43:04 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Linux]]></category>
		<category><![CDATA[Oracle Database]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[cluster]]></category>
		<category><![CDATA[RAC]]></category>
		<category><![CDATA[Standard Edition]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=29</guid>
		<description><![CDATA[I finished today to create a new production environment based on 2 Linux serverX86_64 and running Oracle RAC 10gR2. (I know, there is 11g right now, but I&#8217;m a conservative!)
Wheeew, I just spent a couple of hours applying all the recommended patches!
We choosed 2 nodes with a maximum of 2 multi-core processors each one so [...]]]></description>
			<content:encoded><![CDATA[<p>I finished today to create a new production environment based on 2 Linux serverX86_64 and running Oracle RAC 10gR2. (I know, there is 11g right now, but I&#8217;m a conservative!)<br />
Wheeew, I just spent a couple of hours applying all the recommended patches!<br />
We choosed 2 nodes with a maximum of 2 multi-core processors each one so we can license Standard Edition instead of Enterprise Edition. 64bits addressing allow us to allocate many gigabytes of SGA. I&#8217;m starting with 5Gb but I think we&#8217;ll need more. And a set of 6&#215;300Gb 15krpms disks (it can be expanded with more disks and more shelves).<br />
This configuration keeps low the total cost of ownership but achieves best performance.<br />
Due to disks layout, costs and needed usable storage, we had to configure one huge RAID5 on the SAN with multi-path. I decided anyway to create 2 ASM disk groups (ASM is mandatory for Standard Edition RAC), one for the DB, the second one for the recovery area. With spare disks we should have enough availability and even if it&#8217;s a RAID5 I saw good write performances (&gt;150M/s).</p>
<p>Welcome new RAC, I hope we&#8217;ll feel good together!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/oracle-rac-standard-edition-to-achieve-low-cost-and-high-performance/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>System triggers, stats$user_log and side effects</title>
		<link>http://www.ludovicocaldara.net/dba/system-triggers-statsuser_log-and-side-effects/</link>
		<comments>http://www.ludovicocaldara.net/dba/system-triggers-statsuser_log-and-side-effects/#comments</comments>
		<pubDate>Thu, 09 Oct 2008 15:44:21 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Oracle Database]]></category>
		<category><![CDATA[Performance]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=22</guid>
		<description><![CDATA[Sometimes people get advice from internet: both Metalink or well-known consulting sites.
If people need a fix or a feature, they use to trust advices.
Last week I heard a collegue about a 10g RAC database with performance problems and, since I never lay on my chair, I probed both AWR and ADDM . I suddenly recognized [...]]]></description>
			<content:encoded><![CDATA[<p>Sometimes people get advice from internet: both Metalink or well-known consulting sites.<br />
If people need a fix or a feature, they use to trust advices.</p>
<p>Last week I heard a collegue about a 10g RAC database with performance problems and, since I never lay on my chair, I probed both AWR and ADDM . I suddenly recognized heavy enqueues and physical reads<br />
over a segment named STATS$USER_LOG. &#8220;Strange&#8221;, I said, &#8220;I cannot remember this name in neither perfstat or catalog segments&#8221;.<br />
Then I searched the Internet and the Metalink and found the same thing in BOTH metalink.oracle.com and www.dba-oracle.com: a trick to trace logon and logoffs into a table using system triggers.</p>
<p>Look at this code:</p>
<p><code>create or replace trigger<br />
logon_audit_trigger<br />
AFTER LOGON ON DATABASE<br />
BEGIN<br />
insert into stats$user_log values(<br />
user,<br />
sys_context('USERENV','SESSIONID'),<br />
sys_context('USERENV','HOST'),<br />
sysdate,<br />
to_char(sysdate, 'hh24:mi:ss'),<br />
[...]<br />
);<br />
COMMIT;<br />
END;<br />
/</code></p>
<p>Cool, every single access is kept into stats$user_log.</p>
<p>Let&#8217;s see the logoff trigger:</p>
<p><code>create or replace trigger<br />
logoff_audit_trigger<br />
BEFORE LOGOFF ON DATABASE<br />
BEGIN<br />
-- ***************************************************<br />
-- Update the last action accessed<br />
-- ***************************************************<br />
update stats$user_log [...]<br />
--***************************************************<br />
-- Update the last program accessed<br />
-- ***************************************************<br />
update stats$user_log [...]<br />
-- ***************************************************<br />
[ ... many, many updates ...]<br />
-- ***************************************************<br />
update stats$user_log [...]<br />
-- ***************************************************<br />
-- Compute the elapsed minutes<br />
-- ***************************************************<br />
update stats$user_log set elapsed_minutes =<br />
round((logoff_day - logon_day)*1440)<br />
where<br />
sys_context('USERENV','SESSIONID') = session_id;<br />
COMMIT;<br />
END;<br />
/<br />
</code><br />
That&#8217;s all. It inserts a row when someone logons. It updates MANY rows when someone logoffs.<br />
There is no match between the record inserted and the records updated (but the session_id).<br />
Neither indexes or constraints.</p>
<p>What&#8217;s the matter?</p>
<p>What happens if we have many logons?</p>
<p><code>SQL&gt; select num_rows from dba_tables where table_name='STATS$USER_LOG';</code></p>
<p>NUM_ROWS<br />
&#8212;&#8212;&#8212;-<br />
3053931</p>
<p>What happens if the execution plan does a full scan?</p>
<p>SQL&gt; explain plan for update stats$user_log [...]</p>
<p>Explained.</p>
<p><code>SQL&gt; @?/rdbms/admin/utlxpls<br />
---------------------------------------------<br />
| Id  | Operation          | Name           |<br />
---------------------------------------------<br />
|   0 | UPDATE STATEMENT   |                |<br />
|   1 |  UPDATE            | STATS$USER_LOG |<br />
|   2 |   TABLE ACCESS FULL| STATS$USER_LOG |<br />
---------------------------------------------</code></p>
<p>How many reads should it take?</p>
<p><code>SQL&gt; select bytes/1024/1024 Mb from dba_Segments where segment_name='STATS$USER_LOG';</code></p>
<p>MB<br />
&#8212;&#8212;&#8212;-<br />
237</p>
<p>The database performace will decrease constantly and very slowly&#8230;..<br />
Remember: never trust a solution if it involves a change on the system.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/system-triggers-statsuser_log-and-side-effects/feed/</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Plot Oracle historical statistics within SQL*Plus</title>
		<link>http://www.ludovicocaldara.net/dba/plot-oracle-historical-statistics-within-sqlplus/</link>
		<comments>http://www.ludovicocaldara.net/dba/plot-oracle-historical-statistics-within-sqlplus/#comments</comments>
		<pubDate>Wed, 24 Sep 2008 13:43:42 +0000</pubDate>
		<dc:creator>Ludovico</dc:creator>
				<category><![CDATA[Oracle Database]]></category>
		<category><![CDATA[Performance]]></category>
		<category><![CDATA[awr]]></category>
		<category><![CDATA[dba_hist_sysstat]]></category>
		<category><![CDATA[graph]]></category>
		<category><![CDATA[Oracle]]></category>
		<category><![CDATA[repository]]></category>
		<category><![CDATA[SQL*Plus]]></category>
		<category><![CDATA[workload]]></category>

		<guid isPermaLink="false">http://www.ludovicocaldara.net/dba/?p=10</guid>
		<description><![CDATA[More than often I&#8217;m asked to investigate &#8220;what happened yesterday when performance problems appeared&#8221;.
Sometimes I have the Enterprise Manager DB Console licensed, sometimes not. Sometimes I have direct SQL*Net access to the database that I may use to produce custom reports with my LAMP self-developed application. But it may happen that only an ssh access [...]]]></description>
			<content:encoded><![CDATA[<p>More than often I&#8217;m asked to investigate &#8220;what happened yesterday when performance problems appeared&#8221;.</p>
<p>Sometimes I have the Enterprise Manager DB Console licensed, sometimes not. Sometimes I have direct SQL*Net access to the database that I may use to produce custom reports with my LAMP self-developed application. But it may happen that only an ssh access is granted to the db server.</p>
<p>That&#8217;s why I started to develop some little scripts to plot the trends of database timed statistics.</p>
<p>Let&#8217;s see this one:<br />
<code><br />
SQL&gt; @sysstat.sql<br />
Enter a sysstat to search for: <strong>physical reads</strong></code><br />
<code><br />
STAT_ID     STAT_NAME<br />
----------- ------------------------------------------</code><br />
<strong>2263124246</strong><code> physical reads<br />
4171507801 physical reads cache<br />
297908839  physical reads cache prefetch<br />
2589616721 physical reads direct<br />
2564935310 physical reads direct (lob)<br />
2663793346 physical reads direct temporary tablespace<br />
473165409  physical reads for flashback new<br />
3102888545 physical reads prefetch warmup<br />
531193461  physical reads retry corrupt</code></p>
<p>9 rows selected.</p>
<p>Enter the desired stat_id: <strong>2263124246</strong><br />
<code>Enter the start date (YYYYMMDD) [defaults today] :</code><strong> 20080922</strong><br />
<code>Enter the end date date (YYYYMMDD) [defaults today] :</code><strong> 20080922</strong><br />
<code><br />
STAT_NAME        START    END<br />
---------------- -------- --------<br />
physical reads   20080922 20080922</code></p>
<p><code>BEGIN_INTERVAL_TIME           VALORE PLOTTED_VALUE<br />
------------------------- ---------- -------------------------<br />
22-SEP-08 12.00.12.122 AM          0<br />
22-SEP-08 01.00.28.253 AM     120092<br />
22-SEP-08 02.00.05.039 AM      35780<br />
22-SEP-08 03.00.55.595 AM       4792<br />
22-SEP-08 04.00.43.725 AM       4905<br />
22-SEP-08 05.00.31.855 AM       7300<br />
22-SEP-08 06.00.17.017 AM     234596<br />
22-SEP-08 07.00.08.132 AM      24651<br />
22-SEP-08 08.00.50.936 AM     481884<br />
22-SEP-08 09.00.33.488 AM     130201<br />
22-SEP-08 10.00.03.805 AM    1300306 **<br />
22-SEP-08 11.00.07.764 AM     491857<br />
22-SEP-08 12.00.31.548 PM     304702<br />
22-SEP-08 01.01.04.880 PM    1023664 *<br />
22-SEP-08 02.00.17.822 PM    8588180 ************<br />
22-SEP-08 03.00.36.969 PM    2201615 ***<br />
22-SEP-08 04.01.01.397 PM   17237098 *************************<br />
22-SEP-08 05.00.39.262 PM    1606300 **<br />
22-SEP-08 06.00.03.829 PM     451568<br />
22-SEP-08 07.00.31.461 PM     137684<br />
22-SEP-08 08.00.05.966 PM     203803<br />
22-SEP-08 09.00.24.829 PM     536394<br />
22-SEP-08 10.00.12.945 PM   10209783 **************<br />
22-SEP-08 11.00.35.123 PM    6151663 *********</code></p>
<p>24 rows selected.</p>
<p>Oh! At 4.00 PM we had a lot of physical reads. Nice.</p>
<p>This is the code:</p>

<div class="wp_syntax"><div class="code"><pre class="sql" style="font-family:monospace;"><span style="color: #808080; font-style: italic;">-- display given statistics from DBA_HIST_SYSSTAT</span>
col BEGIN_INTERVAL_TIME <span style="color: #993333; font-weight: bold;">FOR</span> a25
<span style="color: #993333; font-weight: bold;">SET</span> pages <span style="color: #cc66cc;">100</span> <span style="color: #993333; font-weight: bold;">LINES</span> <span style="color: #cc66cc;">130</span>
<span style="color: #993333; font-weight: bold;">SET</span> verify off term <span style="color: #993333; font-weight: bold;">ON</span>
&nbsp;
accept sysstat prompt <span style="color: #ff0000;">'Enter a sysstat to search for: '</span>
<span style="color: #993333; font-weight: bold;">SELECT</span> STAT_ID<span style="color: #66cc66;">,</span> STAT_NAME
  <span style="color: #993333; font-weight: bold;">FROM</span> DBA_HIST_STAT_NAME
   <span style="color: #993333; font-weight: bold;">WHERE</span> lower<span style="color: #66cc66;">&#40;</span>STAT_NAME<span style="color: #66cc66;">&#41;</span> <span style="color: #993333; font-weight: bold;">LIKE</span> lower<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'%&amp;amp;sysstat%'</span><span style="color: #66cc66;">&#41;</span>
  <span style="color: #993333; font-weight: bold;">ORDER</span> <span style="color: #993333; font-weight: bold;">BY</span> stat_name;
&nbsp;
accept stat_id prompt <span style="color: #ff0000;">'Enter the desired stat_id: '</span>
accept startdate prompt <span style="color: #ff0000;">'Start date (YYYYMMDD) [today] : '</span>
accept enddate prompt <span style="color: #ff0000;">'End date date (YYYYMMDD) [today] : '</span>
&nbsp;
<span style="color: #993333; font-weight: bold;">SELECT</span> STAT_NAME<span style="color: #66cc66;">,</span>
  nvl<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'&amp;amp;startdate'</span><span style="color: #66cc66;">,</span>to_char<span style="color: #66cc66;">&#40;</span>sysdate<span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #993333; font-weight: bold;">AS</span> <span style="color: #ff0000;">&quot;START&quot;</span><span style="color: #66cc66;">,</span>
  nvl<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'&amp;amp;enddate'</span><span style="color: #66cc66;">,</span>to_char<span style="color: #66cc66;">&#40;</span>sysdate<span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span> <span style="color: #993333; font-weight: bold;">AS</span> <span style="color: #ff0000;">&quot;END&quot;</span>
 <span style="color: #993333; font-weight: bold;">FROM</span> DBA_HIST_STAT_NAME
<span style="color: #993333; font-weight: bold;">WHERE</span> STAT_ID <span style="color: #66cc66;">=</span> &amp;amp;stat_id;
&nbsp;
<span style="color: #993333; font-weight: bold;">SELECT</span> BEGIN_INTERVAL_TIME<span style="color: #66cc66;">,</span> VALORE<span style="color: #66cc66;">,</span>
  substr<span style="color: #66cc66;">&#40;</span> rpad<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'*'</span><span style="color: #66cc66;">,</span><span style="color: #cc66cc;">40</span><span style="color: #66cc66;">*</span>round<span style="color: #66cc66;">&#40;</span> VALUE<span style="color: #66cc66;">/</span>max<span style="color: #66cc66;">&#40;</span>VALORE<span style="color: #66cc66;">&#41;</span>over<span style="color: #66cc66;">&#40;</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #cc66cc;">2</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #ff0000;">'*'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #cc66cc;">1</span><span style="color: #66cc66;">,</span><span style="color: #cc66cc;">40</span><span style="color: #66cc66;">&#41;</span> PLOTTED_VALORE
  <span style="color: #993333; font-weight: bold;">FROM</span> <span style="color: #66cc66;">&#40;</span>
  <span style="color: #993333; font-weight: bold;">SELECT</span> s<span style="color: #66cc66;">.</span>BEGIN_INTERVAL_TIME BEGIN_INTERVAL_TIME<span style="color: #66cc66;">,</span>
    nvl<span style="color: #66cc66;">&#40;</span>decode<span style="color: #66cc66;">&#40;</span>greatest<span style="color: #66cc66;">&#40;</span>VALUE<span style="color: #66cc66;">,</span> nvl<span style="color: #66cc66;">&#40;</span>lag<span style="color: #66cc66;">&#40;</span>VALUE<span style="color: #66cc66;">&#41;</span> over
      <span style="color: #66cc66;">&#40;</span>partition <span style="color: #993333; font-weight: bold;">BY</span> s<span style="color: #66cc66;">.</span>dbid<span style="color: #66cc66;">,</span> s<span style="color: #66cc66;">.</span>instance_number<span style="color: #66cc66;">,</span> g<span style="color: #66cc66;">.</span>stat_name <span style="color: #993333; font-weight: bold;">ORDER</span> <span style="color: #993333; font-weight: bold;">BY</span> s<span style="color: #66cc66;">.</span>snap_id<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #cc66cc;">0</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span>
    VALUE<span style="color: #66cc66;">,</span>
    VALUE <span style="color: #66cc66;">-</span> lag<span style="color: #66cc66;">&#40;</span>VALUE<span style="color: #66cc66;">&#41;</span> over
      <span style="color: #66cc66;">&#40;</span>partition <span style="color: #993333; font-weight: bold;">BY</span> s<span style="color: #66cc66;">.</span>dbid<span style="color: #66cc66;">,</span> s<span style="color: #66cc66;">.</span>instance_number<span style="color: #66cc66;">,</span> g<span style="color: #66cc66;">.</span>stat_name <span style="color: #993333; font-weight: bold;">ORDER</span> <span style="color: #993333; font-weight: bold;">BY</span> s<span style="color: #66cc66;">.</span>snap_id<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span>VALUE<span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span> <span style="color: #cc66cc;">0</span><span style="color: #66cc66;">&#41;</span> VALORE
  <span style="color: #993333; font-weight: bold;">FROM</span> DBA_HIST_SNAPSHOT s<span style="color: #66cc66;">,</span>
          DBA_HIST_SYSSTAT g<span style="color: #66cc66;">,</span>
          v$instance i
  <span style="color: #993333; font-weight: bold;">WHERE</span> s<span style="color: #66cc66;">.</span>SNAP_ID<span style="color: #66cc66;">=</span>g<span style="color: #66cc66;">.</span>SNAP_ID
  <span style="color: #993333; font-weight: bold;">AND</span> g<span style="color: #66cc66;">.</span>STAT_ID<span style="color: #66cc66;">=</span><span style="color: #ff0000;">'&amp;amp;stat_id'</span>
  <span style="color: #993333; font-weight: bold;">AND</span> s<span style="color: #66cc66;">.</span>BEGIN_INTERVAL_TIME &amp;gt;<span style="color: #66cc66;">=</span>
    trunc<span style="color: #66cc66;">&#40;</span>to_timestamp<span style="color: #66cc66;">&#40;</span>nvl<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'&amp;amp;startdate'</span><span style="color: #66cc66;">,</span>to_char<span style="color: #66cc66;">&#40;</span>sysdate<span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span>
  <span style="color: #993333; font-weight: bold;">AND</span> s<span style="color: #66cc66;">.</span>BEGIN_INTERVAL_TIME &amp;lt; <span style="color: #66cc66;">=</span>
   trunc<span style="color: #66cc66;">&#40;</span>to_timestamp<span style="color: #66cc66;">&#40;</span>nvl<span style="color: #66cc66;">&#40;</span><span style="color: #ff0000;">'&amp;amp;enddate'</span><span style="color: #66cc66;">,</span>to_char<span style="color: #66cc66;">&#40;</span>sysdate<span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">,</span><span style="color: #ff0000;">'YYYYMMDD'</span><span style="color: #66cc66;">&#41;</span><span style="color: #66cc66;">+</span><span style="color: #cc66cc;">1</span><span style="color: #66cc66;">&#41;</span>
  <span style="color: #993333; font-weight: bold;">AND</span> s<span style="color: #66cc66;">.</span>instance_number<span style="color: #66cc66;">=</span>i<span style="color: #66cc66;">.</span>instance_number
  <span style="color: #993333; font-weight: bold;">AND</span> s<span style="color: #66cc66;">.</span>instance_number<span style="color: #66cc66;">=</span>g<span style="color: #66cc66;">.</span>instance_number
  <span style="color: #993333; font-weight: bold;">ORDER</span> <span style="color: #993333; font-weight: bold;">BY</span> <span style="color: #cc66cc;">1</span>
<span style="color: #66cc66;">&#41;</span>;</pre></div></div>

<p>Ciao<br />
&#8211;<br />
Ludovico</p>
]]></content:encoded>
			<wfw:commentRss>http://www.ludovicocaldara.net/dba/plot-oracle-historical-statistics-within-sqlplus/feed/</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
	</channel>
</rss>
