{"id":56,"date":"2009-03-02T19:08:31","date_gmt":"2009-03-02T17:08:31","guid":{"rendered":"http:\/\/www.ludovicocaldara.net\/dba\/?p=56"},"modified":"2020-08-18T17:00:38","modified_gmt":"2020-08-18T15:00:38","slug":"how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool","status":"publish","type":"post","link":"https:\/\/www.ludovicocaldara.net\/dba\/how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool\/","title":{"rendered":"How to collect Oracle Application Server performance data with DMS and RRDtool"},"content":{"rendered":"<p><strong>RRDize everything, chapter 1<\/strong><\/p>\n<p>If you are managing some Application Server deployments you should have wondered how to check and collect performance data.<br \/>\nAs stated in documentation, you can gather performance metrics with the dmstool utility.<br \/>\nAFAIK, this can be done from 9.0.2 release upwards, but i&#8217;m concerned DMS will not work on Weblogic.<\/p>\n<p>Mainly, you should have an external server that acts as collector (it could be a server in the Oracle AS farm as well): copy the dms.jar library from an Oracle AS installation to your collector and use it as you would use dmstool:<\/p>\n<pre lang=\"sh\">java -jar dms.jar [dmstool options]<\/pre>\n<p>There are three basilar methods to get data:<\/p>\n<p>Get all metrics at once:<\/p>\n<pre lang=\"sh\">java -jar dms.jar -dump -a \"youraddress:\/\/...\" [format=xml]<\/pre>\n<p>Get only the interesting metrics:<\/p>\n<pre lang=\"sh\">java -jar dms.jar -a \"youraddress:\/\/...\" metric metric ...<\/pre>\n<p>Get metrics included into specific DMS tables:<\/p>\n<pre lang=\"sh\">java -jar dms.jar -a \"youraddress:\/\/...\" -table table table ...<\/pre>\n<p>What youraddress:\/\/ is, it depends on the component you are trying to connect:<\/p>\n<pre lang=\"sh\">opmn:\/\/asserver:6003\r\nhttp:\/\/asserver:7200\/dms0\/Spy\r\najp13:\/\/asserver:3301\/dmsoc4j\/Spy<\/pre>\n<p>If you are trying to connect to the OHS (Apache), be careful to allow remote access from the collector by editing the dms.conf file.<\/p>\n<p>Now that you can query dms data, you should store it somewhere.<br \/>\nPersonally, I did a first attempt with dmstool -dump format=xml. I wrote a parser in PHP with SimpleXML extension and I did a lot of inserts into a MySQL database. After a few months the whole data collected from tens of servers was too much to be mantained&#8230;<br \/>\nTo avoid the maintenance of a DWH-grade database I investigated and found RRDTool. Now I&#8217;m asking how could I live without it!<\/p>\n<p>I then wrote a parser in awk that parse the output of the dms.jar call and invoke an rrdtool update command.<br \/>\nI always use dms.jar -table command. The output has always the same format:<\/p>\n<pre>###SOF\r\n\r\nMon Mar 02 17:01:19 CET 2009\r\n\r\n---------------\r\nTABLE1_Name\r\n---------------\r\n\r\nrecord1_metric1.name:     value       units\r\nrecord1_metric2.name:     value       units\r\n....\r\n\r\nrecord2_metric1.name:     value       units\r\nrecord2_metric2.name:     value       units\r\n....\r\n\r\n---\r\nTABLE2_Name\r\n---\r\n\r\nrecord1_metric1.name:     value       units\r\nrecord1_metric2.name:     value       units\r\n....\r\n\r\nrecord2_metric1.name:     value       units\r\nrecord2_metric2.name:     value       units\r\n....\r\n\r\n##EOF<\/pre>\n<p>So I written an awk file that works for me.<br \/>\nuse it this way:<\/p>\n<pre lang=\"sh\"> java -jar dms.jar ... | awk -f parse_output.awk<\/pre>\n<pre lang=\"c\">####################\r\n# parse_output.awk #\r\n####################\r\n\r\n#function pl() replaces all non alphanumeric occurrences with an underscore\r\nfunction pl(input) {\r\n        return gensub(\"[^[:alnum:]_-]\",\"_\",\"G\",input);\r\n}\r\n\r\n# function get_rrd_path() returns a path where the rrd files should be placed\r\n# I should rewrite a new path for each dms table... I'll skip many of them\r\nfunction get_rrd_path() {\r\n        if (table == \"mod_oc4j_destination_metrics\")\r\n                return sprintf(\"%s\/%s\/%s\/%s.rrd\", record[\"Host\"],\r\n                    pl(table), pl(record[\"Name.value\"]), pl(var) );\r\n        if (table == \"mod_oc4j_mount_pt_metrics\")\r\n                return sprintf(\"%s\/%s\/%s\/%s\/%s.rrd\", record[\"Host\"],\r\n                    pl(table), pl(record[\"Destination.value\"]), pl(record[\"Name.value\"]), pl(var) );\r\n        if (table == \"ohs_server\")\r\n                return sprintf(\"%s\/%s\/%s.rrd\", record[\"Host\"], pl(table), pl(var) );\r\n        if (table == \"JVM\")\r\n                return sprintf(\"%s\/%s\/%s\/%s.rrd\", record[\"Host\"],\r\n                    pl(table), pl(record[\"Process\"]), pl(var) );\r\n        if (table == \"opmn_process\")\r\n                return sprintf(\"%s\/%s\/%s\/%s\/%s\/%s\/%s\/%s.rrd\", record[\"Host\"], pl(table),\r\n                  pl(record[\"iasInstance.value\"]), pl(record[\"opmn_ias_component\"]),\r\n                  pl(record[\"opmn_process_type\"]),pl(record[\"opmn_process_set\"]),\r\n                  pl(record[\"Name\"]), pl(var) );\r\n\r\n        return sprintf(\"%s\/%s\/%s.rrd\", record[\"Host\"], pl(table), pl(var) );\r\n}\r\n# function process_record actually does the dirty work of invoking the update script\r\nfunction process_record() {\r\n        #every record has a timeStamp.ts metric that I should use to update my rrd\r\n        ts=substr(record[\"timeStamp.ts\"],0,10);\r\n        for ( var in record ) {\r\n        if ( var != \"timeStamp.ts\" &amp;&amp; record[var] ~ \/^[[:digit:]]+$\/ ) {\r\n            if ( var ~ \/\\.(count|completed|time)$\/ ) {\r\n                dstype=\"DERIVE\";\r\n            } else {\r\n                if ( var == \"responseSize.value\" ) {\r\n                    dstype=\"DERIVE\";\r\n                } else {\r\n                    dstype=\"GAUGE\";\r\n                }\r\n            }\r\n            rrdFile=sprintf(\"\/path_to_data\/%s\",get_rrd_path());\r\n            #### update_metric_rrd is a shell script listed below!!!!!\r\n            cmd=sprintf(\"\/path_to_scripts\/update_metric_rrd %s %s %d %d\",\r\n                rrdFile,dstype,ts,record[var]);\r\n            system(cmd);\r\n            }\r\n        }\r\n}\r\n\r\n# parse_record() populates an hash array\r\n# with all metrics belonging to the table record\r\nfunction parse_record() {\r\n    #print \"RRRR -  START OF RECORD (table \" table \")\"\r\n    delete record\r\n    while ( ! \/^$\/ ) {\r\n        # I'm parsing the record as far I'm in this while statement\r\n        # the array hash is the name of the dms metric basename.\r\n        # $1 is the metric name but I have to trim the final \":\"\r\n        key=substr($1,0,length($1)-1)\r\n        record[key]=$2\r\n        getline\r\n    }\r\n    # this function is included in funcions.awk:\r\n    # I invoke it to process the record I've just parsed\r\n    process_record();\r\n}\r\nBEGIN {\r\n    # as far as started is 0, I've never reached the first table\r\n    started=0\r\n}\r\n\r\n#MAIN\r\n{\r\n    # I jump over the first lines until I reach the first table\r\n    if (started==0) {\r\n        while ( ! \/^---\/ ) {\r\n           getline\r\n        }\r\n        started=1\r\n    }\r\n\r\n    # looking for the next occurrence of a table\r\n    # all tables start with:\r\n    # ----------\r\n    # table_name\r\n    # ----------\r\n    if ( \/^---\/ ) {\r\n        # first table reached: the next row is my table name,\r\n        # then I reach again a dashed line -----\r\n        getline table\r\n        getline trash\r\n        #print \"\"\r\n        #print \"##########################\"\r\n        print \"  TABELLA \" table\r\n        #print \"##########################\"\r\n        next\r\n    }\r\n\r\n    if ( ! \/^$\/ ) {\r\n        # reached an empty line: could be the end of a record or the and of a table\r\n        # since a new table is threated in previous \"if\" statement, I'm starting a new record.\r\n        parse_record()\r\n    }\r\n\r\n}\r\n\r\nEND {\r\n}<\/pre>\n<p>And this is the code for update_metric_rrd:<\/p>\n<pre lang=\"bash\">#!\/bin\/bash\r\nRRDFILE=$1\r\nDSTYPE=$2\r\nTS=$3\r\nVALUE=$4\r\n\r\nrrdtool update $RRDFILE ${TS}:${VALUE}\r\n\r\nif [ $? -ne 0 ] ; then\r\n        DIR=`dirname $RRDFILE`\r\n\r\n        [ -d $DIR ] || mkdir -p $DIR\r\n        [ -f $RRDFILE ] || rrdtool create $RRDFILE -b \"now-1month\" -s 1800 \\\r\n                DS:metric:${DSTYPE}:7200:0:U \\\r\n                RRA:AVERAGE:0.5:1:672 \\\r\n                RRA:AVERAGE:0.5:4:1080 \\\r\n                RRA:AVERAGE:0.5:12:1460 \\\r\n                RRA:AVERAGE:0.5:48:1095 \\\r\n                RRA:MAX:0.5:4:1080 \\\r\n                RRA:MAX:0.5:12:1460 \\\r\n                RRA:MAX:0.5:48:1095 \\\r\n                RRA:LAST:0.5:1:672\r\n        rrdtool update $RRDFILE ${TS}:${VALUE}\r\nfi<\/pre>\n<p>Once you have all your rrd files populated, it&#8217;s easy to script automatic reporting. You would probably want a graph with the request count served by your Apache cluster, along with its linear regression:<\/p>\n<pre lang=\"bash\">rrdtool graph - -s \"end-${hours}hours\" -e $end \\\r\n                -v \"Requests Completed\/sec\" \\\r\n        -w 640 -h 240 --slope-mode \\\r\n                -t \"HTTP Requests for www.ludovicocaldara.net\" \\\r\n                DEF:1request_completed=\/data\/wwwserver1\/ohs_server\/request_completed.rrd:metric:AVERAGE \\\r\n                DEF:2request_completed=\/data\/wwwserver2\/ohs_server\/request_completed.rrd:metric:AVERAGE \\\r\n                CDEF:request_completed=1request_completed,2request_completed,+ \\\r\n                VDEF:slope=request_completed,LSLSLOPE \\\r\n                VDEF:lslint=request_completed,LSLINT \\\r\n                CDEF:reg=request_completed,POP,slope,COUNT,*,lslint,+ \\\r\n                LINE1:reg#666666:\"Regression\" \\\r\n                AREA:1request_completed#4040AA:\"wwwserver1\"  \\\r\n                AREA:2request_completed#6666FF:\"wwwserver1\":STACK  \\\r\n        &gt; mygraph.png<\/pre>\n<p>This is the result:<br \/>\n<img decoding=\"async\" src=\"http:\/\/www.ludovicocaldara.net\/images\/dba\/dms_ohs_requests.png\" alt=\"OHS request completed\" \/><br \/>\n<strong>OHHHHHHHHHHHH!!!! COOL!!!!<\/strong><\/p>\n<p>That&#8217;s all for DMS capacity planning. Stay tuned, more about rrdtool is coming!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>RRDize everything, chapter 1 If you are managing some Application Server deployments you should have wondered how to check and collect performance data. As stated in documentation, you can gather performance metrics with the dmstool utility. AFAIK, this can be &hellip; <a href=\"https:\/\/www.ludovicocaldara.net\/dba\/how-to-collect-oracle-application-server-performance-data-with-dms-and-rrdtool\/\">Continue reading <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5,326,6,331,332],"tags":[30,34,286,35],"class_list":["post-56","post","type-post","status-publish","format-standard","hentry","category-linux","category-oracle","category-perf","category-web","category-weblogic","tag-capacity-planning","tag-oracle-application-server","tag-perf","tag-rrdtool"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/56","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/comments?post=56"}],"version-history":[{"count":11,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/56\/revisions"}],"predecessor-version":[{"id":63,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/56\/revisions\/63"}],"wp:attachment":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/media?parent=56"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/categories?post=56"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/tags?post=56"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}