Oracle Grid Infrastructure 18c patching part 2: Independent Local-mode Automaton architecture and activation

The first important step before starting using the new Independent Local-mode Automaton is understanding which are its components inside a cluster.

Resources

Here’s the list of service that you will find when you install a Grid Infrastructure 18c:

# [ oracle@server1:/u01/app/oracle/home [15:14:41] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.MGMT.GHCHKPT.advm
               OFFLINE OFFLINE      server1                STABLE
               OFFLINE OFFLINE      server2                STABLE
ora.MGMT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.OCRVOT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.chad
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.helper
               OFFLINE OFFLINE      server1                IDLE,STABLE
               OFFLINE OFFLINE      server2                STABLE
ora.mgmt.ghchkpt.acfs
               OFFLINE OFFLINE      server1                STABLE
               OFFLINE OFFLINE      server2                STABLE
ora.net1.network
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.ons
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.proxy_advm
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       server2                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       server1                STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       server1                169.254.17.12 10.30.
                                                             200.73,STABLE
ora.asm
      1        ONLINE  ONLINE       server1                Started,STABLE
      2        ONLINE  ONLINE       server2                Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       server1                STABLE
ora.server1.vip
      1        ONLINE  ONLINE       server1                STABLE
ora.server2.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       server1                Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       server1                STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       server1                STABLE
--------------------------------------------------------------------------------

As you can see, there are 4 components that are OFFLINE by default:

Three local resources (that are present on each node):

  • ora.MGMT.GHCHKPT.advm
  • ora.mgmt.ghchkpt.acfs
  • ora.helper

One cluster resource (active on only one server at a time, it can relocate):

  • ora.rhpserver

If you have ever worked with 12c Rapid Home Provisioning, those name should sound familiar.

The GHCHKPT filesystem (ant its relative volume), is used to store some data regarding the ongoing operations across the cluster during the GI home move.

The ora.helper is the process that actually does the operations. It is local because each node needs it to execute some actions at some point.

The rhpserver is the server process that coordinates the operations and delegates them to the helpers.

All those services compose the independent local-mode automaton, that is the default deployment. The full RHP framework (RHP Server and RHP Client) might be configured instead with some additional work.

Important note: Just a few weeks ago Oracle changed the name of Rapid Home Provisioning (RHP) to Fleet Patching and Provisioning (FPP). The name is definitely more appealing now, but it generates again some confusion about product names and acronyms, so beware that in this series sometimes I refer to RHP, sometimes to FPP, but actually it is the same thing.

Tomcat?

You might have noticed that tomcat is deployed now in the GI home, as there are patches specific to it (here I paste the 18.4 version):

$ opatch lspatches
28655963;DBWLM RELEASE UPDATE 18.4.0.0.0 (28655963)
28655784;Database Release Update : 18.4.0.0.181016 (28655784)
28655916;ACFS RELEASE UPDATE 18.4.0.0.0 (28655916)
28656071;OCW RELEASE UPDATE 18.4.0.0.0 (28656071)
28547619;TOMCAT RELEASE UPDATE 18.0.0.0.0 (28547619)
27908644;UPDATE 18.3 DATABASE CLIENT JDK IN ORACLE HOME TO JDK8U171
27923415;OJVM RELEASE UPDATE: 18.3.0.0.180717 (27923415)

 

Indeed Tomcat is registered in the inventory and patched just like any other product inside the OH:

<COMP NAME="oracle.tomcat.crs" VER="18.0.0.0.0" BUILD_NUMBER="0" BUILD_TIME="20180207.193003" REP_VER="0.0.0.0.0" RELEASE="Production" INV_LOC="Components/oracle.tomcat.crs/18.0.0.0.0/1/" LANGS="ALL_LANGS" XML_INV_LOC="Components21/oracle.tomcat.crs/18.0.0.0.0/" ACT_INST_VER="12.2.0.4.0" DEINST_VER="11.2.0.0.0" INSTALL_TIME="2018.Nov.05 13:27:32 CET" INST_LOC="/u01/app/grid/crs1840/tomcat">
   <EXT_NAME>Tomcat Container</EXT_NAME>
   <DESC>Packages files from the Tomcat Container.</DESC>
   <DESCID>COMPONENT_DESC</DESCID>
   <STG_INFO OSP_VER="10.2.0.0.0"/>
   <CMP_JAR_INFO>
      <INFO NAME="filemapObj" VAL="Components/oracle/tomcat/crs/v18_0_0_0_0/filemap.xml"/>
      <INFO NAME="helpDir" VAL="Components/oracle/tomcat/crs/v18_0_0_0_0/help/"/>
      <INFO NAME="actionsClass" VAL="Components.oracle.tomcat.crs.v18_0_0_0_0.CompActions"/>
      <INFO NAME="resourceClass" VAL="Components.oracle.tomcat.crs.v18_0_0_0_0.resources.CompRes"/>
      <INFO NAME="identifiersXML" VAL="Components/oracle/tomcat/crs/v18_0_0_0_0/identifiers.xml"/>
      <INFO NAME="contextClass" VAL="Components.oracle.tomcat.crs.v18_0_0_0_0.CompContext"/>
      <INFO NAME="fastCopyLogXML" VAL="Components/oracle/tomcat/crs/v18_0_0_0_0/fastCopyLog.xml"/>
   </CMP_JAR_INFO>
   <LOC_INFO INST_DFN_LOC="../Scripts" JAR_NAME="install1.jar"/>
   <BOOK NAME="oracle.tomcat.crs.hs"/>
   <PRE_REQ DEF="F"/>
   <PROD_HOME DEF="F"/>
   <LANG_IDX_MAP>
      <LANG LIST="en fr ar bn pt_BR bg fr_CA ca hr cs da nl ar_EG en_GB et fi de el iw hu is in it ja ko es lv lt ms es_MX no pl pt ro ru zh_CN sk sl es_ES sv th zh_TW tr uk vi"/>
      <LANGSET IDX="1" BITSET="{0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44}"/>
   </LANG_IDX_MAP>
   <PLAT_IDX_MAP>
      <PLAT LIST="46"/>
      <PLATSET IDX="1" BITSET="{0}"/>
   </PLAT_IDX_MAP>
   <DST_IDX_MAP>
      <DST LIST="%ORACLE_HOME% %INVENTORY_LOCATION%"/>
   </DST_IDX_MAP>
   <DEP_GRP_LIST/>
   <DEP_LIST/>
   <REF_LIST>
      <REF NAME="oracle.crs" VER="18.0.0.0.0" HOME_IDX="3"/>
   </REF_LIST>
   <INST_TYPE_LIST>
      <INST_TYPE NAME="Complete" NAME_ID="Maximum" DESC_ID=""/>
   </INST_TYPE_LIST>
   <FILESIZEINFO>
      <DEST VOLUME="%ORACLE_HOME%" SPACE_REQ="3375301"/>
      <DEST VOLUME="%INVENTORY_LOCATION%" SPACE_REQ="2000"/>
   </FILESIZEINFO>
</COMP>

 

# [ oracle@server2:/u01/app/grid/crs1830/inventory/Components21/oracle.tomcat.crs/18.0.0.0.0 [08:56:06] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ vi context.xml

<?xml version="1.0" standalone="yes" ?>
<!-- Copyright (c) 1999, 2018, Oracle and/or its affiliates.
All rights reserved. -->
<!-- Do not modify the contents of this file by hand. -->
<COMP_CONTEXT>
   <VAR_LIST SIZE="0">
      <VAR NAME="PROD_HOME" TYPE="String" DESC_RES_ID="" SECURE="F" VAL="/u01/app/grid/crs1830/tomcat" ADV="F" CLONABLE="T" USER_INPUT="DEFAULT"/>
   </VAR_LIST>
   <CONST_LIST SIZE="2">
      <CONST NAME="COMPONENT_DESC" PLAT_SP="F" TYPE="String" TRANS="T" VAL="COMPONENT_DESC_ALL"/>
      <CONST NAME="COMPONENT_NAME" PLAT_SP="F" TYPE="String" TRANS="F" VAL="Tomcat Container"/>
   </CONST_LIST>
</COMP_CONTEXT>

Out of the box, Tomcat is used for the Quality of Services Management (ora.qosmserver resource):

$ ps -eaf | grep tomcat
oracle    58746 142151  0 13:10 pts/1    00:00:00 grep --color=auto tomcat
oracle   108610      1  0 Dec04 ?        00:25:33 /CRS/dbs01/crs1830/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/ORA/dbs01/oracle/crsdata/itrac1602/qos/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=itrac1602.cern.ch -Doracle.http.port=8888 -Doracle.jmx.port=23792 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=false -Djava.security.manager -Djava.security.policy=/ORA/dbs01/oracle/crsdata/itrac1602/qos/conf/catalina.policy -Djava.security.egd=file:/dev/urandom -Dcatalina.home=/CRS/dbs01/crs1840/tomcat -Dcatalina.base=/ORA/dbs01/oracle/crsdata/itrac1602/qos -Djava.io.tmpdir=/ORA/dbs01/oracle/crsdata/itrac1602/qos/temp -Doracle.home=/CRS/dbs01/crs1840 -classpath /CRS/dbs01/crs1840/tomcat/lib/tomcat-juli.jar:/CRS/dbs01/crs1840/tomcat/lib/bootstrap.jar:/CRS/dbs01/crs1840/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

But it is used for the Independent Local Mode Automaton as well, when it is started.

Enabling and starting the independent local-mode automaton

The resources are started using the following commands (as root, the order is quite important):

# /u01/app/grid/crs1830/bin/srvctl enable volume -volume GHCHKPT  -diskgroup mgmt
# /u01/app/grid/crs1830/bin/srvctl enable filesystem -volume GHCHKPT -diskgroup mgmt
# /u01/app/grid/crs1830/bin/srvctl start filesystem -volume GHCHKPT -diskgroup mgmt

Before continuing with the rhpserver resource, you might want to check if the filesystem is mounted:

$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.MGMT.GHCHKPT.advm
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.MGMT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.OCRVOT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.chad
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.helper
               OFFLINE OFFLINE      server1                IDLE,STABLE
               OFFLINE OFFLINE      server2                STABLE
ora.mgmt.ghchkpt.acfs
               ONLINE  ONLINE       server1                mounted on /opt/orac
                                                             le/rhp_images/chkbas
                                                             e,STABLE
               ONLINE  ONLINE       server2                mounted on /opt/orac
                                                             le/rhp_images/chkbas
                                                             e,STABLE
ora.net1.network
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.ons
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.proxy_advm
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       server2                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       server1                STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       server1                169.254.17.12 10.30.
                                                             200.73,STABLE
ora.asm
      1        ONLINE  ONLINE       server1                Started,STABLE
      2        ONLINE  ONLINE       server2                Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       server1                STABLE
ora.server1.vip
      1        ONLINE  ONLINE       server1                STABLE
ora.server2.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       server1                Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       server1                STABLE
ora.rhpserver
      1        OFFLINE OFFLINE                               STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       server1                STABLE
--------------------------------------------------------------------------------


[root@server2 dbs01]# df -k | grep ghchkpt
/dev/asm/ghchkpt-213              1572864   499572   1073292  32% /opt/oracle/rhp_images/chkbase

Now the rhpserver should start without problems, as oracle:

# [ oracle@server1:/u01/app/oracle/home [17:00:49] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ srvctl start rhpserver

Please note that if you omit to activate the filesystem first, the rhpserver will fail to start.

As you can see, now both rhpserver and the helper are online:

# [ oracle@server1:/u01/app/oracle/home [17:00:49] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ srvctl start rhpserver

# [ oracle@server1:/u01/app/oracle/home [17:02:39] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.ASMNET1LSNR_ASM.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.MGMT.GHCHKPT.advm
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.MGMT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.OCRVOT.dg
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.chad
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.helper
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.mgmt.ghchkpt.acfs
               ONLINE  ONLINE       server1                mounted on /opt/orac
                                                             le/rhp_images/chkbas
                                                             e,STABLE
               ONLINE  ONLINE       server2                mounted on /opt/orac
                                                             le/rhp_images/chkbas
                                                             e,STABLE
ora.net1.network
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.ons
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
ora.proxy_advm
               ONLINE  ONLINE       server1                STABLE
               ONLINE  ONLINE       server2                STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       server2                STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       server1                STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       server1                169.254.17.12 10.30.
                                                             200.73,STABLE
ora.asm
      1        ONLINE  ONLINE       server1                Started,STABLE
      2        ONLINE  ONLINE       server2                Started,STABLE
      3        OFFLINE OFFLINE                               STABLE
ora.cvu
      1        ONLINE  ONLINE       server1                STABLE
ora.server1.vip
      1        ONLINE  ONLINE       server1                STABLE
ora.server2.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       server1                Open,STABLE
ora.qosmserver
      1        ONLINE  ONLINE       server1                STABLE
ora.rhpserver
      1        ONLINE  ONLINE       server2                STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       server2                STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       server1                STABLE
--------------------------------------------------------------------------------
# [ oracle@server2:/u01/app/grid/crs1830 [08:59:43] [18.3.0.0.0 [GRID] SID=GRID] 0 ] #
$ ps -eaf | grep tomca
oracle   132330      1 15 08:48 ?        00:01:39 /u01/app/grid/crs1830/jdk/bin/java -server -Xms128M -Xmx384M -Djava.awt.headless=true -Ddisable.checkForUpdate=true -Djava.util.logging.config.file=/u01/app/oracle/crsdata/server2/rhp/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -DTRACING.ENABLED=false -Djava.rmi.server.hostname=server2.cern.ch -Doracle.http.port=8894 -Doracle.jmx.port=23795 -Doracle.tls.enabled=false -Doracle.jwc.tls.http.enabled=true -Doracle.rhp.storagebase=/opt/oracle/rhp_images -Djava.security.egd=file:/dev/urandom -Dcatalina.home=/u01/app/grid/crs1830/tomcat -Dcatalina.base=/u01/app/oracle/crsdata/server2/rhp -Djava.io.tmpdir=/u01/app/oracle/crsdata/server2/rhp/temp -Doracle.home=/u01/app/grid/crs1830 -classpath /u01/app/grid/crs1830/tomcat/lib/tomcat-juli.jar:/u01/app/grid/crs1830/tomcat/lib/bootstrap.jar:/u01/app/grid/crs1830/jlib/jwc-logging.jar org.apache.catalina.startup.Bootstrap start

Now all is set to start using it!

We’ll see how to use it in the next posts.

Ludo

 

The following two tabs change content below.

Ludovico

Oracle ACE Director and Computing Engineer at CERN
Ludovico is an Oracle ACE Director, frequent speaker and community contributor, working as Computing Engineer at CERN, the European Organization for Nuclear Research, in Switzerland.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.