{"id":881,"date":"2015-03-23T23:58:29","date_gmt":"2015-03-23T21:58:29","guid":{"rendered":"http:\/\/www.ludovicocaldara.net\/dba\/?p=881"},"modified":"2020-08-18T16:36:50","modified_gmt":"2020-08-18T14:36:50","slug":"convert-interconnect-to-haip","status":"publish","type":"post","link":"https:\/\/www.ludovicocaldara.net\/dba\/convert-interconnect-to-haip\/","title":{"rendered":"Moving Clusterware Interconnect from single NIC\/Bond to HAIP"},"content":{"rendered":"

Very recently I had to configure a customer’s RAC private interconnect\u00a0from bonding to HAIP to get benefit of both NICs.<\/p>\n

So I would like to recap here what the hypothetic steps would be if you need to do the same.<\/p>\n

In this example I’ll switch from a single-NIC interconnect (eth1) rather than from a bond configuration, so if you are familiar with the RAC Attack!<\/a> environment you can try to put everything in place on your own.<\/p>\n

First, you need to plan the new network configuration in advance, keeping in mind that there are a couple of important restrictions:<\/p>\n

    \n
  1. Your interconnect interface naming must be uniform on all nodes in the cluster. The interconnect uses the interface name in its configuration and it doesn’t support different names on different hosts<\/li>\n
  2. You must bind the different private interconnect interfaces in different subnets (see Note: 1481481.1 – 11gR2 CSS Terminates\/Node Eviction After Unplugging one Network Cable in Redundant Interconnect Environment<\/strong> if you need an explanation)<\/li>\n<\/ol>\n

     <\/p>\n

    Implementation\u00a0<\/strong><\/p>\n

    The RAC Attack book uses one interface per node for the interconnect (eth1, using network 172.16.100.0)<\/p>\n

    To make things a little more complex, we’ll not use the eth1 in the new HAIP configuration,\u00a0so we’ll test also the deletion of the old interface.<\/p>\n

    What you need to do is add two new interfaces (host only in your virtualbox) and configure them as eth2 and eth3, e.g. in networks 172.16.101.0 and 172.16.102.0)<\/p>\n

    eth2      Link encap:Ethernet  HWaddr 08:00:27:32:76:DD\r\n          inet addr:172.16.101.51  Bcast:172.16.101.255  Mask:255.255.255.0\r\n          inet6 addr: fe80::a00:27ff:fe32:76dd\/64 Scope:Link\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1\r\n          RX packets:29 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:25 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:1000\r\n          RX bytes:2044 (1.9 KiB)  TX bytes:1714 (1.6 KiB)\r\n\r\neth3      Link encap:Ethernet  HWaddr 08:00:27:2E:05:4B\r\n          inet addr:172.16.102.61  Bcast:172.16.102.255  Mask:255.255.255.0\r\n          inet6 addr: fe80::a00:27ff:fe2e:54b\/64 Scope:Link\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1\r\n          RX packets:19 errors:0 dropped:0 overruns:0 frame:0\r\n          TX packets:12 errors:0 dropped:0 overruns:0 carrier:0\r\n          collisions:0 txqueuelen:1000\r\n          RX bytes:1140 (1.1 KiB)  TX bytes:720 (720.0 b)\r\n<\/pre>\n
     <\/p>\n
    modify \/var\/named\/racattack in order to use the new addresses (RAC doesn’t care about logical names, it’s just for our convenience):<\/p>\n
    collabn1 A 192.168.78.51\r\ncollabn1-vip A 192.168.78.61\r\ncollabn1-priv A 172.16.100.51\r\ncollabn1-priv1 A 172.16.101.51\r\ncollabn1-priv2 A 172.16.102.61\r\ncollabn2 A 192.168.78.52\r\ncollabn2-vip A 192.168.78.62\r\ncollabn2-priv A 172.16.100.52\r\ncollabn2-priv1 A 172.16.101.52\r\ncollabn2-priv2 A 172.16.102.62<\/pre>\n
    add also the reverse lookup in \u00a0in-addr.arpa:<\/p>\n
    51.101.16.172 PTR collabn1-priv1.racattack.\r\n52.102.16.172 PTR collabn1-priv2.racattack.\r\n61.101.16.172 PTR collabn2-priv1.racattack.\r\n62.102.16.172 PTR collabn2-priv2.racattack.<\/pre>\n
     <\/p>\n
    restart \u00a0named on the first node and check that both nodes can ping all the names correctly:<\/p>\n
    [root@collabn1 named]# ping collabn2-priv1\r\nPING collabn2-priv1.racattack (172.16.101.52) 56(84) bytes of data.\r\n64 bytes from 172.16.101.52: icmp_seq=1 ttl=64 time=1.27 ms\r\n64 bytes from 172.16.101.52: icmp_seq=2 ttl=64 time=0.396 ms\r\n^C\r\n--- collabn2-priv1.racattack ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1293ms\r\nrtt min\/avg\/max\/mdev = 0.396\/0.835\/1.275\/0.440 ms\r\n[root@collabn1 named]# ping collabn2-priv2\r\nPING collabn2-priv2.racattack (172.16.102.62) 56(84) bytes of data.\r\n64 bytes from 172.16.102.62: icmp_seq=1 ttl=64 time=0.924 ms\r\n64 bytes from 172.16.102.62: icmp_seq=2 ttl=64 time=0.251 ms\r\n^C\r\n--- collabn2-priv2.racattack ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1480ms\r\nrtt min\/avg\/max\/mdev = 0.251\/0.587\/0.924\/0.337 ms\r\n[root@collabn1 named]# ping collabn1-priv2\r\nPING collabn1-priv2.racattack (172.16.102.61) 56(84) bytes of data.\r\n64 bytes from 172.16.102.61: icmp_seq=1 ttl=64 time=0.019 ms\r\n64 bytes from 172.16.102.61: icmp_seq=2 ttl=64 time=0.032 ms\r\n^C\r\n--- collabn1-priv2.racattack ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1240ms\r\nrtt min\/avg\/max\/mdev = 0.019\/0.025\/0.032\/0.008 ms\r\n[root@collabn1 named]# ping collabn1-priv1\r\nPING collabn1-priv1.racattack (172.16.101.51) 56(84) bytes of data.\r\n64 bytes from 172.16.101.51: icmp_seq=1 ttl=64 time=0.017 ms\r\n64 bytes from 172.16.101.51: icmp_seq=2 ttl=64 time=0.060 ms\r\n^C\r\n--- collabn1-priv1.racattack ping statistics ---\r\n2 packets transmitted, 2 received, 0% packet loss, time 1224ms\r\nrtt min\/avg\/max\/mdev = 0.017\/0.038\/0.060\/0.022 ms<\/pre>\n
    check the nodes that compose the cluster:<\/p>\n
    [root@collabn1 network-scripts]# olsnodes -s\r\ncollabn1 Active\r\ncollabn2 Active<\/pre>\n
    on all nodes, make a copy of the gpnp profile.xml (just in case, the oifcfg tool does the copy automatically)<\/p>\n
    $ cd $GRID_HOME\/gpnp\/`hostname`\/profiles\/peer\/\r\n$ cp -p profile.xml profile.xml.bk<\/pre>\n
    List the available networks:<\/p>\n
    [root@collabn1 bin]# .\/oifcfg iflist -p -n\r\neth0 192.168.78.0 PRIVATE 255.255.255.0\r\neth1 172.16.100.0 PRIVATE 255.255.255.0\r\neth1 169.254.0.0 UNKNOWN 255.255.0.0\r\neth2 172.16.101.0 PRIVATE 255.255.255.0\r\neth3 172.16.102.0 PRIVATE 255.255.255.0<\/pre>\n
    Get the current ip configuration for the interconnect:<\/p>\n
    [root@collabn1 bin]# .\/oifcfg getif\r\neth0 192.168.78.0 global public\r\neth1 172.16.100.0 global cluster_interconnect<\/pre>\n
    one one node only, set the new interconnect interfaces:<\/p>\n
    [root@collabn1 network-scripts]# oifcfg setif -global eth2\/172.16.101.0:cluster_interconnect\r\n[root@collabn1 network-scripts]# oifcfg setif -global eth3\/172.16.102.0:cluster_interconnect\r\n[root@collabn1 network-scripts]# oifcfg getif\r\neth0 192.168.78.0 global public\r\neth1 172.16.100.0 global cluster_interconnect\r\neth2 172.16.101.0 global cluster_interconnect\r\neth3 172.16.102.0 global cluster_interconnect<\/pre>\n
    check that the other nodes has received the new configuration:<\/p>\n
    [root@collabn2 bin]# .\/oifcfg getif\r\neth0 192.168.78.0 global public\r\neth1 172.16.100.0 global cluster_interconnect\r\neth2 172.16.101.0 global cluster_interconnect\r\neth3 172.16.102.0 global cluster_interconnect<\/pre>\n
    Before deleting the old interface, it would be sensible to stop your cluster resources (in some cases, one of the nodes may be evicted), in any case the cluster must be restarted completely in order to get the new interfaces working.<\/p>\n
    Note:\u00a0<\/strong>having three interfaces in a HAIP interconnect is perfectly working, HAIP works from 2 to 4 interfaces. I’m showing how to delete\u00a0eth1\u00a0just for information!! \ud83d\ude42<\/p>\n
    [root@collabn1 network-scripts]# oifcfg delif -global eth1\/172.16.100.0\r\n[root@collabn1 network-scripts]# oifcfg getif\r\neth0 192.168.78.0 global public\r\neth2 172.16.101.0 global cluster_interconnect\r\neth3 172.16.102.0 global cluster_interconnect<\/pre>\n
    on all nodes, shutdown the CRS:<\/p>\n
    [root@collabn1 network-scripts]# crsctl stop crs\r\nCRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'collabn1'\r\n...<\/pre>\n
    Now you can disable the old interface:<\/p>\n
    [root@collabn1 network-scripts]# ifdown eth1<\/pre>\n
    and modify the parameter ONBOOT=no inside the configuration script of eth1 interface.<\/p>\n
    Start the cluster again:<\/p>\n
    [root@collabn1 network-scripts]# crsctl start crs<\/pre>\n
    And check that the resources are up & running:<\/p>\n
    # crscst stat res -t\r\n--------------------------------------------------------------------------------\r\nNAME TARGET STATE SERVER STATE_DETAILS\r\n--------------------------------------------------------------------------------\r\nLocal Resources\r\n--------------------------------------------------------------------------------\r\nora.DATA.dg\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.LISTENER.lsnr\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.asm\r\nONLINE ONLINE collabn1 Started\r\nONLINE ONLINE collabn2 Started\r\nora.gsd\r\nOFFLINE OFFLINE collabn1\r\nOFFLINE OFFLINE collabn2\r\nora.net1.network\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.ons\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\n--------------------------------------------------------------------------------\r\nCluster Resources\r\n--------------------------------------------------------------------------------\r\nora.LISTENER_SCAN1.lsnr\r\n1 ONLINE ONLINE collabn2\r\nora.LISTENER_SCAN2.lsnr\r\n1 ONLINE ONLINE collabn1\r\nora.LISTENER_SCAN3.lsnr\r\n1 ONLINE ONLINE collabn1\r\nora.collabn1.vip\r\n1 ONLINE ONLINE collabn1\r\nora.collabn2.vip\r\n1 ONLINE ONLINE collabn2\r\nora.cvu\r\n1 ONLINE ONLINE collabn1\r\nora.oc4j\r\n1 ONLINE ONLINE collabn1\r\nora.orcl.db\r\n1 ONLINE ONLINE collabn1 Open\r\n2 ONLINE ONLINE collabn2 Open\r\nora.scan1.vip\r\n1 ONLINE ONLINE collabn2\r\nora.scan2.vip\r\n1 ONLINE ONLINE collabn1\r\nora.scan3.vip\r\n1 ONLINE ONLINE collabn1<\/pre>\n
     <\/p>\n
    \u00a0Testing the high availability<\/strong><\/p>\n
    Disconnect cable from one of the two interfaces (virtually if you’re in virtualbox \ud83d\ude42 )<\/p>\n
    Pay attention at the NO-CARRIER status (in eth2 in this example):<\/p>\n
    # ip l\r\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN\r\nlink\/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\r\n2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000\r\nlink\/ether 08:00:27:07:33:94 brd ff:ff:ff:ff:ff:ff\r\n3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000\r\nlink\/ether 08:00:27:7f:b4:88 brd ff:ff:ff:ff:ff:ff\r\n4: eth2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000\r\nlink\/ether 08:00:27:51:1d:78 brd ff:ff:ff:ff:ff:ff\r\n5: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000\r\nlink\/ether 08:00:27:39:86:f2 brd ff:ff:ff:ff:ff:ff<\/pre>\n
    check that the CRS is still up & running:<\/p>\n
    # crsctl stat res -t\r\n--------------------------------------------------------------------------------\r\nNAME TARGET STATE SERVER STATE_DETAILS\r\n--------------------------------------------------------------------------------\r\nLocal Resources\r\n--------------------------------------------------------------------------------\r\nora.DATA.dg\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.LISTENER.lsnr\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.asm\r\nONLINE ONLINE collabn1 Started\r\nONLINE ONLINE collabn2 Started\r\nora.gsd\r\nOFFLINE OFFLINE collabn1\r\nOFFLINE OFFLINE collabn2\r\nora.net1.network\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\nora.ons\r\nONLINE ONLINE collabn1\r\nONLINE ONLINE collabn2\r\n--------------------------------------------------------------------------------\r\nCluster Resources\r\n--------------------------------------------------------------------------------\r\nora.LISTENER_SCAN1.lsnr\r\n1 ONLINE ONLINE collabn2\r\nora.LISTENER_SCAN2.lsnr\r\n1 ONLINE ONLINE collabn1\r\nora.LISTENER_SCAN3.lsnr\r\n1 ONLINE ONLINE collabn1\r\nora.collabn1.vip\r\n1 ONLINE ONLINE collabn1\r\nora.collabn2.vip\r\n1 ONLINE ONLINE collabn2\r\nora.cvu\r\n1 ONLINE ONLINE collabn1\r\nora.oc4j\r\n1 ONLINE ONLINE collabn1\r\nora.orcl.db\r\n1 ONLINE ONLINE collabn1 Open\r\n2 ONLINE ONLINE collabn2 Open\r\nora.scan1.vip\r\n1 ONLINE ONLINE collabn2\r\nora.scan2.vip\r\n1 ONLINE ONLINE collabn1\r\nora.scan3.vip\r\n1 ONLINE ONLINE collabn1<\/pre>\n
     <\/p>\n
    The virtual interface eth2:1 as failed over on the second interface as eth3:2<\/p>\n
    eth3:1    Link encap:Ethernet  HWaddr 08:00:27:39:86:F2\r\n          inet addr:169.254.185.134  Bcast:169.254.255.255  Mask:255.255.128.0\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1\r\n\r\neth3:2    Link encap:Ethernet  HWaddr 08:00:27:39:86:F2\r\n          inet addr:169.254.104.52  Bcast:169.254.127.255  Mask:255.255.128.0\r\n          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1\r\n<\/pre>\n
     <\/p>\n
    After the cable is reconnected, the virtual interface is back on eth2:<\/p>\n
    eth2:1 Link encap:Ethernet HWaddr 08:00:27:51:1D:78\r\ninet addr:169.254.104.52 Bcast:169.254.127.255 Mask:255.255.128.0\r\nUP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<\/pre>\n
     <\/p>\n
    Further information<\/strong><\/p>\n
    For this post I’ve used a RAC version 11.2, but RAC 12c use the very same procedure.<\/p>\n
    You can discover more here about HAIP:<\/p>\n
    http:\/\/docs.oracle.com\/cd\/E11882_01\/server.112\/e10803\/config_cw.htm#HABPT5279\u00a0<\/a><\/p>\n
    And here about how to set it (beside this post!):<\/p>\n
    https:\/\/docs.oracle.com\/cd\/E11882_01\/rac.112\/e41959\/admin.htm#CWADD90980<\/a><\/p>\n
    https:\/\/docs.oracle.com\/cd\/E11882_01\/rac.112\/e41959\/oifcfg.htm#BCGGEFEI<\/a><\/p>\n
     <\/p>\n
    Cheers<\/p>\n
    —<\/p>\n
    Ludo<\/p>\n","protected":false},"excerpt":{"rendered":"
    Very recently I had to configure a customer’s RAC private interconnect\u00a0from bonding to HAIP to get benefit of both NICs. So I would like to recap here what the hypothetic steps would be if you need to do the same. … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[327,326,3,330,149,132],"tags":[19,22,95,23,103,104],"class_list":["post-881","post","type-post","status-publish","format-standard","hentry","category-oracle-maa","category-oracle","category-oracledb","category-oracle-inst-upg","category-oracle-rac","category-triblog","tag-cluster","tag-oracle-database","tag-oracle-rac","tag-rac","tag-rac-attack","tag-racattack"],"aioseo_notices":[],"_links":{"self":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/881","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/comments?post=881"}],"version-history":[{"count":5,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/881\/revisions"}],"predecessor-version":[{"id":950,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/posts\/881\/revisions\/950"}],"wp:attachment":[{"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/media?parent=881"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/categories?post=881"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ludovicocaldara.net\/dba\/wp-json\/wp\/v2\/tags?post=881"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}