Step by Step Single Node Cluster OpenAIS Suse 11 with Oracle11g & SAPInst

**Myself

OpenAIS Cluster Environment

Cluster information

host01 information :

-          hostname=host01

-          ip= 172.10.10.10

-          virtual IP=172.10.10.11

-          heartbeat ip=No

-          eth0=172.10.10.10 (data no heartbeat)

-          eth1=Not use ,

-          vgshare=qasdatavg

lv=lvol1,lvol2

-          Oracle sid = QAS [Mcod csd,csq,bcd,bcq,j2d,j2q,ecd,ecq]

-          SAP sid = j2d,j2q,trex_90,trex_92

-          disk on qasdatavg= md0/md1/md2/md3/md4

Create OpenAIS cluster  

1. Create a shared volume group.
2. Create a OpenAIS Cluster.
3. Create an application package.
4. Dependency or Constraints of Resource cluster startup.
5. Maintain cluster.

1.     Create a shared volume group

1.       Install multipath and configuration mpio/lvm	(SLES 9 only) Changing system configuration Using an editor of your choice, within /etc/sysconfig/hotplug set this value: HOTPLUG_USE_SUBFS=no On SLES: rpm -ivh multipath-tools-0.X.X-XX.X.i586.rpm On Red Hat: rpm -ivh device-mapper-multipath-0.X.X-XX.elX.x86_64.rpm - Load driver for the HBA is added to INITRD_MODULES in /etc/sysconfig/kernel INITRD_MODULES="cciss reiserfs piix megaraid_sas mptspi siimage processor thermal fan jbd ext3 dm_mod edd dm-multipath qla2xxx" - Ensure that boot.multipath and multipathd are set to start on boot: MPIO Services chkconfig boot.multipath on chkconfig multipathd on The services can be started immediately with:  /etc/init.d/boot.multipath start /etc/init.d/multipathd start To verify  chkconfig --list | grep multipath - Configuring multipath-tools  /etc/multipath.conf. “failback 60”  # incase san not stable Suse if the /etc/multipath.conf file does not exist, copy the example to create the file: cp /usr/share/doc/packages/multipathtools/multipath.conf.synthetic /etc/multipath.conf           Edit Backlist blacklist { devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st|sda)[0-9]*" devnode "^hd[a-z][0-9]*" devnode "^cciss!c[0-9]d[0-9].*" } - MD RAID Preconfiguration chkconfig boot.md off It is still possible to use local MD devices. These can be configured in the file /etc/mdadm.conf.localdevices, which uses the same syntax as the /etc/mdadm.conf. The cluster tools RPM package contains a new initscript called boot.md-localdevices. Copy this file to the /etc/init.d directory and enable it using the command chkconfig boot.md-localdevices on - Using LVM2 on top of the MPIO devices            modify/etc/lvm/lvm.conf filter = [ "a|/dev/sda[1-4]|", "a|/dev/md.*|", "r|/dev/.*|" ] - Reboot
2 Create md Configuration	Check DM multipath  multipath -l 600507680281816384000000000000dc dm-27 ABC,1234 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 3:0:0:4 sdf 8:80  active ready running | `- 4:0:0:4 sdp 8:240 active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 3:0:1:4 sdk 8:160 active ready running   `- 4:0:1:4 sdu 65:64 active ready running 3600507680281816384000000000000cc dm-26 ABC,1234 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 3:0:1:3 sdj 8:144 active ready running | `- 4:0:1:3 sdt 65:48 active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 3:0:0:3 sde 8:64  active ready running   `- 4:0:0:3 sdo 8:224 active ready running 3600507680281816384000000000000cb dm-25 ABC,1234 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 3:0:0:2 sdd 8:48  active ready running | `- 4:0:0:2 sdn 8:208 active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 3:0:1:2 sdi 8:128 active ready running   `- 4:0:1:2 sds 65:32 active ready running 3600507680281816384000000000000ca dm-23 ABC,1234 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 3:0:1:1 sdh 8:112 active ready running | `- 4:0:1:1 sdr 65:16 active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 3:0:0:1 sdc 8:32  active ready running   `- 4:0:0:1 sdm 8:192 active ready running 3600507680281816384000000000000c9 dm-24 ABC,1234 size=200G features='1 queue_if_no_path' hwhandler='0' wp=rw |-+- policy='round-robin 0' prio=50 status=active | |- 3:0:0:0 sdb 8:16  active ready running | `- 4:0:0:0 sdl 8:176 active ready running `-+- policy='round-robin 0' prio=10 status=enabled   |- 3:0:1:0 sdg 8:96  active ready running   `- 4:0:1:0 sdq 65:0  active ready runnin #Create MD arrays using mdadm. mdadm --create /dev/md0 --raid-devices=1 --level=0 --metadata=1.2 --force /dev/mapper/3600507680281816384000000000000c9 mdadm --create /dev/md1 --raid-devices=1 --level=0 --metadata=1.2 --force /dev/mapper/3600507680281816384000000000000ca mdadm --create /dev/md2 --raid-devices=1 --level=0 --metadata=1.2 --force /dev/mapper/3600507680281816384000000000000cb mdadm --create /dev/md3 --raid-devices=1 --level=0 --metadata=1.2 --force /dev/mapper/3600507680281816384000000000000cc mdadm --create /dev/md4 --raid-devices=1 --level=0 --metadata=1.2 --force /dev/mapper/3600507680281816384000000000000dc #Create md for reboot /clusterconf/QAS/mdadm.conf ARRAY /dev/md0 UUID=cb6443ea:7a85f171:3dfca667:723eb9c9 ARRAY /dev/md1 UUID=fd640129:153c4dd5:0a7b2596:44c1584f ARRAY /dev/md2 UUID=e8c6b9d2:14f71cee:f27dc725:c452ea1e ARRAY /dev/md3 UUID=8158487c:5ea824ee:fe01fca9:32dc5cbd ARRAY /dev/md4 UUID=2e152c68:2f72bfb8:4373b348:1164ddc6 #manually start and stop md devices like this: # to stop md scan mdadm --stop -scan # to verify stop md more /proc/mdstat mdadm --detail /dev/md0 for DEVICE in /dev/md0 /dev/md1 /dev/md2 /dev/md3 /dev/md4; do mdadm --assemble "${DEVICE}" --config=/clusterconf/QAS/mdadm.conf; done;   ## to scan md ->  mdadm --assemble --scan
3. create physical volumn	for i in 0 1 2 3 4 do pvcreate -f /dev/md$i done
4. create vg	vgcreate /dev/qasdatavg /dev/md0 /dev/md1 /dev/md2 /dev/md3 /dev/md4
5. create lv	lvcreate -L 20G -n lvorahomeQAS /dev/qasdatavg lvcreate -L 1G -n lvQASoriglogA /dev/qasdatavg lvcreate -L 1G -n lvQASoriglogB /dev/qasdatavg lvcreate -L 1G -n lvQASmirrlogA /dev/qasdatavg lvcreate -L 1G -n lvQASmirrlogB /dev/qasdatavg lvcreate -L 30G -n lvQASoraarch /dev/qasdatavg lvcreate -L 130G -n lvQASsapdata1 /dev/qasdatavg lvcreate -L 130G -n lvQASsapdata2 /dev/qasdatavg lvcreate -L 130G -n lvQASsapdata3 /dev/qasdatavg lvcreate -L 130G -n lvQASsapdata4 /dev/qasdatavg
6. create file system	mkfs.ext3  /dev/qasdatavg/lvorahomeQAS mkfs.ext3 -b 1024 /dev/qasdatavg/lvQASoriglogA mkfs.ext3 -b 1024 /dev/qasdatavg/lvQASoriglogB mkfs.ext3 -b 1024  /dev/qasdatavg/lvQASmirrlogA mkfs.ext3 -b 1024  /dev/qasdatavg/lvQASmirrlogB mkfs.ext3   /dev/qasdatavg/lvQASoraarch mkfs.ext3   /dev/qasdatavg/lvQASsapdata1 mkfs.ext3   /dev/qasdatavg/lvQASsapdata2 mkfs.ext3  /dev/qasdatavg/lvQASsapdata3 mkfs.ext3  /dev/qasdatavg/lvQASsapdata4   #verify blocksize dumpe2fs -h /dev/mapper/qasdatavg-lvQASmirrlogA |grep "Block size"   dumpe2fs -h /dev/mapper/qasdatavg-lvQASoraarch |grep "Block size"
7. create layout fs	mkdir -p /oracle/QAS mkdir -p /oracle/QAS/sapdata1 mkdir -p /oracle/QAS/sapdata2 mkdir -p /oracle/QAS/sapdata3 mkdir -p /oracle/QAS/sapdata4 mkdir -p /oracle/QAS/origlogA mkdir -p /oracle/QAS/origlogB mkdir -p /oracle/QAS/mirrlogA mkdir -p /oracle/QAS/mirrlogB mkdir -p /oracle/QAS/oraarch

2.    Create a OpenAIS Cluster

1. Create file authentication  key  2 node ( ทำ ที่ host01 )	host01# ssh-keygen -t rsa   # Copy from put to authorized cp -rp /root/ssh/id_rsa.pub /root/ssh/authorized_keys # Test know host ssh host01
2. configuration cluster information	Yast2->other->cluster   Generate       chkconfig csync2 on chkconfig xinetd on   # start OpenAIS rcopenais start
3. Set hacluster  password connection	host01# passwd hacluster password
4. Set stonith and quorum to single node	crm configure property stonith-enabled="false" crm configure property expected-quorum-votes="1" crm configure show   property $id="cib-bootstrap-options" \         dc-version="1.1.2-ecb1e2ea172ba2551f0bd763e557fccde68c849b" \         cluster-infrastructure="openais" \         expected-quorum-votes="1" \         stonith-enabled="false"     #restart rcopenais stop; rcopenais start
5. Start pacemaker configuration	Yast2->other->pacemaker --or-- hb_gui
6. Add Resource  virtual ip	Resources->Add->group   primitive       ADD   OK->Apply
7. Add Resource  disk resource [md disk]	Resources->Add->group     primitive       ADD       Name:Raidconf Value:/clusterconf/QAS/mdadm.conf   Name:raiddev Value:/dev/md0     Add  primitive md1,md2,md3,md4       OK->Apply   Cancel->apply
8. Add volume group and logical volume  resource [lvm]	Resources->Add->group   ora_vg_qasdb     primitive         ADD       Name:volgrpname Value:qasdatavg OK->Apply Cancel->apply   #### Add File system map logical volume Resources ora_vg_qasdb->Add->primitive   Device: /dev/qasdatavg/lvorahomeQAS Directory: /oracle/QAS Fstype: ext3   ##Add all lv/FS   ID=qas_lvQASoriglogA [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASoriglogA Directory: /oracle/QAS/origlogA Fstype: ext3   ID=qas_lvQASoriglogB [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASoriglogB Directory: /oracle/QAS/origlogB Fstype: ext3   ID=qas_lvQASmirrlogA [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASmirrlogA Directory: /oracle/QAS/mirrlogA Fstype: ext3   ID=qas_lvQASmirrlogB [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASmirrlogB Directory: /oracle/QAS/mirrlogB Fstype: ext3   ID=qas_lvQASoraarch [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASoraarch Directory /oracle/QAS/oraarch Fstype: ext3   ID=qas_lvQASsapdata1 [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASsapdata1 Directory /oracle/QAS/sapdata1 Fstype: ext3   ID=qas_lvQASsapdata2 [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASsapdata2 Directory /oracle/QAS/sapdata2 Fstype: ext3   ID=qas_lvQASsapdata3 [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASsapdata3 Directory /oracle/QAS/sapdata3 Fstype: ext3   ID=qas_lvQASsapdata4 [ocf,heartbeat,Filesystem] Device: /dev/qasdatavg/lvQASsapdata4 Directory /oracle/QAS/sapdata4 Fstype: ext3
9. Add logical volume group   resource [lvm]	ora_vg_qasdb     primitive           ADD       Name:volgrpname Value:qasdatavg OK->Apply Cancel->apply

3.    Create an application package.

1. Add primitive oracle listener  ora_list_vhost1	Resources-> ora_list_vhost1 ->Edit-> primitive->add     ID: listener_crmqasdb Class: ocf Provider:heartbeat Type: oralsnr     SID=QAS   Add Name=listener Value=LISTENER   Name=home Value=/oracle/QAS/112_64   OK   #Start listener resource host01:~ # crm resource start  ora_list_crmqasdb
2. Create New Group Oracle Instant [QAS]	Resources-> Add->group->   ora_inst_QAS_crmqasdb     primitive       ora_QAS_crmqasdb   ADD       Sid:QAS Home:/oracle/QAS/112_64 User:oraqas Shutdown_method:immediate   OK->Apply Cancel->apply   #Start oracle instant resource host01:~ # crm resource start  ora_inst_QAS_crmqasdb
3. [Option] Create New Group SAP CI/ACS [J2D and TREX] Instant	Note 995116 - Backward porting of sapstartsrv for earlier releases Manual postprocessing in Release 6.40 for non-Windows platforms 1. Set the "SAPSYSTEM" profile parameter in the start profile in accordance with the value in the relevant instance profile for each instance. 2. The Unicode version of sapstartsrv requires the shared library librfcum. In Release 6.40, this was originally not included in all replication lists (for example, scs.lst and scsclient.lst). After you import Support Package 20, these lists are updated and exist in librfcum.  If you want to activate sapstartsrv only by applying the relevant kernel patch or importing sapexe.sar, you can manually copy librfcum until you import Support Package 20. 3. Set up the autostart of sapstartsrv in accordance with Note 823941. You can do this by once installing sapinit on each application server (the required scripts are available in the attachment to this note) and enter the start of sapstartsrv in /usr/sap/sapservices for each instance. During the following OS boot, the start services are started. You can also start the services by executing /usr/sap/sapservices. All entered sapstartsrv are executed as root and started for all systems and instances. The start services are executed as <sid>adm and started for the relevant system. You can also use "sapcontrol -nr <instance no.> -function StartService <SID>" to manually start individual services. For Release 6.40, we currently do not intend to provide an autostart of sapstartsrv using startsap (as in Release 700). Note 823941 - SAP start service on Unix platforms Summary Symptom After an SAP upgrade, the SAP Start Service (sapstartsrv) on UNIX platforms is not automatically started when the operating system is started. Other terms sapstartsrv, sapinit, installsapinit.sh, configure sapstartsrv as a boot service, sapstartsrv as an SYSV service. Solution In order to configure the SAP Start Service (sapstartsrv) as boot service, you must make some manual changes after the upgrade. First, you must copy the 'sapinit' script to the correct location. This takes place automatically by using the 'installsapinit.sh' script. In the case of IBM i, you can skip this step.            Log on to the operating system as user 'root'.            Execute the following command in the 'DIR_PUT/bin' directory (or /usr/sap/jupgrade/exe in the case of a Java upgrade): ./installsapinit.sh            The 'sapinit' script was now copied to the correct location and the runlevel directories updated accordingly, so that sapinit is called automatically when the operating system is restarted and starts sapstartsrv.                       If sapinit/installsapinit.sh does not exist, download the attached sapinit.tgz (or sapinit.sar). The archive            contains both scripts. Extract the archive to a temporary            directory and proceed as described above.            Create a file called 'sapservices' in the '/usr/sap/' directory. The file should contain the corresponding commands to be able to start the SAP Start Service (sapstartsrv) for each installed SAP instance on the server.            Since 'sapservices' is read by 'sapinit', the file permissions should be set to 0755 (rwxr-xr-x).            For IBM i, you must set the owner to QSECOFR and the file permissions to 0644 (-rw-r--r--). With an ABAP-only installation, the file should look as follows:                     #!/bin/sh                     LD_LIBRARY_PATH=<DIR_EXECUTABLE>:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;<DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<INSTANCE_START_profile> -D -u <sid>adm                     # if exists:                     LD_LIBRARY_PATH=<DIR_EXECUTABLE>/:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;<DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<ASCS_START_profile> -D -u <sid>adm With a J2EE-only installation, the file should look as follows:                     #!/bin/sh                     LD_LIBRARY_PATH=<DIR_EXECUTABLE>:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;<DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<INSTANCE_START_PROFILE> -D -u <sid>adm                     LD_LIBRARY_PATH=<DIR_EXECUTABLE>:$LD_LIBRARY_PATH;export LD_LIBRARY_PATH;<DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<SCS_START_profile> -D -u <sid>adm                     Replace <DIR_EXECUTABLE> and <DIR_PROFILE> with the value of SAP profile parameters DIR_EXECUTABLE or DIR_PROFILE, and replace the profile files with the relevant file names.                     This syntax is applicable to Linux and Solaris/SunOS operating systems. On AIX, replace 'LD_LIBRARY_PATH' with 'LIBPATH', for HP-UX replace 'LD_LIBRARY_PATH' with 'SHLIB_PATH'.                                         For IBM i, the following settings are sufficient: In the case of an ABAP-only installation:                     < DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<INSTANCE_START_profile> -D -u <sid>adm                     If available:                     < DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<ASCS_START_profile> -D -u <sid>adm In the case of J2EE-only or dual-stack installation:                     < DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<INSTANCE_START_PROFILE> -D -u <sid>adm                     < DIR_EXECUTABLE>/sapstartsrv pf=<DIR_PROFILE>/<SCS_START_profile> -D -u <sid>adm                     < DIR_EXECUTABLE> and <DIR_PROFILE> must be replaced accordingly.                                         After that, sapstartsrv  is fully functional as start service. Comments:                     The entry for starting an SAP start service in /usr/sap/sapservices must be made in a single text line without other line breaks.                     Instead of making the entry yourself manually in /usr/sap/sapservices, you can also use the sapstartsrv as of Kernel Release 7.01 by executing "sapstartsrv pf=<INSTANCE_START_PROFILE> -reg".                     As of Release 7.10, start profiles are no longer used in general.  The start entries are in the instance profile instead. In this case, you must specify the relevant instance profile instead of the start profile. However, the start profiles may still be active depending on the upgrade history and upgrade options that were used. Proceed as follows to determine which profile is to be specified.  If there are entries in the instance profile "Start_Program" or "Restart_Program", specify the instance profile. Otherwise, specify the start profile.Example for /usr/sap/sapservices: ABAP-only (Linux/Solaris-Syntax):            #!/bin/sh            LD_LIBRARY_PATH=/usr/sap/LNX/DVEBMGS00/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH;/usr/sap/LNX/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/LNX/SYS/profile/START_DVEBMGS00_lnxhost -D -u lnxadm Java-only (Linux/Solaris-Syntax):            #!/bin/sh            LD_LIBRARY_PATH=/usr/sap/LNX/JC00/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH; /usr/sap/LNX/JC00/exe/sapstartsrv pf=/usr/sap/LNX/SYS/profile/START_JC00_lnxhost -D -u lnxadm            LD_LIBRARY_PATH=/usr/sap/LNX/SCS01/exe:$LD_LIBRARY_PATH; export LD_LIBRARY_PATH;/usr/sap/LNX/SCS01/exe/sapstartsrv pf=/usr/sap/LNX/SYS/profile/START_SCS01_lnxhost -D -u lnxadm Examples for /usr/sap/sapservices for IBM i: ABAP-only:            /usr/sap/DB4/DVEBMGS00/exe/sapstartsrv pf=/usr/sap/DB4/SYS/profile/START_DVEBMGS00_db4host -D -u db4adm Java-only or dual-stack:            /usr/sap/DB4/JC00/exe/sapstartsrv pf=/usr/sap/DB4/SYS/profile/START_JC00_db4host -D -u db4adm            /usr/sap/DB4/SCS01/exe/sapstartsrv pf=/usr/sap/DB4/SYS/profile/START_SCS01_db4host -D -u db4adm     #Need to download sapstartsrv/ sapcontrol  #Need download sapinit and installsapinit.sh to /tmp #From Root user 1)       chmod 777 installsapinit.sh 2)       ./installsapinit.sh   #create file sapservices in the '/usr/sap/'   3)         #!/bin/sh #!/bin/sh LD_LIBRARY_PATH=/usr/sap/J2D/SYS/exe/run; export LD_LIBRARY_PATH; /usr/sap/J2D/SYS/exe/run/sapstartsrv pf=/usr/sap/J2D/SYS/profile/S TART_SCS71_vhost1 -D -u j2dadm     LD_LIBRARY_PATH=/usr/sap/J2D/SYS/exe/run; export LD_LIBRARY_PATH; /usr/sap/J2D/SYS/exe/run/sapstartsrv pf=/usr/sap/J2D/SYS/profile/S TART_JC80_vhost1 -D -u j2dadm   4)       chmod 755 sapservices #Extract from download or copy from CSD sapstartsrv/sapcontrol /librfccm.so 5)       cp /sapmnt/CSD/exe/sapstartsrv  /sapmnt/CSD/exe/sapcontrol /sapmnt/CSD/exe/librfcum.so /sapmnt/J2D/exe 6)       chown j2dadm:sapsys /sapmnt/CSD/exe/sapstartsrv  /sapmnt/CSD/exe/sapcontrol /sapmnt/CSD/exe/librfcum.so               Resources-> Add->group->   sap_inst_CI_QAS_crmqasdb       primitive   ### ADD  SCS of J2D   sap_inst_SCS_J2D InstanceName J2D_SCS71_vhost1 START_PROFILE  /sapmnt/J2D/profile/START_SCS71_vhost1     ### ADD J2D and TREX For CI of J2D and script start TREX   sap_inst_CI_J2D_TREX         ADD     InstanceName     J2D_JC80_vhost1 PRE_START_USEREXIT   /clusterconf/QAS/script/startTREX90J2D.sh   START_PROFILE  /sapmnt/J2D/profile/START_JC80_vhost1   PRE_STOP_USEREXIT  /clusterconf/QAS/script/stopTREX90J2D.sh     AUTOMATIC_RECOVER     true           Remove monitoring resource Set start and stop timeout=480     OK->Apply Cancel->apply   #Start SAP J2D instant resource crm resource start  sap_inst_CI_QAS_crmqasdb   !!Disable sap_inst_J2D_TREX_crmqasdb resource group

4.    Dependency or Constraints of Resource cluster startup.

!!Disable Resource SAP Instant and start manual

Order Constraints

1. Add Order Constraints MD disk to VG->LV->FS	Constraints-> add resource order   Disk_fs ora_data_qasdb to ora_vg_qasdb
2. Add Order Constraints Oracle Instant	Constraints-> add resource order   Oracle_QAS_Service ora_vg_qasdb to ora_inst_QAS_crmqasdb
3. Add Order Constraints Oracle Listener Service	Constraints-> add resource order   Oracle_Listener_Service ora_vg_qasdb to ora_list_crmqasdb
4. Add Order Constraints SAP J2D/TREX Service	Constraints-> add resource order     sap_J2D_TREX_List_service   ora_list_crmqasdb to sap_inst_J2D_TREX_crmqasdb       and Constraints-> add resource order sap_J2D_TREX_Ora_service   ora_list_crmqasdb to sap_inst_J2D_TREX_crmqasdb   Summary constraint

5.    Maintain cluster.

Normal Status : crm_mon

Maintain

Detail	Command
Start cluster	rcopenais start
Start SAP after cluster all resource on	su - j2dadm startsap r3 vhost1 exit   su – trexadm TREX start exit
Stop cluster	rcopenais stop
Stop SAP	su - j2dadm stopsap r3 vhost1 exit   su – trexadm TREX stop exit
Start resource [qas]	# DM [Disk raid] crm resource start  ora_data_qasdb # FS/LV/VG crm resource start  qas_qasdatavg   # Oracle instant [QAS] crm resource start  ora_inst_QAS_crmqasdb   # Virtual IP and Oracle listener [Disk raid] crm resource start  ora_list_crmqasdb   # resource disable SAP  [J2D,TREX90] crm resource start  sap_inst_J2D_TREX_crmqasdb
Stop resource [qas]	# resource disable SAP  [J2D,TREX90] crm resource stop  sap_inst_J2D_TREX_crmqasdb # Virtual IP and Oracle listener [Disk raid] crm resource stop  ora_list_crmqasdb # Oracle instant [QAS] crm resource stop  ora_inst_QAS_crmqasdb # FS/LV/VG crm resource stop  qas_qasdatavg   # DM [Disk raid] crm resource stop  ora_data_qasdb
Cleanup resource  [qas] # Case change status not save/un commit/abnormal	# FS/LV/VG crm resource cleanup  qas_qasdatavg   # DM [Disk raid] crm resource cleanup  ora_data_qasdb # Virtual IP and Oracle listener [Disk raid] crm resource cleanup  ora_list_crmqasdb   # Oracle instant [QAS] crm resource cleanup  ora_inst_QAS_crmqasdb   # resource disable SAP  [J2D,TREX90] crm resource cleanup  sap_inst_J2D_TREX_crmqasdb

Cluster Commands

Useful crm commands :

• cibadmin -Q : List the xml configuration of the cluster


• cibadmin -Q > cib-backup.xml : Save the configuration of the cluster in the given file


• cibadmin -E –force : Delete the complete cluster information (immediately)

If you want to reset the whole configuration you should also delete /var/lib/heartbeat/crm/* on all nodes

• crm_mon : Displays the cluster state of hosts and resources

• crm_verifiy -LV : Check the xml for errors or orphaned resources

• crm_resource -M -r resource_virtual_ip -H hacluster1 : Migrate the resource resource_irtual_ip to hacluster1

Script lib agent path

/usr/lib/ocf/resource.d/heartbeat/