Oracle RAC 19c Patching on Linux – Complete Step-by-Step Guide

A complete production-ready SOP for patching Oracle RAC 19c on Linux. Covers both rolling and non-rolling patch methods, Grid Infrastructure patching, ASM patching, database patching, datapatch execution, and full post-patch validation across all nodes — with real commands, expected outputs, and consultant-level notes for both standard OFA and enterprise custom path conventions.

1. Document Info

2. RAC Patching — Concepts You Must Know First

📝 Why is RAC patching more complex than standalone? In standalone patching you shut down one database, patch one Oracle Home, run datapatch, done. In RAC you have multiple Oracle Homes (one per node), two types of software to patch (Grid Infrastructure AND Database), and a choice between rolling and non-rolling methods. Getting the sequence wrong can bring down the entire cluster.

Key RAC Patching Concepts

RAC Patching — Which Home Gets Patched in What Order

⚠️ IMPORTANT: Sequence of patching in RAC must ALWAYS follow this order. Never patch DB home before GI home on the same node.

For each node (one at a time for rolling):
  Step 1 → Patch Grid Infrastructure Home (GI Home) first
  Step 2 → Patch Database Oracle Home second
  Step 3 → Run datapatch (only once — after ALL nodes are patched)

Rolling vs Non-Rolling — When to Use Which

📝 Always check the patch README first. The README explicitly states whether rolling patching is supported for that specific patch.

3. Path Conventions

📝 All examples use Convention A. Substitute Convention B paths where applicable.

4. Pre-Patching Checks — Run on All Nodes

⚠️ IMPORTANT: Every pre-check must be verified on ALL nodes before starting any patching activity. A healthy cluster before patching means a smooth patching experience. Never start patching if the cluster has any resource in a non-online state.

4.1 — Confirm Current Version and Patch Level on All Nodes

📝 Why? You must know the exact current patch level on ALL nodes before deciding which patch to apply. In a RAC environment, all nodes must be on the same patch level before you start. If nodes are on different patch levels, fix that first.

-- Connect from any node
sqlplus / as sysdba

set linesize 200
set pagesize 100
col inst_id       for 999
col instance_name for a15
col host_name     for a20
col version_full  for a20
col status        for a12

-- Shows version on all nodes simultaneously
SELECT inst_id, instance_name, host_name,
       version, version_full, status
FROM   gv$instance
ORDER BY inst_id;

# Check GI patch level on ALL nodes
su - grid

# Node 1
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Grid|Patch"

# Node 2 — run same command via SSH
ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory | grep -E 'Oracle Grid|Patch'"

# Check DB home patch level on ALL nodes
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch"
ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory | grep -E 'Oracle Database|Patch'"

4.2 — Check Cluster Health — All Resources Must Be Online

📝 Why? Patching a cluster that already has resources in a degraded state is extremely risky. Fix any cluster issues BEFORE patching. Never patch a cluster in a partially degraded state.

su - grid

# Full cluster status — all nodes
$ORACLE_HOME/bin/crsctl check cluster -all

# All CRS resources — look for OFFLINE or INTERMEDIATE
$ORACLE_HOME/bin/crsctl stat res -t

# Specifically grep for any non-online resources
$ORACLE_HOME/bin/crsctl stat res -t | grep -E "OFFLINE|INTERMEDIATE|UNKNOWN"

# Check database status on all nodes
$ORACLE_HOME/bin/srvctl status database -db RACDB

# Check ASM status on all nodes
$ORACLE_HOME/bin/srvctl status asm

# Check listener status on all nodes
$ORACLE_HOME/bin/srvctl status listener

# Check SCAN listener
$ORACLE_HOME/bin/srvctl status scan_listener

# Check voting disks — all must be ONLINE
$ORACLE_HOME/bin/crsctl query css votedisk

⚠️ IMPORTANT: If ANY resource shows OFFLINE that should be online — stop. Fix the cluster issue first and get the cluster fully healthy before attempting any patching.

4.3 — Verify OPatch Version on All Nodes and Upgrade if Needed

📝 Why? OPatch version must meet the minimum required by the patch on ALL nodes. Both GI home and DB home have their own OPatch. Both must be at the correct version.

# Check OPatch version in GI Home — ALL nodes
su - grid

# Node 1 — GI OPatch version
$ORACLE_HOME/OPatch/opatch version

# Node 2 — GI OPatch version
ssh racnode2 "/u01/app/19.3.0/grid/OPatch/opatch version"

# Check OPatch version in DB Home — ALL nodes
su - oracle

# Node 1 — DB OPatch version
$ORACLE_HOME/OPatch/opatch version

# Node 2 — DB OPatch version
ssh racnode2 "/u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatch version"

If OPatch needs upgrading — do this on ALL nodes, for BOTH GI and DB homes:

# Upgrade OPatch in GI Home on ALL nodes
su - grid

# Node 1 — GI OPatch upgrade
mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_backup_$(date +%Y%m%d)
unzip -q /stage/patches/p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch version

# Node 2 — GI OPatch upgrade (via SSH or directly on node2)
ssh racnode2 "mv /u01/app/19.3.0/grid/OPatch /u01/app/19.3.0/grid/OPatch_backup_$(date +%Y%m%d)"
scp -r /u01/app/19.3.0/grid/OPatch racnode2:/u01/app/19.3.0/grid/OPatch
ssh racnode2 "/u01/app/19.3.0/grid/OPatch/opatch version"

# Upgrade OPatch in DB Home on ALL nodes
su - oracle

# Node 1 — DB OPatch upgrade
mv $ORACLE_HOME/OPatch $ORACLE_HOME/OPatch_backup_$(date +%Y%m%d)
unzip -q /stage/patches/p6880880_190000_Linux-x86-64.zip -d $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch version

# Node 2 — DB OPatch upgrade
ssh racnode2 "mv /u01/app/oracle/product/19.3.0/dbhome_1/OPatch /u01/app/oracle/product/19.3.0/dbhome_1/OPatch_backup_$(date +%Y%m%d)"
scp -r $ORACLE_HOME/OPatch racnode2:$ORACLE_HOME/OPatch
ssh racnode2 "$ORACLE_HOME/OPatch/opatch version"

📎 Download latest OPatch: MOS Doc ID 6880880.1

4.4 — Download and Stage Patch on All Nodes

📝 Why stage on all nodes? OPatch must find the patch files locally on each node it patches. The patch must be unzipped and accessible on every node before patching starts.

# Copy patch zip to all nodes (from node1)
scp /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
    racnode2:/stage/patches/

# Verify patch is present on ALL nodes
ls -lh /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip
ssh racnode2 "ls -lh /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip"

# Unzip patch on NODE 1
mkdir -p /stage/patches/unzipped
unzip -q /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
      -d /stage/patches/unzipped

# Unzip patch on NODE 2
ssh racnode2 "mkdir -p /stage/patches/unzipped && \
unzip -q /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
-d /stage/patches/unzipped"

# Verify on both nodes
ls -l /stage/patches/unzipped/
ssh racnode2 "ls -l /stage/patches/unzipped/"

4.5 — Read Patch README — Confirm Rolling Support

# Read the README before anything else
less /stage/patches/unzipped/<PATCHNUM>/README.html
# OR
less /stage/patches/unzipped/<PATCHNUM>/README.txt

📝 What to look for in README:

• Is rolling patch supported? Look for text like "This patch can be applied in a rolling manner"
• Minimum OPatch version required
• Any pre-patch steps specific to this patch
• Any post-patch steps beyond standard datapatch
• Any known issues with this patch and RAC

4.6 — Run OPatch Conflict Check on All Nodes

📝 Why? Conflict check must be run against BOTH GI Home and DB Home on ALL nodes. A conflict on any single node will cause patching to fail on that node.

# Check conflicts against GI Home — ALL nodes
su - grid

# Node 1 — GI Home conflict check
/u01/app/19.3.0/grid/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>

# Node 2 — GI Home conflict check
ssh racnode2 "/u01/app/19.3.0/grid/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>"

# Check conflicts against DB Home — ALL nodes
su - oracle

# Node 1 — DB Home conflict check
/u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>

# Node 2 — DB Home conflict check
ssh racnode2 "/u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>"

What to look for: All nodes, both homes must show OPatch succeeded.

⚠️ IMPORTANT: If any conflict is found on any node or any home — stop. Investigate and resolve before proceeding. Never proceed with a known conflict.

4.7 — Check Disk Space on All Nodes

# Run on ALL nodes
df -hP $ORACLE_HOME     # DB Home filesystem
df -hP /u01/app/19.3.0  # GI Home filesystem
df -hP /tmp

# SSH to node2 and check
ssh racnode2 "df -hP /u01/app/19.3.0/grid; df -hP /u01/app/oracle/product/19.3.0/dbhome_1"

4.8 — Check Active Sessions and Coordinate Maintenance Window

sqlplus / as sysdba

set linesize 180
set pagesize 100
col inst_id  for 999
col username for a20
col machine  for a25
col program  for a30
col status   for a10

-- Active sessions across ALL nodes
SELECT inst_id, username, machine, program, status, count(*)
FROM   gv$session
WHERE  type    = 'USER'
AND    username IS NOT NULL
GROUP BY inst_id, username, machine, program, status
ORDER BY inst_id, count(*) DESC;

-- Quick count per node
SELECT inst_id, COUNT(*) active_sessions
FROM   gv$session
WHERE  type   = 'USER'
AND    status = 'ACTIVE'
GROUP BY inst_id
ORDER BY inst_id;

4.9 — Take RMAN Backup Before Patching

📝 Why? Same as standalone — your safety net before any patching activity on production.

su - oracle
rman target /

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;
RMAN> LIST BACKUP SUMMARY;
RMAN> EXIT;

4.10 — Record Pre-Patch Baseline

sqlplus / as sysdba

-- Record component versions
set linesize 200
set pagesize 100
col comp_name for a50
col version   for a15
col status    for a12

SELECT comp_name, version, status
FROM   dba_registry
ORDER BY comp_name;

-- Record invalid object count
SELECT COUNT(*) pre_patch_invalid_count
FROM   dba_objects
WHERE  status = 'INVALID';

-- Record current datapatch history
set linesize 200
set pagesize 100
col patch_id    for 9999999999
col version     for a15
col status      for a15
col description for a55
col action_time for a25

SELECT patch_id, version, status, description, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

# Save OPatch inventory from ALL nodes and BOTH homes
su - grid
/u01/app/19.3.0/grid/OPatch/opatch lsinventory \
    > /tmp/pre_patch_GI_node1_$(date +%Y%m%d).txt

ssh racnode2 "/u01/app/19.3.0/grid/OPatch/opatch lsinventory" \
    > /tmp/pre_patch_GI_node2_$(date +%Y%m%d).txt

su - oracle
/u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatch lsinventory \
    > /tmp/pre_patch_DB_node1_$(date +%Y%m%d).txt

ssh racnode2 "/u01/app/oracle/product/19.3.0/dbhome_1/OPatch/opatch lsinventory" \
    > /tmp/pre_patch_DB_node2_$(date +%Y%m%d).txt

5. Rolling Patch — Method 1 (Preferred for Production)

📝 What happens during rolling patch? You patch one node at a time. While node1 is being patched (database instance on node1 is down), node2 continues serving database connections. Clients experience no downtime — they either stay connected to node2 or their connection is automatically failed over by the SCAN listener.

📝 Sequence overview:

Node1: Stop instance → Patch GI Home → Patch DB Home → Start instance
Node2: Stop instance → Patch GI Home → Patch DB Home → Start instance
Both:  Run datapatch once (on any node, after both nodes are patched)

5.1 — Relocate Services Away from Node 1

📝 Why? Before stopping the node1 instance, relocate all services to node2 so client connections are not disrupted. This is the key to zero-downtime rolling patching.

su - oracle

# Check which services are running on node1
$ORACLE_HOME/bin/srvctl status service -db RACDB

# Relocate each service from node1 to node2
$ORACLE_HOME/bin/srvctl relocate service \
    -db RACDB \
    -service RACDB_APP \
    -oldinst RACDB1 \
    -newinst RACDB2

# Verify services are now on node2
$ORACLE_HOME/bin/srvctl status service -db RACDB

5.2 — Stop Database Instance on Node 1

📝 Why? The database instance on node1 must be down before patching the GI and DB homes on node1. Node2 instance continues running throughout.

su - oracle

# Stop instance on node1 only (node2 stays running)
$ORACLE_HOME/bin/srvctl stop instance \
    -db RACDB \
    -instance RACDB1 \
    -stopoption immediate

# Verify node1 instance is down, node2 is still up
$ORACLE_HOME/bin/srvctl status database -db RACDB

Expected output:

Instance RACDB1 is not running on node racnode1
Instance RACDB2 is running on node racnode2

5.3 — Stop Node 1 Listener

su - grid

# Stop local listener on node1 only
$ORACLE_HOME/bin/srvctl stop listener -n racnode1

# Verify
$ORACLE_HOME/bin/srvctl status listener

5.4 — Patch Grid Infrastructure Home on Node 1

📝 Why GI first? Grid Infrastructure must always be patched before the database on the same node. GI controls ASM which the database depends on. If you patch DB home first and the patch requires a specific GI version, ASM may misbehave.

⚠️ IMPORTANT: GI patching MUST be done as root. OPatch for GI requires root because it modifies Clusterware files and restarts cluster resources.

# Switch to root on NODE 1
su - root

# Set GI environment
export ORACLE_HOME=/u01/app/19.3.0/grid
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

# Apply patch to GI Home
# -local flag tells OPatch to patch only the local node (not all nodes)
# This is critical for rolling patches — without -local it tries to patch all nodes
$ORACLE_HOME/OPatch/opatch apply \
    -local \
    -oh $ORACLE_HOME \
    -silent \
    /stage/patches/unzipped/<PATCHNUM>

📝 Monitor GI patching log in another terminal:

tail -100f /tmp/OPatch/opatch_<date>_<time>.log

Expected output:

Patching component oracle.crs, 19.x.x.x.x...
Patch <PATCHNUM> successfully applied.
OPatch succeeded.

📝 After GI patching OPatch automatically restarts CRS resources on node1. Wait for this to complete before proceeding.

# Verify CRS restarted and is running on node1
$ORACLE_HOME/bin/crsctl check crs
$ORACLE_HOME/bin/crsctl stat res -t | grep -i node1

5.5 — Patch Database Home on Node 1

📝 Why? After GI home is patched on node1, now patch the DB home on node1. This is done as the oracle user.

# As oracle user on NODE 1
su - oracle

# Apply patch to DB Home — -local flag patches only node1
$ORACLE_HOME/OPatch/opatch apply \
    -local \
    -oh $ORACLE_HOME \
    -silent \
    /stage/patches/unzipped/<PATCHNUM>

Expected output:

Patching component oracle.rdbms, 19.x.x.x.x...
Patch <PATCHNUM> successfully applied.
OPatch succeeded.

5.6 — Start Node 1 Instance and Listener

su - oracle

# Start listener on node1
$ORACLE_HOME/bin/srvctl start listener -n racnode1

# Start database instance on node1
$ORACLE_HOME/bin/srvctl start instance \
    -db RACDB \
    -instance RACDB1

# Verify both nodes are now running
$ORACLE_HOME/bin/srvctl status database -db RACDB
$ORACLE_HOME/bin/srvctl status listener

📝 At this point node1 is running the new patched binaries and node2 is still running the old binaries. The database is in a mixed patch state — this is normal and expected during rolling patching. The database will remain operational.

5.7 — Relocate Services Away from Node 2

su - oracle

# Move services back to node1 (or keep on node2 — either is fine)
# Here we move them to node1 so node2 can be patched
$ORACLE_HOME/bin/srvctl relocate service \
    -db RACDB \
    -service RACDB_APP \
    -oldinst RACDB2 \
    -newinst RACDB1

# Verify
$ORACLE_HOME/bin/srvctl status service -db RACDB

5.8 — Stop Database Instance on Node 2

su - oracle

# Stop instance on node2 only (node1 stays running)
$ORACLE_HOME/bin/srvctl stop instance \
    -db RACDB \
    -instance RACDB2 \
    -stopoption immediate

# Verify
$ORACLE_HOME/bin/srvctl status database -db RACDB

5.9 — Stop Node 2 Listener

su - grid

# Stop local listener on node2 via SSH
ssh racnode2 "$ORACLE_HOME/bin/srvctl stop listener -n racnode2"

# Verify
$ORACLE_HOME/bin/srvctl status listener

5.10 — Patch Grid Infrastructure Home on Node 2

# As root on NODE 2 — either SSH or directly on node2
ssh racnode2 "su - root"
# OR log directly into node2

su - root

export ORACLE_HOME=/u01/app/19.3.0/grid
export PATH=$ORACLE_HOME/bin:$ORACLE_HOME/OPatch:$PATH

# Patch GI Home on node2
$ORACLE_HOME/OPatch/opatch apply \
    -local \
    -oh $ORACLE_HOME \
    -silent \
    /stage/patches/unzipped/<PATCHNUM>

# Verify CRS restarted on node2
$ORACLE_HOME/bin/crsctl check crs

5.11 — Patch Database Home on Node 2

# As oracle user on NODE 2
ssh racnode2
su - oracle

$ORACLE_HOME/OPatch/opatch apply \
    -local \
    -oh $ORACLE_HOME \
    -silent \
    /stage/patches/unzipped/<PATCHNUM>

5.12 — Start Node 2 Instance and Listener

# On node2 as oracle user
su - oracle

$ORACLE_HOME/bin/srvctl start listener -n racnode2

$ORACLE_HOME/bin/srvctl start instance \
    -db RACDB \
    -instance RACDB2

# Verify all instances are running on all nodes
$ORACLE_HOME/bin/srvctl status database -db RACDB

Expected output:

Instance RACDB1 is running on node racnode1
Instance RACDB2 is running on node racnode2

📝 Both nodes are now running the new patched binaries. Proceed to datapatch.

6. Non-Rolling Patch — Method 2 (Full Downtime)

📝 When to use? When the patch README says rolling is not supported, or when you prefer simplicity over zero-downtime in non-production environments. All nodes are patched simultaneously while the database is completely down.

6.1 — Stop All Services and Database on All Nodes

su - oracle

# Stop all services
$ORACLE_HOME/bin/srvctl stop service -db RACDB

# Stop entire database (all instances on all nodes)
$ORACLE_HOME/bin/srvctl stop database \
    -db RACDB \
    -stopoption immediate

# Stop all listeners on all nodes
$ORACLE_HOME/bin/srvctl stop listener

# Verify everything is down
$ORACLE_HOME/bin/srvctl status database -db RACDB
$ORACLE_HOME/bin/srvctl status listener

6.2 — Patch GI Home on All Nodes Simultaneously

# As root on NODE 1
su - root
export ORACLE_HOME=/u01/app/19.3.0/grid
$ORACLE_HOME/OPatch/opatch apply \
    -local -oh $ORACLE_HOME -silent \
    /stage/patches/unzipped/<PATCHNUM>

# As root on NODE 2 — run at same time as node1
# Log into node2 directly or use parallel SSH tool
su - root
export ORACLE_HOME=/u01/app/19.3.0/grid
$ORACLE_HOME/OPatch/opatch apply \
    -local -oh $ORACLE_HOME -silent \
    /stage/patches/unzipped/<PATCHNUM>

6.3 — Patch DB Home on All Nodes Simultaneously

# As oracle user on NODE 1
su - oracle
$ORACLE_HOME/OPatch/opatch apply \
    -local -oh $ORACLE_HOME -silent \
    /stage/patches/unzipped/<PATCHNUM>

# As oracle user on NODE 2 — run at same time
su - oracle
$ORACLE_HOME/OPatch/opatch apply \
    -local -oh $ORACLE_HOME -silent \
    /stage/patches/unzipped/<PATCHNUM>

6.4 — Start Everything on All Nodes

su - oracle

# Start database on all nodes
$ORACLE_HOME/bin/srvctl start database -db RACDB

# Start listeners on all nodes
$ORACLE_HOME/bin/srvctl start listener

# Start services
$ORACLE_HOME/bin/srvctl start service -db RACDB

# Verify
$ORACLE_HOME/bin/srvctl status database -db RACDB
$ORACLE_HOME/bin/srvctl status listener
$ORACLE_HOME/bin/srvctl status service -db RACDB

7. Run datapatch (After All Nodes Are Patched — Both Methods)

⚠️ IMPORTANT: datapatch is run ONLY ONCE regardless of rolling or non-rolling method. Run it from any ONE node after ALL nodes have been patched and ALL instances are running. datapatch updates the data dictionary for the entire database — it applies to all instances automatically.

📝 Why must all instances be running? datapatch in a RAC environment requires all instances to be up so it can coordinate the dictionary changes across all nodes. If any instance is down when you run datapatch, it will warn you or fail.

7.1 — Verify All Instances Are Running Before datapatch

sqlplus / as sysdba

set linesize 200
set pagesize 50
col inst_id       for 999
col instance_name for a15
col host_name     for a20
col status        for a12

SELECT inst_id, instance_name, host_name, status
FROM   gv$instance
ORDER BY inst_id;

All instances must show OPEN before running datapatch.

7.2 — Run datapatch

# Run from NODE 1 as oracle user — only run once from one node
su - oracle
cd $ORACLE_HOME/OPatch

# -verbose shows detailed progress of every SQL script applied
./datapatch -verbose

📝 Monitor datapatch log for progress and errors:

# In another terminal — find and monitor the active datapatch log
ls -lrt /u01/app/oracle/cfgtoollogs/sqlpatch/
tail -100f /u01/app/oracle/cfgtoollogs/sqlpatch/<latest_dir>/<latest_log>

Expected successful output:

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done
Adding patches to installation queue...done
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...done
Patch <PATCHNUM> apply: SUCCESS
datapatch: ALL PATCHES APPLIED SUCCESSFULLY.

⚠️ IMPORTANT: If datapatch fails — do NOT restart it immediately. Check the log file for the specific error. Common issues in RAC:

• One instance is not open — start it first
• TEMP tablespace is full — extend it
• INVALID objects are blocking compilation — run utlrp.sql first then rerun datapatch

7.3 — Relocate Services Back to Normal Distribution

su - oracle

# If services were all moved to one node during rolling patch
# Move them back to their preferred nodes for load balancing
$ORACLE_HOME/bin/srvctl relocate service \
    -db RACDB \
    -service RACDB_APP \
    -oldinst RACDB1 \
    -newinst RACDB2

# Or simply stop and start the service — CRS places it on preferred node
$ORACLE_HOME/bin/srvctl stop  service -db RACDB -service RACDB_APP
$ORACLE_HOME/bin/srvctl start service -db RACDB -service RACDB_APP

# Verify balanced distribution
$ORACLE_HOME/bin/srvctl status service -db RACDB

8. Post-Patching Checks

⚠️ IMPORTANT: Post-patch checks in RAC must cover ALL nodes. Never sign off on RAC patching without verifying every item below on every node.

8.1 — Verify New Patch Version on All Nodes

sqlplus / as sysdba

set linesize 200
set pagesize 100
col inst_id       for 999
col instance_name for a15
col host_name     for a20
col version_full  for a20

-- Confirm new version on all nodes
SELECT inst_id, instance_name, host_name,
       version, version_full
FROM   gv$instance
ORDER BY inst_id;

# Verify patch in OPatch inventory — GI Home on ALL nodes
su - grid
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Grid|Patch"
ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory | grep -E 'Oracle Grid|Patch'"

# Verify patch in OPatch inventory — DB Home on ALL nodes
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch"
ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory | grep -E 'Oracle Database|Patch'"

8.2 — Verify datapatch Applied Successfully

sqlplus / as sysdba

set linesize 200
set pagesize 100
col patch_id    for 9999999999
col version     for a15
col status      for a15
col description for a55
col action_time for a25

-- Confirm patch applied successfully in data dictionary
SELECT patch_id, version, status, description, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

What to look for: Your patch number must appear with SUCCESS in status column.

8.3 — Check Cluster Health After Patching

su - grid

# Full cluster status
$ORACLE_HOME/bin/crsctl check cluster -all

# All resources must be online
$ORACLE_HOME/bin/crsctl stat res -t

# Check for any offline resources
$ORACLE_HOME/bin/crsctl stat res -t | grep -E "OFFLINE|INTERMEDIATE"

# Check cluster nodes
$ORACLE_HOME/bin/olsnodes -n -i -s -t

8.4 — Check All Database Instances on All Nodes

sqlplus / as sysdba

set linesize 200
set pagesize 100
col inst_id           for 999
col instance_name     for a15
col host_name         for a20
col status            for a12
col database_status   for a18

SELECT inst_id, instance_name, host_name,
       status, database_status
FROM   gv$instance
ORDER BY inst_id;

-- Check open mode on all nodes
SELECT inst_id, name, open_mode, log_mode
FROM   gv$database
ORDER BY inst_id;

8.5 — Check All Datafiles Online on All Nodes

set linesize 200
set pagesize 100
col file#           for 999
col tablespace_name for a25
col status          for a12
col name            for a70

SELECT file#, tablespace_name, status, name
FROM   v$datafile
ORDER BY file#;

8.6 — Check ASM Diskgroups on All Nodes After Patching

su - grid
sqlplus / as sysasm

set linesize 200
set pagesize 100
col inst_id      for 999
col name         for a12
col state        for a12
col total_gb     for 99999.99
col free_gb      for 99999.99
col offline_disks for 999

SELECT inst_id, name, state,
       total_mb/1024  total_gb,
       free_mb/1024   free_gb,
       offline_disks
FROM   gv$asm_diskgroup
ORDER BY name, inst_id;

What to look for: All diskgroups MOUNTED on all nodes. offline_disks = 0.

8.7 — Check for INVALID Objects After Patching

sqlplus / as sysdba

set linesize 180
set pagesize 100
col owner       for a20
col object_name for a45
col object_type for a25
col status      for a10

SELECT owner, object_name, object_type, status
FROM   dba_objects
WHERE  status = 'INVALID'
ORDER BY owner, object_type, object_name;

SELECT COUNT(*) post_patch_invalid_count
FROM   dba_objects
WHERE  status = 'INVALID';

-- If count is higher than pre-patch baseline — recompile
@?/rdbms/admin/utlrp.sql

-- Re-verify
SELECT COUNT(*) FROM dba_objects WHERE status = 'INVALID';

8.8 — Verify All Components Are VALID

set linesize 200
set pagesize 100
col comp_name for a50
col version   for a15
col status    for a12

SELECT comp_name, version, status
FROM   dba_registry
ORDER BY comp_name;

8.9 — Check Voting Disks After Patching

su - grid

# Voting disks must all be ONLINE after patching
$ORACLE_HOME/bin/crsctl query css votedisk

8.10 — Check OCR Integrity After Patching

su - grid

# OCR check must succeed
$ORACLE_HOME/bin/ocrcheck

# Check OCR backup was taken post-patching
$ORACLE_HOME/bin/ocrconfig -showbackup

8.11 — Check Alert Logs on All Nodes for Errors

# CRS alert log — ALL nodes
tail -200 /u01/app/grid/diag/crs/racnode1/crs/trace/alert.log \
     | grep -E "ORA-|Error|WARNING|EVICTED"

ssh racnode2 "tail -200 /u01/app/grid/diag/crs/racnode2/crs/trace/alert.log \
     | grep -E 'ORA-|Error|WARNING|EVICTED'"

# ASM alert log — ALL nodes
tail -200 /u01/app/oracle/diag/asm/+asm/+ASM1/trace/alert_+ASM1.log \
     | grep -E "ORA-|Error|WARNING"

ssh racnode2 "tail -200 /u01/app/oracle/diag/asm/+asm/+ASM2/trace/alert_+ASM2.log \
     | grep -E 'ORA-|Error|WARNING'"

# DB alert log — ALL nodes
tail -200 /u01/app/oracle/diag/rdbms/racdb/RACDB1/trace/alert_RACDB1.log \
     | grep -E "ORA-|Error|WARNING"

ssh racnode2 "tail -200 /u01/app/oracle/diag/rdbms/racdb/RACDB2/trace/alert_RACDB2.log \
     | grep -E 'ORA-|Error|WARNING'"

8.12 — Verify SCAN and Services

su - grid

# SCAN and SCAN listener
$ORACLE_HOME/bin/srvctl status scan
$ORACLE_HOME/bin/srvctl status scan_listener

# All services
$ORACLE_HOME/bin/srvctl status service -db RACDB

# Test SCAN connectivity end-to-end
sqlplus system/Oracle_123@rac-scan:1521/RACDB_APP

8.13 — Save Post-Patch Inventory from All Nodes

# GI Home inventory — all nodes
su - grid
$ORACLE_HOME/OPatch/opatch lsinventory \
    > /tmp/post_patch_GI_node1_$(date +%Y%m%d).txt

ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory" \
    > /tmp/post_patch_GI_node2_$(date +%Y%m%d).txt

# DB Home inventory — all nodes
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory \
    > /tmp/post_patch_DB_node1_$(date +%Y%m%d).txt

ssh racnode2 "$ORACLE_HOME/OPatch/opatch lsinventory" \
    > /tmp/post_patch_DB_node2_$(date +%Y%m%d).txt

# Compare pre vs post for each node and each home
diff /tmp/pre_patch_GI_node1_$(date +%Y%m%d).txt \
     /tmp/post_patch_GI_node1_$(date +%Y%m%d).txt

diff /tmp/pre_patch_DB_node1_$(date +%Y%m%d).txt \
     /tmp/post_patch_DB_node1_$(date +%Y%m%d).txt

9. Patch Rollback Procedure for RAC

📝 When to use? If patching causes critical issues and you need to return to the previous state. RAC rollback must be done in reverse order on all nodes.

⚠️ IMPORTANT: Rollback sequence — datapatch rollback first (DB open, all instances up) → then stop DB → then OPatch rollback on all nodes → then start DB.

9.1 — Run datapatch Rollback (All Instances Must Be Running)

# Run from NODE 1 as oracle — only once
su - oracle
cd $ORACLE_HOME/OPatch

./datapatch -verbose -rollback <PATCHNUM>

9.2 — Stop Database and Listeners on All Nodes

su - oracle

$ORACLE_HOME/bin/srvctl stop service  -db RACDB
$ORACLE_HOME/bin/srvctl stop database -db RACDB -stopoption immediate
$ORACLE_HOME/bin/srvctl stop listener

# Verify all down
$ORACLE_HOME/bin/srvctl status database -db RACDB

9.3 — Rollback DB Home on All Nodes

# Node 1 — as oracle
su - oracle
$ORACLE_HOME/OPatch/opatch rollback \
    -id <PATCHNUM> -local -silent \
    -oh $ORACLE_HOME

# Node 2 — as oracle
ssh racnode2 "su - oracle -c \
    '$ORACLE_HOME/OPatch/opatch rollback \
    -id <PATCHNUM> -local -silent \
    -oh $ORACLE_HOME'"

9.4 — Rollback GI Home on All Nodes

# Node 1 — as root
su - root
/u01/app/19.3.0/grid/OPatch/opatch rollback \
    -id <PATCHNUM> -local -silent \
    -oh /u01/app/19.3.0/grid

# Node 2 — as root
ssh racnode2 "su - root -c \
    '/u01/app/19.3.0/grid/OPatch/opatch rollback \
    -id <PATCHNUM> -local -silent \
    -oh /u01/app/19.3.0/grid'"

9.5 — Start Everything and Verify Rollback

su - oracle

$ORACLE_HOME/bin/srvctl start database -db RACDB
$ORACLE_HOME/bin/srvctl start listener
$ORACLE_HOME/bin/srvctl start service  -db RACDB

# Verify old version is back
sqlplus / as sysdba

SELECT inst_id, instance_name, version_full
FROM   gv$instance
ORDER BY inst_id;

-- Confirm patch is no longer listed
SELECT patch_id, status, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

10. Quick Reference Card

This SOP covers everything you need to patch Oracle RAC 19c on Linux without referring to any other source. Always read the patch README to confirm rolling support, always patch GI Home before DB Home on every node, run datapatch only once after all nodes are patched, and always verify cluster health before and after every patching activity.