Oracle Data Guard 19c Patching on Linux – Complete Step-by-Step Guide

A complete production-ready SOP for patching Oracle Data Guard 19c on Linux. Covers the rolling patch method using the standby-first approach, Grid Infrastructure patching in DG environments, datapatch execution, switchover after patching, and full post-patch validation on both primary and standby — with real commands, expected outputs, and consultant-level notes for both standard OFA and enterprise custom path conventions.

1. Document Info

2. Data Guard Patching — Concepts You Must Know First

📝 Why is DG patching different from standalone? In standalone patching you have one database, one Oracle Home, and patching requires full downtime. In a Data Guard environment you have two databases (primary and standby) each with their own Oracle Home. The power of Data Guard patching is that you can patch the standby first while the primary stays fully online, then do a planned switchover so the newly patched standby becomes the primary — resulting in near-zero downtime for the application.

Key DG Patching Concepts

DG Patching Sequence Overview

High Level Steps — Standby-First Rolling Approach:

PHASE 1 — Patch Standby (Primary stays online — zero application downtime)
  Step 1  → Pre-patch checks on both servers
  Step 2  → Stop MRP on standby
  Step 3  → Patch Oracle Home on standby server
  Step 4  → Start MRP on standby — let it catch up
  Step 5  → Verify standby is healthy and in sync

PHASE 2 — Switchover (Brief application reconnect — seconds to minutes)
  Step 6  → Validate DG and standby is ready for switchover
  Step 7  → Perform switchover (standby becomes new primary)
  Step 8  → Run datapatch on new primary (was standby)
  Step 9  → Verify new primary is healthy

PHASE 3 — Patch Old Primary (Now Standby — no application impact)
  Step 10 → Patch Oracle Home on old primary (now standby)
  Step 11 → Start MRP on old primary standby
  Step 12 → Verify full DG configuration is healthy
  Step 13 → Optional switchover back to original primary

📝 Total application downtime with this approach is only the switchover time — typically 30 seconds to 5 minutes depending on active sessions and redo volume in transit.

Convention B — Key Advantage During DG Patching

📝 Why Convention B is better for patching? In Convention B, primary uses /oracle/RDBMS/19.31 and standby uses /oracle/RDBMS/19.31Standby. These are completely separate directories. You can:

• Pre-patch the standby home (19.31Standby) at any time without touching the primary home (19.31)
• After switchover, patch the old primary home (19.31) while it runs as standby
• Both homes can have different patch levels simultaneously without any conflict
• Rollback is as simple as pointing back to the old home

📝 In Convention A (same home name on both servers), the homes are still separate (on different servers) — patching is still independent. The difference is mostly organizational clarity.

3. Path Conventions

📝 All examples use Convention A. For Convention B substitute the respective paths. The steps are identical — only directory paths differ.

4. Pre-Patching Checks — Both Servers

⚠️ IMPORTANT: Every pre-check must be completed on BOTH primary and standby before starting. Patching a Data Guard configuration that is not fully healthy is one of the most common causes of patching failures and extended outages.

4.1 — Confirm Current Patch Level on Both Servers

📝 Why? You must document exactly what patch level both primary and standby are on before starting. Both should be on the same patch level before you begin. If they are already on different patch levels — fix that first.

# On PRIMARY — check Oracle Home patch inventory
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory

# Quick summary
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch "

# On STANDBY — check Oracle Home patch inventory
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch "

-- On PRIMARY — confirm DB version and patch level
sqlplus / as sysdba

set linesize 150
set pagesize 50
col banner for a80

SELECT banner FROM v$version WHERE banner LIKE 'Oracle%';

SELECT version, version_full FROM v$instance;

4.2 — Verify Data Guard Configuration is Fully Healthy

📝 Why? Data Guard must be completely healthy before patching. Any existing errors, apply lag, or archive gaps will be amplified during patching. Fix all DG issues before touching any patch.

-- Connect to Broker from PRIMARY
dgmgrl sys/Oracle_123@ORCL

-- Overall configuration health
DGMGRL> SHOW CONFIGURATION;
-- Must show: SUCCESS

-- Primary database details
DGMGRL> SHOW DATABASE VERBOSE 'ORCL';

-- Standby database details
DGMGRL> SHOW DATABASE VERBOSE 'ORCL_STBY';

-- Validate standby is ready
DGMGRL> VALIDATE DATABASE 'ORCL_STBY';

-- Check lag
DGMGRL> SHOW DATABASE 'ORCL_STBY' 'ApplyLagSecs';
DGMGRL> SHOW DATABASE 'ORCL_STBY' 'TransportLagSecs';

DGMGRL> EXIT;

⚠️ IMPORTANT: If Broker shows WARNING or ERROR — stop. Resolve the DG issue first. Common issues to resolve before patching:

• Archive destination error on primary
• MRP not running on standby
• Archive log gap on standby
• Transport lag above acceptable threshold

-- Additional DG health checks directly on each DB
-- On PRIMARY
sqlplus / as sysdba

set linesize 200
set pagesize 50
col dest_id   for 999
col dest_name for a20
col status    for a12
col error     for a45

-- Check archive destinations — DEST_ID 2 must be VALID with no error
SELECT dest_id, dest_name, status, error
FROM   v$archive_dest
WHERE  dest_id IN (1,2)
ORDER BY dest_id;

-- On STANDBY — check MRP and apply lag
sqlplus / as sysdba

set linesize 200
set pagesize 50
col process   for a12
col status    for a20
col sequence# for 999999

-- MRP must be running
SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

-- Apply lag must be minimal
set linesize 200
col name  for a30
col value for a40
col unit  for a20

SELECT name, value, unit
FROM   v$dataguard_stats
WHERE  name IN ('transport lag','apply lag')
ORDER BY name;

4.3 — Check for Archive Log Gaps on Standby

-- On STANDBY
set linesize 150
set pagesize 50
col low_sequence#  for 999999
col high_sequence# for 999999

-- No rows = no gaps (good to proceed)
-- Any rows = gaps exist — resolve before patching
SELECT thread#, low_sequence#, high_sequence#
FROM   v$archive_gap;

⚠️ IMPORTANT: If archive gaps exist, the standby will not be current. After patching and switchover, the new primary (was standby) may be missing transactions. Resolve all gaps before starting the patching process.

4.4 — Verify OPatch Version on Both Servers and Upgrade if Needed

# On PRIMARY
su - oracle
$ORACLE_HOME/OPatch/opatch version

# On STANDBY
su - oracle
$ORACLE_HOME/OPatch/opatch version

📝 Compare both versions against the minimum required OPatch version in the patch README.

Upgrade OPatch on standby if needed (do standby first):

# On STANDBY as oracle user
su - oracle

# Backup existing OPatch
mv $ORACLE_HOME/OPatch \
   $ORACLE_HOME/OPatch_backup_$(date +%Y%m%d)

# Unzip new OPatch into Oracle Home
unzip -q /stage/patches/p6880880_190000_Linux-x86-64.zip \
      -d $ORACLE_HOME

# Verify new version
$ORACLE_HOME/OPatch/opatch version

# On PRIMARY as oracle user — upgrade OPatch here too
su - oracle
mv $ORACLE_HOME/OPatch \
   $ORACLE_HOME/OPatch_backup_$(date +%Y%m%d)
unzip -q /stage/patches/p6880880_190000_Linux-x86-64.zip \
      -d $ORACLE_HOME
$ORACLE_HOME/OPatch/opatch version

📎 Download latest OPatch: MOS Doc ID 6880880.1

4.5 — Download Patch and Stage on Both Servers

# Download patch on PRIMARY
ls -lh /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip

# Verify checksum
sha256sum /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip

# Copy patch to STANDBY
scp /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
    oracle@dbserver02:/stage/patches/

# Verify on STANDBY
ssh oracle@dbserver02 \
    "ls -lh /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip"

# Unzip on STANDBY first (we patch standby first)
ssh oracle@dbserver02 \
    "mkdir -p /stage/patches/unzipped && \
     unzip -q /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
     -d /stage/patches/unzipped"

# Unzip on PRIMARY (for later)
mkdir -p /stage/patches/unzipped
unzip -q /stage/patches/p<PATCHNUM>_190000_Linux-x86-64.zip \
      -d /stage/patches/unzipped

# Verify on both
ls -l /stage/patches/unzipped/
ssh oracle@dbserver02 "ls -l /stage/patches/unzipped/"

4.6 — Read Patch README

# Read the README on STANDBY
less /stage/patches/unzipped/<PATCHNUM>/README.html

📝 What to look for:

• Does this patch support Data Guard rolling patching? Look for "Data Guard Standby-First Patching" section
• Minimum OPatch version required
• Any pre-patch steps specific to this patch
• Any post-patch steps beyond standard datapatch
• Any known issues in Data Guard environments

4.7 — Run OPatch Conflict Check on Both Servers

# On STANDBY — conflict check first (we patch standby first)
su - oracle
$ORACLE_HOME/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>

# On PRIMARY — conflict check
su - oracle
$ORACLE_HOME/OPatch/opatch prereq \
    CheckConflictAgainstOHWithDetail \
    -ph /stage/patches/unzipped/<PATCHNUM>

Both must show OPatch succeeded.

4.8 — Check Disk Space on Both Servers

# On PRIMARY
df -hP $ORACLE_HOME
df -hP /tmp

# On STANDBY
ssh oracle@dbserver02 "df -hP $ORACLE_HOME; df -hP /tmp"

📝 Both servers need minimum 2-3 GB free in the Oracle Home filesystem for OPatch to store backup of replaced files.

4.9 — Take RMAN Backup on Primary Before Patching

su - oracle
rman target /

RMAN> BACKUP DATABASE PLUS ARCHIVELOG;
RMAN> LIST BACKUP SUMMARY;
RMAN> EXIT;

4.10 — Record Pre-Patch Baseline on Both Servers

-- On PRIMARY
sqlplus / as sysdba

set linesize 200
set pagesize 100
col comp_name for a50
col version   for a15
col status    for a12

SELECT comp_name, version, status
FROM   dba_registry
ORDER BY comp_name;

SELECT COUNT(*) pre_patch_invalid_count
FROM   dba_objects
WHERE  status = 'INVALID';

-- Record datapatch history
set linesize 200
set pagesize 100
col patch_id    for 9999999999
col version     for a15
col status      for a15
col description for a55
col action_time for a25

SELECT patch_id, version, status, description, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

# Save OPatch inventory from BOTH servers
su - oracle

# Primary inventory
$ORACLE_HOME/OPatch/opatch lsinventory \
    > /tmp/pre_patch_PRIMARY_$(date +%Y%m%d).txt

# Standby inventory
ssh oracle@dbserver02 \
    "$ORACLE_HOME/OPatch/opatch lsinventory" \
    > /tmp/pre_patch_STANDBY_$(date +%Y%m%d).txt

cat /tmp/pre_patch_PRIMARY_$(date +%Y%m%d).txt
cat /tmp/pre_patch_STANDBY_$(date +%Y%m%d).txt

5. Phase 1 — Patch Standby Oracle Home

📝 What happens in Phase 1? The standby Oracle Home is patched while the primary database continues running and serving application traffic. During this phase the standby goes through a brief period where MRP is stopped — archivelogs from primary continue to accumulate on the standby but are not applied. After patching the standby home, MRP is restarted and the standby catches up.

⚠️ IMPORTANT: All steps in Phase 1 are performed on the STANDBY server unless stated otherwise. The primary database is NOT touched during Phase 1.

5.1 — Stop MRP on Standby

📝 Why? MRP (Managed Recovery Process) reads from the Oracle Home binaries to apply redo. If we replace those binaries while MRP is running, MRP may crash or produce errors. Always stop MRP before patching the standby Oracle Home.

-- On STANDBY as oracle user
sqlplus / as sysdba

-- Check MRP is currently running
set linesize 180
set pagesize 50
col process  for a12
col status   for a20
col sequence# for 999999

SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

-- Stop MRP gracefully
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

-- Confirm MRP has stopped — should return no rows
SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

What to look for after cancel: No MRP0 row returned = MRP is stopped.

📝 Note: While MRP is stopped, the primary continues shipping archivelogs to the standby. They accumulate in the standby’s archive destination but are not applied. This is fine — they will all be applied when MRP is restarted after patching.

5.2 — Verify Standby Database is in MOUNT State or Open Read-Only

📝 Why? We do not need to shut down the standby database completely to patch the Oracle Home. The standby instance can remain running in MOUNT state while we patch. This saves significant time compared to a full shutdown and startup cycle.

-- On STANDBY
set linesize 150
set pagesize 50
col name           for a12
col db_unique_name for a15
col open_mode      for a15
col database_role  for a20

SELECT name, db_unique_name, open_mode, database_role
FROM   v$database;

📝 Standby can be in MOUNTED or READ ONLY WITH APPLY (Active Data Guard) state during patching. Either is fine — MRP is already stopped so no apply is happening.

5.3 — Confirm No Oracle Processes Are Using the Home (Check Before Patching)

# On STANDBY as oracle user
# Check what processes are running from this ORACLE_HOME
ps -ef | grep $ORACLE_HOME | grep -v grep

# You should see standby background processes (smon, pmon, etc.)
# These are OK — OPatch can patch while the instance is mounted
# OPatch will stop and restart them automatically during patching

5.4 — Apply Patch to Standby Oracle Home

📝 Why opatch apply while standby is mounted? OPatch is smart enough to handle a running Oracle instance during patching. It stops any Oracle processes that have open file handles on the files it needs to replace, replaces the files, then restarts those processes. The instance itself stays up in MOUNT state throughout.

⚠️ IMPORTANT: Run OPatch as the oracle user. Never run OPatch as root for the database home.

# On STANDBY as oracle user
su - oracle

# Navigate to patch directory
cd /stage/patches/unzipped/<PATCHNUM>

# Apply patch to standby Oracle Home
# Note: No -local flag needed here — this is standalone (not RAC)
$ORACLE_HOME/OPatch/opatch apply \
    -oh $ORACLE_HOME \
    -silent

# Monitor progress in another terminal
tail -100f /tmp/OPatch/opatch_<date>_<time>.log

Expected output:

Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   <PATCHNUM>
Do you want to proceed? [y|n]   (auto-answered yes with -silent)
User Responded with: Y
Backing up files...
Patching component oracle.rdbms, 19.x.x.x.x...
Patching component oracle.rdbms.util, 19.x.x.x.x...
Patch <PATCHNUM> successfully applied.
Log file location: /tmp/OPatch/opatch_<date>_<time>.log
OPatch succeeded.

⚠️ IMPORTANT: If OPatch fails on standby — do NOT proceed to Phase 2. Run OPatch rollback on the standby to restore the original home, investigate the error, and fix it before retrying.

# If patch fails — rollback on standby
$ORACLE_HOME/OPatch/opatch rollback \
    -id <PATCHNUM> \
    -oh $ORACLE_HOME \
    -silent

5.5 — Verify Patch Applied Successfully on Standby

# On STANDBY
su - oracle

# Confirm patch is now in inventory
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch "

# Detailed check
$ORACLE_HOME/OPatch/opatch lsinventory -detail | grep -A3 "<PATCHNUM>"

5.6 — Restart MRP on Standby

📝 Why? Now that the Oracle Home is patched, restart MRP with the new patched binaries. MRP will automatically catch up on all archivelogs that accumulated while it was stopped during patching.

-- On STANDBY
sqlplus / as sysdba

-- Start real-time apply with new patched binaries
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE
    DISCONNECT FROM SESSION;

-- Verify MRP is running
set linesize 180
set pagesize 50
col process  for a12
col status   for a20
col sequence# for 999999

SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

What to look for: MRP0 must show APPLYING_LOG or WAIT_FOR_LOG.

5.7 — Wait for Standby to Catch Up After Patching

📝 Why? During the time MRP was stopped, archivelogs accumulated on the standby but were not applied. Before proceeding to Phase 2 (switchover), confirm the standby has caught up and apply lag is near zero.

-- On STANDBY — monitor catch-up progress
-- Run this repeatedly until apply lag is 0

set linesize 200
set pagesize 50
col name  for a30
col value for a40
col unit  for a20

SELECT name, value, unit
FROM   v$dataguard_stats
WHERE  name IN ('transport lag', 'apply lag')
ORDER BY name;

-- Check last applied sequence number on standby
set linesize 150
set pagesize 50

SELECT thread#, max(sequence#) last_applied
FROM   v$log_history
GROUP BY thread#;

-- Cross-check with primary — what is the last archived sequence on primary
-- Run this on PRIMARY
SELECT thread#, max(sequence#) last_archived
FROM   v$archived_log
WHERE  dest_id = 1
GROUP BY thread#;

⚠️ IMPORTANT: Do not proceed to Phase 2 until apply lag is zero or near zero (within 1-2 minutes). Switching over with a large lag means the new primary (was standby) is missing recent transactions.

5.8 — Verify Standby is Healthy After Phase 1

-- Connect to Broker from PRIMARY
dgmgrl sys/Oracle_123@ORCL

DGMGRL> SHOW CONFIGURATION;
-- Must show: SUCCESS

DGMGRL> SHOW DATABASE VERBOSE 'ORCL_STBY';

-- Validate standby is ready for switchover
DGMGRL> VALIDATE DATABASE 'ORCL_STBY';
-- Look for: Ready for Switchover: Yes

DGMGRL> SHOW DATABASE 'ORCL_STBY' 'ApplyLagSecs';
-- Should show 0 or very low number

DGMGRL> EXIT;

📝 At this point: standby is on NEW patch level, primary is still on OLD patch level. Data Guard configuration is healthy in a mixed-patch state. This is supported and expected.

6. Phase 2 — Switchover and datapatch

📝 What happens in Phase 2? We perform a planned switchover — the newly patched standby becomes the new primary, and the old primary becomes the new standby. Then we run datapatch on the new primary to update the data dictionary to the new patch level.

📝 Application impact: During switchover, active application connections to the old primary will be disconnected. Applications must reconnect to the new primary. With proper connection pooling and SCAN/service configuration, this reconnect happens automatically within seconds to a few minutes.

6.1 — Pre-Switchover Final Checks

-- Connect to Broker on PRIMARY
dgmgrl sys/Oracle_123@ORCL

-- Final configuration check
DGMGRL> SHOW CONFIGURATION;

-- Final validation of standby
DGMGRL> VALIDATE DATABASE 'ORCL_STBY';

-- Check primary has no outstanding redo not yet shipped
-- On PRIMARY
sqlplus / as sysdba

set linesize 200
set pagesize 50
col dest_id for 999
col status  for a12
col error   for a45

-- Archive dest 2 (standby) must be VALID and error-free
SELECT dest_id, status, error,
       target, archiver
FROM   v$archive_dest
WHERE  dest_id = 2;

-- Confirm standby received latest archivelog
-- Note the sequence number
SELECT thread#, max(sequence#) last_on_primary
FROM   v$archived_log
WHERE  dest_id = 1
GROUP BY thread#;

-- On STANDBY — confirm same sequence is applied
SELECT thread#, max(sequence#) last_applied
FROM   v$log_history
GROUP BY thread#;

-- Both numbers must match before switchover

6.2 — Perform Switchover Using Broker

📝 Why Broker for switchover? Broker handles the entire switchover sequence automatically:

• Flushes all remaining redo from primary to standby
• Converts primary role to standby
• Converts standby role to primary
• Opens new primary in READ WRITE mode
• Starts MRP on new standby (old primary)
  All in one command with automatic error checking.

-- Connect to Broker on PRIMARY (connect before it becomes standby)
dgmgrl sys/Oracle_123@ORCL

-- Perform switchover
-- This takes 1-5 minutes depending on active sessions and redo in transit
DGMGRL> SWITCHOVER TO 'ORCL_STBY';

-- Monitor switchover progress in Broker output
-- You will see messages like:
-- Performing switchover NOW, please wait...
-- Operation requires a connection to database "ORCL_STBY"
-- Switchover succeeded, new primary is "ORCL_STBY"

📝 After switchover completes:

• ORCL_STBY (dbserver02) is now the NEW PRIMARY — running on PATCHED Oracle Home
• ORCL (dbserver01) is now the NEW STANDBY — running on OLD (unpatched) Oracle Home

-- Verify new configuration
DGMGRL> SHOW CONFIGURATION;

Expected output after switchover:

Configuration - ORCL_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  ORCL_STBY - Primary database
  ORCL      - Physical standby database

Configuration Status:
SUCCESS

DGMGRL> EXIT;

6.3 — Verify New Primary is Open and Healthy

-- On dbserver02 (NEW PRIMARY — was standby)
sqlplus / as sysdba

set linesize 200
set pagesize 50
col name           for a12
col db_unique_name for a15
col open_mode      for a15
col database_role  for a20
col log_mode       for a15

SELECT name, db_unique_name, open_mode,
       database_role, log_mode
FROM   v$database;

Expected output:

NAME    DB_UNIQUE_NAME  OPEN_MODE   DATABASE_ROLE    LOG_MODE
------- --------------- ----------- ---------------- ------------
ORCL    ORCL_STBY       READ WRITE  PRIMARY          ARCHIVELOG

6.4 — Run datapatch on New Primary

⚠️ IMPORTANT: datapatch must be run on the NEW PRIMARY (dbserver02 — was standby). The new primary is running the patched Oracle Home so datapatch will apply the SQL-level changes for the new patch. These changes replicate to the new standby (old primary) via redo automatically.

📝 Why run datapatch only on the new primary? datapatch updates the data dictionary. In a Data Guard configuration the standby’s data dictionary is kept synchronized with the primary via redo apply. So you run datapatch on the primary and redo carries those changes to the standby automatically.

# On dbserver02 (NEW PRIMARY) as oracle user
su - oracle

# Set ORACLE_SID to the new primary SID
export ORACLE_SID=ORCL_STBY

# Navigate to OPatch directory
cd $ORACLE_HOME/OPatch

# Run datapatch on new primary
./datapatch -verbose

📝 Monitor datapatch log in another terminal:

ls -lrt /u01/app/oracle/cfgtoollogs/sqlpatch/
tail -100f /u01/app/oracle/cfgtoollogs/sqlpatch/<latest_dir>/<latest>.log

# Convention B
ls -lrt /oracle/cfgtoollogs/sqlpatch/
tail -100f /oracle/cfgtoollogs/sqlpatch/<latest_dir>/<latest>.log

Expected output:

Connecting to database...OK
Bootstrapping registry and package to current versions...done
Determining current state...done
Adding patches to installation queue...done
Installing patches...
Patch installation complete. Total patches installed: 1
Validating logfiles...done
Patch <PATCHNUM> apply: SUCCESS
datapatch: ALL PATCHES APPLIED SUCCESSFULLY.

⚠️ IMPORTANT: If datapatch fails — check the log for specific errors. Common issues:

• Database not fully open — confirm open_mode = READ WRITE
• TEMP tablespace full — extend it
• INVALID objects blocking compilation — run @?/rdbms/admin/utlrp.sql first then rerun datapatch

6.5 — Verify datapatch Applied Successfully

-- On NEW PRIMARY (dbserver02)
sqlplus / as sysdba

set linesize 200
set pagesize 100
col patch_id    for 9999999999
col version     for a15
col status      for a15
col description for a55
col action_time for a25

SELECT patch_id, version, status, description, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

What to look for: Patch number shows with SUCCESS status.

6.6 — Check for INVALID Objects on New Primary

-- On NEW PRIMARY
set linesize 180
set pagesize 100
col owner       for a20
col object_name for a45
col object_type for a25
col status      for a10

SELECT owner, object_name, object_type, status
FROM   dba_objects
WHERE  status = 'INVALID'
ORDER BY owner, object_type, object_name;

SELECT COUNT(*) post_patch_invalid_count
FROM   dba_objects
WHERE  status = 'INVALID';

-- Recompile if count is higher than pre-patch baseline
@?/rdbms/admin/utlrp.sql

-- Re-verify
SELECT COUNT(*) FROM dba_objects WHERE status = 'INVALID';

6.7 — Verify All Components Are VALID on New Primary

set linesize 200
set pagesize 100
col comp_name for a50
col version   for a15
col status    for a12

SELECT comp_name, version, status
FROM   dba_registry
ORDER BY comp_name;

7. Phase 3 — Patch Old Primary Oracle Home (Now Standby)

📝 What happens in Phase 3? The old primary (dbserver01) is now running as the standby database. Its Oracle Home is still on the OLD patch level. We now patch it to the same patch level as the new primary. This is done while the new primary continues serving applications — zero additional downtime.

⚠️ IMPORTANT: All steps in Phase 3 are performed on dbserver01 (OLD PRIMARY — NOW STANDBY) unless stated otherwise.

7.1 — Verify New Standby (Old Primary) is Running and Receiving Redo

-- On dbserver01 (NEW STANDBY — old primary)
sqlplus / as sysdba

set linesize 200
set pagesize 50
col name           for a12
col db_unique_name for a15
col open_mode      for a15
col database_role  for a20

-- Confirm it is now a standby
SELECT name, db_unique_name, open_mode, database_role
FROM   v$database;

-- Check MRP is running and applying redo
set linesize 180
set pagesize 50
col process  for a12
col status   for a20
col sequence# for 999999

SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

7.2 — Stop MRP on New Standby (Old Primary)

-- On dbserver01 (NEW STANDBY)
sqlplus / as sysdba

-- Stop MRP before patching
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

-- Confirm MRP stopped
SELECT process, status FROM v$managed_standby WHERE process = 'MRP0';
-- Should return no rows

7.3 — Apply Patch to Old Primary Oracle Home (Now Standby)

# On dbserver01 (NEW STANDBY) as oracle user
su - oracle

# Navigate to patch directory
cd /stage/patches/unzipped/<PATCHNUM>

# Apply patch to Oracle Home
$ORACLE_HOME/OPatch/opatch apply \
    -oh $ORACLE_HOME \
    -silent

# Monitor progress
tail -100f /tmp/OPatch/opatch_<date>_<time>.log

Expected output:

Patching component oracle.rdbms, 19.x.x.x.x...
Patch <PATCHNUM> successfully applied.
OPatch succeeded.

7.4 — Verify Patch Applied Successfully on Old Primary Home

# On dbserver01 (NEW STANDBY)
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch "

7.5 — Restart MRP on New Standby

-- On dbserver01 (NEW STANDBY)
sqlplus / as sysdba

-- Restart MRP with new patched binaries
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE
    DISCONNECT FROM SESSION;

-- Verify MRP is running
set linesize 180
set pagesize 50
col process  for a12
col status   for a20
col sequence# for 999999

SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

7.6 — Wait for New Standby to Catch Up

-- On dbserver01 (NEW STANDBY)
-- Monitor catch-up progress — run repeatedly
set linesize 200
set pagesize 50
col name  for a30
col value for a40
col unit  for a20

SELECT name, value, unit
FROM   v$dataguard_stats
WHERE  name IN ('transport lag','apply lag')
ORDER BY name;

⚠️ IMPORTANT: Wait for apply lag to reach 0 before performing any further operations or declaring patching complete.

8. Optional — Switchover Back to Original Primary

📝 Why switchover back? After patching, some teams prefer to return the database to its original primary server (dbserver01). This is entirely optional — there is no technical reason the database cannot permanently run on dbserver02 as primary. Decide based on your team’s preference and documentation.

📝 When to do this? Only after Phase 3 is complete and the new standby (old primary) has fully caught up. Both Oracle Homes are now on the same patch level so the switchover is completely clean.

-- Connect to Broker on NEW PRIMARY (dbserver02)
dgmgrl sys/Oracle_123@ORCL_STBY

-- Verify configuration is fully healthy
DGMGRL> SHOW CONFIGURATION;
DGMGRL> VALIDATE DATABASE 'ORCL';
-- Ready for Switchover: Yes

-- Switchover back to original primary
DGMGRL> SWITCHOVER TO 'ORCL';

-- Verify configuration after switchover back
DGMGRL> SHOW CONFIGURATION;

Expected output after switchover back:

Configuration - ORCL_DG_CONFIG

  Protection Mode: MaxPerformance
  Members:
  ORCL      - Primary database
  ORCL_STBY - Physical standby database

Configuration Status:
SUCCESS

DGMGRL> EXIT;

9. Post-Patching Checks — Both Servers

⚠️ IMPORTANT: Run all post-patch checks after Phase 3 completes (and after optional switchback if performed). These checks confirm both Oracle Homes are on the same patch level and Data Guard is fully healthy.

9.1 — Confirm Patch Level is Same on Both Servers

# On PRIMARY (whichever server is primary now)
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory | grep -E "Oracle Database|Patch "

# On STANDBY
ssh oracle@dbserver02 \
    "$ORACLE_HOME/OPatch/opatch lsinventory | grep -E 'Oracle Database|Patch '"

Both must show the SAME patch number.

9.2 — Verify Data Guard Broker Configuration is Healthy

dgmgrl sys/Oracle_123@ORCL

DGMGRL> SHOW CONFIGURATION;
-- Must show: SUCCESS

DGMGRL> SHOW DATABASE VERBOSE 'ORCL';
DGMGRL> SHOW DATABASE VERBOSE 'ORCL_STBY';

DGMGRL> SHOW DATABASE 'ORCL_STBY' 'ApplyLagSecs';
DGMGRL> SHOW DATABASE 'ORCL_STBY' 'TransportLagSecs';

DGMGRL> VALIDATE DATABASE 'ORCL_STBY';

DGMGRL> EXIT;

9.3 — Check Database Roles and Open Mode on Both Servers

-- On PRIMARY
sqlplus / as sysdba

set linesize 200
set pagesize 50
col name            for a12
col db_unique_name  for a15
col open_mode       for a15
col database_role   for a20
col protection_mode for a22

SELECT name, db_unique_name, open_mode,
       database_role, protection_mode
FROM   v$database;

-- On STANDBY
sqlplus / as sysdba

SELECT name, db_unique_name, open_mode, database_role
FROM   v$database;

9.4 — Verify New Version on Both Servers

-- On PRIMARY
sqlplus / as sysdba

set linesize 150
set pagesize 50
col banner for a80

SELECT banner FROM v$version WHERE banner LIKE 'Oracle%';

SELECT version, version_full FROM v$instance;

-- On STANDBY (standby must be in MOUNT or READ ONLY state to query)
-- Use v$instance which is always accessible
SELECT version, version_full FROM v$instance;

9.5 — Verify datapatch Applied Successfully

-- On PRIMARY only
set linesize 200
set pagesize 100
col patch_id    for 9999999999
col version     for a15
col status      for a15
col description for a55
col action_time for a25

SELECT patch_id, version, status, description, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;

9.6 — Check MRP Status and Apply Lag on Standby

-- On STANDBY
set linesize 200
set pagesize 50
col process  for a12
col status   for a20
col sequence# for 999999

SELECT process, status, sequence#
FROM   v$managed_standby
WHERE  process = 'MRP0';

set linesize 200
col name  for a30
col value for a40
col unit  for a20

SELECT name, value, unit
FROM   v$dataguard_stats
WHERE  name IN ('transport lag','apply lag')
ORDER BY name;

9.7 — Check Archive Destination on Primary is Error-Free

-- On PRIMARY
set linesize 200
set pagesize 50
col dest_id   for 999
col dest_name for a20
col status    for a12
col error     for a50

SELECT dest_id, dest_name, status, error
FROM   v$archive_dest
WHERE  dest_id IN (1,2)
ORDER BY dest_id;

What to look for: Both destinations must show VALID with empty error column.

9.8 — Check for Archive Log Gaps on Standby

-- On STANDBY
set linesize 150
set pagesize 50
col low_sequence#  for 999999
col high_sequence# for 999999

-- Should return no rows after successful patching
SELECT thread#, low_sequence#, high_sequence#
FROM   v$archive_gap;

9.9 — Check All Datafiles Online on Primary

-- On PRIMARY
set linesize 200
set pagesize 100
col file#           for 999
col tablespace_name for a25
col status          for a12
col name            for a70

SELECT file#, tablespace_name, status, name
FROM   v$datafile
ORDER BY file#;

9.10 — Check Invalid Objects on Primary

sql

-- On PRIMARY
set linesize 180
set pagesize 100
col owner       for a20
col object_name for a45
col object_type for a25

SELECT owner, object_name, object_type, status
FROM   dba_objects
WHERE  status = 'INVALID'
ORDER BY owner, object_type, object_name;

SELECT COUNT(*) final_invalid_count
FROM   dba_objects
WHERE  status = 'INVALID';

9.11 — Check All Components Are VALID on Primary

set linesize 200
set pagesize 100
col comp_name for a50
col version   for a15
col status    for a12

SELECT comp_name, version, status
FROM   dba_registry
ORDER BY comp_name;

9.12 — Check Alert Logs on Both Servers

# PRIMARY alert log
tail -300 /u01/app/oracle/diag/rdbms/orcl/ORCL/trace/alert_ORCL.log \
     | grep -E "ORA-|Error|WARNING|Gap|ARC|MRP|switchover"

# Convention B primary alert log
tail -300 /oracle/diag/rdbms/orcl/ORCL/trace/alert_ORCL.log \
     | grep -E "ORA-|Error|WARNING|Gap|ARC|MRP|switchover"

# STANDBY alert log
ssh oracle@dbserver02 \
    "tail -300 /u01/app/oracle/diag/rdbms/orcl_stby/ORCL_STBY/trace/alert_ORCL_STBY.log \
     | grep -E 'ORA-|Error|WARNING|Gap|MRP|apply'"

# Convention B standby alert log
ssh oracle@dbserver02 \
    "tail -300 /oracle/diag/rdbms/orcl_stby/ORCL_STBY/trace/alert_ORCL_STBY.log \
     | grep -E 'ORA-|Error|WARNING|Gap|MRP|apply'"

9.13 — End-to-End Redo Flow Verification

📝 Why? Force a log switch on the new primary and verify the archivelog is received and applied on the standby. This confirms the complete redo transport pipeline is working end-to-end with the new patched binaries on both sides.

-- On PRIMARY — force log switch
sqlplus / as sysdba
ALTER SYSTEM SWITCH LOGFILE;
ALTER SYSTEM ARCHIVE LOG CURRENT;

-- Note the sequence number
SELECT thread#, max(sequence#) last_archived
FROM   v$archived_log
WHERE  dest_id = 1
GROUP BY thread#;

-- On STANDBY — wait 60 seconds then verify same sequence applied
SELECT thread#, max(sequence#) last_applied
FROM   v$log_history
GROUP BY thread#;

-- Sequence numbers must match

9.14 — Save Post-Patch Inventory from Both Servers

# Save final inventory from PRIMARY
su - oracle
$ORACLE_HOME/OPatch/opatch lsinventory \
    > /tmp/post_patch_PRIMARY_$(date +%Y%m%d).txt

# Save final inventory from STANDBY
ssh oracle@dbserver02 \
    "$ORACLE_HOME/OPatch/opatch lsinventory" \
    > /tmp/post_patch_STANDBY_$(date +%Y%m%d).txt

# Compare pre vs post for both servers
diff /tmp/pre_patch_PRIMARY_$(date +%Y%m%d).txt \
     /tmp/post_patch_PRIMARY_$(date +%Y%m%d).txt

diff /tmp/pre_patch_STANDBY_$(date +%Y%m%d).txt \
     /tmp/post_patch_STANDBY_$(date +%Y%m%d).txt

10. Patch Rollback Procedure for Data Guard

📝 When to use? If the patch causes critical issues after completing all phases. Rollback in DG is more complex because you need to rollback on both servers and coordinate the DG configuration.

📝 Rollback sequence:

1. Run datapatch rollback on current PRIMARY
2. Switchover back to original primary (if switched)
3. Stop MRP on standby
4. OPatch rollback on standby Oracle Home
5. Restart MRP on standby
6. OPatch rollback on primary Oracle Home
7. Verify DG is healthy

10.1 — Run datapatch Rollback on Current Primary

su - oracle
cd $ORACLE_HOME/OPatch

# Rollback datapatch changes from data dictionary
./datapatch -verbose -rollback <PATCHNUM>

10.2 — Stop MRP on Standby

-- On STANDBY
sqlplus / as sysdba
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE CANCEL;

10.3 — OPatch Rollback on Standby Oracle Home

# On STANDBY as oracle user
su - oracle

$ORACLE_HOME/OPatch/opatch rollback \
    -id <PATCHNUM> \
    -oh $ORACLE_HOME \
    -silent

10.4 — Restart MRP on Standby After Rollback

-- On STANDBY
sqlplus / as sysdba
ALTER DATABASE RECOVER MANAGED STANDBY DATABASE
    USING CURRENT LOGFILE DISCONNECT FROM SESSION;

10.5 — OPatch Rollback on Primary Oracle Home

# On PRIMARY as oracle user
su - oracle

$ORACLE_HOME/OPatch/opatch rollback \
    -id <PATCHNUM> \
    -oh $ORACLE_HOME \
    -silent

10.6 — Verify Rollback and DG Health

# Verify patch is removed from both homes
$ORACLE_HOME/OPatch/opatch lsinventory | grep <PATCHNUM>
# Should return no output

ssh oracle@dbserver02 \
    "$ORACLE_HOME/OPatch/opatch lsinventory | grep <PATCHNUM>"
# Should return no output

-- Verify Broker configuration is healthy after rollback
dgmgrl sys/Oracle_123@ORCL

DGMGRL> SHOW CONFIGURATION;
DGMGRL> SHOW DATABASE 'ORCL_STBY' 'ApplyLagSecs';
DGMGRL> EXIT;

-- Verify datapatch rollback in data dictionary
sqlplus / as sysdba

SELECT patch_id, status, action_time
FROM   dba_registry_sqlpatch
ORDER BY action_time DESC;
-- Patch should show ROLLED BACK status

11. Quick Reference Card

This SOP covers everything you need to patch Oracle Data Guard 19c on Linux using the standby-first rolling approach without referring to any other source. Always verify DG is fully healthy before starting, always patch standby first, always run datapatch only on the primary after all switchovers are complete, and always verify end-to-end redo flow after patching both Oracle Homes.