Database PSU via opatchauto failing in production

Posted byIndraneil SealPosted inUncategorized

Problem Summary:
We were in the middle of July release PSU patches using opatchauto in our EBS oracle home(our grid and DB are both on 19.10 RU), when It failed with multiple errors in the opatch log.

Errors:
2021-08-21 15:04:59,783 INFO [1] oracle.glcm.opatch.common.impl.SingletonPatch – /stage/PATCHES/PSUJUL2021/32900083/32895426/32904851/etc/config/checksum.xml does not exist
2021-08-21 15:04:59,783 INFO [1] oracle.glcm.opatch.common.impl.SingletonPatch – /stage/PATCHES/PSUJUL2021/32900083/32895426/32904851/etc/config/automation.xml does not exist
2021-08-21 15:04:59,783 INFO [1] oracle.glcm.opatch.common.impl.SingletonPatch – Patch Deploy artifact does not exist
2021-08-21 15:04:59,783 INFO [1] oracle.glcm.opatch.common.impl.SingletonPatch – /stage/PATCHES/PSUJUL2021/32900083/32895426/32904851/etc/config/maintenance_alias.xml does not exist
2021-08-21 15:05:27,167 INFO [1] oracle.opatchauto.core.binary.OACLogger –
Applying /stage/PATCHES/PSUJUL2021/32900083/32895426/32904851 …
2021-08-21 15:05:32,121 FINE [1] oracle.opatch.util.perfmonitor.MethodTrackerObj – Error while ending the tracker. Message – null

==Following patches FAILED in apply:

Patch: /stage/PATCHES/PSUJUL2021/32900083/32895426/32904851
Log: /u01/app/oracle/PROD/db/tech_st/19c/cfgtoollogs/opatchauto/core/opatch/opatch2021-08-21_15-04-22PM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target “client_sharedlib”.
Re-link fails on target “client_sharedlib”.
Re-link fails on target “ldapsearch”.

Issue & resoluton:

After digging through the log (/u01/app/oracle/PROD/db/tech_st/19c/cfgtoollogs/opatchauto/core/opatch/opatch2021-08-21_15-04-22PM_1.log), we found that re-linking was failing with below message –

Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Re-link fails on target “client_sharedlib”.
Re-link fails on target “client_sharedlib”.
Re-link fails on target “ldapsearch”.

On digging further, we found out that the file $ORACLE_HOME/network/admin/shrept.lst was missing on both EBS RAC nodes causing the relinking to fail and subsequently the entire opatchauto failed.

Subsequently, we followed below steps to resolve the issue:

Step 1. Recreated the file $ORACLE_HOME/network/admin/shrept.lst manually on both DB nodes

Step 2. Copy the folder $ORACLE_HOME/jdk from lower env to PROD(as patch session was removing the files from $ORACLE_HOME/jdk/bin everytime patch failed)

Step 3. Manually ran ‘relink all’ command.

Step 4. As root, run – opatchauto resume

Published by Indraneil Seal

I originally hail from Kolkata, India, and I've dedicated a significant portion of my professional journey to both India and the United States before relocating to Canada during the pandemic. I’m a member of MongoDB's esteemed Technical Services team. Before joining this exceptional group, I held the role of Senior Apps DBA at the Government of Ontario. Prior to that, I spent many years honing my technical(DBA/Cloud) and soft skills with TCS, KBACE Technologies(which later got acquired by Cognizant Technology Solutions) and Oracle Corporation. Throughout my career, I was deeply immersed in day-to-day operations and spearheaded significant projects, including the modernization of platforms, Oracle application and database upgrades. In my last stint at the Government of Ontario, I was also responsible for overseeing various automation initiatives including out-of-place patching, automated EBS Application patching. Outside of my professional life, I have a strong passion for reading, blogging, spending quality time with my family, and my feline buddies, Smokey & Louis. I also relish solitary walks and jogging as personal interests. As a proponent of open source technologies, I'm looking forward to sharing my knowledge and expertise as well as contribute as much as possible to the success of the IT fraternity thereby perpetually expanding my skillset.

Leave a comment