[ons-secure] Connection OHS SSL handshake failed

Posted byIndraneil SealPosted inOracle EBusiness SuiteTags:,

Any inbound Connections to an enterprise application like EBS (user accessing the application through Oracle Forms or mobile phone communications via a REST service) follows the below set of steps before a successful communication(SSL/TLS handshake) is established.

Below is a simplified diagram depicting the steps

What could go wrong if the certificates are expired?

SSL Handshake could fail due to expired certificates. CA signed certificates expire every year and have to be renewed annually preventing outages associated with expired certs. Certificates could be either In-House and Self-Signed Certificates or CA signed certificates.

Last week during one such certificate renewal activity, I encountered a weird issue which at this point seems undocumented. Hence, I decided to write about it.

Oracle has simplified the maintenance of wallets in EBS R12.2 as per the latest version of the Doc ID 2143101.1 (Enabling SSL or TLS in Oracle E-Business Suite Release 12.2). Unlike previously, we need to update the CA signed certificate only in the following current wallet location:

$FMW_HOME/webtier/instances/<s_ohs_instance>/config/OHS/<s_ohs_component>/keystores/default

And create/copy a self-sign certificate for the OPMN Remote Port and OHS Admin Port (used for FMW internal communication) in the following locations:

$FMW_HOME/webtier/instances/config/OHS//proxy-wallet
$FMW_HOME/webtier/instances/config/OPMN/opmn/wallet
$EBS_DOMAIN_HOME/opmn/wallet
$EBS_DOMAIN_HOME/opmn/wallet

While adding self-signed cert in the above locations, I noticed the Apache was unable to start.

[applmgr@appsnode scripts]$ ./adopmnctl.sh startall
You are running adopmnctl.sh version 120.0.12020000.2
Starting Apache…
EXIT CODE is 152. Please check the log file for more details.
adopmnctl.sh: exiting with status 152
adopmnctl.sh: check the logfile /ebs/APPS/R122/fs1/inst/apps/APPS_appsnode/logs/appl/admin/log/adopmnctl.txt for more information …

The following log gave a bit more details.

/ebs/APPS/R122/fs1/FMW_Home/webtier/instances/EBS_web_OHS2/diagnostics/logs/OPMN/opmn/opmn.log
[2023–01–07T21:47:56–05:00] [opmn] [ERROR:1] [] [ons-secure] Connection OHS SSL handshake failed (29024)
[2023–01–07T21:47:58–05:00] [opmn] [ERROR:1] [] [ons-secure] Connection OHS SSL handshake failed (29024)
[2023–01–07T21:48:00–05:00] [opmn] [ERROR:1] [] [ons-secure] Connection OHS SSL handshake failed (29024)

I decided to copy the CA signed cert instead and finally was able to bring up the apache.

During my research I found that due to a bug the internal opmn communication to remote port 6200 which uses the self-signed certificate generated by FMW was failing.

34067016 — POST MES4.5 UPG NZOS_HANDSHAKE DOES NOT SEND CERTIFICATE CA NAMES WHEN REQUESTING MUTUAL_AUTH

This is resolved via Patch 34067016(part of Oct 2022 CPU) on webtier. Once the fix is applied, we have to copy the self-signed certificates in the Oracle recommended locations & recycle the EBS httpd service.

This has already been implemented in a lower environment Dev and it is working fine.

To validate that the patch worked in DEV( vs Prod where no patch is applied), I ran below tests.

*** DEV (patch 34067016 was applied) ***

[applmgr@devappsnode ~]$ openssl s_client -connect localhost:6201 2>&1 | grep client

Acceptable client certificate CA names <== (requests a client certificate meaning it works !!)

*** PROD( patch 34067016 was NOT applied) ***

[applmgr@prodappsnode~]$ openssl s_client -connect localhost:6200 2>&1 | grep client

No client certificate CA names sent <== (it does not request a client certificate)

Below are some references:

  • Fusion Middleware Control Shows Oracle HTTP Server in Incorrect Down Status With 29024/29019 & Optic Startphonecall Failed After Applying OSS Patch 32287205 (Doc ID 2868698.1)
  • ** UPDATED** — After OCT 2021 CPU or Later: Forms Getting Disconnected With [ FRM-92102 : A network error has occurred ] [ javax.net.ssl.SSLException: javax.crypto.AEADBadTagException: Tag mismatch! ] (Doc ID 2836058.1)
  • After Patching E-Business FMW With CPU October 2021 or Later, Both OAM and Grid EM Console Shows OHS As Down (Doc ID 2884854.1)

Published by Indraneil Seal

I originally hail from Kolkata, India, and I've dedicated a significant portion of my professional journey to both India and the United States before relocating to Canada during the pandemic. I’m a member of MongoDB's esteemed Technical Services team. Before joining this exceptional group, I held the role of Senior Apps DBA at the Government of Ontario. Prior to that, I spent many years honing my technical(DBA/Cloud) and soft skills with TCS, KBACE Technologies(which later got acquired by Cognizant Technology Solutions) and Oracle Corporation. Throughout my career, I was deeply immersed in day-to-day operations and spearheaded significant projects, including the modernization of platforms, Oracle application and database upgrades. In my last stint at the Government of Ontario, I was also responsible for overseeing various automation initiatives including out-of-place patching, automated EBS Application patching. Outside of my professional life, I have a strong passion for reading, blogging, spending quality time with my family, and my feline buddies, Smokey & Louis. I also relish solitary walks and jogging as personal interests. As a proponent of open source technologies, I'm looking forward to sharing my knowledge and expertise as well as contribute as much as possible to the success of the IT fraternity thereby perpetually expanding my skillset.

Leave a comment