Recently, I am working on a Enovia PLM project and after the upgrade to 2013xE, we also upgrade apache httpd server to 2.4.6 and webagent 12.52.sp01.cr09. and after everything is on production, we found the apache server was not stable to process clients' request. and sometimes, "a ssl peer shutdown unexpectedly" happened etc, the apache server process keeps crashing and restarting. and the following illustrates how we diagnose the root cause of the problem.1. When our enoviaplm application is working, we often see the apache logs like below and after analysis, we can see that the apache processes crashed due to a exception[19/Aug/2019:05:50:17] [Info] [CA WebAgent IPC] [32754] [CSmSharedSegment::smalloc] Attached to shared memory segment 688134 using key 0x6947862b
[19/Aug/2019:05:50:17] [Information] SiteMinder Agent
SiteMinder agent is running.
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct null not valid
[Mon Aug 19 05:51:00.242945 2019] [core:notice] [pid 2886:tid 140178461149312] AH00052: child pid 32754 exit signal Aborted (6)
[19/Aug/2019:05:53:08] [Information] SiteMinder Agent
SiteMinder agent is enabled.
[19/Aug/2019:05:53:08] [Information] SiteMinder Agent
Configuration file path:
'/etc/httpd/conf/WebAgent.conf'.
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 2162710 using key 0xc847862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1736711 using key 0x6b47862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 622596 using key 0x6c47862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1933325 using key 0x6647862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1966094 using key 0x6747862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1998865 using key 0x6c47862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 786441 using key 0x6547862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 2031634 using key 0x6847862b
[19/Aug/2019:05:53:09] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 2064403 using key 0x6947862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1802249 using key 0x3247862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 655365 using key 0x6147862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1867787 using key 0x3347862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 720903 using key 0x6247862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1900556 using key 0x3447862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 753672 using key 0x6347862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSem::getSem] Attached to semaphore 1835018 using key 0x6a47862b
[19/Aug/2019:05:53:10] [Info] [CA WebAgent IPC] [678] [CSmSharedSegment::smalloc] Attached to shared memory segment 688134 using key 0x6947862b
[19/Aug/2019:05:53:10] [Information] SiteMinder Agent
SiteMinder agent is running.
terminate called after throwing an instance of 'std::logic_error'
what(): basic_string::_S_construct null not valid
[Mon Aug 19 05:54:48.465566 2019] [core:notice] [pid 2886:tid 140178461149312] AH00052: child pid 32627 exit signal Aborted (6)
[19/Aug/2019:05:55:26] [Information] SiteMinder Agent
SiteMinder agent is enabled.
[19/Aug/2019:05:55:26] [Information] SiteMinder Agent
Configuration file path:
'/etc/httpd/conf/WebAgent.conf'.
[19/Aug/2019:05:55:27] [Info] [CA WebAgent IPC] [914] [CSmSem::getSem] Attached to semaphore 2162710 using key 0xc847862b
[19/Aug/2019:05:55:27] [Info] [CA WebAgent IPC] [914] [CSmSem::getSem] Attached to semaphore 1736711 using key 0x6b47862b
2. We tried to enable the core dump for apache server according to the guide https://httpd.apache.org/dev/debugging.html (Debugging intermittent crashes) and when the crashed happened, you will see a crash core dump there in folder /tmp
add CoreDumpDirectory /tmp to httpd.conf
ulimit -c unlimited
3. we need a gdb tool to help analyze the core dump, use the following command sudo gdb /usr/sbin/httpd /tmp/systemd-private-9811289cf7d44396a1bbc7e776cafa01-httpd.service-SXVLFM/tmp/core.19620 and you will see the error stack below[enovia@esenoaqv66 (QA APP1) tmp]$ sudo gdb /usr/sbin/httpd /tmp/systemd-private-9811289cf7d44396a1bbc7e776cafa01-httpd.service-SXVLFM/tmp/core.19620GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-110.el7Copyright (C) 2013 Free Software Foundation, Inc.License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.There is NO WARRANTY, to the extent permitted by law. Type "show copying"and "show warranty" for details.This GDB was configured as "x86_64-redhat-linux-gnu".For bug reporting instructions, please see:<http://www.gnu.org/software/gdb/bugs/>
Reading symbols from /usr/sbin/httpdReading symbols from /usr/sbin/httpd(no debugging symbols found)done.(no debugging symbols found)done.[New LWP 19632][New LWP 19646][New LWP 19622][New LWP 19645][New LWP 19623][New LWP 19644][New LWP 19620][New LWP 19625][New LWP 19643][New LWP 19627][New LWP 19642][New LWP 19624][New LWP 19628][New LWP 19629][New LWP 19626][New LWP 19631][New LWP 19634][New LWP 19641][New LWP 19633][New LWP 19647][New LWP 19630][New LWP 19635][New LWP 19640][New LWP 19639][New LWP 19638][New LWP 19637][New LWP 19636][Thread debugging using libthread_db enabled]Using host libthread_db library "/lib64/libthread_db.so.1".Core was generated by `/usr/sbin/httpd -DFOREGROUND'.Program terminated with signal 6, Aborted.#0 0x00007f2ecbba2207 in raise () from /usr/lib64/libc.so.6Missing separate debuginfos, use: debuginfo-install httpd-2.4.6-90.el7.x86_64(gdb) bt#0 0x00007f2ecbba2207 in raise () from /usr/lib64/libc.so.6#1 0x00007f2ecbba38f8 in abort () from /usr/lib64/libc.so.6#2 0x00007f2ec911f7d5 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib64/libstdc++.so.6#3 0x00007f2ec911d746 in ?? () from /usr/lib64/libstdc++.so.6#4 0x00007f2ec911d773 in std::terminate() () from /usr/lib64/libstdc++.so.6#5 0x00007f2ec911d993 in __cxa_throw () from /usr/lib64/libstdc++.so.6#6 0x00007f2ec9172597 in std::__throw_logic_error(char const*) () from /usr/lib64/libstdc++.so.6#7 0x00007f2ec917e3c1 in char* std::string::_S_construct<char const*>(char const*, char const*, std::allocator<char> const&, std::forward_iterator_tag) () from /usr/lib64/libstdc++.so.6#8 0x00007f2ec917e778 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, std::allocator<char> const&) () from /usr/lib64/libstdc++.so.6#9 0x00007f2ecb447472 in ModifyLogTransaction(request_rec*) () from /app/CA/webagent/bin/libmod_sm24.so#10 0x000055cd5a3d3640 in ap_run_log_transaction ()#11 0x000055cd5a3d4738 in ap_read_request ()#12 0x000055cd5a3fa0de in ap_process_http_connection ()#13 0x000055cd5a3f2180 in ap_run_process_connection ()#14 0x00007f2ec09f0bd5 in worker_thread () from /etc/httpd/modules/mod_mpm_event.so#15 0x00007f2ecc144dd5 in start_thread () from /usr/lib64/libpthread.so.0#16 0x00007f2ecbc6ab3d in clone () from /usr/lib64/libc.so.6 4. We can see It is sso module problem and It crashed in method ModifyLogTransaction(request_rec*) () You can send the error logs together with the core dump to CA siteminder team for their analysis, but finally our sso team help resolved my problem by upgrading the webagent to 12.52.sp01.cr10
You can find the details in the url https://bz.apache.org/bugzilla/show_bug.cgi?id=63670