JVM is not reachable with jstat and jstack
Summary
Generating Jira thread dumps using "jstack" returns error "Unable to open socket file"
Environment
Jira Software/Service Management running on Linux hosts with Java versions 8 or 11
Diagnosis
Trying to generate Jira thread dumps using jstack, the command returns the following error:
$ sudo -u jira /usr/lib/jvm/jdk-11.0.2+7/bin/jstack 1455
1455: Unable to open socket file /proc/1455/root/tmp/.java_pid1455: target process 1455 doesn't respond within 10500ms or HotSpot VM not loadedThe path "/proc/<PID>/root" is a symbolic link to the root path ( / )
ls -l /proc/<PID>/root
lrwxrwxrwx 1 jira jira 0 dez 3 18:29 /proc/<PID>/root -> /So the path "/proc/<PID>/root/tmp/" is pointing to "/tmp" to store temporary files for the processes.
Checking the "/tmp", the Java socket file was not present:
ls -l /tmp/.java_pid<PID>
ls: cannot access '/tmp/.java_pid<PID>': No such file or directory Cause
There are some external factors that can cause the socket file to be deleted from "/tmp":
- External routine tools to clean files older than X days;
- Linux
"systemd-tmpfiles-clean"removing Java Hooks
(More information can be found in Red Hat Bugzilla – Bug 1704608 but this is not limited to RedHat) - Manual removal of the files from
"/tmp"
Solution
1 - If any external tool is being used to delete old files from "/tmp", configure the tool to ignore the patterns below:
/tmp/hsperfdata_*
/tmp/.java_pid*2 - If it is found to be the Linux Systemd Cleanup task deleting the files, do the following:
Edit the file "/usr/lib/tmpfiles.d/tmp.conf" and add the 2 lines below:
(The "x" in both lines are to indicate that the patterns must be excluded from the cleanup task)
# Exclude java-hooks
x /tmp/hsperfdata_*
x /tmp/.java_pid*Workaround
If the socket file was deleted and for a troubleshooting situation it is necessary to generate thread dumps before restarting the application, the command below can be used to send thread dumps to "Catalina.out" log.
See more details in the document Generating a thread dump under the section "Linux/Unix/OS X environment" > "Linux/Unix command line"
kill -3 <PID>Additional Information
Additional information can be found in the following links:
HotSpot Dynamic Attach Mechanism
Why jstack not working when the /tmp/.java_pid<num> socket file has been deleted
Target process xxxx doesn't respond within 10500ms or HotSpot VM not loaded
Red Hat Bugzilla – Bug 1704608 - systemd-tmpfiles-clean deletes /tmp/hsperfdata_*