Use Jmap command:
jmap.exe -dump:format=b,file=HeapDump.hprof <pid>, pid can be found from system task manager or ps(linux)
Generate automatically when Java heap space out of memory
Add JVM options, -XX:-HeapDumpOnOutOfMemoryError
Where the file loaction?
After that, you can use visualVM or MAT(Eclipse), or other tools to open it.
VisualVM heap memory setting
on Windows, if you want to increase the memory for the Visual VM tool, you must can do it from in the file
%JDK_HOME%\lib\visualvm\etc\visualvm.conf
changing the default_options property