JVM 崩溃分析
首先查看 syslog（可以看到是否被系统kill等其他信息） 发现没权限...
只有crash日志，那从上往下如下
错误的概要介绍 分析 ------
SIGSEGV (0xb) at pc=0x00007fce0a3e3bcf, pid=9562, tid=140508990830336
其中 重要的几个参数
1)错误的信号类型
这里是 SIGSEGV
表示jvm crash时正在执行jni代码，而不是在执行java或者jvm的代码，如果没有在应用程序里手动调用jni代码，那么很可能是JIT动态编译时导致的该错误
（常见的还有其他几种,比如“EXCEPTION_ACCESS_VIOLATION”，该描述表示jvm crash时正在执行jvm自身的代码，这往往是因为jvm的bug导致的crash；另一种常见的描述是“EXCEPTION_STACK_OVERFLOW”，该描述表示这是个栈溢出导致的错误，这往往是应用程序中存在深层递归导致的。)
其他几个是一些地址和进程id，在这里对我们帮助不大~
2)Problematic frame里面比较重要的
V [libjvm.so+0x77cbcf]
这里前缀是大V。指在vmframe.所以应该和jvm代码有关。还有其他几种 解释如下
C: Native C frame
j: Interpreted Java frame
V: VMframe
v: VMgenerated stub frame
J: Other frame types, including compiled Java frames
这边如果是J ，就是JIT动态编译导致的该错误。
查阅资料发现：此异常是由于jdk JIT compiler optimization 导致，bug id 8021898
libjvm.so+0x77cbcf 这个指执行 在 libjvm.so 库 + 偏移量 的位置， 和前面的 pc=0x00007fce0a3e3bcf 一样，只是表示方式不一样而已(所以这里可以通过
objectdump -d -C libjvm.so >> jvmsodisass.dump
查找偏移地址 0x77cbcf ， 就是当时的执行的汇编，然后结合上下文，源码推测出问题的语句。比较麻烦，我没有采取这种方式定位
Thread 线程分析-------
比较重要的在 JavaThread "ChannelEventProcessor-thread-22222" daemon [_thread_in_vm, id=8752, stack(0x00007fcacc6c7000,0x00007fcacc7c8000)]
中，_thread_in_vm的含义是 线程正在执行虚拟机代码
可以有其他类型，如下
_thread_in_native：线程当前状态，状态枚举包括：
_thread_uninitialized：线程还没有创建，它只在内存原因崩溃的时候才出现
_thread_new：线程已经被创建，但是还没有启动
_thread_in_native：线程正在执行本地代码，一般这种情况很可能是本地代码有问题
_thread_in_vm：线程正在执行虚拟机代码
_thread_in_Java：线程正在执行解释或者编译后的Java代码
_thread_blocked：线程处于阻塞状态
…_trans：以_trans结尾，线程正处于要切换到其它状态的中间状态
在Instructions 里面（栈顶程序计数器旁的操作码），可以使用 udis库里带的udcli工具来反汇编
echo 'ff 48 63 c2 55 48 01 c6 48 89 e5 0f b7 06 0f b6' | udcli -intel -x -64 -o 0x00007fce0a3e3baf
也会显示相关的执行语句~ 特麻烦...可以忽略这点
再往下 Stack: [0x00007fcacc6c7000,0x00007fcacc7c8000], sp=0x00007fcacc7c6110, free space=1020k
这边比较重要的是 free space ， 如果比较小 ，比如2k，4k 。那就可能存在栈溢出
前面种种情况分析大概知道是 jvm应该不是执行普通java代码crash的
再往下看
VM state:synchronizing (normal execution)
这行比较重要
虚拟机状态。包括：
not at a safepoint：正常运行状态；
at safepoint：所有线程都因为虚拟机等待状态而阻塞，等待一个虚拟机操作完成；
synchronizing：一个特殊的虚拟机操作，要求虚拟机内的其它线程保持等待状态。
这里看到最重要的jvm正在执行一个特殊的操作。而且stop the world
再往下看Heap 内容
concurrent-mark-sweep perm gen total 327680K, used 327678K [0x00000007ec000000, 0x0000000800000000, 0x0000000800000000)
这行可以看到 perm gen 已经使用率很接近100%了，只差2k。看到cms 收集器正在收集 永久待，查看 cms 确实会在某些阶段stop the world 符合前面vm的状态
所以大概猜测到 在permsize空间几乎耗完时候，回收压缩perm空间出错导致的crash。
这里的建议，适当增加-XX:MaxPermSize=512M ，之前为320M
但是除了该种情况，还有几种原因可能导致这种类型的crash
1) jre bug 。实际上就算是 perm gen溢出 也应该是jre bug，应该 抛出 exception 而不是 直接crash
2) 物理硬件出错。比如Ram正好该位置损坏
其他建议。增加permsize 后再崩溃，系统可以配合做一个coredump，这样能更好的分析~
下面是 crash日志,因为是线上环境，所以系统环境和其他敏感信息已经去除，另外process内部线程信息因为过多而且有些敏感所以也已去除
#
A fatal error has been detected by the Java Runtime Environment:
#
SIGSEGV (0xb) at pc=0x00007fce0a3e3bcf, pid=9562, tid=140508990830336
#
JRE version: 6.0_26-b03
Java VM: Java HotSpot(TM) 64-Bit Server VM (20.1-b02 mixed mode linux-amd64 compressed oops)
Problematic frame:
V [libjvm.so+0x77cbcf] Rewriter::rewrite_member_reference(unsigned char*, int)+0x1f
#
If you would like to submit a bug report, please visit:
http://java.sun.com/webapps/bugreport/crash.jsp
#
--------------- T H R E A D ---------------
Current thread (0x00007fcca0577000): JavaThread "ChannelEventProcessor-thread-22222" daemon [_thread_in_vm, id=8752, stack(0x00007fcacc6c7000,0x00007fcacc7c8000)]
siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x00007fcca05b77bc
Registers:
RAX=0x000000000000ffff, RBX=0x00000000000000b7, RCX=0x00007fcca05777c0, RDX=0x000000000000ff00
RSP=0x00007fcacc7c6110, RBP=0x00007fcacc7c6110, RSI=0x00000007fffff6f3, RDI=0x00007fcacc7c6230
R8 =0x000000000000ffff, R9 =0x0000000000000001, R10=0x000000000000000c, R11=0x00000007fffff560
R12=0x00000007fffff6f2, R13=0x0000000000000003, R14=0x0000000000000002, R15=0x0000000000000000
RIP=0x00007fce0a3e3bcf, EFLAGS=0x0000000000010206, CSGSFS=0x0000000000000033, ERR=0x0000000000000004
TRAPNO=0x000000000000000e
Top of Stack: (sp=0x00007fcacc7c6110)
0x00007fcacc7c6110: 00007fcacc7c61b0 00007fce0a3e3f98
0x00007fcacc7c6120: 00007fcca02324e0 00000006a0577c20
0x00007fcacc7c6130: 00000007fffff6f0 00000000003e30ea
0x00007fcacc7c6140: 00000007fffff700 00007fcacc7c6230
0x00007fcacc7c6150: 00007fcacc7c61b0 00007fce0a3e35dc
0x00007fcacc7c6160: 00007fcacc7c6187 00000000a05777b0
0x00007fcacc7c6170: 00007fcacc7c6258 00007fcacc7c6248
0x00007fcacc7c6180: 01007fcca0577c20 0000000000000002
0x00007fcacc7c6190: 0000000000000001 0000000000000000
0x00007fcacc7c61a0: 00007fcacc7c6230 00000007fffff570
0x00007fcacc7c61b0: 00007fcacc7c6210 00007fce0a3e4843
0x00007fcacc7c61c0: 0000000000000000 00007fcacc7c6230
0x00007fcacc7c61d0: 00007fcca0577000 00007fce09ed91a5
0x00007fcacc7c61e0: 00007fcca0577c20 00000007fffff430
0x00007fcacc7c61f0: 00007fcca0577c38 00007fcca0577000
0x00007fcacc7c6200: 00007fcacc7c6220 00007fcacc7c6380
0x00007fcacc7c6210: 00007fcacc7c62c0 00007fce0a3e41e6
0x00007fcacc7c6220: 00007fcca0577c48 00007fcca0577c40
0x00007fcacc7c6230: 00007fcca0577c38 00007fcca0577c48
0x00007fcacc7c6240: 00007fcca0577c40 00007fcc00000010
0x00007fcacc7c6250: 00007fcca05777c0 00007fca00000002
0x00007fcacc7c6260: 00007fcca0577800 00007fce00000008
0x00007fcacc7c6270: 00000000560baa00 00000000000cdc29
0x00007fcacc7c6280: 00007fcca02324e0 00007fcca05777b0
0x00007fcacc7c6290: 00007fcca05777c0 00007fcca0577b98
0x00007fcacc7c62a0: 00007fcacc7c6360 00007fcca0577c20
0x00007fcacc7c62b0: 00007fcca0577000 00007fcacc7c6310
0x00007fcacc7c62c0: 00007fcacc7c6400 00007fce0a11412b
0x00007fcacc7c62d0: 00007fcacc7c62f0 00007fce0a0e40ed
0x00007fcacc7c62e0: 00007fcacc7c6320 01007fce00000000
0x00007fcacc7c62f0: 00007fcca0577c38 00007fce0a04f5f6
0x00007fcacc7c6300: 00007fcca0577c30 00007fcca0577c28
Instructions: (pc=0x00007fce0a3e3bcf)
0x00007fce0a3e3baf: ff 48 63 c2 55 48 01 c6 48 89 e5 0f b7 06 0f b6
0x00007fce0a3e3bbf: d0 0f b6 cc c1 e2 08 09 d1 4c 63 c1 48 8b 4f 20
0x00007fce0a3e3bcf: 42 8b 04 81 66 89 06 c9 c3 66 66 66 90 66 66 66
0x00007fce0a3e3bdf: 90 55 48 63 c2 48 89 e5 48 01 c6 41 57 41 56 41
Register to memory mapping:
RAX=0x000000000000ffff is an unknown value
RBX=0x00000000000000b7 is an unknown value
RCX=0x00007fcca05777c0 is an unknown value
RDX=0x000000000000ff00 is an unknown value
RSP=0x00007fcacc7c6110 is pointing into the stack for thread: 0x00007fcca0577000
RBP=0x00007fcacc7c6110 is pointing into the stack for thread: 0x00007fcca0577000
RSI=0x00000007fffff6f3 is an oop
{constMethod}
- klass: {other class}
- method: 0x00000007fffff700 {method} '
' '(Ljava/lang/String;)V' in 'java/lang/IllegalThreadStateException' - exceptions: 0x00000007ec001ea8
bci_from(0x7fffff6f3) = 3; print_codes():
RDI=0x00007fcacc7c6230 is pointing into the stack for thread: 0x00007fcca0577000
R8 =0x000000000000ffff is an unknown value
R9 =0x0000000000000001 is an unknown value
R10=0x000000000000000c is an unknown value
R11=0x00000007fffff560 is an oop
[[I - klass: 'java/lang/Object'[]
- length: 2
R12=0x00000007fffff6f2 is an oop
{constMethod} - klass: {other class}
- method: 0x00000007fffff700 {method} '
' '(Ljava/lang/String;)V' in 'java/lang/IllegalThreadStateException' - exceptions: 0x00000007ec001ea8
bci_from(0x7fffff6f2) = 2; print_codes():
R13=0x0000000000000003 is an unknown value
R14=0x0000000000000002 is an unknown value
R15=0x0000000000000000 is an unknown value
Stack: [0x00007fcacc6c7000,0x00007fcacc7c8000], sp=0x00007fcacc7c6110, free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0x77cbcf] Rewriter::rewrite_member_reference(unsigned char*, int)+0x1f
V [libjvm.so+0x77cf98] Rewriter::scan_method(methodOopDesc)+0x1f8
V [libjvm.so+0x77d843] Rewriter::Rewriter(instanceKlassHandle, constantPoolHandle, objArrayHandle, Thread)+0xf3
V [libjvm.so+0x77d1e6] Rewriter::rewrite(instanceKlassHandle, Thread)+0xe6
V [libjvm.so+0x4ad12b] instanceKlass::link_class_impl(instanceKlassHandle, bool, Thread)+0x6cb
V [libjvm.so+0x4ac9bc] instanceKlass::link_class(Thread)+0x4c
V [libjvm.so+0x4ad290] instanceKlass::initialize_impl(instanceKlassHandle, Thread)+0x30
V [libjvm.so+0x4ac91a] instanceKlass::initialize(Thread)+0x6a
V [libjvm.so+0x4d3dfc] InterpreterRuntime::_new(JavaThread, constantPoolOopDesc*, int)+0xbc
j java.lang.Thread.start()V+15
j org.jboss.netty.util.HashedWheelTimer.start()V+34
j org.jboss.netty.util.HashedWheelTimer.newTimeout(Lorg/jboss/netty/util/TimerTask;JLjava/util/concurrent/TimeUnit;)Lorg/jboss/netty/util/Timeout;+45
j org.summercool.hsf.util.HandshakeUtil.resetHandshakeTimeout(Lorg/summercool/hsf/netty/channel/HsfChannel;)V+58
j org.summercool.hsf.netty.listener.impl.AcceptorGroupMessageEventListener.channelConnected(Lorg/jboss/netty/channel/ChannelHandlerContext;Lorg/summercool/hsf/netty/channel/HsfChannel;Lorg/jboss/netty/channel/ChannelStateEvent;)Lorg/summercool/hsf/netty/listener/EventBehavior;+1
j org.summercool.hsf.netty.event.EventDispatcher$2.run()V+122
Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j java.lang.Thread.start()V+15
j org.jboss.netty.util.HashedWheelTimer.start()V+34
j org.jboss.netty.util.HashedWheelTimer.newTimeout(Lorg/jboss/netty/util/TimerTask;JLjava/util/concurrent/TimeUnit;)Lorg/jboss/netty/util/Timeout;+45
j org.summercool.hsf.util.HandshakeUtil.resetHandshakeTimeout(Lorg/summercool/hsf/netty/channel/HsfChannel;)V+58
j org.summercool.hsf.netty.listener.impl.AcceptorGroupMessageEventListener.channelConnected(Lorg/jboss/netty/channel/ChannelHandlerContext;Lorg/summercool/hsf/netty/channel/HsfChannel;Lorg/jboss/netty/channel/ChannelStateEvent;)Lorg/summercool/hsf/netty/listener/EventBehavior;+1
j org.summercool.hsf.netty.event.EventDispatcher$2.run()V+122
J java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Ljava/lang/Runnable;)V
J java.util.concurrent.ThreadPoolExecutor$Worker.run()V
j java.lang.Thread.run()V+11
v ~StubRoutines::call_stub
--------------- P R O C E S S ---------------
Other Threads:
0x00007fce04269000 VMThread [stack: 0x00007fcd99c78000,0x00007fcd99d79000] [id=9588]
0x00007fce04a57000 WatcherThread [stack: 0x00007fcd98ace000,0x00007fcd98bcf000] [id=9600]
VM state:synchronizing (normal execution)
VM Mutex/Monitor currently owned by a thread: ([mutex/lock_event])
[0x00007fce04007010] Threads_lock - owner thread: 0x00007fce04269000
[0x00007fce04007590] Heap_lock - owner thread: 0x00007fce04222800
Heap
par new generation total 1152000K, used 4054K [0x00000006ec000000, 0x0000000749c00000, 0x0000000749c00000)
eden space 768000K, 0% used [0x00000006ec000000, 0x00000006ec3f5808, 0x000000071ae00000)
from space 384000K, 0% used [0x0000000732500000, 0x0000000732500000, 0x0000000749c00000)
to space 384000K, 0% used [0x000000071ae00000, 0x000000071ae00000, 0x0000000732500000)
concurrent mark-sweep generation total 2658304K, used 1279402K [0x0000000749c00000, 0x00000007ec000000, 0x00000007ec000000)
concurrent-mark-sweep perm gen total 327680K, used 327678K [0x00000007ec000000, 0x0000000800000000, 0x0000000800000000)
Code Cache [0x00007fce01000000, 0x00007fce018f0000, 0x00007fce04000000)
total_blobs=3012 nmethods=2580 adapters=383 free_code_cache=41450560 largest_free_block=52480
« git遇到的问题 Java堆内内存分析 »
No Comments filed.