Apologies for the large SVG
I got inspired by a Devoxx talk about flame graphs and how they can visualize what is happening on a JVM process.
Getting a graph is actually quite simple. You only need a recent enough Java 8 JDK, a running subject JVM process running on linux, perf which is part of kernel utils in most distributions and a couple of simple profiling tools which are open source. Detailed infomation can be found in a blog post by Nitsan Wakart here http://psy-lob-saw.blogspot.fi/2017/02/flamegraphs-intro-fire-for-everyone.html
So what is in that SVG.
It is illustrating what was happening on a Scala app I'm running on a DigitalOcean pod sampled 100 times a second during a 40 second period. The bars describe call stacks and the topmost item is always the one running in the CPU. More details how to read the graph can be found here http://www.brendangregg.com/flamegraphs.html
In this case the stacks are divided to threads. The leftmost stuff (thread) contains a lot of GC operations although upon closer inspection you can see that is happening on other threads as well. Then right to that thread there are a lot of sample points spent on JVM methods related to JIT (at least i assume that from the method names which contain C1 and C2 keywords). As you can see quite a lot of the total samples were related to JIT compiling which most probably tells that the application was still warming up so to speak.
On the other hand when looking at the application code there is already some in-lining visible marked with aqua color. I'm not 100% sure on how to interpret those stacks though. I assume that the methods call on top of each other bounded by green (scala) method calls are in-lined to each other.
What can I learn from this flame graph? I'm certainly not an expert in JVM nor in performance tuning but some of my suspicions where I could improve the performance are visible. I could reduce the amount of disk IO by not reading the configs from disk each time. Also I could look into caching the results of XML transform operations which this particular application does a lot. In another graph where the treads are squashed together, that becomes more visible. Obviously I knew those operations are relatively slow beforehand and also i could have just used visualvm or some other more familiar tool to see that as well.
Finding hotspots in the Scala code is not necessarily the most interesting part of this graph. What I found fascinating is the visibility of the whole linux process with kernel calls included. Interestingly enough in the graph the embedded RocksDB stack presents itself as a surprisingly minor CPU consumer. Have to say it is also surprising that almost 40% of the calls are not any code related to the application code itself.
https://jompanakumpana.fi/flames.svg
I got inspired by a Devoxx talk about flame graphs and how they can visualize what is happening on a JVM process.
Getting a graph is actually quite simple. You only need a recent enough Java 8 JDK, a running subject JVM process running on linux, perf which is part of kernel utils in most distributions and a couple of simple profiling tools which are open source. Detailed infomation can be found in a blog post by Nitsan Wakart here http://psy-lob-saw.blogspot.fi/2017/02/flamegraphs-intro-fire-for-everyone.html
So what is in that SVG.
It is illustrating what was happening on a Scala app I'm running on a DigitalOcean pod sampled 100 times a second during a 40 second period. The bars describe call stacks and the topmost item is always the one running in the CPU. More details how to read the graph can be found here http://www.brendangregg.com/flamegraphs.html
In this case the stacks are divided to threads. The leftmost stuff (thread) contains a lot of GC operations although upon closer inspection you can see that is happening on other threads as well. Then right to that thread there are a lot of sample points spent on JVM methods related to JIT (at least i assume that from the method names which contain C1 and C2 keywords). As you can see quite a lot of the total samples were related to JIT compiling which most probably tells that the application was still warming up so to speak.
On the other hand when looking at the application code there is already some in-lining visible marked with aqua color. I'm not 100% sure on how to interpret those stacks though. I assume that the methods call on top of each other bounded by green (scala) method calls are in-lined to each other.
What can I learn from this flame graph? I'm certainly not an expert in JVM nor in performance tuning but some of my suspicions where I could improve the performance are visible. I could reduce the amount of disk IO by not reading the configs from disk each time. Also I could look into caching the results of XML transform operations which this particular application does a lot. In another graph where the treads are squashed together, that becomes more visible. Obviously I knew those operations are relatively slow beforehand and also i could have just used visualvm or some other more familiar tool to see that as well.
Finding hotspots in the Scala code is not necessarily the most interesting part of this graph. What I found fascinating is the visibility of the whole linux process with kernel calls included. Interestingly enough in the graph the embedded RocksDB stack presents itself as a surprisingly minor CPU consumer. Have to say it is also surprising that almost 40% of the calls are not any code related to the application code itself.
https://jompanakumpana.fi/flames.svg
Comments
Post a Comment