Skip to main content

Flame graph from a Scala app

Apologies for the large SVG


I got inspired by a Devoxx talk about flame graphs and how they can visualize what is happening on a JVM process.

Getting a graph is actually quite simple. You only need a recent enough Java 8 JDK, a running subject JVM process running on linux, perf which is part of kernel utils in most distributions and a couple of simple profiling tools which are open source. Detailed infomation can be found in a blog post by Nitsan Wakart here http://psy-lob-saw.blogspot.fi/2017/02/flamegraphs-intro-fire-for-everyone.html

So what is in that SVG.

It is illustrating what was happening on a Scala app I'm running on a DigitalOcean pod sampled 100 times a second during a 40 second period. The bars describe call stacks and the topmost item is always the one running in the CPU. More details how to read the graph can be found here http://www.brendangregg.com/flamegraphs.html

In this case the stacks are divided to threads. The leftmost stuff (thread) contains a lot of GC operations although upon closer inspection you can see that is happening on other threads as well. Then right to that thread there are a lot of sample points spent on JVM methods related to JIT (at least i assume that from the method names which contain C1 and C2 keywords). As you can see quite a lot of the total samples were related to JIT compiling which most probably tells that the application was still warming up so to speak.

On the other hand when looking at the application code there is already some in-lining visible marked with aqua color. I'm not 100% sure on how to interpret those stacks though. I assume that the methods call on top of each other bounded by green (scala) method calls are in-lined to each other.

What can I learn from this flame graph? I'm certainly not an expert in JVM nor in performance tuning but some of my suspicions where I could improve the performance are visible. I could reduce the amount of disk IO by not reading the configs from disk each time. Also I could look into caching the results of XML transform operations which this particular application does a lot. In another graph where the treads are squashed together, that becomes more visible. Obviously I knew those operations are relatively slow beforehand and also i could have just used visualvm or some other more familiar tool to see that as well.

Finding hotspots in the Scala code is not necessarily the most interesting part of this graph. What I found fascinating is the visibility of the whole linux process with kernel calls included. Interestingly enough in the graph the embedded RocksDB stack presents itself as a surprisingly minor CPU consumer. Have to say it is also surprising that almost 40% of the calls are not any code related to the application code itself.

https://jompanakumpana.fi/flames.svg


Labels

Labels:

Comments

Post a Comment

Popular posts from this blog

I'm not a passionate developer

A family friend of mine is an airlane pilot. A dream job for most, right? As a child, I certainly thought so. Now that I can have grown-up talks with him, I have discovered a more accurate description of his profession. He says that the truth about the job is that it is boring. To me, that is not that surprising. Airplanes are cool and all, but when you are in the middle of the Atlantic sitting next to the colleague you have been talking to past five years, how stimulating can that be? When he says the job is boring, it is not a bad kind of boring. It is a very specific boring. The "boring" you would want as a passenger. Uneventful.  Yet, he loves his job. According to him, an experienced pilot is most pleased when each and every tiny thing in the flight plan - goes according to plan. Passengers in the cabin of an expert pilot sit in the comfort of not even noticing who is flying. As someone employed in a field where being boring is not exactly in high demand, this sounds pro...
Post a Comment
Read more

Canyon Precede:ON 7

I bought or technically leased a Canyon Precede:ON 7 (2022) electric bike last fall. This post is about my experiences with it after riding for about 2000 km this winter. The season was a bit colder than usual, and we had more snow than in years, so I properly put the bike through its paces. I've been cycling for almost 20 years. I've never owned a car nor used public transport regularly. I pedal all distances below 30km in all seasons. Besides commuting, I've mountain biked and raced BMX, and I still actively ride my road bike during the spring and summer months. I've owned a handful of bikes and kept them until their frames failed. Buying new bikes or gear has not been a major part of my hobby, and frankly, I'm quite sceptical about the benefits of updating bikes or gear frequently. I've never owned an E-bike before, but I've rented one a couple of times. The bike arrived in a hilariously large box. I suppose there's no need to worry about damage durin...
Post a Comment
Read more

Emit structured Postgres data change events with wal2json

A common thing I see in an enterprise system is that when an end-user does some action, say add a user, the underlying web of subsystems adds the user to multiple databases in separate transactions. Each of these transactions may happen in varying order and, even worse, can fail, leaving the system in an inconsistent state. A better way could be to write the user data to some main database and then other subsystems like search indexes, pull/push the data to other interested parties, thus eliminating the need for multiple end-user originating boundary transactions. That's the theory part; how about a technical solution. The idea of this post came from the koodia pinnan alla podcast about event-driven systems and CDC . One of the discussion topics in the show is emitting events from Postgres transaction logs.  I built an utterly simple change emitter and reader using Postgres with the wal2json transaction decoding plugin and a custom go event parser. I'll stick to the boring ...
Post a Comment
Read more