Skip to main content

Is your VM really running out of memory?

Every so often, I get fooled by the scary-looking rising memory graphs on my puny minimum sized VM running on Digital Ocean. The same thing seems to happen to my colleagues and I have to go into this discussion on memory usage on virtual machines and on work laptops. I'm not an expert on the topic but I have gathered here a couple of things to demystify the issue.

The first question I ask myself or a fellow developer is: what do you mean by running out of memory? You are most likely talking about memory pressure, right? What are the symptoms? Is it just a gut feeling or is there some verifiable performance degradation?

After that, we could talk about a few seemingly counter-intuitive topics

  • The "raw" memory usage should be high. Especially on a busy VM, it is good that frequently used files are mapped to memory.
  • The OS does not know when it is out of memory. The Linux kernel allocates memory until it can't.
  • You probably should have swap on

Before one goes digging into the system, a few concepts need to be at least somewhat familiar to the user here.

  • RSS
  • Virtual memory
  • Cache, buffers == page cache
  • Memory pages
Then it is time to run some CLI tools

Do the usual stuff with free -h and check the buffered and cached sizes. Note that you might want a lot of files cached. This might be something that could cause performance issues. A lot of the cached files could be "freed" due to high memory pressure. A better option is to look vmstat -a 1 and check the inactive vs active memory ratio. If there is little inactive memory compared to the active, could be a signal of memory pressure.

vmstat -aw 1
vmstat -aw 1

A more detailed output can be fetched from cat /proc/meminfo

What is interesting here is the AnonPages (or Active(anon)). In a gross oversimplification, this means all the memory your applications are using in their heap. The anonymous means it is not backed by a file so it was for example created with malloc.

You could also look into the running processes with pmap -x <pid> and check the heap size and files mapped to memory


For example in the java application pmap above, the heap size is the dominating part, mapped to anonymous memory. To dig in more deeply to the heap, a more specialized tool is needed, like a heap dump analyzer.


Are you using docker? That really does not change the process that much. The containers are processes almost just like any other. Note that if you check the memory consumption with docker stats and compare it ps aux <pid> the results may be different since docker stats removes the shared memory consumption from its output.

Using these few tools will give you the data but the hard part obviously is how to interpret that. I don't have any answers to give here on an easy metric to determine if you are running out of memory or rather is the lack of memory causing performance problems. But what I can tell you is that it is a harder issue than one might think.


Much of this blogpost is based on Brendan Gregg's Systems Performance book http://www.brendangregg.com/sysperfbook.html





Labels:

Comments

Post a Comment

Popular posts from this blog

I'm not a passionate developer

A family friend of mine is an airlane pilot. A dream job for most, right? As a child, I certainly thought so. Now that I can have grown-up talks with him, I have discovered a more accurate description of his profession. He says that the truth about the job is that it is boring. To me, that is not that surprising. Airplanes are cool and all, but when you are in the middle of the Atlantic sitting next to the colleague you have been talking to past five years, how stimulating can that be? When he says the job is boring, it is not a bad kind of boring. It is a very specific boring. The "boring" you would want as a passenger. Uneventful.  Yet, he loves his job. According to him, an experienced pilot is most pleased when each and every tiny thing in the flight plan - goes according to plan. Passengers in the cabin of an expert pilot sit in the comfort of not even noticing who is flying. As someone employed in a field where being boring is not exactly in high demand, this sounds pro...
Post a Comment
Read more

Canyon Precede:ON 7

I bought or technically leased a Canyon Precede:ON 7 (2022) electric bike last fall. This post is about my experiences with it after riding for about 2000 km this winter. The season was a bit colder than usual, and we had more snow than in years, so I properly put the bike through its paces. I've been cycling for almost 20 years. I've never owned a car nor used public transport regularly. I pedal all distances below 30km in all seasons. Besides commuting, I've mountain biked and raced BMX, and I still actively ride my road bike during the spring and summer months. I've owned a handful of bikes and kept them until their frames failed. Buying new bikes or gear has not been a major part of my hobby, and frankly, I'm quite sceptical about the benefits of updating bikes or gear frequently. I've never owned an E-bike before, but I've rented one a couple of times. The bike arrived in a hilariously large box. I suppose there's no need to worry about damage durin...
Post a Comment
Read more

Extracting object properties from an IFC file with IfcOpenShell

Besides the object geometry information, IFC files may contain properties for the IFC objects. The properties can be, for example, some predefined dimension information such as an object volume or a choice of material. Some of the properties are predefined in the IFC standards, but custom ones can be added. IFC files can be massive and resource-intensive to process, so in some cases, it helps to separate the object properties from the geometry data. IfcOpenShell  is a toolset for processing IFC files. It is written mostly in C++ but also provides a Python interface. To read an IFC file >>> ifc_file = ifcopenshell.open("model.ifc") Fetch all objects of type IfcSlab >>> slab = ifc_file.by_type("IfcSlab")[1] Get the list of properties >>> slab.IsDefinedBy (#145075=IfcRelDefinesByType('2_fok0__fAcBZmMlQcYwie',#1,$,$,(#27,#59),#145074), #145140=IfcRelDefinesByProperties('3U2LyORgXC2f_hWf6I16C1',#1,$,$,(#27,#59),#145141), #145142...
Post a Comment
Read more