Skip to main content

OpenShift logging issues

I have been digging into some logging issues in an OpenShift production system. The first problem what we noticed was that the pod logs viewed from the web console were clearly missing some lines. Initially, we thought that this was due to some rate limiting for the web console itself but it turned out to be an issue at the OS level. Another issue what we initially thought was related to the first one was that the Elasticsearch cluster which contains the aggregated logs from all nodes was missing some logs as well and we even had the Elasticsearch cluster members crashing a couple of times without being able to recover the cluster health.

It turned out that we had two separate issues with similar symptoms

First thing was to check why the web console was missing logs. Openshift (kubernetes) is logging the container logs to journald. After tailing the journald logs a while, it seemed fine. Upon closer inspection, I saw something strange though. It seems that the containers that produced a lot of log lines wrote those in chunks and then the whole log seemed to freeze for a while.

After some googling around I found out that there is a rate limit in journald which is by default 1K lines of logs in 30 seconds. All exceeding lines will be dropped out. After tailing the journald logs with journald unit name itself, I saw that this was indeed the issue. I found out soon after that the OpenShift "high load" guide warned about this and described some example configurations for both journald and rsyslogd to mitigate the issue.

However, we could not simply increase the rate limit in production system since we were not sure what would happen to the already fragile Elasticsearch logging cluster. The master machines had quite high CPU and IO usage already.

First, we had to fix the Elasticsearch cluster since we could not log into Kibana to view the logs. The problem was that there were too many indices out of sync and when the ES cluster members started, the authentication part timed out. Since we had already lost a lot of logs due to the journald issue, we simply removed all indexes and gave the ES's a "fresh start" so to speak.

Now when we had the aggregated logging working again, we one by one fixed the rate limits in each of the nodes. We also modified the buffering configurations in node Fluentd's to decrease the rate which they are sending the logs from each node to the ES cluster. We also gave a bit more resources to the page cache on the host machines.

Anyone running OpenShift with its logging feature enabled, I recommend reading this https://docs.openshift.com/enterprise/3.2/install_config/aggregate_logging_sizing.html
Labels:

Comments

Post a Comment

Popular posts from this blog

I'm not a passionate developer

A family friend of mine is an airlane pilot. A dream job for most, right? As a child, I certainly thought so. Now that I can have grown-up talks with him, I have discovered a more accurate description of his profession. He says that the truth about the job is that it is boring. To me, that is not that surprising. Airplanes are cool and all, but when you are in the middle of the Atlantic sitting next to the colleague you have been talking to past five years, how stimulating can that be? When he says the job is boring, it is not a bad kind of boring. It is a very specific boring. The "boring" you would want as a passenger. Uneventful.  Yet, he loves his job. According to him, an experienced pilot is most pleased when each and every tiny thing in the flight plan - goes according to plan. Passengers in the cabin of an expert pilot sit in the comfort of not even noticing who is flying. As someone employed in a field where being boring is not exactly in high demand, this sounds pro...
Post a Comment
Read more

Bird is causing high CPU on my macOS

There is no lack of people complaining about MacOS Tahoe, mostly about rounded corners and inconsistent design decisions. I have not paid that much attentention to that, but there is one mac bug i have paid attention to. It hasn't been a visual or ux but rather few system processes pegging the CPU. trustd , alongside with ecosystemd and ecosystemanalyticsd all reported high CPU usage. I can't exactly recall when this started, it might have predated my Tahoe upgrade but anyway, the trio of processes all had high CPU usage. Sure, it may have been the virtual efficiency cores and whether it affeced battery life or slowed down other processes i don't know to be hon...
Post a Comment
Read more

Ousterhout's law

Back in the day, everyone was using Winamp. It is a music player with the user interface of a mixing console. The bloody thing had an equalizer on the front page! As an amateur music producer, I know that EQ is a powerful tool but making changes that sound good is difficult, to say the least. Mixing engineers spend considerable effort with the artist to balance the frequency ranges to arrive at the desired musical outcome. I bet the 15-year-old me butchered a lot of songs with the thing. Now we are using Spotify, a player with basically a search bar and a play button. I ran into something called Ousterhout's Law on the  Operating Systems: Three Easy Pieces book . Here is a quote from the book TIP: AVOID VOO-DOO CONSTANTS (OUSTERHOUT’S LAW) Avoiding voo-doo constants is a good idea whenever possible. Unfortunately, as in the example above, it is often difficult. One could try to make the system learn a good value, but that too is not straightforward. The frequent result: a configura...
Post a Comment
Read more