Koodihommia

I have been digging into some logging issues in an OpenShift production system. The first  problem what we noticed was that the pod logs viewed from the web console were clearly missing some lines. Initially, we thought that this was due to some rate limiting for the web console itself but it turned out to be an issue at the OS level. Another issue what we initially thought was related to the first one was that the Elasticsearch cluster which contains the aggregated logs from all nodes was missing some logs as well and we even had the Elasticsearch cluster members crashing a couple of times without being able to recover the cluster health. It turned out that we had two separate issues with similar symptoms First thing was to check why the web console was missing logs. Openshift (kubernetes) is logging the container logs to journald. After tailing the journald logs a while, it seemed fine. Upon closer inspection, I saw something strange though. It seems that the contai...

A few tips on setting up an embedded OrientDB to run on OpenShift / Kubernetes cluster. Set the ORIENTDB_NODE_NAME system property or environment variable. If your database volumes are host volumes, you can use the downwards API spec.nodeName . If the node name contains dots, replace those with dashes for example. If you use something like OpenShift persistence volumes, make sure that the running pods ORIENTDB_NODE_NAME  matches with the node name values it reads from the DB. Use at least 3 replicas and don't use even number due to split-brain clustering issues If you use rolling upgrade strategy, give the pods some time to start up the DBs so no more than one pod is unavailable at a time. This way syncing up the cluster status becomes smoother. Use the newNodeStrategy dynamic OrientDb distribution configuration parameter   so unreachable nodes don't break up the write quorum so easily. Use Hazelcast to discover the cluster members. There is a library  fo...