Skip to main content

Using actors as a throttle

I've been working on a web shop integration project from time to time for last six months. The use case is that there are several different instances of a webshop (slave) and one master shop. All product information should be integrated from the master shop to the slaves on regular intervals. There are also some business rules applied to the products upon integration.

The integration is set to happen on a certain time of day. It is a relatively long process because it checks through all the products in all shops. That's fine though, there is no requirements on how quick the integration should be. It should rather be a resource constrained process so it would not affect the users using the web shops.

Originally I thought this would be a perfect use case for serverless application running in AWS Lambda for example. I also tried out OpenWhisk from IBM which can be run as a self-hosted serverless platform. While it would have been interesting to try out those technologies I ended up using my DigitalOcean virtual server purely to save costs.

I wrote the app with Scala. I wanted to know the language better and also wanted to see how Akka works in the language it is built in (if all you have is a hammer...). The application is really simple, one just triggers the integration task, it will connect to the webshops to fetch all their product information, cache that to RocksDB, run the transformation rules and the updates the slave webshops.

The application has a configurable amount of workers which do the operations (fetch product data, store it...). The workers are essentially actors which do a lot of blocking operations. This is usually a problem especially if the thread pool for Akka is the default one. The operations using REST APIs of the shops are not exactly fast which can result in quick drainage of Akka threads.

The Akka default pool can be configured to have a fixed number of threads on startup. The amount of workers is derived from the maximum threads available. Each shop interface has a set of workers which is roughly calculated from the amount of threads Akka has divided by the number of shops minus a constant to give some processing time to the aggregate actors. This ends up working pretty well especially when I want to limit the load to the webshops and also keeps the integration application itself from hogging all my 1 core virtual machine resources.
Labels:

Comments

Post a Comment

Popular posts from this blog

I'm not a passionate developer

A family friend of mine is an airlane pilot. A dream job for most, right? As a child, I certainly thought so. Now that I can have grown-up talks with him, I have discovered a more accurate description of his profession. He says that the truth about the job is that it is boring. To me, that is not that surprising. Airplanes are cool and all, but when you are in the middle of the Atlantic sitting next to the colleague you have been talking to past five years, how stimulating can that be? When he says the job is boring, it is not a bad kind of boring. It is a very specific boring. The "boring" you would want as a passenger. Uneventful.  Yet, he loves his job. According to him, an experienced pilot is most pleased when each and every tiny thing in the flight plan - goes according to plan. Passengers in the cabin of an expert pilot sit in the comfort of not even noticing who is flying. As someone employed in a field where being boring is not exactly in high demand, this sounds pro
Post a Comment
Read more

Extracting object properties from an IFC file with IfcOpenShell

Besides the object geometry information, IFC files may contain properties for the IFC objects. The properties can be, for example, some predefined dimension information such as an object volume or a choice of material. Some of the properties are predefined in the IFC standards, but custom ones can be added. IFC files can be massive and resource-intensive to process, so in some cases, it helps to separate the object properties from the geometry data. IfcOpenShell  is a toolset for processing IFC files. It is written mostly in C++ but also provides a Python interface. To read an IFC file >>> ifc_file = ifcopenshell.open("model.ifc") Fetch all objects of type IfcSlab >>> slab = ifc_file.by_type("IfcSlab")[1] Get the list of properties >>> slab.IsDefinedBy (#145075=IfcRelDefinesByType('2_fok0__fAcBZmMlQcYwie',#1,$,$,(#27,#59),#145074), #145140=IfcRelDefinesByProperties('3U2LyORgXC2f_hWf6I16C1',#1,$,$,(#27,#59),#145141), #145142
Post a Comment
Read more

Hubristic developer

Almost half of any Finnish generation goes through a shared experience: the conscript army. An integral part of that experience is learning military slang, a set way people in the army talk. The stories told with said jargon often spread outside of the barracks. It is not uncommon to hear strangers bonding together over beers reminiscing and feeling nostalgic about freezing cold nights spent in tents. There is a similar phenomenon detectable among us coders. To be part of the coder tribe, there is at least one type of story that one must master. That is - of course - ranting about legacy codebases. "Can you believe how much of a mess the previous coders left? Hear, hear!" There is no better way to onboard a new team member than to blame some previous B-team for all the murky parts of the system at hand. This can be seen as harmless, a subject for a good  meme . Rarely do we hold real grudges against "the legacy folk" and can be the best of friends in a social gather
Post a Comment
Read more