Thoughts on using Docker in the cloud for side projects

For most companies nowadays, Kubernetes and all its flavors is all the rage. And for others, virtual machines still do the job quite nicely.

Some context

For all my side projects, I usually fall into one of 2 categories :

• I use a super cheap Digital Ocean virtual machine to run my stuff the 'old' way. Most of those machines barely see any load in their life
• I use the free tier of any of the serverless (take that word in a liberal way) out there. I have a VERY soft spot for Netlify but I use Firebase a lot as well for anything authentication related because of how plug and play it is. And for specific things, some of the AWS APIs (like rekognition) are also super nice.

For one of my latest projects, I was in need of a time series database. I settled on InfluxDb, which seemed to be pretty much the industry standard. The actual volume of data received by the database will be very low (couple points per hour), but I wanted long term retention (years). I looked around but found no free solution that would give me more than 30 days of retention policy, while allowing me to export the data if needed.

Running and managing a Docker instance in the cloud

Instead of using a Virtual Machine as usual, I decided to look seriously into finding a way to host a Docker image in the cloud. The requirements are simple :

• I don't want to lose data if the image crashes
• I want the image to be as cheap as possible for my use case
• I want to have as little maintenance as possible

What that translates into is :

• I need to be able to have a persistent volume next to my Docker image (for restarts, settings, ....)
• I need to be able to run backup script purely for the data

Using Docker myself in a Digital Ocean virtual machine would defy the purpose of using Docker in my opinion. Yes, it would typically tick most boxes, but create additional maintenance that I don't want to have. I might just as well run InfluxDb straight in the VM.

The main reason for this blog is that I was surprised to see how expensive running a simple Docker container on the main cloud providers is!

• Using the Azure calculator, we come to a price of over 30$ / month for a 1Gb, 1 CPU linux machine.
• Amazon has similar prices as well for the same capabilities

There are many articles about the best Docker cloud providers out there. I have nothing against Kubernetes per se but I don't want to have several containers running just to support my infrastructure so the list slims down pretty quick.

• I gave a shot at Clever Cloud for a little while. I'm a big fan of the team there and I like supporting smaller companies as soon as I can. For 1CPU and 1Gb, their pricing is about 15$ a month, which I'm willing to pay.

Giving a shot at Clever Cloud

Clever Cloud can be a little misleading at first in terms of expectations (just like Netlify actually). They want you to deploy apps, or add-ons and offer a layer on top of the actual tech stack. This is actually quite refreshing to see.

The workflow is as such

• Choose your tech stack (in my case docker)

• Choose your scaling options (vertical, horizontal and the threshold)
• Choose the origin of your app. You can even pick a Docker repo or simply push manually yourself. Anytihng you push will automagically deployed
• The configuration is done via environment variables (exposed port, location of the Dockerfile, ...)

The whole setup goes flawlessly, and their staff is super welcoming and helpful. Now, I run into a bit of an issue for my specific use case

• They don't support volumes at the moment
• The Docker container would become unreachable and restart itself once in a while, for no apparent reason.

After having to recreate my setup for the third time, I decided to stop the experiment.

Don't get me wrong, I really like Clever Cloud and I'm definitely considering them first for each of my side projects. It's nice to see folks simplifying our workflows in innovative ways.

I realize this part is not essential to the blog, I just wanted to give a shoutout to a product I really like :).

So how is it running now ?

Well, after having tried quite a few things, I ended going back to square 1, and rented a cheap DO VM to host my database. It's working flawlessly, and I can easily script backups twice a day.

So far the whole setup has never reached more than 30% CPU usage and disk I/O is more than minimal

So why, Cloud Providers?

I find it interesting (and frustrating) to see that the main cloud providers are so expensive for such a simple setup. Given that all of them support the capability, and also already can benefit from the economy of scale.

More and more developers makes use of AWS, GCP or Azure in their daily work life, and many developers also run side projects. So why not take advantadge of this and binds them to the platform even more?  They surely should be able to compete with a Digital Ocean or a Clever Cloud, should they want to.

I wonder if it's not simply because cloud providers want to push you in a given direction, and pricing is a very efficient way to do it. That's most likely why all of them have an always free serverless bundle (together with a storage solution).

The downside of this is that for some use cases, like mine, it isn't really a good use case :). I find it a shame, personally. The usage I make of cloud providers in my private lie is basically limited to very short lived and then destroyed experiments. And in the meantime I'm still bound to making update to Virtual Machines that will spend most of their lifespan underutilized. Azure, you could get my 30$ / month, should you want to. . .

Do you see other reasons? I'm curious, let me know on Twitter.