Stavros' Stuff

Angry rants of programming and other things.

How to deploy Django with Docker

Finally, Django, with Docker, on production!

I finally managed to deploy Django in a Docker container on production! I’ve been trying to switch to a full Docker development/production model since Docker came out, but only recently did the ecosystem mature enough to allow me to easily use Docker both for development (where it excels, in my opinion) and on production (where it’s pretty okay and quite useful).

In this post, I will quickly give you all the relevant details and files you need to go from a newly checked-out repository to a full development environment in one command, and to deploy that service to production. As a bonus, I’ll show you how to use Gitlab (which is awesome) to build your containers and store them in the Gitlab registry.

Let’s begin!


Continue reading…

Building a cheap home sensor/controller

Sense light and motion in every room!

After designing my first PCB, I went on a designing spree. It turns out that making PCBs (printed circuit boards, basically a piece of plastic that includes all the connections of your components in it. It helps make your project smaller and cut down on the amount of wires floating around) is so enjoyable, I’m PCBing all the things! The next victim for PCBfication is a circuit I had originally built on an Arduino and subsequently migrated to an ESP8266.

The circuit is a home sensor and controller. It can sense light, temperature, humidity and motion, and includes an RF controller (at 433 MHz) and an infrared LED so you can control your TV and other home devices. In this post, I’ll go into some detail about the build and how it connects to other sensors and controllers around the house.

This post is also a test of my new Expounder concept library. Throughout the post, various terms will be underlined like this (with a dashed underline), and you can click on them if you’re unfamiliar with the underlined term. After clicking, some text will expand and explain the term.

Let’s continue to

Continue reading…

Introducing: String Phone

"Nothing is as secure as a string phone" –The NSA

As you can probably tell from previous posts, I’ve been pretty into hardware lately. I’ve especially been building things like home sensors and controllers, so I have a central computer reading motion, temperature, humidity, light and other values in the house and deciding whether the lights or air conditioning need to be on or off.

I also want to be able to turn these on and off from my mobile phone, from anywhere in the world. The problem with that is that I need a way to ensure that only my phone can turn things on in my house. I wouldn’t want someone to be able to turn the heating on in my house at full blast when I’m not there and waste all my electricity bill (or set fire to something).

TLS is a pretty good solution, as it ensures confidentiality between client and server, but it does nothing for verifying the client or securing communications against a malicious server. I needed something better, and I couldn’t find something readily available. So I set out to write it. Thus, string phone was born.

String phone is a

Continue reading…

Shufflecast - Your own TV station

Your own, personal TV station

I very recently discovered the very useful pychromecast library, and, being the owner of a Chromecast, I started wondering what I could do with it. Since there aren’t that many things a Chromecast can do (it pretty much only plays videos over the network, although there are some other things as well), I decided to make my own TV station. Thus, Shufflecast was born.

Continue reading…

Show page generation time in Django

Ever longed for the good old PHP days? No? Good.

Earlier today, i.e. a few minutes ago, I was working on my latest guinea pig, TiThess. It’s an events guide for my city, and the latest project I’m trying everything on. I like having a project I can try new things on, as it helps keep my skills sharp.

As I was testing page load times with the excellent Web Page Test, trying to get them down to the absolute minimum, I was getting an F in time-to-first-byte. This is very odd, because the whole site is supposed to be cached, so I was wondering whether the cache is doing something wrong and slowing page generation down.

To make sure, I needed a simple way to show how long page generation took, like the old “page generated in X seconds” footer that was all the rage with PHP sites way back when. Here’s how I did it:

Continue reading…

On API authentication

Make your API unhackable, like the Titanic

This post needs no introduction, so it doesn’t have one. If you want to write an API and need to know how to make it secure, and have the requests authenticate against a server or a client, look no further! Well, do look a bit further, because I’m going to tell you how to do all these things in this post.

Use cases

As with most other things, your API authentication method will depend on your use case. I will detail a few common ones, along with the best authentication scheme for each one:

Continue reading…

Standalone Django scripts: The definitive guide

Jeez, why is this so hard to find info on?

You know the deal, you have your fantastic Django application and it’s working great and everything, but you need to make a small change which is too cumbersome to do in the shell, so you figure “duh, I’ll just write a script to do it”. You write your external script in two minutes and then struggle for two hours to figure out how to load the models and the rest of the context so it will work with your app’s settings and all your Django goodness.

You visit StackOverflow and a bunch more sites, and they either tell you to use a management command, which is great advice, except your thing is a one-off and you don’t want to have to check it to git and go through all that hassle to get it deployed just to do this simple thing, or they give you some arcane lines that just don’t work.

Fear not, for I am here. I will give you five simple lines that will make everything work perfectly. Perfectly, I say!

Without further ado (all the previous ado was just so I could fill the paragraph so the side-box doesn’t look weird with short text), I give you the magic commands! Here they are:

Continue reading…

Calling single-argument methods in Django templates

No longer are your methods confined to bare calls!

One of my pet peeves when it comes to Django is that you can’t call methods that require arguments in templates. While this is fine most of the time, it does mean that you need to have one property or method per call you want to make, which sometimes gets very cumbersome.

I needed a way to define various dynamic permissions that are calculated at runtime (for irrelevant reasons, Django’s permissions framework wasn’t a good fit), and writing properties like can_register, can_add_tags, can_subscribe got tedious. These tended to be defined all over the place, rather than in one central spot, and it was hard to add more checks without cluttering the classes.

I would much prefer to have a single method (let’s call it can()) that accepted a string with the permission I wanted to check, and return True or False, depending. This is easy to do in the views, but templates would never be able to call it with an argument.

However, since Django can do dictionary-style attribute lookups, I could add a dictionary interface over the method, and allow

Continue reading…

Writing a FUSE filesystem in Python

Turns out FUSE filesystems are ridiculously easy!

If you’re a regular reader, you might have noticed that I’ve been on a quest for the perfect backup program, and ended up writing my own encryption layer over bup.

While writing encbup, I wasn’t very satisfied with having to download the entire huge archive just to restore a file, and still wished that I could use EncFS together with rdiff-backup to have true remote-mountable, encrypted, deduplicated, versioned backups.

Trying obnam again (spoiler: it’s still pretty slow), I noticed that it included a mount command. Looking at it, I discovered fuse-python and fusepy, and realized that writing a FUSE filesystem in Python is pretty much trivial.

The astute observer will have already realized where I’m going with this: I decided to write an encrypted filesystem layer in Python! This layer would be very similar to EncFS, with a few crucial differences:

Continue reading…

Encrypted, deduplicated remote backups

Why are secure backups so hard?

Note: Be sure to check the sequel to this post, about the program that will supersede this one and be compatible with all backup utilities.

Backing things up is important, and, luckily, there are many high-quality services geared to everyday people that are very easy to use and cheap. Unfortunately, I am not everyday people, as I am very paranoid and insist that absolutely nobody be able to see my photos of my dog and lawn. It’s a matter of privacy.

To that end, I’ve long been looking for a secure/encrypted backups service, but I haven’t managed to find a single service or tool that fulfils my requirements:

Django's per-site caching doesn't work

Surprise! Your cache doesn't.

A few days ago, I wrote a post about a peculiar piece of code that a friend of mine had sent me. Since it was interesting bit of code, I thought Hacker News would enjoy it, so I posted it there. To my great pleasure, the post shot up to the first place in a few minutes and continued there for a full day, bringing just over 50,000 visitors to this blog, in total.

I was very happy that people were liking and discussing this post (and the discussion was very interesting in its own right), but I noticed that AppEngine, where this blog is hosted, was struggling to serve it. I had to create new instances because the average latency was about ten seconds(!), even though this blog is pretty much only text and static media, and I use Django’s per-site cache to cache every single page.

Continue reading…

Brilliant or insane code?

A Moste Wonderfull Tale of Optimizations and Legibility

I came upon a very interesting and cryptic snippet of code somewhere nameless, and I can’t decide if it is brilliant or completely insane. It is a very obscure way of accomplishing the required task, but it’s around four times faster than the alternatives I’ve tried, so I have to admit that it’s not completely without merit. Still, I cringe a bit at seeing it, since it packs around four unusual Python concepts in almost as many characters.

This is the snippet in question:

Continue reading…

Faster installs with pip

Today’s tip is brought to you by the color green and the number g.

If you’re an industrious Python programmer, you probably use pip and virtualenv to install your packages in an isolated location, which is great, and good job to you. However, having to install Django, pandas and ipython every time you want to experiment with something is a bit tiresome, since they’re all big files and downloading them takes a while, plus what if you’re on a data-limited connection and bla bla bla.

Here’s a very easy way to both speed up your downloads and avoid having to do them in the first place:

Continue reading…

Django template projects

If you haven’t heard yet, let me break the good news to you: Django 1.6 is getting template apps/projects. This means that, when starting a new project, you don’t have to copy your old directories and change stuff around any more, you can just put them into reusable directories (or even repositories) and Django will be able to use them very simply: startproject --template=/djtemplates/default myapp

I’ve been needing a feature like that for ages, and now I’m giddy with excitement, so, without further ado,

Continue reading…

Monitoring your system's temperatures with Python

I have suddenly noticed that my computer’s fans have been more noisy than when I first bought it, and now it’s driving me crazy. I wondered if I could cut the fans’ power without increasing the temperatures inside the case too much, or even leave the case open. To test it, I spent five minutes writing a small script to monitor temperatures in Python, using smartmontools and lm-sensors.

This is that script:

Continue reading…

How to replace a Django model field with a property

If you use Django for your web apps, you will probably have come across a situation where you need to run some custom code on assigning a value to a model field. You’ve probably hacked around this by overriding the save() method or some other arcane hackery, but this is not only unnecessary, but it’s more complicated than it should be.

You can use a Python Continue reading…

How to highlight code in Python

I’ve recently been looking at syntax highlighters, both for this blog and for Instahero, the new product my company is developing. I used to use a JS-based solution for the blog, but it broke with many inputs, it was fiddly to work with, didn’t parse correctly sometimes, and slowed the site down considerably. I especially disliked the fact that

Continue reading…

The curious case of the range() function

If you’ve ever programmed in Python, you have a 96.2% chance of having used the range() function. As you have a 96.1% chance of knowing, said function returns a list containing an arithmetic progression of integers, exactly as the documentation says. For example, range(4) returns [0, 1 2, 3].

If you’ve ever bothered to look at the docstring, you will see that it goes thus:

Continue reading…

A schemaless layer over SQLite

I’ve been hard at work these days, and to relax I decided to develop a small application/side project. As with many applications, I needed a reliable, persistent store, so I went with the most reliable, easy to set up, performant, and generally awesome store I knew: SQLite.

However, since this was a very experimental prototype, I got tired of the frequent schema changes and wished

Continue reading…

How to handle Python package installation on GAE

As I’ve mentioned many times on this blog, I’m a big fan of Google AppEngine. I like the free tier, the effortless scaling, the zero administration, and how I can run Django almost unmodified and deploy in seconds. It has its downsides, but overall I like it very much.

The one part of deployment I didn’t like was dependency handling. Due to the way GAE works, you need t

Continue reading…

How to extend the Django admin site with custom views

Often I find that I need to include administration views in my site, such as statistics, management, etc etc. When these are more than just model CRUD views, a good idea is to extend the builtin Django admin site, as it’s the easiest to do.

While looking at the docs, however, I realised they were a bit cryptic and lacked good examples. It didn’t take me long to figure out how

Continue reading…

Printing "Hello world!" using curve fitting techniques (or: The "Hello world!" function)

Well, I have a computer architecture exam in six hours and can’t be bothered, so I figured I would realize a lifelong dream of mine, and make a program that prints “Hello world!” using curve fitting techniques. Enlisting the help of a good friend with numerous mathematical papers under his belt (ostensibly because he could not afford a tighter belt), MATLAB and a longing for p

Continue reading…