Proper Cleanup of Resources in Python using the With Statement

The python with statement is damn useful! Learn the intricacies of the with statement.

“Accept who you are. Unless you’re a serial killer.” ― Ellen DeGeneres, Seriously… I’m Kidding

1. Introduction

Resource acquisition and cleanup is a very crucial aspect of programs. Resources need to be properly cleaned and released as soon as their usage is complete. Failing which scare resources might not be available to other programs running on the same machine. This is especially true for long-running programs such as servers and daemons.

Other languages have built-in support for automatic resource cleanup. C++ has support and constructs for RAII (Resource-Acquistion Is Initialization), a programming idiom that helps programmers cleanup resources. Java 8 has support for the try-with-resources block which also performs the same functionality.

Fortunately, python has also had support for resource cleanup from ancient versions, even though many programmers might not know about it. This is the with statement and allows you to safe-guard scarce resources. In this article, we look at some aspects of this statement and how you can use it in your programs.

2. Opening a File

The most common example of using the with statement is when opening a file. The following block illustrates the usage.

with open('readme.txt', 'r'):
    print 'process file here.'

Now let us look into doing something with the opened file, such as counting lines. We need to assign the value returned from open() to a variable so we can access the file methods.

n = 0
with open('readme.txt', 'r') as f:
    while f.readline():
        n += 1
print n, 'lines.'

3. Maintaining a Directory Stack

Let us use this concept to implement a directory stack – where you change the current directory, run some task and change back to the original directory. We implement a class called workdir which works like this – whether a normal exit (via a return) or an abnormal exit (via an exception), the directories are changed back normally.

Here is how the should look like.

with workdir(...):
    # do your task in the new directory here
# here you are now back to the original directory

Here is the implementation of the class.

import os

class workdir:
    def __init__(self, newdir):
        self.ndir = newdir
        self.odir = os.getcwd()

    def __enter__(self):
        os.chdir(self.ndir)
        return self

    def __exit__(self, extype, exvalue, tb):
        os.chdir(self.odir)

This class is used as follows:

with workdir('ext'):
    print 'curdir -> ', os.getcwd()
    with workdir('1'):
        print 'curdir -> ', os.getcwd()
print '   now -> ', os.getcwd()
# prints
curdir ->  .../ext
curdir ->  .../ext/1
   now ->  ...

What happens on an abnormal exit (exception)? The directories are popped in succession and you are back to the original.

try:
    with workdir('ext'):
        print 'curdir -> ', os.getcwd()
        with workdir('1'):
            print 'curdir -> ', os.getcwd()
            open('readme.txt', 'r')
        print '   now -> ', os.getcwd()
except Exception as x:
    print x
print 'now -> ', os.getcwd()
# prints
again
curdir ->  .../ext
curdir ->  .../ext/1
[Errno 2] No such file or directory: 'readme.txt'
   now ->  ...

Depending on where you catch the exception, the position is maintained.

with workdir('ext'):
    print 'curdir -> ', os.getcwd()
    try:
        with workdir('1'):
            print 'curdir -> ', os.getcwd()
            open('joe.txt', 'r')
        print '   now -> ', os.getcwd()
    except Exception as x:
        print x
    print '   now -> ', os.getcwd()
print '   now -> ', os.getcwd()
# prints
curdir ->  .../ext
curdir ->  .../ext/1
[Errno 2] No such file or directory: 'joe.txt'
now ->  .../ext
now ->  ...

4. Locking Out Critical Sections of Code

Using the python with statement, it is possible to implement critical sections of code which need to be executed by just one thread at a time. A critical section is implemented using a thread lock – one thread acquires the lock and another thread can enter the section only when the first thread releases it. Here is a simple class which implements this concept.

The class acquires the lock when the thread enters the section and releases it when exiting the section, whether normally or abnormally.

import threading as t

class critical_section:
    def __init__(self):
        self.lock = t.Lock()

    def __enter__(self):
        self.lock.acquire()

    def __exit__(self, extype, exvalue, tb):
        self.lock.release()

And here is a sample client code which needs to be executed by a single thread at a time as it contains business-critical logic (updating a bank customer’s balance).

For the sake of illustrating normal and abnormal exits, the code fetches a random number and raises an exception if the number is above 10. And to illustrate random amounts of time spent in the business code, it sleeps for the random amount of time before completing the update.

import time, random

def runner(crit, incr):
    global balance
    try:
        with crit:
            x = random.randint(1, 15)
            if x > 10:
                raise Exception('too much sleep: ' + str(x))
            else:
                time.sleep(x)
                balance += incr
            print 'update', incr, 'done - balance is', balance
    except Exception as x:
        print x, ', update cannot be applied: ', incr

The following is the sequence of events which simulates multiple threads attempting to update the customer balance. Five threads are created, each of which attempts to make a different update. But only one thread can execute the update at a time, thanks to the critical_section object.

balance = 0
crit = t.Lock()
t.Thread(target = runner, args = (crit,-100)).start()
t.Thread(target = runner, args = (crit,-200)).start()
t.Thread(target = runner, args = (crit,200)).start()
t.Thread(target = runner, args = (crit,400)).start()
t.Thread(target = runner, args = (crit,-100)).start()
# prints
update -100 done - balance is -100
update -200 done - balance is -300
update 200 done - balance is -100
update 400 done - balance is 300
too much sleep: 11 , update cannot be applied:  -100

To lookup the balance at any time, the following code can be used. This code also uses the critical_section object for safe access to the global variable.

def check_bal(crit):
    global balance
    with crit:
        print 'current balance: ', balance
        return balance

Conclusion

In this article, we examined the use of the python with statement in various contexts. It is a useful little construct which allows you to cleanly dispose of used resources regardless of whether an exception occurred or not.