“Accept who you are. Unless you’re a serial killer.” ― Ellen DeGeneres, Seriously… I’m Kidding
Resource acquisition and cleanup is a very crucial aspect of programs. Resources need to be properly cleaned and released as soon as their usage is complete. Failing which scare resources might not be available to other programs running on the same machine. This is especially true for long-running programs such as servers and daemons.
Other languages have built-in support for automatic resource cleanup. C++ has support and constructs for RAII (Resource-Acquistion Is Initialization), a programming idiom that helps programmers cleanup resources. Java 8 has support for the try-with-resources block which also performs the same functionality.
Fortunately, python has also had support for resource cleanup from ancient versions, even though many programmers might not know about it. This is the with statement and allows you to safe-guard scarce resources. In this article, we look at some aspects of this statement and how you can use it in your programs.
2. Opening a File
The most common example of using the with statement is when opening a file. The following block illustrates the usage.
with open('readme.txt', 'r'): print 'process file here.'
Now let us look into doing something with the opened file, such as counting lines. We need to assign the value returned from open() to a variable so we can access the file methods.
n = 0 with open('readme.txt', 'r') as f: while f.readline(): n += 1 print n, 'lines.'
3. Maintaining a Directory Stack
Let us use this concept to implement a directory stack – where you change the current directory, run some task and change back to the original directory. We implement a class called workdir which works like this – whether a normal exit (via a return) or an abnormal exit (via an exception), the directories are changed back normally.
Here is how the should look like.
with workdir(...): # do your task in the new directory here # here you are now back to the original directory
Here is the implementation of the class.
import os class workdir: def __init__(self, newdir): self.ndir = newdir self.odir = os.getcwd() def __enter__(self): os.chdir(self.ndir) return self def __exit__(self, extype, exvalue, tb): os.chdir(self.odir)
This class is used as follows:
with workdir('ext'): print 'curdir -> ', os.getcwd() with workdir('1'): print 'curdir -> ', os.getcwd() print ' now -> ', os.getcwd() # prints curdir -> .../ext curdir -> .../ext/1 now -> ...
What happens on an abnormal exit (exception)? The directories are popped in succession and you are back to the original.
try: with workdir('ext'): print 'curdir -> ', os.getcwd() with workdir('1'): print 'curdir -> ', os.getcwd() open('readme.txt', 'r') print ' now -> ', os.getcwd() except Exception as x: print x print 'now -> ', os.getcwd() # prints again curdir -> .../ext curdir -> .../ext/1 [Errno 2] No such file or directory: 'readme.txt' now -> ...
Depending on where you catch the exception, the position is maintained.
with workdir('ext'): print 'curdir -> ', os.getcwd() try: with workdir('1'): print 'curdir -> ', os.getcwd() open('joe.txt', 'r') print ' now -> ', os.getcwd() except Exception as x: print x print ' now -> ', os.getcwd() print ' now -> ', os.getcwd() # prints curdir -> .../ext curdir -> .../ext/1 [Errno 2] No such file or directory: 'joe.txt' now -> .../ext now -> ...
4. Locking Out Critical Sections of Code
Using the python with statement, it is possible to implement critical sections of code which need to be executed by just one thread at a time. A critical section is implemented using a thread lock – one thread acquires the lock and another thread can enter the section only when the first thread releases it. Here is a simple class which implements this concept.
The class acquires the lock when the thread enters the section and releases it when exiting the section, whether normally or abnormally.
import threading as t class critical_section: def __init__(self): self.lock = t.Lock() def __enter__(self): self.lock.acquire() def __exit__(self, extype, exvalue, tb): self.lock.release()
And here is a sample client code which needs to be executed by a single thread at a time as it contains business-critical logic (updating a bank customer’s balance).
For the sake of illustrating normal and abnormal exits, the code fetches a random number and raises an exception if the number is above 10. And to illustrate random amounts of time spent in the business code, it sleeps for the random amount of time before completing the update.
import time, random def runner(crit, incr): global balance try: with crit: x = random.randint(1, 15) if x > 10: raise Exception('too much sleep: ' + str(x)) else: time.sleep(x) balance += incr print 'update', incr, 'done - balance is', balance except Exception as x: print x, ', update cannot be applied: ', incr
The following is the sequence of events which simulates multiple threads attempting to update the customer balance. Five threads are created, each of which attempts to make a different update. But only one thread can execute the update at a time, thanks to the critical_section object.
balance = 0 crit = t.Lock() t.Thread(target = runner, args = (crit,-100)).start() t.Thread(target = runner, args = (crit,-200)).start() t.Thread(target = runner, args = (crit,200)).start() t.Thread(target = runner, args = (crit,400)).start() t.Thread(target = runner, args = (crit,-100)).start() # prints update -100 done - balance is -100 update -200 done - balance is -300 update 200 done - balance is -100 update 400 done - balance is 300 too much sleep: 11 , update cannot be applied: -100
To lookup the balance at any time, the following code can be used. This code also uses the critical_section object for safe access to the global variable.
def check_bal(crit): global balance with crit: print 'current balance: ', balance return balance
In this article, we examined the use of the python with statement in various contexts. It is a useful little construct which allows you to cleanly dispose of used resources regardless of whether an exception occurred or not.