“The world is a book and those who do not travel read only one page.” ― Augustine of Hippo
Contents
1. Introduction
In a previous article, we looked at calling an external command in python using subprocess.call(). That method allows you to accomplish several tasks during the invocation:
- Invoke a command and pass it command line arguments
- Redirect the invoked process output (stdout and stderr) to a file.
- Pass data from an open file to the process input as stdin.
- Invoke a shell command specified as a string.
- Capture output from the process into a string.
- Capture child process exit code, and error message, if any.
Read on to learn more about using another method to call an external command in python – using subprocess.Popen().
2. Simple Example of Popen
Here is a simple example of using subprocess.Popen(). This is just like using subprocess.call(). However, you get a little bit more control over the child process – method poll() checks the status of the external command and returns the exit code if it has exited. Method wait(), on the other hand, waits for the process to complete and returns the exit code.
p = subprocess.Popen(['ls', '-l']) print 'poll?', p.poll() print 'wait:', p.wait() # prints poll? None total 4 -rwxr--r-- 1 xxxxxxx xxxxxxx 333 Jan 27 13:41 README.txt wait: 0
3. Reading Output from the External Command
Let us now create a pipe from the child STDOUT using the parameter stdout=subprocess.PIPE. In this case, the output from the external process does not automatically appear on the invoker’s output. You can instead obtain the output as a stuple from the method communicate() as shown. The first member of the tuple is out from stdout and the second part is the output from stderr. Since we did not specify pipe creation from stderr, that output is None below.
p = subprocess.Popen(['ls', '-l'], stdout=subprocess.PIPE)
print 'poll?', p.poll()
print 'wait:', p.wait()
print 'comm:', p.communicate()
# prints
poll? None
wait: 0
comm: ('total 4\n-rwxr--r-- 1 xxxxxxx xxxxxxx 333 Jan 27 13:50 README.txt\n', None)
4. Two-Way Communication and Deadlocks
The Popen class allows you to create a duplex pipe (piping both stdin and stdout) to th external process. This pipe can be used for two-way communication with the child, but you should be careful to avoid deadlocks. A deadlock can occur when the parent expects output from the child, while the child is waiting for input from the parent. And vice versa.
Two-way communication with an external process requires you to be aware of how the child works, when it generates output, and when it waits for input. With this knowledge, the parent must interleave child reads and writes appropriately.
5. Sending input via STDIN
Let us look at an example of sending some input to the child process and reading some output from it.
The following example starts an instance of the bash shell with both stdin and stdout pipes to the parent. The parent then writes an echo message and commands the child to exit. The output from the child is then read read by the parent.
p = subprocess.Popen(['/bin/bash'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write('echo hello bash; exit 2\n')
p.stdin.flush()
print p.stdout.read()
print 'exit code:', p.wait()
# prints
hello bash
exit code: 2
6. Avoiding deadlock when using stdin and stdout
As mentioned above, a deadlock is possible when attempting to use stdin and stdout directly. This is illustrated by the following example. We start cat in the child, write some data to it and expect to read it back.
p = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write('hello world')
p.stdin.flush()
print p.stdout.read()
p.terminate()
print 'poll?', p.poll()
print 'wait:', p.wait()
# program hangs ...
Things do not go as smoothly as we expect. This is because when reading output from the child, read() expects to see an EOF (End-Of-File). However, the child process echoes the text back and waits for more input. This causes a deadlock.
We can remedy this in this situation by using os.read() which does not wait for EOF, rather returns when it can read something.
p = subprocess.Popen(['cat'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
p.stdin.write('hello world')
p.stdin.flush()
print os.read(p.stdout.fileno(), 100)
p.terminate()
print 'poll?', p.poll()
print 'wait:', p.wait()
# prints
hello world
exit code: -15
And note that the exit code is -15 since the parent terminated the child.
Conclusion
This article examined the use of subprocess.Popen() in running an external command in python when you have more complex requirements such as reading from and writing to the child process. Some simple scenarios have been presented here. In the next article, we shall deal with more complex situations, including reading and writing from multiple threads.
The previous part of this article can be found here.