Python Subprocess: Giving Stdin, Reading Stdout, Then Giving More Stdin

I'm working with a piece of scientific software called Chimera. For some of the code downstream of this question, it requires that I use Python 2.7. I want to call a process, give

Solution 1:

you don't need to use process.communicate in your example.

Simply read and write using process.stdin.write and process.stdout.read. Also make sure to send a newline, otherwise read won't return. And when you read from stdin, you also have to handle newlines coming from echo.

Note: process.stdout.read will block until EOF.

# talk_with_example_input.pyimport subprocess

process = subprocess.Popen(["./exampleInput.sh"], 
                        stdin = subprocess.PIPE,
                        stdout = subprocess.PIPE)

process.stdin.write("5\n")
stdout = process.stdout.readline()
print(stdout)

if stdout == "25\n":
    process.stdin.write("yes\n")
    print(process.stdout.readline())

$ python2 test.py
25

Okay, moving on...

---

Update

When communicating with an program in that way, you have to pay special attention to what the application is actually writing. Best is to analyze the output in a hex editor:

$ chimera --nogui 2>&1 | hexdump -C

Please note that readline only reads to the next newline (\n). In your case you have to call readline at least four times to get that first block of output.

If you just want to read everything up until the subprocess stops printing, you have to read byte by byte and implement a timeout. Sadly, neither read nor readline does provide such a timeout mechanism. This is probably because the underlying read syscall (Linux) does not provide one either.

On Linux we can write a single-threaded read_with_timeout() using poll / select. For an example see .

from select import epoll, EPOLLIN

defread_with_timeout(fd, timeout__s):
    """Reads from fd until there is no new data for at least timeout__s seconds.

    This only works on linux > 2.5.44.
    """
    buf = []
    e = epoll()
    e.register(fd, EPOLLIN)
    whileTrue:
        ret = e.poll(timeout__s)
        ifnot ret or ret[0][1] isnot EPOLLIN:
            break
        buf.append(
            fd.read(1)
        )
    return''.join(buf)

In case you need a reliable way to read non blocking under Windows and Linux, this answer might be helpful.

---

from the python 2 docs:

readline(limit=-1)

Read and return one line from the stream. If limit is specified, at most limit bytes will be read.

The line terminator is always b'\n' for binary files; for text files, the newline argument to open() can be used to select the line terminator(s) recognized.

from man 2 read:

#include<unistd.h>ssize_tread(int fd, void *buf, size_t count);

example

$ tree
.
├── prog.py
└── prog.sh

prog.sh

#!/usr/bin/env bashfor i in $(seq 3); doecho"${RANDOM}"sleep 1
donesleep 3
echo"${RANDOM}"

prog.py

# talk_with_example_input.pyimport subprocess
from select import epoll, EPOLLIN

defread_with_timeout(fd, timeout__s):
    """Reads from f until there is no new data for at least timeout__s seconds.

    This only works on linux > 2.5.44.
    """
    buf = []
    e = epoll()
    e.register(fd, EPOLLIN)
    whileTrue:
        ret = e.poll(timeout__s)
        ifnot ret or ret[0][1] isnot EPOLLIN:
            break
        buf.append(
            fd.read(1)
        )
    return''.join(buf)

process = subprocess.Popen(
    ["./prog.sh"],
    stdin = subprocess.PIPE,
    stdout = subprocess.PIPE
)

print(read_with_timeout(process.stdout, 1.5))
print('-----')
print(read_with_timeout(process.stdout, 3))

$ python2 prog.py 
6194
14508
11293

-----
10506