Subprocess Stdout/stderr To Finite Size Logfile

I have a process which chats a lot to stderr, and I want to log that stuff to a file. foo 2> /tmp/foo.log Actually I'm launching it with python subprocess.Popen, but it may a

Solution 1:

You should be able to use the stdlib logging package to do this. Instead of connecting the subprocess' output directly to a file, you can do something like this:

import logging

logger = logging.getLogger('foo')

defstream_reader(stream):
    whileTrue:
        line = stream.readline()
        logger.debug('%s', line.strip())

This just logs every line received from the stream, and you can configure logging with a RotatingFileHandler which provides log file rotation. You then arrange to read this data and log it.

foo_proc = subprocess.Popen(['foo'], stderr=subprocess.PIPE)

thread = threading.Thread(target=stream_reader, args=(foo_proc.stderr,))
thread.setDaemon(True) # optional 
thread.start()

# do other stuff

thread.join() # await thread termination (optionalfor daemons)

Of course you can call stream_reader(foo_proc.stderr) too, but I'm assuming you might have other work to do while the foo subprocess does its stuff.

Here's one way you could configure logging (code that should only be executed once):

import logging, logging.handlershandler= logging.handlers.RotatingFileHandler('/tmp/foo.log', 'a', 100000, 10)
logging.getLogger().addHandler(handler)
logging.getLogger('foo').setLevel(logging.DEBUG)

This will create up to 10 files of 100K named foo.log (and after rotation foo.log.1, foo.log.2 etc., where foo.log is the latest). You could also pass in 1000000, 1 to give you just foo.log and foo.log.1, where the rotation happens when the file would exceed 1000000 bytes in size.

Solution 2:

The way with circular buffer would be hard to implement, as you would constantly have to rewrite the whole file as soon as something falls out.

The approach with logrotate or something would be your way to go. In this case, you simply would do similiar to this:

import subprocess
import signal

defhupsignal(signum, frame):
    global logfile
    logfile.close()
    logfile = open('/tmp/foo.log', 'a')

logfile = open('/tmp/foo.log', 'a')
signal.signal()
foo_proc = subprocess.Popen(['foo'], stderr=subprocess.PIPE)
for chunk initer(lambda: foo_proc.stderr.read(8192), ''):
    # iterate until EOF occurs
    logfile.write(chunk)
    # or do you want to rotate yourself?# Then omit the signal stuff and do it here.# if logfile.tell() > MAX_FILE_SIZE:#     logfile.close()#     logfile = open('/tmp/foo.log', 'a')

It is not a complete solution; think of it as pseudocode as it is untested and I am not sure about the syntax in the one or other place. Probably it needs some modification for making it work. But you should get the idea.

As well, it is an example of how to make it work with logrotate. Of course, you can rotate your logfile yourself, if needed.

Solution 3:

You may be able to use the properties of 'open file descriptions' (distinct from, but closely related to, 'open file descriptors'). In particular, the current write position is associated with the open file description, so two processes that share an single open file description can each adjust the write position.

So, in context, the original process could retain the file descriptor for standard error of the child process, and periodically, when the position reaches your 1 MiB size, reposition the pointer to the start of the file, thus achieving your required circular buffer effect.

The biggest problem is determining where the current messages are being written, so that you can read from the oldest material (just in front of the file position) to the newest material. It is unlikely that new lines overwriting the old will match exactly, so there'd be some debris. You might be able to follow each line from the child with a known character sequence (say 'XXXXXX'), and then have each write from the child reposition to overwrite the previous marker...but that definitely requires control over the program that's being run. If it is not under your control, or cannot be modified, that option vanishes.

An alternative would be to periodically truncate the file (maybe after copying it), and to have the child process write in append mode (because the file is opened in the parent in append mode). You could arrange to copy the material from the file to a spare file before truncating to preserve the previous 1 MiB of data. You might use up to 2 MiB that way, which is a lot better than 500 MiB and the sizes could be configured if you're actually short of space.

Have fun!