Sorting File In Place With Python On Unix System

I'm sorting a text file from Python using a custom unix command that takes a filename as input (or reads from stdin) and writes to stdout. I'd like to sort myfile and keep the sort

Solution 1:

If you don't want to create temporary files, you can use subprocess as in:

import sys
import subprocess

fname = sys.argv[1]
proc = subprocess.Popen(['sort', fname], stdout=subprocess.PIPE)
stdout, _ = proc.communicate()
with open(fname, 'w') as f:
    f.write(stdout)

Solution 2:

You either create a temporary file, or you'll have to read the whole file into memory and pipe it to your command.

Solution 3:

The best solution is to use os.replace because it would work on Windows too.

This is not really what I regards as "in-place sorting" though. Usually, in-place sorting means that you actually exchange single elements in the lists without doing copies. You are making a copy since the sorted list has to get completely built before you can overwrite the original. If your files get very large, this obviously won't work anymore. You'd probably need to choose between atomicity and in-place-ity at that point.

If your Python is too old to have os.replace, there are lots of resources in the bug adding os.replace.

For other uses of temporary files, you can consider using the tempfile module, but I don't think it would gain you much in this case.

Solution 4:

You could try a truncate-write pattern:

withopen(filename, 'r') as f:
   model.read(f)
model.process()
withopen(filename, 'w') as f:
   model.write(f)

Note this is non-atomic

This entry describes some pros/cons of updating files in Python: http://blog.gocept.com/2013/07/15/reliable-file-updates-with-python/