Read Git Clone's Output In Real Time

I need to read the status ( Receiving objects XXX%) of a git clone process, but can't figure out how. I am using subprocess.Popen but I cant' figure out how to grab the line I need

Solution 1:

[edited as per the comments] Calling git clone with the --progress flag redirects the output ok.

This works for me.

import subprocess

popen = subprocess.Popen(["git", "clone", "--progress", "git@bitbucket.org:xxx/yyy.git"], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line inpopen.stdout:
    print line,

print"done"

Gives the output

$ python test.py 
Cloning into 'yyy'...
remote: Counting objects: 1076, done.
remote: Compressing objects: 100% (761/761), done.
remote: Total 1076 (delta 488), reused 576 (delta 227)
Receiving objects: 100% (1076/1076), 6.24 MiB | 260.00 KiB/s, done.
Resolving deltas: 100% (488/488), done.
Checking connectivity... done.
done

[edit] Like Cristian Ciupitu points out you don't need the iter() just for line in popen.stdout: works just as well (or not depending on version).

Solution 2:

I assume that you want the real-time communication to display some sort of progress while the process is still running.

The problem is that the normal stdout stream is buffered. What you need is the unbuffered stream. You can obtain it using the os module, for example:

  fd = proc.stdout.fileno()
  while proc.returncode isNone:
      l = os.read(fd, 1000)   # Read a bit of dataprint l

Solution 3:

I think a subprocess.communicate() should do the trick.

I usually do it like this:

process=subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdoutput,stderror=process.communicate()
for line in stdoutput:
    if line.startswith('Receiving objects'):
        print line

Solution 4:

First of all, I suggest combining stdout and stderr to have everything in one place i.e. call Popen with stderr=subprocess.STDOUT.

Second, you need to execute git with --progress because the git-clone man page says:

--progress

Progress status is reported on the standard error stream bydefaultwhen
it is attached to a terminal, unless -q is specified.  This flag forces
progress status even if the standard error stream isnot directed to a
terminal.

and your git is not attached to a terminal.

So the code should look like this:

popen = subprocess.Popen(["git", "clone", "--progress", url],
                         stdout=subprocess.PIPE,
                         stderr=subprocess.STDOUT)
for line inpopen.stdout:
  print"out newline:"print line
print"done"

I suggest reading the answers to the "Getting realtime output using subprocess" question to improve your program.

Solution 5:

Tagging on to other responses that have described stderr and --progress, here is a complete minimal working example that will print the git clone command in real time:

import os
import re
import subprocess

def test():

    url = 'http://github.com/octocat/Hello-World/'output = subprocess.Popen(['git', 'clone', '--progress', url], stderr=subprocess.PIPE, stdout=subprocess.PIPE)

    fd = output.stderr.fileno()
    while True:
        lines = os.read(fd,1000).decode('utf-8')
        lines = re.split('\n|\r', lines)
        for l inlines:
            if l != '':
                print(l)
        iflen(lines) == 1:
                breakprint('Press enter to continue.')