Beginner To Scraping, Keep On Getting Empty Lists

I've decided to take a swing at web scraping using Python (with lxml and requests). The webpage I'm trying to scrape to learn is: http://www.football-lineups.com/season/Real_Madrid

Solution 1:

If you inspect the row source of the page you will see that the lineup table is not there. It is fed after loading the page using AJAX so you wont be able to fetch it only by getting http://www.football-lineups.com/season/Real_Madrid/2013-2014 since the JS won't be interpreted and thus the AJAX not executed.

The AJAX request is the following:

• URL: http://www.football-lineups.com/ajax/get_sectf.php
• method: POST
• data: d1=3&d2=-2013&d3=0&d4=1&d5=0&d6=1&d7=20&d8=0&d9=&d10=0&d11=0&d12=undefined

Maybe you can forge the request to get this data. I'll let you analyse what are those well named dX arguments :)

Solution 2:

Here, I give full code which fulfill your requirement:

from selenium import webdriver
import csv
url="http://www.football-lineups.com/season/Real_Madrid/2013-2014"
driver=webdriver.Chrome('./chromedriver.exe')
driver.get(url)
myfile = open('demo.csv', 'wb')
wr = csv.writer(myfile, quoting=csv.QUOTE_ALL)
tr_list=driver.find_elements_by_xpath("//span[@id='sptf']/table/tbody/tr")
    for tr in tr_list:
    lst=[]
    for td in tr.find_elements_by_tag_name('td'):
        lst.append(td.text)
    wr.writerow(lst)
 driver.quit()
 myfile.close()