Can't Collect Profile Names From Identical Urls Using Requests

I'm trying to log into facebook using selenium and then transfer cookies to requests module so that I can collect the profile name from the two urls using requests. The profile nam

Solution 1:

Although the content you are interested in are not dynamic, they are commented out. Try the following to achieve that:

forlinkin links:
    content = requests.get(link,headers={'User-Agent':'Mozilla/5.0'},cookies=c).text
    comment = content.replace("-->", "").replace("<!--", "")
    soup = BeautifulSoup(comment,"lxml")
    name = soup.select_one("#fb-timeline-cover-name > a").text
    print(name)

In my opinion, using session is what you wanna do to make it robust:

s = requests.Session()
[s.cookies.set(cookie['name'],cookie['value']) for cookie in driver.get_cookies()]

forlinkin links:
    content = s.get(link,headers={'User-Agent':'Mozilla/5.0'}).text
    comment = content.replace("-->", "").replace("<!--", "")
    soup = BeautifulSoup(comment,"lxml")
    name = soup.select_one("#fb-timeline-cover-name > a").text
    print(name)