How To Get Browser Network Logs Using Python Selenium

I'm trying to get browser network logs using selenium to debug request/responses. Could you please help me to find out a way. And I'm using selenium 3.14.0 and latest Chrome browse

Solution 1:

Using python + selenium + firefox

Don't set up a proxy unless you have to- in order to get outbound API requests I used the solution from this answer, but in python: https://stackoverflow.com/a/45859018/14244758

test = driver.execute_script("var performance = window.performance || window.mozPerformance || window.msPerformance || window.webkitPerformance || {}; var network = performance.getEntries() || {}; return network;")

for item intest:
  print(item)

You get an array of dicts.

This allows me to see all the network requests made. I'm using it to parse out a parameter from one of the requests so that I can use it to make my own requests against the API.

Solution 2:

Using Python and ChromeDriver

To get network logs, you need to install BrowserMobProxy as well along with selenium in python

pip install browsermob-proxy

Then we need to download the browsermobproxy zip from https://bmp.lightbody.net/.

Unzip it to any folder(For e.g. path/to/extracted_folder). This folder contains the browsermob-proxy binary file. We need to mention this path while calling Server() in python code

You need to start browser proxy and configure the proxy in chrome option of chrome driver,

from browsermobproxy import Server
from selenium import webdriver

server = Server("path/to/extracted_folder/bin/browsermob-proxy")
server.start()
proxy = server.create_proxy()

# Configure the browser proxy in chrome options
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--proxy-server={0}".format(proxy.proxy))
browser = webdriver.Chrome(chrome_options = chrome_options)

#tag the har(network logs) with a name
proxy.new_har("google")

Then you can navigate to page using selenium

browser.get("http://www.google.co.in")

After navigation, you can get the network logs in json format from the proxy

print(proxy.har) # returns a Network logs (HAR) as JSON 

Also before quitting the driver, stop the proxy server as well at the end,

server.stop()
browser.quit()

Solution 3:

Try selenium-wire, I think this is a better way which also provides undetected-chromedriver against bot detection.

Solution 4:

For the latest python selenium version 4.1.0, webdriver.get_log(self, log_type) only have 4 type logs

driver.get_log('browser')
driver.get_log('driver')
driver.get_log('client')
driver.get_log('server')

can't get performace log by driver.get_log function

Solution 5:

To get only the network logs up until the page has finished loading (no ajax/async network logs during the main usage of the page), you can get the Performance Log: http://chromedriver.chromium.org/logging/performance-log

To enable the Performance Logging for the ChromeDriver, for example,

DesiredCapabilitiescap= DesiredCapabilities.chrome();
LoggingPreferenceslogPrefs=newLoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability(CapabilityType.LOGGING_PREFS, logPrefs);
RemoteWebDriverdriver=newRemoteWebDriver(newURL("http://127.0.0.1:9515"), cap);

The chromium performance-log page also links to this complete example https://gist.github.com/klepikov/5457750 which has Java and python code to get the Performance Logs.

Again, it's important to keep in mind that this will only get the network requests up until the point that the page is finished loading. After that, the driver will only return the same performance logs until the page reloads.

---

If you want to get network logs asynchronously throughout the usage of the page, you can use BrowserMobProxy to act as a proxy server for your Selenium driver and capture all those network requests. Then, you can get those captured requests from BrowserMobProxy's generated HAR file: https://github.com/lightbody/browsermob-proxy#using-with-selenium

// start the proxyBrowserMobProxyproxy=newBrowserMobProxyServer();
proxy.start(0);

// get the Selenium proxy objectProxyseleniumProxy= ClientUtil.createSeleniumProxy(proxy);

// configure it as a desired capabilityDesiredCapabilitiescapabilities=newDesiredCapabilities();
capabilities.setCapability(CapabilityType.PROXY, seleniumProxy);

// start the browser upWebDriverdriver=newFirefoxDriver(capabilities);

// enable more detailed HAR capture, if desired (see CaptureType for the complete list)
proxy.enableHarCaptureTypes(CaptureType.REQUEST_CONTENT, CaptureType.RESPONSE_CONTENT);

// create a new HAR with the label "yahoo.com"
proxy.newHar("yahoo.com");

// open yahoo.com
driver.get("http://yahoo.com");

// get the HAR dataHarhar= proxy.getHar();

Once you have the HAR file, it is a JSON like list of network events that you can work with.