Batch/bulk Dns Lookup In Python?

I've got a script that gets DNS (CNAME, MX, NS) data in the following way: from dns import resolver ... def resolve_dns(url): response_dict = {} print '\nResolving DNS for

Solution 1:

You can put the work into a thread pool. Your resolve_dns does 3 requests serially so I created a slightly more generic worker that does only 1 query and used collections.product to generate all combinations. In the thread pool I set chunksize to 1 to reduce thread pool batching, which can increase exec time if some queries take a long time.

import dns
from dns import resolver
import itertools
import collections
import multiprocessing.pool

def worker(arg):
    """query dns for (hostname, qname) and return (qname, [rdata,...])"""
    try:
        url, qname = arg
        rdatalist = [rdata for rdata in resolver.query(url, qname)]
        return qname, rdatalist
    except dns.exception.DNSException, e:
        return qname, []

def resolve_dns(url_list):
    """Given a list of hosts, return dict that maps qname to
    returned rdata records.
    """
    response_dict = collections.defaultdict(list)
    # create pool for querys but cap max number of threads
    pool = multiprocessing.pool.ThreadPool(processes=min(len(url_list)*3, 60))
    # run for all combinations of hosts and qnames
    for qname, rdatalist in pool.imap(
            worker, 
            itertools.product(url_list, ('CNAME', 'MX', 'NS')),
            chunksize=1):
        response_dict[qname].extend(rdatalist)
    pool.close()
    return response_dict

url_list = ['example.com', 'stackoverflow.com']
result = resolve_dns(url_list)
for qname, rdatalist in result.items():
    print qname
    for rdata in rdatalist:
        print '   ', rdata