Elastic Search Performance Using Pyes

Sorry for cross posting.The following question is also posted on Elastic Search's google group. In short I am trying to find out why I am not able to get optimal performance while

Solution 1:

Isn't it just a matter of concurrency?

You're doing all your queries in sequence. So a query has to finish before the next one can come in to play. If you have a 1ms RTT to the server, this will limit you to 1000 requests per second.

Try to run a few instances of your script in parallel and see what kind of performance you got.

Solution 2:

There are severeal ways to improve that with using pyes.

• First of all try to get rid of the DottedDict class/object which is used to generat from every json/dict to an object for every result you get.
• Second switch the json encoder to ujson.

These two things brought up a lot of performance. This has the disadvantage that you have to use the ways to access dicts instead of the dotted version ("result.facets.attribute.term" instead you have to use something like "result.facets['attribute']['term']" or "result.facets.get('attribute', {}).get('term', None)" )

I did this through extending the ES class and replace the "_send_request" function.