Query With Paging By Cursor Causes Error Because Of Limitations For "in Filter" In Cursor() Method... What Should Be The Alternative?

July 20, 2023 Post a Comment

I am developing a twitter like microblogging system by using the following models: class Member(db.Model): user = db.UserProperty(required=True) follower_count = db.Int

Solution 1:

A quick and nasty way...

Download this pagintor.py import it into your project.

Then you can do something like this for pagination

from paginator import Paginator, InvalidPage, EmptyPage
     model = Member.all().fetch(100)
     paginator = Paginator(model,5)

                if(self.request.GET):
                    page = int(self.request.GET.get('page', '1'))
                    if(page isnot None):
                        try:
                            page = int(self.request.GET.get('page', '1'))
                        except ValueError:
                            page = 1# If page request (9999) is out of range, deliver last page of results.try:
                            paginator = paginator.page(page)
                        except (EmptyPage, InvalidPage):
                            paginator = paginator.page(paginator.num_pages)
    return self.response.out.write( template.render(path+'.html',{'paginator':paginator}))


#In templates

{% if paginator.object_list %}

{% for values in paginator.object_list %}

#do your tasks

{% endfor %}
<div  align="right"class="pagination" >
        {% if paginator.has_previous %}
            <a  id="previous" href="{{ paginator.previous_page_number }}">Previous</a>
        {% else %}
         <span class="page-nulled" >
            Previous
        </span>
        {% endif %}

        <span class="current" id="pagenum" title="{{ paginator.number }}">
            &nbsp;&nbsp;&nbsp;Page {{ paginator.number }} of {{paginator.paginator.num_pages }}&nbsp;&nbsp;&nbsp;
        </span>

        {% if paginator.has_next %}
            <a  id="next" href="{{ paginator.next_page_number }}"> Next </a>
            {% else %}
         <span class="page-nulled" >
            Next 
        </span>
        {% endif %}

</div>

on click of next or previous , take the href val() and pass to the url as get variable like http://someurl?page=.

more reference here

Solution 2:

The reason for this restriction is that IN and != queries are executed by splitting the query into multiple underlying queries, which are executed individually by the datastore, then merged together in sorted order.

If you want to do a query like this in a paginated fashion, you would have to execute the queries yourself, and do the merge yourself. To fetch a cursor, you'd need to fetch cursors from the individual sub-queries and concatenate them together. Further, you'd need to keep track of how many fetched but un-consumed results you had, so you can pick up exactly from where you left off.

As you can see, this is complicated and leads to excessively long cursor values, which is why it isn't implemented by the SDK currently. Unfortunately, it's the only practical way to do it, unless you can find a way to avoid the use of an IN clause, or drop your requirement for ordering by another one (in which case you could just execute the queries serially, paginating each).

Solution 3:

My solution is using the date value like a cursor as I had described as a comment to Nick Johnson's answer... It is like this:

if cursor: # This is not actually a cursor! It is base64 datetime string
  cursordate = _strptime(base64.b64decode(cursor)) # _strptime is a local method that converts str to datetime# IN has a limit for lists: 30 items allowed
listofNewsLists = []
listofMemberLists = [followed_member_list[i:i+30] for i inrange(0, len(followed_member_list), 30)]
for eachList in listofMemberLists:
   query = NewsItem.all()
   query.filter('posted_by IN', eachList).filter('status =', 1)
   if cursor:
      query.filter('posted_on <', cursordate)
   query.order('-posted_on')                        
   listofNewsLists.append(query.fetch(PAGE_SIZE))

  newsList = []
  if listofNewsLists:
    emptyListCount = 0whilelen(newsList) < PAGE_SIZE and emptyListCount < len(listofNewsLists):
      max = datetime.datetime.min
      maxInd = -1
      emptyListCount = 0for i inrange(len(listofNewsLists)):
        if listofNewsLists[i] == []:
          emptyListCount += 1elif listofNewsLists[i][0].posted_on > max:
          max = listofNewsLists[i][0].posted_on
          maxInd = i
      ifmax > datetime.datetime.min:
        newsList.append(listofNewsLists[maxInd].pop(0))

template_values['cursor'] = base64.b64encode(newsList[-1].posted_on.isoformat())

That is; I store the last displayed item's date value as the starting point of the new list...

This works well (I guess) unless I have items with same posted_on value...

Baca Juga

Solution 4:

Consider you have a member object named member_obj. From your model you can do something like this

To get the list of followings,

member_obj.followings will give you the list of keys , simply pass it to

followings_keys = member_obj.followings
Member.get(following_keys).

Then just loop through the members and display their messages.. This post on Modeling in appengine will help you a lot.

Solution 5:

I've been able to solve this problem using the search API.

https://developers.google.com/appengine/docs/python/search/

You need to mirror the relevant bits of your objects in a search document, and save it to an index in _pre_put_hook or _post_put_hook (and you can clean these out in your _pre_delete_hook or _post_delete_hook). Use a serialization of your key as the search doc_id.

Once you've got this going, you can do the kind of searching using Search, that you're trying to do above. It's fast! Just return doc_ids, then use them to get() your datastore objects.

Getting Started with Python