Skip to content Skip to sidebar Skip to footer

Searching + Reverse Seeking A Pickled File , Values Getting Skipped

Minimum reproducible example, only goto_index() is being used in my code. The rest is self-explanatory : import pickle,os def goto_index(idx_str,src,dest=False) : '''Go to ind

Solution 1:

The problem was a combination of :

  • obj_cnt was not persistent during function calls so always started from scratch even while the file position was modified in each call, so goto_idx() acted as though it was at BOF but would instead be much ahead.
  • Seeking to start of object at index (src.seek(-len(pickle.dumps(obj)),os.SEEK_CUR)) caused the next read to read the same object it did before - if the previous bug was fixed, this would lead to goto_index()always going to and returning the object at the index from it's first call ever.

I fixed it by a) putting the function in a class where it can access a count variable, b) adding an additional flag fp_set and only seeking back if it is set to a true value, c) providing a reset() method in the class so as to reset obj_cnt to -1 when done with an ordered series of queries.

Keep in mind I am very new to OOP in python and something is probably weird in the below code :

classgoto_index:
    obj_cnt = -1# 0-based countdefsorted(idx_str,src,dest=None,fp_set=False) :
    #Use if going to indexes in ascending order in loop# idx_str = comma-seperated index , eg : "7,8" like foo[7][8]# src     = file object to search in, from it's current position# dest    = if True, will copy all objects until obj @ idx_str found OR EOF# fp_set  = if True, will seek such that next read will return obj @ idx_str
    
        index = [int(subidx)-1for subidx in idx_str.split(',')]
        # Make 0-based int list from 1-based csv string 
        val = Nonetry :
            whileTrue :                            # EOFError if not found
                obj = pickle.load(src)
                goto_index.obj_cnt += 1# increment counterif goto_index.obj_cnt == index[0] : # 1st element of index is object number
                    val = obj
                    for subidx in index[1::] :      # Index the object itself
                        val = val[subidx]           # IndexError if illegal indexif fp_set : src.seek(-len(pickle.dumps(obj)),os.SEEK_CUR)
                    # Seek back to begining of object in srcreturn val                      # Return value @ indexelif dest : pickle.dump(obj,dest)   # Copy object to destexcept (EOFError, IndexError) : raise# Caller handles these defreset():
        goto_index.obj_cnt = -1defrandom(idx_str,src,dest=None,fp_set=False) :
        goto_index.reset() # Just in case
        src.seek(0)        # Just in case       
        goto_index.sorted(idx_str,src,dest=None,fp_set=False)
        goto_index.reset() # Clear count

Where the question's other function are basically the same except fetch_elements() :

deffetch_elements(f) :
    elements = []
    for idx_str in ('1','2','3') : # Indexes are passed sorted
        elements.append(goto_index.sorted(idx_str,f))
    goto_index.reset()            # Required if using the methods laterreturn elements

Post a Comment for "Searching + Reverse Seeking A Pickled File , Values Getting Skipped"