Searching + Reverse Seeking A Pickled File , Values Getting Skipped
Minimum reproducible example, only goto_index() is being used in my code. The rest is self-explanatory : import pickle,os def goto_index(idx_str,src,dest=False) : '''Go to ind
Solution 1:
The problem was a combination of :
obj_cnt
was not persistent during function calls so always started from scratch even while the file position was modified in each call, sogoto_idx()
acted as though it was at BOF but would instead be much ahead.- Seeking to start of object at index (
src.seek(-len(pickle.dumps(obj)),os.SEEK_CUR)
) caused the next read to read the same object it did before - if the previous bug was fixed, this would lead togoto_index()
always going to and returning the object at the index from it's first call ever.
I fixed it by a) putting the function in a class where it can access a count variable, b) adding an additional flag fp_set
and only seeking back if it is set to a true value, c) providing a reset()
method in the class so as to reset obj_cnt
to -1
when done with an ordered series of queries.
Keep in mind I am very new to OOP in python and something is probably weird in the below code :
classgoto_index:
obj_cnt = -1# 0-based countdefsorted(idx_str,src,dest=None,fp_set=False) :
#Use if going to indexes in ascending order in loop# idx_str = comma-seperated index , eg : "7,8" like foo[7][8]# src = file object to search in, from it's current position# dest = if True, will copy all objects until obj @ idx_str found OR EOF# fp_set = if True, will seek such that next read will return obj @ idx_str
index = [int(subidx)-1for subidx in idx_str.split(',')]
# Make 0-based int list from 1-based csv string
val = Nonetry :
whileTrue : # EOFError if not found
obj = pickle.load(src)
goto_index.obj_cnt += 1# increment counterif goto_index.obj_cnt == index[0] : # 1st element of index is object number
val = obj
for subidx in index[1::] : # Index the object itself
val = val[subidx] # IndexError if illegal indexif fp_set : src.seek(-len(pickle.dumps(obj)),os.SEEK_CUR)
# Seek back to begining of object in srcreturn val # Return value @ indexelif dest : pickle.dump(obj,dest) # Copy object to destexcept (EOFError, IndexError) : raise# Caller handles these defreset():
goto_index.obj_cnt = -1defrandom(idx_str,src,dest=None,fp_set=False) :
goto_index.reset() # Just in case
src.seek(0) # Just in case
goto_index.sorted(idx_str,src,dest=None,fp_set=False)
goto_index.reset() # Clear count
Where the question's other function are basically the same except fetch_elements()
:
deffetch_elements(f) :
elements = []
for idx_str in ('1','2','3') : # Indexes are passed sorted
elements.append(goto_index.sorted(idx_str,f))
goto_index.reset() # Required if using the methods laterreturn elements
Post a Comment for "Searching + Reverse Seeking A Pickled File , Values Getting Skipped"