Numpy Genfromtxt() Skip Invalid Lines

I am using Numpy's genfromtxt() function to get large amounts of txt data as array. The data is provided in following format: 2020-05-20 16:54:01.807645 1033.074 2392.555 25

Solution 1:

According to the manual you can also pass in a generator generating bytestrings (lines) to np.genfromtxt().

fname

• file, str, pathlib.Path, list of str, generator

File, filename, list, or generator to read. If the filename extension is gz or bz2, the file is first decompressed. Note that generators must return byte strings. The strings in a list or produced by a generator are treated as lines.

defskip_stopped_lines(fi):
    for line in fi:
        ifb"DUT stopped"in line:
            # Don't yield this line, we don't want itcontinueyield line


# NB: not using `with` to avoid the file being closed#     while the generators are active

fi = open("somefile.txt", "rb")  # note binary mode
skipper = skip_stopped_lines(fi)
values = np.genfromtxt(
    skipper,
    dtype="S32,f8,f8,f8,f8,f8,f8,f8,f8,f8,f8,f8",
    missing_values="",
    delimiter="\t",
    invalid_raise=False,
    filling_values=0,
)

might do the trick for you.