python - Read the continuously growing file -

i ran command program 2> log.txt,

and write output log.txt continuously.

and output in same line, because every output has \r in end.

i'm trying read log.txt in way, truncate clear data i've read.

    read_in_file = open(in_file, 'r')     records = []     new_item in self.__get_matched_line(read_in_file):         records.append(new_item)      read_in_file.truncate()     read_in_file.close()     return records 

but has problem, still have-read data sometime.

the program may run weeks.and outputs every 0.5 seconds

your program have open file handle on output file; therefore if truncate file outside, file handle write old position, no matter how file had been truncated. result file of original length before truncation (the gap filled zeros, maybe read sparse files more details if choose path). can circumvent closing file handle , opening anew each line. best done within program of course. if cannot change that, can read line-by-line (delimited '\r') , append whole lines log file:

program |& while ifs= read -e -d $'\r' line; echo "$line" >> log.txt; done 

in case want stderr logged way, use pattern:

(program 2>&1 1>&3 | while ifs= read -e -d $'\r' line; echo "$line" >> log.txt; done) 3>&1 

this way, truncate on log file have desired effect.

but consider using truncate remove stuff end of file while keeping beginning. there no way keep end while removing beginning if use 1 file. using truncate remove parts read makes sense if truncate whole file (set 0 bytes).

so might want consider log rotation. that's well-defined process may have advantages on approach (and there facilities you).