Python - Processing Large Text Files One Line At A Time
Python - Processing Large Text Files One Line At A Time
Submitted by Corey Goldberg on Wed, 07/11/2007 - 20:48.I want to process some very large text files one line at a time. Normally when I process text files, I slurp them into a list using the readlines() method. However, sometimes the files are huge and it isn't feasible or optimal to read the entire content into memory upfront. In this case, it makes sense to process them one line at a time.
The best solution I can come up with is this:
fh = open('foo.txt', 'r') line = fh.readline() while
line: # do something here line = fh.readline()
It doesn't feel very pythonic/idiomatic. Anyone have a better solution?
Update
Thanks to the comments below, I found a few different ways to do it. The best and
most Pythonic way seems to be this:
for line in open('foo.txt', 'r'): # do something here
Python file objects support the iterator protocol, so you can just open it and go. This is the same as using a while loop and calling readline() but more compact.
