close

How do I read a huge file line by line in Python, without loading the entire thing into memory first?
http://www.yak.net/fqa/171.html
by rupe

How do I read a huge file line by line in Python, without loading the entire thing into memory first?


In Python, the most common way to read lines from a file is to do the following: 

for line in open('myfile','r').readlines():
do_something(line)


When this is done, however, the readlines() function loads the entire file into memory as it runs. A better approach for large files is to use the fileinput module, as follows:

import fileinput
for line in fileinput.input(['myfile']):
do_something(line)


the fileinput.input() call reads lines sequentially, but doesn't keep them in memory after they've been read.

 


Annotation by enki :

as 'file' is iteratable, why not simply iterate on it? (use iter() when you need more control over the iterator's state)

ex:

for line in open('myfile','r'):
  doSomething(line)

 


 
arrow
arrow
    全站熱搜
    創作者介紹
    創作者 peicheng 的頭像
    peicheng

    FLASHC

    peicheng 發表在 痞客邦 留言(0) 人氣()