How to iterate over a file in chunks in Python?

Published on Aug. 22, 2023, 12:20 p.m.

To iterate over a file in chunks in Python, you can use a combination of the with keyword, the open() function, and a loop that reads a fixed number of bytes from the file. Here is an example:

chunk_size = 1024 # size of each chunk in bytes
with open('myfile.txt', 'rb') as file:
    while True:
        data = file.read(chunk_size)
        if not data:
            break
        # process the current chunk of data
        print(data)

In this example, we open the file ‘myfile.txt’ in binary mode (‘rb’), and then use a while loop to read chunks of data from the file using the read() method. If there is no more data to read, the loop exits. Inside the loop, you can perform whatever processing is necessary on the current chunk of data.

Note that you can adjust the chunk_size variable to control the size of each chunk of data read from the file.

Tags: