What is the 'stream' parameter in Requests, and when should I use it?

The stream parameter in the requests library in Python is an optional argument that you can pass to various request methods like get, post, put, delete, etc. When set to True, it alters the way the response is handled by the library.

By default, when you make a request with requests.get() or similar methods, the requests library will immediately download the entire content of the response from the server before it returns the Response object. This is because stream defaults to False.

However, when you're dealing with large files or slow connections, you might want to download the content in chunks rather than all at once. This is where the stream parameter comes in. Setting stream=True means that requests will not download the whole response immediately. Instead, it will provide a Response object and you can iterate over the response data in chunks as needed.

Here's a simple example of how to use the stream parameter:

import requests

# Make a request with stream=True
response = requests.get('http://example.com/bigfile', stream=True)

# Check if the request was successful
if response.status_code == 200:
    with open('bigfile', 'wb') as fd:
        for chunk in response.iter_content(chunk_size=128):
            fd.write(chunk)

In the example above, iter_content is a method that allows you to iterate over the response data in chunks of a specified size (in this case, 128 bytes). Writing the chunks directly to a file can be much more memory-efficient for large files.

You should use the stream parameter when:

1. You're dealing with large files that you don't want to load into memory all at once.
2. You want to download content in chunks and process them as they arrive (for example, if you want to display the progress of a download).
3. You're working with streaming APIs that send real-time data in chunks (like Twitter's streaming API).

Keep in mind that when you use stream=True, you need to ensure that the content is properly consumed and the connections are closed. If you don't consume the content or fail to close the connection, you can potentially leak sockets and create issues with connection pooling in requests. This is especially important in long-running processes. To properly close the connection, you should either fully consume the data or call response.close().

Also, when streaming, the responsibility of handling the download and ensuring data integrity (like checking for complete download, handling network errors, etc.) falls on your code. You need to be prepared to handle these scenarios manually.