How do I handle cookies with the Requests library?

Cookies are often used by websites to store session information. When using the Python requests library, you can handle cookies quite easily either by sending cookies to the server or by handling cookies set by the server.

Sending Cookies to the Server

You can send cookies to a server by including them in the headers of your request or by using the cookies parameter.

Using the cookies parameter:

import requests

url = 'http://example.com/some/cookie/setting/url'
cookies = {'session_token': '123456789', 'visited': 'yes'}
response = requests.get(url, cookies=cookies)

print(response.text)

Setting cookies in the headers manually:

import requests

url = 'http://example.com/some/cookie/setting/url'
headers = {
    'Cookie': 'session_token=123456789; visited=yes'
}
response = requests.get(url, headers=headers)

print(response.text)

Handling Cookies Set by the Server

When you make a request to a server that sets cookies, the requests library will automatically handle the cookies for you. You can access the cookies from the response object.

import requests

url = 'http://example.com/cookies'
response = requests.get(url)

# Accessing cookies set by the server
cookies = response.cookies

# You can access cookies for a specific domain or path
for cookie in cookies:
    print(cookie.name, cookie.value)

# Or you can convert them to a dictionary
cookie_dict = requests.utils.dict_from_cookiejar(response.cookies)
print(cookie_dict)

Maintaining a Session with Cookies

If you want to maintain a session across multiple requests, you can use a requests.Session() object. This will keep track of cookies between your requests.

import requests

# Start a session
session = requests.Session()

# Make a request
response = session.get('http://example.com/cookies')

# Cookies saved within the session
print(session.cookies)

# Make another request (cookies will be sent automatically)
response = session.get('http://example.com/use/cookies')

# Inspecting cookies received in the second request
print(response.text)

Saving and Loading Cookies

You might want to save cookies to a file and load them later to maintain the session even after your script is restarted.

Saving cookies to a file:

import requests
import pickle

# Create a session
session = requests.Session()
session.get('http://example.com/cookies')

# Save cookies to a file
with open('cookies.pkl', 'wb') as f:
    pickle.dump(session.cookies, f)

Loading cookies from a file:

import requests
import pickle

# Create a session
session = requests.Session()

# Load cookies from a file and set them in the session
with open('cookies.pkl', 'rb') as f:
    cookies = pickle.load(f)
    session.cookies.update(cookies)

# Now you can make requests with your saved session
response = session.get('http://example.com/use/cookies')
print(response.text)

Please note that when dealing with cookies, you should be aware of the website's terms of service and privacy policy. Some websites do not allow automated access or web scraping, so always ensure that you have permission to interact with a website programmatically.