Is there a way to set custom cookies when making a request with Requests?

Yes, there is a way to set custom cookies when making HTTP requests with the requests library in Python. You can use the cookies parameter in the request methods (get, post, put, etc.) to send cookies to the server.

Here is an example of how to set custom cookies when making a GET request with the requests library:

import requests

# Define the URL you want to scrape
url = 'http://example.com'

# Define your custom cookies
cookies = {
    'cookie_name1': 'cookie_value1',
    'cookie_name2': 'cookie_value2',
}

# Make the GET request, passing the custom cookies
response = requests.get(url, cookies=cookies)

# Inspect the response
print(response.text)

Similarly, for a POST request, you would pass the cookies in the same way:

import requests

# Define the URL you want to scrape
url = 'http://example.com'

# The data you want to send in the POST request
data = {
    'key1': 'value1',
    'key2': 'value2',
}

# Define your custom cookies
cookies = {
    'cookie_name1': 'cookie_value1',
    'cookie_name2': 'cookie_value2',
}

# Make the POST request, passing the custom cookies
response = requests.post(url, data=data, cookies=cookies)

# Inspect the response
print(response.text)

Remember to abide by the terms of service of the website you are scraping and ensure that you are not violating any privacy laws or regulations when using cookies.

Also, note that when you're scraping websites that require authentication or sessions, you might need to handle session cookies that are set after you log in. The requests.Session object can help maintain a session across multiple requests and handle cookies automatically:

import requests

# Create a session object
session = requests.Session()

# Define the URL for login
login_url = 'http://example.com/login'

# Login credentials
credentials = {
    'username': 'your_username',
    'password': 'your_password',
}

# Perform login
response = session.post(login_url, data=credentials)

# Now you can make authenticated requests with the session object
# The session will handle cookies for you
response = session.get('http://example.com/protected_page')

# Inspect the response from the protected page
print(response.text)

In this session-based example, you don't need to set the cookies manually since the session object will handle them for you. However, you can still pass additional custom cookies as required.