How do I upgrade from urllib2 to urllib3?

urllib2 is a module available in Python 2.x for opening and reading URLs. However, Python 3.x deprecated urllib2 and introduced urllib.request for the same purpose. urllib3, on the other hand, is a third-party library that offers more features than the built-in modules. It provides connection pooling, thread safety, client-side SSL/TLS verification, file post support, etc.

To upgrade from urllib2 to urllib3, you need to install urllib3 and adjust your code to use its interface. Below are the steps and examples to guide you through the process.

Step 1: Install urllib3

First, you need to install the urllib3 library. You can do this using pip:

pip install urllib3

Step 2: Replace urllib2 with urllib3 in your code

You will have to modify your existing code to use the urllib3 API. Here's an example of how you might convert a simple urllib2 request to urllib3.

Using urllib2 (Python 2.x):

import urllib2

response = urllib2.urlopen('http://httpbin.org/ip')
data = response.read()
print(data)

Using urllib3 (Python 3.x):

import urllib3

http = urllib3.PoolManager()
response = http.request('GET', 'http://httpbin.org/ip')
data = response.data.decode('utf-8')  # Decode from bytes to string if necessary
print(data)

Step 3: Handle Exceptions Differently

urllib3 has its exception classes; you’ll need to update your exception handling to reflect this. Here is an example of handling exceptions in both libraries.

Using urllib2 (Python 2.x):

import urllib2
from urllib2 import HTTPError, URLError

try:
    response = urllib2.urlopen('http://httpbin.org/status/404')
    data = response.read()
except HTTPError as e:
    print('HTTP error:', e.code)
except URLError as e:
    print('URL error:', e.reason)

Using urllib3 (Python 3.x):

import urllib3
from urllib3.exceptions import HTTPError, RequestError

http = urllib3.PoolManager()

try:
    response = http.request('GET', 'http://httpbin.org/status/404')
    data = response.data
except HTTPError as e:
    print('HTTP error:', e.response.status)
except RequestError as e:
    print('Request error:', e.reason)

Step 4: SSL/TLS Verification

urllib3 by default verifies SSL certificates for HTTPS requests. If you need to disable SSL warnings (not recommended for production), you can suppress them as follows:

import urllib3

# Disable SSL warnings (not recommended for production code)
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

http = urllib3.PoolManager(cert_reqs='CERT_NONE')
response = http.request('GET', 'https://self-signed.badssl.com/')
print(response.data.decode('utf-8'))

Step 5: Encoding Data for POST Requests

If you need to send POST data, you will encode it with urllib3 differently than with urllib2.

Using urllib2 (Python 2.x):

import urllib2
import urllib

data = urllib.urlencode({'field': 'value'})
response = urllib2.urlopen('http://httpbin.org/post', data)
print(response.read())

Using urllib3 (Python 3.x):

import urllib3

http = urllib3.PoolManager()
response = http.request(
    'POST',
    'http://httpbin.org/post',
    fields={'field': 'value'}
)
print(response.data.decode('utf-8'))

Step 6: File Uploads

urllib3 also simplifies file uploads:

import urllib3

http = urllib3.PoolManager()

with open('example.txt', 'rb') as f:
    file_data = f.read()

response = http.request(
    'POST',
    'http://httpbin.org/post',
    fields={
        'filefield': ('example.txt', file_data, 'text/plain')
    }
)
print(response.data.decode('utf-8'))

By following these steps and adjusting your code accordingly, you can upgrade from urllib2 to urllib3 successfully. Remember that urllib3 is a powerful library and offers a lot more than what is covered in this quick guide, so be sure to read its documentation to take full advantage of its features.