What is the difference between urllib and urllib3 in Python?

urllib and urllib3 are both Python modules that allow you to work with URLs and perform HTTP requests. However, they are quite different in terms of their features, API design, and history.

urllib

urllib is a package that is part of the Python Standard Library, meaning it is included with Python and does not need to be installed separately. The urllib package is actually a collection of several modules:

• urllib.request for opening and reading URLs
• urllib.error for containing the exceptions raised by urllib.request
• urllib.parse for parsing URLs
• urllib.robotparser for parsing robots.txt files

urllib provides a very basic interface for making HTTP requests and dealing with URL-related functionality. It's suitable for simple tasks, but it lacks many features that are required for more complex web interactions, such as handling HTTP connection pooling, thread safety, file uploads, or automatic handling of cookies and redirects.

Here is an example of how you might use urllib.request to make a simple GET request:

import urllib.request

response = urllib.request.urlopen('http://httpbin.org/get')
print(response.read())

urllib3

urllib3, on the other hand, is a third-party HTTP client for Python that provides much more functionality than urllib. It is not included in the Python Standard Library, so it must be installed separately using pip. urllib3 offers features such as:

• Connection pooling
• Thread safety
• Full control over the connection re-use
• Support for file uploads
• Support for automatic handling of HTTP redirections and retries
• Support for gzip and deflate encoding
• SSL/TLS verification
• Chunked request support

urllib3 is a more powerful tool for complex web scraping and web interaction tasks. It's the library that underpins the popular requests module, which provides a higher-level HTTP client interface.

Here is an equivalent example using urllib3:

import urllib3

http = urllib3.PoolManager()
response = http.request('GET', 'http://httpbin.org/get')
print(response.data)

To install urllib3, you would typically use pip:

pip install urllib3

Summary

• urllib is part of the Python Standard Library and offers basic functionality for working with URLs and HTTP requests.
• urllib3 is a third-party library that provides a more extensive feature set for making HTTP requests and is suitable for more complex web interaction tasks. It must be installed separately.
• urllib3 is often preferred for serious web scraping and HTTP interaction due to its advanced features and performance advantages.
• If you need a higher-level HTTP client interface with an even simpler API, you might consider using the requests module, which is built on top of urllib3.