How do you extract and store data scraped by Mechanize?

Mechanize is a library in Python that acts as a programmable web browser, allowing you to perform web scraping, form submission, and other tasks typically done through a web browser. To extract and store data scraped by Mechanize, you typically follow these steps:

1. Install the Mechanize library if you haven't already.
2. Create a new Mechanize browser object.
3. Navigate to the webpage you want to scrape.
4. Use Mechanize methods to extract the data.
5. Store the data in a structured format, such as CSV, JSON, or a database.

Step 1: Install Mechanize

If you don't have Mechanize installed, you can install it via pip:

pip install mechanize

Step 2: Create Mechanize Browser Object

import mechanize

# Create a browser object
br = mechanize.Browser()

# Browser options
br.set_handle_equiv(True)
br.set_handle_gzip(True)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)

# Specify the user-agent
br.addheaders = [('User-agent', 'Firefox')]

Step 3: Navigate to the Webpage

# Navigate to the webpage
url = 'http://example.com'
br.open(url)

Step 4: Extract Data

You can use various methods to extract the data. For example, you can search for specific elements, find forms, follow links, or use regular expressions.

# Read the page content and do something with it
html = br.response().read()

# Use BeautifulSoup to parse the HTML if needed
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')

# Extract data
data = soup.find_all('div', {'class': 'data-container'})

Step 5: Store the Data

Once you have extracted the data, you can store it in various formats. Below are examples to store data in CSV and JSON formats.

Storing Data in CSV

import csv

# Assuming `data` is a list of dictionaries
keys = data[0].keys()  # Get the keys for CSV column names

with open('data.csv', 'w', newline='') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(data)

Storing Data in JSON

import json

# Assuming `data` is a list of dictionaries
with open('data.json', 'w') as json_file:
    json.dump(data, json_file)

Storing Data in a Database

Here is an example of storing data in an SQLite database using Python's sqlite3 module:

import sqlite3

# Connect to the SQLite database (or create it if it doesn't exist)
conn = sqlite3.connect('scraped_data.db')
c = conn.cursor()

# Create a table (if it is the first run)
c.execute('''CREATE TABLE IF NOT EXISTS data (id INTEGER PRIMARY KEY, name TEXT, value TEXT)''')

# Assuming `data` is a list of tuples (name, value)
c.executemany('INSERT INTO data (name, value) VALUES (?, ?)', data)

# Commit the transaction and close the connection
conn.commit()
conn.close()

Remember that web scraping should be done responsibly and in accordance with the terms of service of the website you are scraping. Always check robots.txt for the site's policy on web scraping and do not overload the website with too many requests in a short period.