How do I scrape and compare SEO-friendly URLs?

Scraping and comparing SEO-friendly URLs involves a few steps such as collecting URLs from websites, parsing them for comparison, and then performing the actual comparison. Here's how to approach this task in a systematic manner:

Step 1: Identifying the URLs

First, you need to scrape the URLs from the target websites. You can do this using web scraping libraries like requests and BeautifulSoup in Python, or using tools like Puppeteer or Cheerio in Node.js.

Python Example with requests and BeautifulSoup:

import requests
from bs4 import BeautifulSoup

def get_seo_friendly_urls(url):
    r = requests.get(url)
    soup = BeautifulSoup(r.text, 'html.parser')
    urls = [a['href'] for a in soup.find_all('a', href=True) if a['href'].startswith('http')]
    return urls

seo_urls = get_seo_friendly_urls('https://example.com')
print(seo_urls)

Node.js Example with Puppeteer:

const puppeteer = require('puppeteer');

async function getSeoFriendlyUrls(url) {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url);

    const urls = await page.$$eval('a[href]', anchors => {
        return anchors.map(anchor => anchor.href).filter(href => href.startsWith('http'));
    });

    await browser.close();
    return urls;
}

getSeoFriendlyUrls('https://example.com').then(urls => {
    console.log(urls);
});

Step 2: Parsing URLs

After scraping the URLs, you should parse them to extract the paths and parameters which will be important for the comparison. The Python urllib.parse module and the Node.js url module can be used for parsing URLs.

Python Example with urllib.parse:

from urllib.parse import urlparse

parsed_urls = [urlparse(url) for url in seo_urls]
for parsed_url in parsed_urls:
    print(parsed_url.path)  # You can also access query parameters with parsed_url.query

Node.js Example with URL:

const { URL } = require('url');

seo_urls.forEach(url => {
    const parsedUrl = new URL(url);
    console.log(parsedUrl.pathname);  // You can also access query parameters with parsedUrl.searchParams
});

Step 3: Comparing URLs

To compare SEO-friendly URLs, you'll want to consider what aspects of the URLs are important for SEO. Typically, these might include the URL path, usage of keywords, URL length, and whether the URL uses a query string or is more RESTful.

Here's a simple comparison by path:

Python Example:

def compare_urls(url1, url2):
    parsed_url1 = urlparse(url1)
    parsed_url2 = urlparse(url2)

    return parsed_url1.path == parsed_url2.path

# Example usage:
url_comparison = compare_urls('https://example.com/page1', 'https://example.com/page1?query=123')
print(url_comparison)  # Output: True or False based on the paths being equal

Node.js Example:

function compareUrls(url1, url2) {
    const parsedUrl1 = new URL(url1);
    const parsedUrl2 = new URL(url2);

    return parsedUrl1.pathname === parsedUrl2.pathname;
}

// Example usage:
const urlComparison = compareUrls('https://example.com/page1', 'https://example.com/page1?query=123');
console.log(urlComparison);  // Output: true or false based on the paths being equal

Remember, comparing SEO-friendly URLs for SEO purposes is not just about string comparison. You might need a more sophisticated approach that involves checking for keyword usage, URL structure, and other SEO best practices. For a more thorough comparison, consider using dedicated SEO tools or developing custom logic that aligns with the specific SEO guidelines you're trying to follow.