What user-agent should I use for scraping Idealista?

When scraping a website like Idealista, it's important to be respectful of their terms of service and to ensure that your scraping activities don't overload their servers. As for the user-agent string, it is typically used to identify the type of device and browser making the request. Websites might serve different content or behave differently depending on the user-agent string.

To scrape a website without running afoul of their terms of service, you should:

1. Check Idealista's robots.txt file: This file, usually found at https://www.idealista.com/robots.txt, will tell you which parts of the site the webmasters prefer that automated agents do not access.
2. Read the Terms of Service: Ensure that scraping is not against their terms. If it is, you should not proceed with scraping.
3. Rate Limiting: Make requests at a reasonable pace to avoid placing too much load on the site's servers.
4. User-Agent String: Use a legitimate user-agent string that identifies your bot as a scraper. This can be the user-agent of a popular browser, but it should be honest about your intentions. Some websites may block requests that do not have a user-agent string or that use a known scraper user-agent.

Here's how you might set a user-agent in Python using the requests library and in JavaScript using axios:

Python Example with requests:

import requests

url = 'https://www.idealista.com'
headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; YourBot/1.0; +http://yourbotdomain.com)'
}

response = requests.get(url, headers=headers)

# Make sure to handle status codes and potential errors
if response.status_code == 200:
    html_content = response.text
    # Continue with processing the response
else:
    print(f"Failed to retrieve content, status code: {response.status_code}")

JavaScript Example with axios:

const axios = require('axios');

const url = 'https://www.idealista.com';
const headers = {
    'User-Agent': 'Mozilla/5.0 (compatible; YourBot/1.0; +http://yourbotdomain.com)'
};

axios.get(url, { headers })
    .then(response => {
        const html_content = response.data;
        // Continue with processing the response
    })
    .catch(error => {
        console.error(`Failed to retrieve content: ${error}`);
    });

Remember to replace 'YourBot/1.0; +http://yourbotdomain.com' with the user-agent for your own bot, including a link to a webpage that explains the purpose of your bot and provides contact information, which is a common courtesy in the scraping community.

However, if Idealista has an API, that's usually the best option to obtain data from the site, both legally and technically. Always prefer using an API over scraping if one is available and it meets your data needs. If you do scrape, make sure to follow good practices and avoid disrupting the service.