Website search and extract/compare agent

James,
Very interesting that everyone seems to be egging you on while some technical aspects of what you have requested might be impossible. Circumventing captcha and similar scrapping protections might not work regardless of what approach you try. This systems have spend a lot of money just to counter this! This is the hard truth.

However, some of the apps you have listed provide real API with search endpoints that should work. Others, like eBay, Craigslist, and Vinted all have native "notify me of new listings" or RSS feeds for saved searches which gets you the exact "only show me what's new" behavior natively, no scraping needed, no CAPTCHA to fight.

For sites with no API or alerts, a human-in-the-loop browser session (you log in, Claude assists with reading/organizing results you're already looking at) is very different from an unattended daily bot login - but it would be a viable approach.

All in all, you will need different approaches to get close to what you have requested, since the 80+ apps would never share the same architecture and approach.

This brings the question of budget earlier asked back to focus. Considering the different implementations expected, how flexible is your budget?

Kind regards,
Robert.

Hi James, could you share the complete list of approximately 80 websites you want included so we can assess the access constraints and quote the scope accurately?

Hi James,

A few points I would confirm before finalising the scope:

1. Do you already have the search terms, filters, and locations documented for each website?

2. How many listings are typically returned by each daily search?

3. Do any websites use two-factor authentication or emailed login codes?

4. Should removed, sold, or changed listings also be tracked?

5. Would you prefer the system to run on your own computer, a cloud server, or through a managed browser environment?

6. Which 3–5 websites should be included in the initial proof of concept?

7. Do you have an allocated budget in mind for the first phase or the complete rollout?

Based on the complexity and the number of websites involved, I don't believe a complete 80-site solution would be achievable within a £750 budget. However, I'd be happy to deliver a robust proof of concept covering 3–5 priority websites, including secure login, scheduled searches, comparison against previous results, and a dashboard showing only new listings. Once validated, we can expand the platform in phases.

Looking forward to discussing it further.