Webscraper needs to be built

Cex scraper

Need to have the following tables:

Products -
Item no, make, model, category, sku, colour, capacity, last updated? frequent?

Sell-pricing -
Item no. grade a price, grade b price, Grade c price

Buy-pricing -
Item no. grade a price, grade b price, Grade c price

Voucher-pricing -
Item no. grade a price, grade b price, Grade c price

Grades-
Grade, active?

Categories -
Category, active?

URLs -
URL scraped, date/time last scrape, success?

The web scraper should scrape the website www.cex.co.uk. First of all it should find URLs to scrape but only add them to the URLs table if they are unique. This part of the scraper should be running 24/7

The second part of the scraper finds the data. So it will take one of the URL's then complete everything in the pricing tables and the products table. This should keep running through all of the products in order and do 10 scrapes a minute on this data.

The final is the frequent scraper, if a product is marked as frequent it should scrape the data for this product every 15 mins on a cron.

Each one of the scrapers needs to be hosted on a separate EC2 instance on Amazon AWS with different IP addresses, it should then all feed back into the same database.