Domain Scraper - Python to Web App - Phase 2
- or -
Post a project like this3426
$48/hr
- Posted:
- Proposals: 1
- Remote
- #615640
- Awarded
Description
Experience Level: Expert
Estimated project duration: Ongoing
General information for the website: Domain Scraper - Python to Web App - Phase 2
Description of requirements/features: Build web app around python script that Jonathan built.
I'm picturing how this would work as a web app. I'm seeing views which navigate through the collected data. Filtering the collected domains, manually adding new ones, kicking of a quick manual check for a domain's availability.
Configuration about how domains are checked - including any necessary credentials. Editing the list of sites to scrape. Being able to drill down and see the collected pages (i.e. sitemap) for each site map and exclude certain pages, perhaps exclude pages based on a filter, perhaps exclude based on content-type of a page.
Configuring scheduled scrapes as well. Perhaps a way of recording each scrape process and viewing the log.
User management would also be critical as members would be paying members; the need to simply add/remove/suspend users based on payment statuses is crucial. Payment via PayPal & Stripe will be required.
Web app is required to be built on our servers so we can monitor progress, and my system admin will be able to tighten the firewall requirements as the application is built due to the fact that there's going to be transactions made - we can't have any potential "holes" in the system.
Integration to fetch domain stats from a few external providers will also be required; we'll firstly start with the Moz.com API; some stats required from here include Domain Authority, Page Authority, MozRank, MozTrust & Inbound Links.
API credentials will be provided when required; I have just purchased their commercial API and will continue to upgrade as required.
The Majestic.com API will then come next once Moz is done; this will require the CF & TF metrics being retrieved & display on a per domain basis.
I'm also envisioning registering domains directly from the app via NameCheap.com - this shouldn't be overly difficult however could be extensive as the user may require features such as transferring the domains, changing nameservers etc.
Extra notes:
Description of requirements/features: Build web app around python script that Jonathan built.
I'm picturing how this would work as a web app. I'm seeing views which navigate through the collected data. Filtering the collected domains, manually adding new ones, kicking of a quick manual check for a domain's availability.
Configuration about how domains are checked - including any necessary credentials. Editing the list of sites to scrape. Being able to drill down and see the collected pages (i.e. sitemap) for each site map and exclude certain pages, perhaps exclude pages based on a filter, perhaps exclude based on content-type of a page.
Configuring scheduled scrapes as well. Perhaps a way of recording each scrape process and viewing the log.
User management would also be critical as members would be paying members; the need to simply add/remove/suspend users based on payment statuses is crucial. Payment via PayPal & Stripe will be required.
Web app is required to be built on our servers so we can monitor progress, and my system admin will be able to tighten the firewall requirements as the application is built due to the fact that there's going to be transactions made - we can't have any potential "holes" in the system.
Integration to fetch domain stats from a few external providers will also be required; we'll firstly start with the Moz.com API; some stats required from here include Domain Authority, Page Authority, MozRank, MozTrust & Inbound Links.
API credentials will be provided when required; I have just purchased their commercial API and will continue to upgrade as required.
The Majestic.com API will then come next once Moz is done; this will require the CF & TF metrics being retrieved & display on a per domain basis.
I'm also envisioning registering domains directly from the app via NameCheap.com - this shouldn't be overly difficult however could be extensive as the user may require features such as transferring the domains, changing nameservers etc.
Extra notes:
Guy E.
100% (8)Projects Completed
11
Freelancers worked with
9
Projects awarded
36%
Last project
11 Dec 2014
Australia
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
There are no clarification messages.
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies