Scale Our Database To Handle 30M to 300M Rows (Possibly via MSSQL)
- or -
Post a project like this1986
$$$
- Posted:
- Proposals: 4
- Remote
- #2178849
- Awarded
Description
Experience Level: Expert
Estimated project duration: less than 1 week
I have a Flex Cloud Server that is scalable up to 16 CPU Cores and 128GB RAM, based on Windows 2012 and a MySQL Database, but we have the capability to use MSSQL as well via Plesk.
We now have 1 Million Products and about 10 million variations in the product_variations table, so our database has about 13 Million rows total. Our website page loading speed slows to a crawl when we have that many products and I am looking for a database expert to implement a strategy that will allow us to keep our current Cloud Server with Plesk and still be able to have the front end pages load quickly.
I want to improve the page loading time and Searches when the customer searches for a product. We currently are web-scraping to get our products, and those tables are huge. There are about 5 tables, and 2 of them are already 6 Million rows.
We then have an import script that moves the Raw data from one database into our website's products tables and that now is only processing 29 records per minute, and we have millions of products to update, so that Import Script would need 20 days to run fully. We need that much faster.
We are thinking about possible using MySQL Fabric to automatically Shard the 2 or three large database product tables without us having to change our coding/application. But any other strategy is welcome.
Please provide a quote that includes cost, time to implement, and how soon you can get started.
Thank you,
Joseph
We now have 1 Million Products and about 10 million variations in the product_variations table, so our database has about 13 Million rows total. Our website page loading speed slows to a crawl when we have that many products and I am looking for a database expert to implement a strategy that will allow us to keep our current Cloud Server with Plesk and still be able to have the front end pages load quickly.
I want to improve the page loading time and Searches when the customer searches for a product. We currently are web-scraping to get our products, and those tables are huge. There are about 5 tables, and 2 of them are already 6 Million rows.
We then have an import script that moves the Raw data from one database into our website's products tables and that now is only processing 29 records per minute, and we have millions of products to update, so that Import Script would need 20 days to run fully. We need that much faster.
We are thinking about possible using MySQL Fabric to automatically Shard the 2 or three large database product tables without us having to change our coding/application. But any other strategy is welcome.
Please provide a quote that includes cost, time to implement, and how soon you can get started.
Thank you,
Joseph
Joseph B.
100% (46)Projects Completed
15
Freelancers worked with
15
Projects awarded
47%
Last project
6 Nov 2018
United States
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
Hi Joseph;
What is the programing language that you used in your web site?
is it ASP:net or Php?
Thanks;
Selçuk K.
ASsoft
725490
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies