Parse large number of text files to mysql database
- or -
Post a project like this€250(approx. $269)
- Posted:
- Proposals: 14
- Remote
- #2589138
- OPPORTUNITY
- Expired
Software Engineer | Java Developer | PHP Developer | Data Scraping | Web Apps. | Desktop Apps | BI Reports and Dashboards
Anantapur
12290012687251412352153735517722421888380216813323711802503281255683128411513231113
Description
Experience Level: Expert
I am looking for parsing over 1000 files. Each 25 mb in size. The parser should work in linux environment. Parsing should be done in under 5 min (speed is of importance here). We have a system with 16 core and 30 gb ram with ssd storage.
The parser should parallel process all files in one folder and populated database. I would need the parser to write to mysql db. The parser should be able to create the schema on first run. On subsequent runs if any new fields are found then the parser should be able to include this to the db. The parser should also use multi threading .I have attached as example one such file and one table created.
Tables should be indexed, there are a few other details which I will provide later. The developer must be willing to tune the parser to customer satisfaction. If interested please let me know how you plan to implement this and timeframe you need
The parser should parallel process all files in one folder and populated database. I would need the parser to write to mysql db. The parser should be able to create the schema on first run. On subsequent runs if any new fields are found then the parser should be able to include this to the db. The parser should also use multi threading .I have attached as example one such file and one table created.
Tables should be indexed, there are a few other details which I will provide later. The developer must be willing to tune the parser to customer satisfaction. If interested please let me know how you plan to implement this and timeframe you need
Three (PPH Corporate) A.
100% (41)Projects Completed
36
Freelancers worked with
18
Projects awarded
62%
Last project
15 Apr 2021
Ireland
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
That's a lot of data.
Is the database on the same machine as the machine where the parsing will happen? Because I would consider putting your database on multiple hosts so they can process more data at the same time. Also, could you share with us the data model? Since either you're inserting lots of records, or you're sending a lot of data in a column. Both need different solutions... -
What kind of files are these, JSON/CSV/XML/TAB-Delimited?
-
Hello
I'm interested on your project but seems not able to see the sample files you attached.
Could you please resend them ?
Thank you
864151861923860726
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies