
Parse text file to mysql db in parallel using multiple cores
- or -
Post a project like this2062
€250(approx. $284)
- Posted:
- Proposals: 4
- Remote
- #2495079
- OPPORTUNITY
- Completed
Description
Experience Level: Intermediate
Hi,
I am looking to parse a large text file into mysql database, below are some of the parser criteria
1. Since file is large parsing should be done in parallel
2. The parser should go into file and build a schema when parsing for the first time
3. subsequent parsing the parser should truncate all existing data and be able to find new fields and add to tables if found in the file
4. tables should be indexed
5. While parsing the parser should display time taken to parse
6. Once parsing is finished total parsing time should be displayed
7. all tables and fields should have a name prefix (eg. t_ for tables and f_ for fields) to avoid mysql reserved names. The prefix and database connection may be specified in a config file
8. the parser will be run on a linux machine with 16 cores. parser should be built using java
9. parsing should be done under 15min for a large file of 2 Gb
Sample files and parsing requirements will be provided
I am looking to parse a large text file into mysql database, below are some of the parser criteria
1. Since file is large parsing should be done in parallel
2. The parser should go into file and build a schema when parsing for the first time
3. subsequent parsing the parser should truncate all existing data and be able to find new fields and add to tables if found in the file
4. tables should be indexed
5. While parsing the parser should display time taken to parse
6. Once parsing is finished total parsing time should be displayed
7. all tables and fields should have a name prefix (eg. t_ for tables and f_ for fields) to avoid mysql reserved names. The prefix and database connection may be specified in a config file
8. the parser will be run on a linux machine with 16 cores. parser should be built using java
9. parsing should be done under 15min for a large file of 2 Gb
Sample files and parsing requirements will be provided

Three (PPH Corporate) A.
100% (41)Projects Completed
36
Freelancers worked with
18
Projects awarded
62%
Last project
15 Apr 2021
Ireland
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
Hi Three,
Does the parser have to written in Java? I can create an app using NodeJS, which would be just as fast and you will be able to use it from external devices. -
Hi. Whether a grammar for the parser will have to be evolved, or whether the lines in the input file will follow a strict file format? Also a small sample would have helped estimate the complexity of the assignment if it is a fixed cost project. Thanks
835804835790
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies