Parse files into mysql database in parallel using multiple cores

  • Posted:
  • Proposals: 13
  • Remote
  • #2475635
  • OPPORTUNITY
  • Awarded
Marembo I.
Md. Mahedi H.
Dustin C.
Rajesh Kumar T.
SAMUEL O. + 8 others have already sent a proposal.
  • 13

Description

Experience Level: Expert
Hi,

I am looking to parse a few thousand text files into mysql database, below are some of the parser criteria

1. Since file are large in number parsing should be done in parallel
2. The parser should go into each file and build a schema when parsing for the first time
3. subsequent parsing the parser should truncate all existing data and be able to find new fields and add to tables if found in the files
4. tables should be indexed
5. While parsing the parser should display time taken to parse each file
6. Once parsing is finished total parsing time should be displayed
7. all tables and fields should have a name prefix (eg. t_ for tables and f_ for fields) to avoid mysql reserved names. The prefix and database connection may be specified in a config file
8. the parser will be run on a linux machine with 16 cores. parser should be built using java

Sample files and parsing requirements will be provided
we currently have a similar java parser for xml files (can provide source code if required)

New Proposal

Create an account now and send a proposal now to get this project.

Sign up

Clarification Board Ask a Question

  • TSOLUTIONS

    Three

    Is this a one off or cron need to be setup and is your budget flexible?