Scala Developer Needed
- or -
Post a project like this$100
- Posted:
- Proposals: 1
- Remote
- #1804606
- Expired
Description
Experience Level: Intermediate
General information for the business: I need a scala developer to convert a piece of python code
Description of requirements/functionality: The functionality required is below
* Read the input files (.csv) and create parquet file
-The logic has been written in python which needs to convert into scala . I will walk through the Pyton logic.
- most of the generic logic has already been written in scala which can be used from other project.
- The .csv file convert into dataframe and save into hdfs as parquet file
- The code needs to be parameterized where ever possible
- Main and utility program (which has most of the functions) needs to be converted.
* partitioning the date eg.2017-11-21
*Log null values, anomalies (substation name with ; example) and number of records
*Include error handling where ever possible
*Add tests for anomalies
*check if the new data was created sucessfully without any anomalies
*externalize schema of the csv inputs
*track metrics like duration of the job
* Logging (trait created as template)
Thanks
Extra notes:
Description of requirements/functionality: The functionality required is below
* Read the input files (.csv) and create parquet file
-The logic has been written in python which needs to convert into scala . I will walk through the Pyton logic.
- most of the generic logic has already been written in scala which can be used from other project.
- The .csv file convert into dataframe and save into hdfs as parquet file
- The code needs to be parameterized where ever possible
- Main and utility program (which has most of the functions) needs to be converted.
* partitioning the date eg.2017-11-21
*Log null values, anomalies (substation name with ; example) and number of records
*Include error handling where ever possible
*Add tests for anomalies
*check if the new data was created sucessfully without any anomalies
*externalize schema of the csv inputs
*track metrics like duration of the job
* Logging (trait created as template)
Thanks
Extra notes:
MEET SOLUTIONS LIMITED
99% (27)Projects Completed
33
Freelancers worked with
30
Projects awarded
43%
Last project
2 Apr 2023
United Kingdom
New Proposal
Login to your account and send a proposal now to get this project.
Log inClarification Board Ask a Question
-
There are no clarification messages.
We collect cookies to enable the proper functioning and security of our website, and to enhance your experience. By clicking on 'Accept All Cookies', you consent to the use of these cookies. You can change your 'Cookies Settings' at any time. For more information, please read ourCookie Policy
Cookie Settings
Accept All Cookies