Another important note:
When script is initiated, there is a table where changes are logged. We'll have to log in the same database and get its max journal_id and save it in the same directory -- name it latest_journal_entry.dat
I will provide a list of its table names and description separated by comma:
But, the output file should be:
2. must be tab delimited
3. option to include/exclude header
4. no "" enclosure for each field. If a field has the following values:
Eminem - "Rap God"
the final output should be the same. Not
"Eminem - ""Rap God"", or
"Eminem - "Rap God"", etc.
5. NULL values will be NULL. Not "", or " ", or null, or ^@
6. Ready to be consumed by other DB application, e.g. MS SQL Server
7. Make sure the tab delimiter is actually tabbed, not a series of empty spaces. Excel should be able to interpret it.
All output files to be loaded to a target S3 bucket endpoint AFTER all tables extracted/dumped successfully. Where possible, use multiprocessing or multi-threading.
Suggest to use mysqldump, or mysqldbexport.
NB. mysqldump file is not readily consumed. See attached on something I found that might help you.
The end goal of this process is to automate a scheduled run. When schedule is triggered, it knows what tables to dump (95 in total, probably read in from file per line), logs in and dump each table to its file (named by its table name exclude schema; eg. tablename.sql), data format transformation, compress each file. When done, push every content in the directory to S3 -- directory will be cleared first.
Is this something you can do? Please estimate hours and cost.
Python 3.5 or greater please.
Danny T.100% (2)
Create an account now and send a proposal now to get this job.Sign up
Clarification Board Ask a Question
Are you free to choose shell scripting instead of python?
ThanksDanny T.05 Apr 2017
If we choose shell instead, we may have to download AWS CLI to push data to S3 endpoint. Is that right?Khalid A.05 Apr 2017
Yes...you need to configure AWS CLI.
You can use Cron job to schedule it. It will take backup then push it to s3 and remove the old backup directory. Or directly write the backup in s3.