Data Engineer

As a Data Engineer, you will be responsible for designing, developing, and maintaining the systems and infrastructure needed to process and manage large volumes of data. Your role will involve working with various data sources, transforming data into usable formats, and ensuring data quality and reliability. Additionally, you will collaborate with cross-functional teams to develop scalable data solutions and optimize data workflows.

Key Responsibilities:

Data Pipeline Development: Design and implement data pipelines to extract, transform, and load (ETL) data from various sources into databases or data warehouses. Develop efficient data integration processes that handle large volumes of structured and unstructured data.

Database Design and Optimization: Work closely with data architects and analysts to design and optimize database structures and schemas for efficient data storage and retrieval. Ensure proper indexing, partitioning, and data organization to maximize query performance.

Data Transformation and Cleansing: Develop scripts and workflows to clean, transform, and preprocess raw data into usable formats. Apply data quality checks and validation techniques to ensure accuracy and consistency of data.

Data Modeling and Warehousing: Design and implement data models for data warehousing and reporting purposes. Develop and maintain data marts and dimensional models to support analytical queries and reporting needs.

Performance Tuning and Optimization: Monitor and analyze database performance, identifying and resolving performance bottlenecks. Optimize SQL queries, indexes, and database configurations to enhance system performance and scalability.

Data Security and Governance: Implement data security measures, access controls, and data encryption techniques to protect sensitive information. Ensure compliance with data governance policies and regulations.

Collaboration and Integration: Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and develop solutions that meet business needs. Integrate data from various systems and platforms to create a unified view of data.

Documentation and Documentation: Maintain comprehensive documentation of database designs, data models, ETL processes, and workflows. Document data standards, data dictionaries, and data lineage for effective data management and governance.

Qualifications:

Bachelor's degree in Computer Science, Information Systems, or a related field (or equivalent work experience).
Proven experience as a Data Engineer or in a similar role.
Strong proficiency in SQL and experience with relational databases (e.g., Oracle, MySQL, Microsoft SQL Server).
Familiarity with big data technologies (e.g., Hadoop, Spark, Hive) and NoSQL databases (e.g., MongoDB, Cassandra).
Experience with ETL tools and data integration frameworks.
Knowledge of data modeling concepts and dimensional modeling.
Understanding of data warehousing and business intelligence concepts.
Proficiency in programming languages like Python, Java, or Scala.
Familiarity with cloud-based data platforms (e.g., AWS, Azure, GCP).
Strong analytical and problem-solving skills.
Excellent communication and collaboration skills.