Senior Data Engineer
Data Engineers implement methods to improve data reliability and quality. They combine raw information from different sources to create consistent and machine-readable formats. They also develop and test architectures that enable data extraction and transformation for predictive or prescriptive modeling, and loading of data from a wide variety of data sources using SQL.
They put architecture models into production and build the infrastructure required for optimal extraction, and transformation. Choosing optimal solutions to use for collection, storing, processing, and analyzing huge sets of data, then maintaining, implementing, and monitoring them. The DD is also responsible for integrating them with the architecture used across the company.
Key responsibilities
- Works with stakeholder teams to assist with data-related technical issues and support their data infrastructure needs.
- Evaluates business needs and objectives.
- Builds the infrastructure required for optimal extraction, transformation, and loading of data from a wide variety of data sources using SQL.
- Identifies, designs, and implements internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Works on legacy data migration.
- Selects and integrates Data tools and frameworks required to provide requested capabilities.
- Conduct query performance tuning/optimizations and data loads to meet the SLAs for both batch and real-time data uses cases.
- Partake in the creation of artifacts to align with the global SDLC process, Governance clearance, conduct peer code reviews and Unit Test Results
- Monitors performance and advises any necessary infrastructure changes.
- Ensures database optimization, integrity, consistency, security, and privacy.
- Participates in SQL code reviews, writes, and deploys SQL patches.
- Continually evaluates software components and tools for software applications.
- Recommends ways to improve data reliability, efficiency, and quality.
- Remains up to date with industry standards and technological advancements that will improve the quality of outputs.
- Works with the engineering teams to resolve issues, evaluate software modules and develop specifications.
Requirements
- University degree in Computer Engineering, Informatics or a related field
- Hands-on experience with CI/CD deployment Pipelines and data-readiness (data quality) design, development, implementation and address code or data issues.
- Strong Data warehousing Concepts, ETL tools such as Talend Cloud Data Integration tool
- Working with relational databases, query authoring (SQL) as well as working familiarity with a variety of databases.
- Investigating data sources, understanding, and documenting data behavior.
- Stored procedure writing, query optimization, and indexes.
- Query troubleshooting and query execution optimization.
- Development of the ETL data pipelines is considered a plus
- Experience on Microsoft Azure cloud and Snowflake SQL, database queryingly/performance tuning is considered a plus
- Exposure to financial domain knowledge is considered a plus
- Modelling and designing DWH.
- Redshift based DWH using SQL to develop complex and stored procedures is a plus.
- Designing and developing different kinds of reports such as analysis reports, ad hoc and standard reports.
What we offer
- Full-time employment
- Working from home possibility
- Private health insurance
- Monthly Gym membership
- Access to education and professional improvement
- Working with great people in a culture that supports both autonomy and collaboration
- Office perks: great location, modern equipment unlimited coffee & tea, fruits, spacious lounge area
- Team building activities
- Competitive compensation package
