We are seeking a skilled and motivated Data Engineer to join our growing data team. The Data Engineer will be responsible for designing, building, and maintaining robust data pipelines, data systems, and architectures that support data analysis, machine learning, and decision-making processes within the organization. The ideal candidate will have experience with cloud platforms, data integration, ETL processes, and database management. Key Responsibilities: Data Pipeline Development: Design, implement, and optimize scalable and reliable data pipelines that transform, clean, and load data from various sources into data warehouses or other data storage systems. ETL Process Management: Develop and manage Extract, Transform, Load (ETL) processes that support the integration of data from diverse sources such as APIs, databases, flat files, and third-party systems. Data Warehousing: Work with data architects to build and maintain enterprise-level data warehouses or data lakes for both structured and unstructured data. Database Management: Ensure optimal performance, scalability, and reliability of the databases, and create efficient query mechanisms to retrieve data. Data Integration: Collaborate with business analysts, data scientists, and software engineers to integrate new data sources, ensuring that all data is accurate and accessible for business needs. Automation & Monitoring: Automate data pipelines where possible and monitor the performance of data systems to ensure reliability and timely data processing. Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and ensure that solutions meet organizational needs. Data Governance & Quality: Ensure high data quality, security, and compliance by implementing validation checks, data validation rules, and adhering to data governance best practices. Technical Skills: Strong proficiency in SQL and experience with relational databases (e.g., PostgreSQL, MySQL, MS SQL Server). Familiarity with NoSQL databases (e.g., MongoDB, Cassandra) and cloud-based data storage solutions (e.g., Amazon S3, Google Cloud Storage). Proficiency in Python, Java, Scala, or other programming languages used for data engineering tasks. Experience with data processing frameworks like Apache Spark, Hadoop, or similar big data technologies. Experience with cloud platforms such as AWS, GCP, or Azure and their data services (e.g., AWS Redshift, BigQuery). Familiarity with containerization and orchestration tools like Docker, Kubernetes, Airflow, or similar.
