Title: Senior Data Engineer OR Architect

100% REMOTE – Latam

About the Role

We are looking for an experienced Senior Data Engineer to design, build, and maintain scalable data platforms and pipelines that support analytics, reporting, data science, and AI initiatives.

The ideal candidate has strong hands-on experience with Databricks, Apache Spark, Python, SQL, and cloud data platforms, along with a solid understanding of data architecture, ETL/ELT, data quality, and performance optimization.

Key Responsibilities

Design, develop, and maintain highly scalable data pipelines and ETL/ELT workflows.

Build and optimize data engineering solutions using Databricks and Apache Spark.

Develop robust batch and streaming data pipelines using Python and SQL.

Implement data processing and transformation using PySpark.

Design and maintain data lake/lakehouse architectures using technologies such as Delta Lake.

Develop reliable data ingestion pipelines from databases, APIs, files, and cloud-based sources.

Build and maintain data models for analytics, reporting, and downstream applications.

Implement data quality, validation, monitoring, and error-handling frameworks.

Optimize Spark jobs, SQL queries, clusters, and data pipelines for performance and cost.

Develop workflows and orchestration using Databricks Workflows, Airflow, or similar tools.

Work with cloud data platforms such as AWS, Azure, or GCP.

Implement CI/CD and automated deployment practices for data pipelines.

Collaborate with data scientists, analysts, software engineers, architects, and business stakeholders.

Participate in architecture and design discussions and provide technical leadership to junior engineers.

Troubleshoot production data issues and ensure high availability and reliability of data pipelines.

Establish and enforce data engineering best practices, coding standards, and documentation.

Required Qualifications

10+years of professional experience in data engineering.

Strong hands-on experience with Databricks.

Strong experience with Apache Spark / PySpark.

Advanced SQL skills, including query optimization and complex transformations.

Strong programming experience with Python.

Experience designing and implementing ETL/ELT pipelines.

Strong understanding of data lake and lakehouse architectures.

Hands-on experience with Delta Lake / Delta tables.

Experience working with relational and NoSQL databases.

Experience with at least one major cloud platform: AWS, Azure, or GCP.

Experience with workflow orchestration tools such as Airflow, Databricks Workflows, or Azure Data Factory.

Experience with Git and CI/CD practices.

Strong understanding of data modeling, data warehousing, and dimensional modeling.

Experience with data quality, governance, security, and monitoring.

Strong analytical, troubleshooting, and communication skills.

Preferred Qualifications

Experience with Azure Databricks and Microsoft Azure services.

Experience with AWS Databricks, S3, Glue, EMR, or related services.

Experience with Unity Catalog and Databricks governance.

Experience implementing Medallion Architecture (Bronze, Silver, Gold).

Experience with real-time/streaming technologies such as Kafka, Spark Structured Streaming, or Event Hubs.

Experience with Snowflake, BigQuery, Redshift, Synapse, or similar data warehouses.

Experience with dbt or other modern data transformation frameworks.

Experience with Terraform or Infrastructure as Code.

Knowledge of data governance, lineage, cataloging, and access-control frameworks.

Experience supporting machine learning or AI data pipelines.

Databricks certifications are a plus.

Technical Skills

Languages: Python, PySpark, SQL

Big Data: Apache Spark, Spark SQL

Data Platform: Databricks, Delta Lake, Lakehouse

Cloud: AWS / Azure / GCP

Orchestration: Databricks Workflows, Airflow, ADF

Streaming: Kafka, Spark Structured Streaming, Event Hubs

Databases: PostgreSQL, SQL Server, MySQL, MongoDB

Warehousing: Snowflake, Redshift, BigQuery, Synapse

DevOps: Git, CI/CD, Docker, Terraform

Governance: Unity Catalog, Data Quality, Data Lineage, Security

Thanks & Regards

Nitin Kushwaha

Eros Technologies Inc

Email: Nitin.k@erostechnologies.com

EROS Technologies Inc

Data Architect