Data Engineering Team Leader

Lusha is growing, and we seek an experienced Data Infra TL to join us!

If you are a highly skilled software engineer experienced in solving complex data challenges, with a proven track record in scaling data-driven organizations, a strong knowledge of a wide range of technologies and solutions in data engineering, and someone who thrives in a cross-functional, fast-paced environment - continue reading 🙂

Team Objective:

Our mission is to establish a state-of-the-art data infrastructure that seamlessly supports the ingestion, processing, and lifecycle management of external and internal data sources. We aim to ensure that data is not only accessible and secure but also efficiently managed across its lifecycle.

Team Scope:

You'll be at the forefront of managing data from its initial raw state in S3 (Data Lake) to its transformation into a structured, query-able bronze layer. Your leadership will guide the implementation of data governance and cataloging frameworks (Unity Catalog), oversee the publication of data assets within the organization, and maintain cutting-edge data infrastructure tools and practices.

You will manage a team with responsibilities:

- Ingestion of External and Internal Sources: Spearheaded the design and implementation of robust data ingestion pipelines, ensuring the integrity and availability of data for processing and analysis.
- Raw to Bronze Transformation: Craft and maintain pipelines that elevate raw data into the bronze layer, adhering to our rigorous data modeling standards.
- Catalog Implementation: Champion data governance by integrating the data catalog and making data assets cataloged, described, and easily discoverable.
- Data Asset Publication: Define protocols for disseminating information about data assets to ensure stakeholders are well-informed and can leverage data effectively.
- Metadata Management: Develop a comprehensive strategy for metadata management to boost data discoverability, quality, and governance.
- Data Monitoring and Lineage Tools: Implement and manage tools for ongoing data quality, traceability, and compliance monitoring.
- Tables Performance Optimization: Continuously monitor and enhance data table performance for optimized data access and analysis.
- Infrastructure and Tools Implementation:
- Airflow Infrastructure: Oversee Airflow to enable automated data workflows, developing custom solutions as needed.
- Data Testing Infrastructure: Lead the establishment of data testing best practices, enhancing data quality and integrity.
- Python Infrastructure Management: Manage the Python environment, ensuring a state-of-the-art development environment.
- General Data Infrastructure Tools POC: Drive proof-of-concept initiatives for new tools, evaluating their integration into our ecosystem.

You are:

- Autodidact. Likes a challenge, motivated
- Ambitious. You are eager to make an impact and lead by example. You want to grow, you thrive in a fast-moving, dynamic growth environment, and you can operate at the strategic level and the micro-detail to get things right
- Independent. You pick things up quickly and can execute independently from ideation to implementation
- Results-driven. You have a can-do approach, drive to exceed your goals
- Data-oriented. You like analyzing and working with data
- Organized. You are able to manage highly interdependent, multifaceted projects and keep various work streams on track - even those with many moving parts
- A team player. You will collaborate across the data team and with other teams as well

Lusha is the place for you!

REQUIREMENTS

Requirements:

- 3+ years of experience in leading people.
- 3+ years experience in data engineering.
- 5+ years of experience as a software developer (Python preferable).
- Proficient in the composition of Advanced SQL (analytical functions) and query performance tuning skills.
- Experience working within the AWS ecosystem (AWS S3, Glue, EMR, Redshift, Athena) or comparable cloud environments (Azure/GCP).
- Experience extracting, ingesting, and transforming large data sets.
- Experience working with multiple databases (relational, document storage, key-value, time-series) and their associated query languages, specifically understanding when to use one vs. another.
- Experience with big data platforms such as Spark, Databricks, and Kafka.