About the role
We are looking for a Data Engineer to build data collection and processing pipelines that gather business information from multiple authorized sources. The candidate will develop systems to discover companies, normalize business information, remove duplicate records and organize lead data for AI-based analysis.
What you'll do
- Develop multi-source data ingestion pipelines.
- Integrate search APIs, business directories and licensed B2B data providers.
- Build country, state and city-based business discovery pipelines.
- Extract and normalize company information.
- Implement data deduplication and entity resolution.
- Develop ETL and ELT pipelines.
- Process structured and semi-structured data.
- Implement data quality checks and validation.
- Maintain source attribution and data freshness.
What we're looking for
- Python.
- SQL and PostgreSQL.
- ETL/ELT pipelines.
- Pandas.
- REST APIs.
- Data cleaning and transformation.
- Data modeling.
- Preferred: Apache Airflow, Apache Kafka, PySpark, AWS S3, OpenSearch, Data pipeline orchestration.
Key Skills
Python
SQL
PostgreSQL
ETL/ELT
Pandas
Airflow
Kafka
Data Modeling
Who can apply
Degree in Computer Science, Data Science, Engineering or related technical domain.