Data Engineering - Web Scraping

*Roles and Resposibilities:
*

1. Work on data sourcing and use web scrapers like BeautifulSoup, Selenium, etc.
2. Implement the latest AI techniques to augment large-scale data scraping
3. Deeply understand and implement data-mesh architecture, especially in relation to Elastic Search and Vector Search
4. Ensure data standards and work with tools like Elastic search and vector database.
5. Manage data normalization and standards validation
6. Automate and parametrize the scrapers
7. Monitor data sanity and ensure data availability and reliability
8. Understand business drivers and derive insights through data
9. Collaborate with stakeholders to establish current and ongoing data support and reporting needs

*Skills and Qualifications:
*

1. Graduate/Post Graduate degree in Computer Science or Engineering
2. 1-3 years of Hands-on experience with AWS.
3. Exceptional skills in data-mesh architecture, especially Elastic Search and Vector Search
4. Passion and interest in ETL jobs
5. Good English speaking and communication skills
6. Ability to work with a global remote culture
7. Initiative and entrepreneurship skills
8. Experience with microservices architecture and writing REST APIs
9. Knowledge of Kubernetes, Docker, and Airflow
10. Prior experience with Python, Django, and Gunicorn
11. Independent work ethic with an ability to thrive in a fast-paced environment