Python Developer with Airflow, Supabase, and GCP Experience

Зарплата

от 4000 $

Местоположение и тип занятости

Полный рабочий деньМожно удаленно

Компания

Мы занимаемся дизайном социальных сервисов и информационных проектов в интернете

Описание вакансии

О компании и команде

We are looking for a Python Developer with expertise in handling large datasets and working with modern data platforms. This role will involve data scraping, API integration, and the use of third-party scraping tools to manage records, which are expected to scale into the tens of millions. Efficient data processing, categorization, and optimization are key components of this project.


Key Responsibilities:
  • Data Management & Scraping: Handle large datasets by developing efficient data pipelines, including scraping data from various sources (APIs, third-party scraping tools), with a focus on extracting reused items from various marketplaces and consolidating the data for further analysis.
  • Airflow Automation: Use Apache Airflow to schedule Python jobs for regular data cleaning, optimization, and scraping tasks. While prior Airflow experience is preferred, strong Python skills are sufficient for quick upskilling.
  • Supabase & Data Integration: Integrate data into Supabase for the US platform, mirroring the existing UK platform setup. Experience with PostgreSQL and Supabase will be crucial for this task.
  • GCP Experience: Work within a Google Cloud Platform (GCP) environment, managing cloud storage, computing resources, and database solutions.
  • Frontend Collaboration (Next.js): Collaborate with the frontend team working with Next.js to ensure backend integration aligns with frontend requirements, particularly in mirroring the UK platform’s structure.

Ожидания от кандидата

Required Skills:

  • Python Development: Strong experience in Python, particularly in data processing, API integration, and managing large datasets.
  • SQL & PostgreSQL: Advanced proficiency in SQL, including writing complex queries and managing databases optimized for large-scale data.
  • Apache Airflow (Preferred): Experience using Airflow for scheduling, automation, and managing Python-based workflows.
  • Supabase & PostgreSQL: Familiarity with Supabase as a backend service and working knowledge of PostgreSQL databases.
  • Google Cloud Platform (GCP): Experience using GCP services like Cloud Storage, Cloud Functions, or Cloud SQL in data-heavy environments.
  • Git & Vercel: Ability to work with version control (Git) and deploy on platforms like Vercel as part of a modern DevOps pipeline.
  • Next.js (Bonus): Although not the primary focus, knowledge of Next.js would be an advantage for better backend/frontend integration.

Desirable Traits:

  • Knowledge of working with product data and affiliate product data, including geolocation data and techniques for categorizing data.
  • Experience in extracting reused items from various marketplaces and consolidating the data.
  • Strong ability to communicate clearly and effectively, particularly when collaborating with frontend teams.
  • Familiarity with third-party scraping tools for large-scale data extraction.
  • Ability to work independently, while actively collaborating with other teams.
  • Expertise in data cleaning and optimization techniques to ensure high-performance environments.

Условия работы

This work is for an international company on an English-speaking team. English Upper Intermediate and higher is a must. There is a trial period of 3 months, full-time on an hourly paid basis, freelance contract.