Lead Data Engineer

Lead Data Engineer: Sourcing designs and owns the end-to-end extraction and ETL pipeline that turns unstructured web data into the world's most accurate B2B dataset across 13 markets. You set the architectural standards: extractor patterns, proxy strategy, LLM infrastructure as production systems, agentic escalation workflows, and cost-aware orchestration. This is not a pipeline maintenance or incremental optimisation role. You're architecting the framework other engineers ship into, building for the hardest extraction problems (anti-bot defences, JS-heavy rendering, schema drift), and owning the sourcing layer end to end. ~80% hands-on architecture and reference implementation: designing extractor frameworks, shipping the hardest extractors, building agentic pipelines, writing eval harnesses and labelled datasets. ~20% cross-functional voice: technical input on sourcing decisions with data platform, product, and analytics; setting standards for coverage, schema, and accuracy.

• Skills library: versioned specs for recurring extraction patterns, invocable by engineers and agents alike • Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling • 6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery • Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo. • Built agents and tool-calling pipelines: architected agent workflows, written specs others depend on, run tool-calling systems in production • Sharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team • Expert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation • Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale • Shipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it • Architecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics • You live and breathe AI tools. Structured outputs, evals, traces, and LLM tracing (Logfire, OpenTelemetry, similar) as your default way of working, not a productivity experiment. In 2026, this is how data engineers build extraction at scale and you need to already be doing it.

Firmable is the market-leading B2B sales intelligence platform in Asia-Pacific. Our competitive moat is our data: the deepest, most localised company and people dataset in every market we operate in. We've proven the model in ANZ. Now we're scaling it across Southeast Asia and the US. Every record Firmable sells starts in a sourcing pipeline. This role architects that pipeline. The Role Lead Data Engineer: Sourcing designs and owns the end-to-end extraction and ETL pipeline that turns unstructured web data into the world's most accurate B2B dataset across 13 markets. You set the architectural standards: extractor patterns, proxy strategy, LLM infrastructure as production systems, agentic escalation workflows, and cost-aware orchestration. This is not a pipeline maintenance or incremental optimisation role. You're architecting the framework other engineers ship into, building for the hardest extraction problems (anti-bot defences, JS-heavy rendering, schema drift), and owning the sourcing layer end to end. ~80% hands-on architecture and reference implementation: designing extractor frameworks, shipping the hardest extractors, building agentic pipelines, writing eval harnesses and labelled datasets. ~20% cross-functional voice: technical input on sourcing decisions with data platform, product, and analytics; setting standards for coverage, schema, and accuracy. What You'll Own Data Sourcing and ETL architecture • End-to-end pipeline design: extraction, normalisation, deduplication, validation, and load; plus cost, performance, and reliability of the whole layer • Sourcing layer standards: coverage, accuracy, schema design; the rule-vs-LLM standard codified so the team applies it without you • Cost and performance optimisation: cost ceilings with the token math behind them, incremental processing, recovery design, scheduling Extraction systems and frameworks • Extractor framework: patterns, abstractions, and tooling other engineers ship into; new extractors are fast to build and reliable to run • Hard-source extractors: anti-bot defences, JS-heavy rendering, schema drift, low-quality structure; plus the proxy and IP rotation strategy behind them • Agentic extraction pipelines: rule-based triage, LLM escalation, structured-output validation, retries, human-review queues LLM infrastructure and observability • LLMs as production systems: versioned prompts, labelled eval sets, measured precision and recall, prompt versioning you can roll back, judges debugged on real data • Eval and observability scaffolding: eval frameworks, prompt versioning, traces, drift detection when a vendor silently updates a model • Model-choice playbook: cheap models for classification, stronger models for nuanced extraction, frontier models for hard edge cases; revised as model economics shift Skills library and orchestration • Skills library: versioned specs for recurring extraction patterns, invocable by engineers and agents alike • Orchestration: Airflow or equivalent patterns that scale with data volumes and source counts; dependency management, recovery, cost-aware scheduling What We're Looking For Must Haves • 6-10 years shipping production extraction, ETL, or data pipeline systems in business-critical environments; deep web extraction at scale with experience designing around anti-bot defences, proxy architecture, JS rendering, schema drift, and recovery • Shipped LLMs inside extraction pipelines as production systems: structured outputs, versioned prompts, labelled eval sets, logged traces, judges debugged on precision/recall, drift detected on vendor model updates. You can show the repo. • Built agents and tool-calling pipelines: architected agent workflows, written specs others depend on, run tool-calling systems in production • Sharp judgement on rules vs. LLMs: reach for deterministic logic first, defend the call either way, and have codified the standard for a team • Expert Python: production-grade, performance-aware, comfortable with concurrency and scale; plus advanced SQL for complex transformations and performance optimisation • Extensive Airflow or equivalent: orchestration, dependency management, recovery patterns, cost optimisation at production scale • Shipped with agentic IDEs: Claude Code, Cursor, or equivalent as your default mode, with shipped extraction systems to show for it • Architecture judgement and a product mindset: when to refactor, optimise, ship, or start again; you care about coverage and accuracy landing with customers, not uptime metrics • You live and breathe AI tools. Structured outputs, evals, traces, and LLM tracing (Logfire, OpenTelemetry, similar) as your default way of working, not a productivity experiment. In 2026, this is how data engineers build extraction at scale and you need to already be doing it. Highly Valued • Cloud data platforms: Snowflake, Redshift; AWS for pipeline deployment (Lambda, S3, ECS, Glue) • Vector databases, embeddings, or retrieval patterns for matching and deduplication • Eval frameworks like Braintrust, Promptfoo, or Inspect; LLM tracing tooling • Data quality frameworks with automated testing and anomaly detection at scale • B2B data: firmographics, people data, company registries across markets • Data privacy and compliance: GDPR, CCPA • Startup or scaleup experience where you shipped fast and owned outcomes end to end The Environment Firmable runs lean and ships fast: small senior teams, no layers, minimal process. Sourcing is a core competitive advantage; this role sets the pace for how fast and accurate the entire data pipeline moves. Weekly releases moving toward daily; no fixed hours, full autonomy on architecture choices. We are an AI-native organisation. That means AI isn't a tool we reach for: it's the default operating mode. Agentic extraction pipelines, LLM-powered classification with eval sets, structured-output validation, labelled datasets versioned with prompt runs, traces logged from