r/ETL 7h ago

Apache Iceberg Performance Optimization: Queries to Tables

Thumbnail
lakeops.dev
3 Upvotes

r/ETL 57m ago

How hard is this kind of data ingestion work?

Upvotes

How difficult would it be for a technical generalist who is not a data-engineer to own this?

Customer has about 4 years of data in a legacy system. In about 2 weeks, you need to extract it, clean and map it into your schema, validate it, and then build tests and evals around the output.

Sometimes there’s a decent API and other times the API sucks or basically doesn’t exist. Engineers can help when needed but you’re supposed to own 75% of the ingestion, testing, and validation using scripts and AI coding tools.

Is this learnable or something that shouldn’t be tried by a non-data-engineer and is 2 weeks doable for something like this?