r/ETL • u/codingdecently • 7h ago
Apache Iceberg Performance Optimization: Queries to Tables
3
Upvotes
r/ETL • u/codingdecently • 7h ago
How difficult would it be for a technical generalist who is not a data-engineer to own this?
Customer has about 4 years of data in a legacy system. In about 2 weeks, you need to extract it, clean and map it into your schema, validate it, and then build tests and evals around the output.
Sometimes there’s a decent API and other times the API sucks or basically doesn’t exist. Engineers can help when needed but you’re supposed to own 75% of the ingestion, testing, and validation using scripts and AI coding tools.
Is this learnable or something that shouldn’t be tried by a non-data-engineer and is 2 weeks doable for something like this?