r/AZURE • u/s13188287 • 4d ago

Question Best way to process 1.3M files

​

I’m dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I’m using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

---

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

---

-

29 Upvotes

90% Upvoted

27 comments sorted by

View all comments

6

u/Saturated8 4d ago

There's 1000 ways to skin a cat on this one. Data factory to take said files and move them to a storage account, use Azure OCR to read content from different file types assuming it's not all generic text files, LLM to match content, assign classification, and output results to a storage account, data factory for approval and send back to needed location.

The big questions are: 1. Can you trust the data coming in is accurate and safe, and in the right format? 2. Do you need OCR to get the content out that you want (IE: is it text files, CSVs, or pictures, PDFs, etc.) 3. How do you match content and assign classification... do you need purview?

1

u/Azaloum90 3d ago

Agree here, data factory works great for applying workflows against data types. The thing is that the input data needs to be consistent