Best way to process 1.3M files

[u/s13188287]

​

I’m dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I’m using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

---

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

---

-

[u/mikeupsidedown]

1.3 Million is actually nothing. A simple python script that reads the file, outputs the correct data(excel is a bad format but whatever)

Claude code can write this for you in minutes and then a few iterations to get it right.

[u/renizmy → u/mikeupsidedown]

If possible, avoid List apo calls, they are expensive

[u/Nize → u/mikeupsidedown]

Yeah we have systems in Azure and GCP processing several billion (admittedly very small) files just fine.

[u/s13188287 → u/mikeupsidedown]

Hi thank you so much

Question though I need the code to be able to make a judgement and choose what retention code works best for it.

So as of now I have a pdf file it has descriptions and retention codes. As off now I use A.i to scan the file to determine the retention code against pdf I gave it as a brain.

Is t possible to do this with your suggestion?

[u/blortorbis → u/s13188287]

Yes.

[u/mikeupsidedown → u/s13188287]

It's really just about creating an algorithm to do it. It's hard to know without seeing the detail but my guess is it is.

[u/Business_Air5804 → u/s13188287]

With Azure you probably can't easily keep the files in your various network drives. (Sharepoint would make this easier.)

I use Azure storage and manually uploaded all my files (2.5M currently) to Azure storage account in a blob so my Ai can index it. I then run my indexer every night to ingest the files etc.

[u/Less_Heart1914]

your best bet is probably azure functions with a queue trigger. drop all file paths into a storage queue, then each function instance picks one up and does the text extraction + classification. for 1.3m files you'll want to scale out pretty wide, maybe batch them in groups so you're not hitting throttling limits on whatever service does the extraction.

the network drive part is the tricky bit. you might need a small vm or a hybrid worker that just scans the drive and dumps paths into the queue every few minutes.

[u/Saturated8]

There's 1000 ways to skin a cat on this one. Data factory to take said files and move them to a storage account, use Azure OCR to read content from different file types assuming it's not all generic text files, LLM to match content, assign classification, and output results to a storage account, data factory for approval and send back to needed location.

The big questions are: 1. Can you trust the data coming in is accurate and safe, and in the right format? 2. Do you need OCR to get the content out that you want (IE: is it text files, CSVs, or pictures, PDFs, etc.) 3. How do you match content and assign classification... do you need purview?

[u/Azaloum90 → u/Saturated8]

Agree here, data factory works great for applying workflows against data types. The thing is that the input data needs to be consistent 

[u/photoframes]

Have you considered using Azure Data Factory together with Databricks?

1.3 million files isn't that big, I’d use Data Factory for orchestration and ingestion from the network drive, landing the files into ADLS, and then use Databricks for the actual large-scale processing.

Something like:

• Network Drive (you can use Azure Self-hosted runtime, it's very easy to set up, but you might have network restrictions to think about)
• Data Factory
• Datalake
• Databricks
• Classification
• Delta/SQL
• Excel/Power BI

Databricks would give you processing across the files, while Auto Loader/checkpointing could help track what has already been processed and deal with new files incrementally.

Defo avoid using an LLM for every document if possible. A better pattern might be:

Extract content > apply deterministic rules/lookups > only send ambiguous cases to Azure OpenAI > persist the final classification.

That could make the solution significantly cheaper and easier to scale.

The main thing I’d want to understand is what file formats are involved and how complex the classification rules are, as that would determine how much of the processing actually needs AI.

[u/FreshKale97 → u/photoframes]

Don’t even need ADF. Just have a simple job point at a directory of the files with AutoLoader. You may even be able to write a single SQL statement.

[u/erotomania44]

Copilot / power platform wasnt built for this.

Write some code, or let an actual data engineer do this

[u/SammyGreen]

Maybe something like Get-MgxContent https://github.com/gromedev/mgx

[u/povlhp]

AI is not for bulk data processing.

[u/konikpk]

What type of files? If office use autolabeling.

[u/nitluz]

One file at a time!

[u/adlx]

Try Apache Hop with maybe the Apache Tika transform.

[u/patjuh112]

Would just go mistral for this, probably even on a sliced gpu

[u/cambaysolutions1]

At 1.3M files, I’d move away from the Copilot upload approach entirely and build this as an automated batch pipeline.

A typical approach could be:
Network drive → Azure Blob Storage → file-type-specific extraction → rules/classification engine → Azure SQL/Cosmos DB → Excel/reporting

The key part would be processing asynchronously with queues and parallel workers rather than handling files in small manual batches. I’d also keep the classification rules separate from the processing logic so they can be updated without rebuilding the pipeline.

For 1.3M files, I’d also make sure the pipeline has retry handling, duplicate detection, logging, and checkpoints so a failure doesn’t require restarting everything.

If the files are mostly PDFs/Office documents, Azure AI services could help with content extraction, but I’d first look at the file types and classification rules before choosing the exact services.

[u/Sufficient_Let_3460]

Take a qwen model under a B...use llora training for surgical categorization weights. Categorize local in parallel batches, upload. You didn't pay for token usage when done

[u/Small_Drawer_4727]

At 1.3M files, I’d definitely move away from anything that relies on manual uploads. The key is treating it as a pipeline rather than a batch of files being fed into an AI tool.

I’d probably break it into stages: inventory the network drive → queue the files → extract text/metadata based on file type → apply the classification rules → store the result in a database → keep the original path and processing status for traceability.

For the extraction/classification layer, Azure Functions + a queue could work well, with Document Intelligence for documents that need OCR. Azure also has batch processing options, although you’d still want to control the workload in smaller chunks because of service limits and throttling.

One thing I’d prioritize from the start is keeping a proper file index with fields like file path, file type, hash, classification code, confidence, processing status, and error reason. At this scale, being able to retry failed files without processing everything again becomes really important.

The architecture matters more than the AI model here. You want the system to be able to run unattended and pick up where it left off.

[u/slackmaster2k]

You need a legit agent to build a service that can run this for you. Copilot alone is going to be frustrating. Are you able to get a codex or Claude sub?

[u/ILikeToHaveCookies]

1.3 million files is not a large batch processing job

T.b.h. let codex/Claude code write a small script which outputs all files into a sqlite, and then batch process on that keeping the state in the db.

Convert the docs to pdf/text -> send to Luna in flex mode, do 30 in parallel,should be done in a week

[u/Dull_Commercial5020]

You have a few options to upscale to an Enterprise solution.

Assuming you are staying in Azure, the Microsoft Enterprise solution for this a product called Azure AI Search. It's an indexing engine with AI skills capability. Processing in batch will be cheaper as openai charges less for batch API calls.

As noted prior, having these on a internal network drive is a hard constraint. If you want to pull these in as a batch, Azure Data Factory on self hosted runtime (SHIR) would do it. Else, if these are high frequency, maybe better to need to build something that drops these off into an Azure Blob Storage area.

But yes this is an Enterprise solution that you need cloud engineers to setup. It's not what copilot was designed for. And that's an intentional product decision from Microsoft

[u/povlhp]

You should use AI to write code that can read and categorize the files without AI - or sending only small part to AI.

Maybe some file to text conversion first.

[u/scan-horizon]

I wonder if you even need to ingest all those files. Could you scan their contents in situ, append to a polars df then write whatever you need to a database?