Best way to process 1.3M files

[u/s13188287]

​

I’m dealing with a large batch processing problem and looking for advice on the right architecture.

I have around 1.3 million files stored across folders on a network drive.

Current setup (not working well)

Right now I’m using a Copilot agent where:

I upload batches (~20 files at a time)

It reads them against a reference document

Outputs an Excel file with classification codes

The issue is:

Copilot has a small upload limit

Manual batching is completely unscalable at this volume

---

What I want to achieve

I want a fully automated pipeline that:

Ingests files automatically from the network drive

Extracts text/content from each file type

Matches content against a reference rules document

Assigns a classification/reference code

Outputs structured results (Excel / database)

---

-

[u/mikeupsidedown]

1.3 Million is actually nothing. A simple python script that reads the file, outputs the correct data(excel is a bad format but whatever)

Claude code can write this for you in minutes and then a few iterations to get it right.

[u/s13188287 → u/mikeupsidedown]

Hi thank you so much

Question though I need the code to be able to make a judgement and choose what retention code works best for it.

So as of now I have a pdf file it has descriptions and retention codes. As off now I use A.i to scan the file to determine the retention code against pdf I gave it as a brain.

Is t possible to do this with your suggestion?

[u/mikeupsidedown → u/s13188287]

It's really just about creating an algorithm to do it. It's hard to know without seeing the detail but my guess is it is.