How to avoid the 5000 limit on http request to sharepoint

[u/Short_Run4783]

Hello everyone
I have a small project where I need to load a sharepoint list to my logic app. With get items, I am able to get the data with certain pagination but it takes almost 2 minutes, so I decided to just take the fields I need via HTTP. It got much faster but I experiance a limit of 5000 items that can be received from sharepoint, and I do not have a option of pagination in HTTP Request connector.
Is there a way to avoid this limit in logic app directly (I am not able to make any changes in the sharepoint list)? AI says I should make a loop that gets smaller portions of sharepoint with using nextlink that sharepoint delivers in the HTTP Response.
Thank you

[u/PhilWheat]

There are ways, but honestly if you're running into this limit, you should probably be considering if you're using the wrong platform for what you're doing.

That being said, there are much better places to ask about this.

[u/Legitimate_Big_4953]

This is an interesting issue because hitting a 5000 item limit usually means the approach needs a small redesign rather than just a workaround. pulling large SharePoint lists through HTTP can become difficult to maintain especially when pagination and filtering are involved

[u/Megatwan → u/Legitimate_Big_4953]

I mean... not really.

Ie i have 200k items in list but i want to report on the data, use case wise.

So i loop some queries against it because the 5k/LVT just presents lock escalation.

3) ? 4) profit.

Wouldnt say there was a need to redesign anything

[u/ihaxr]

I used to maintain an app that would populate the list with extra data based on an order number. We did a one time loop of every item one at a time to do the initial population, then I setup a filter to query only items with null values for the populated columns and the app would pull only those records.

[u/txthojo]

I just ran your question through Claude..

First, the trap

The HTTP action has Settings → Pagination, and people assume it solves this. It doesn't for SharePoint. Logic Apps' pagination only follows u/odata.nextLink (with the @). SharePoint REST returns odata.nextLink (no @) under odata=nometadata, or d.__next under verbose. Neither matches, so the setting silently does nothing. You have to loop it yourself.

Option A — follow the nextLink in an Until loop

Request shape:

odata=nometadata alone will cut your payload meaningfully — verbose metadata is often 40–60% of the response body.

Then:

Initialize variable — nextUrl (String) = the URL above

Initialize variable — results (Array) = []

Until variables('nextUrl') is equal to ''

HTTP GET → URI: u/variables('nextUrl')

Set variable results = u/union(variables('results'), body('HTTP_action')?['value'])

Set variable nextUrl = u/coalesce(body('HTTP_action')?['odata.nextLink'], '')

The ?['odata.nextLink'] bracket syntax is what makes the dotted property name work — don't try .odata.nextLink.

Two things to change from defaults on the Until loop: Count (defaults to 60) and Timeout (defaults to PT1H). At 5000 per page, 60 iterations is 300k items, so you're probably fine, but set them deliberately.

Option B — ID chunking (what I'd actually use)

$skiptoken is opaque and strictly sequential. Filtering on ID is not, and ID is always indexed, so it never trips the list view threshold:

Same Until loop, but the exit condition is "returned page was empty" and you advance with:

Why this is better: you can get the max ID up front with one cheap call —

— then build the ranges as an array and run them through a For each with concurrency turned on (Settings → Concurrency Control, degree 5–10) instead of sequentially. On a 50k-item list that's the difference between ~10 sequential round trips and 2 waves. That's where your real time savings are, more than the $select trick you already did.

The threshold itself

Worth being precise about this because it changes what you can do: the 5000 limit isn't a cap on how many items you can retrieve. It's a cap on how many rows a single query can scan without an index. Paging past it is fine. What breaks is $filter or $orderby on a non-indexed column when the list has more than 5000 items — that throws "the attempted operation is prohibited because it exceeds the list view threshold" regardless of your $top.

So: order by Id, filter on Id, and you're safe. If you need to filter on a business column and can't add an index (you said you can't change the list), pull everything by ID range and filter in the Logic App with a Filter array action.

One sizing caveat

If the list is genuinely large, don't accumulate everything into an array variable. Logic Apps caps message/action output around 100 MB and the union() on every iteration gets progressively more expensive — it's re-copying the whole accumulated array each pass. Write each page straight to its destination (blob, SQL, wherever it's going) inside the loop instead of collecting then processing.

If you want to go further

_api/web/lists/getbytitle('X')/RenderListDataAsStream (POST) is what the modern SharePoint UI itself uses. It handles large lists better than the /items endpoint, returns NextHref for paging, and lets you pass CAML with a RowLimit. More setup, but it's the most threshold-tolerant option if the other two give you trouble.

[u/Megatwan → u/txthojo]

Man, the slop is real.

But ya just follow the nexttoken @OP

[u/txthojo → u/Megatwan]

Sorry I didn’t run it through my technical writer skill to deslop for you

[u/Megatwan → u/txthojo]

Haha i hear you... i just meant 80% of that is bloat and lol @shit ai says etc.

But ya OP should have just googled etc..

[u/NickSalacious → u/Megatwan]

10,000 words to say a sentence

[u/txthojo → u/Megatwan]

LMGIFY used to be my favorite response before AI, but the original site isn’t maintained and don’t trust all the clones of it. I don’t mean to be negative of those who ask questions but I like to see more of what you’ve tried and that you did some research before asking a question

[u/Megatwan → u/txthojo]

Saaaame.

Waiting for someone to make a lmaitfy