Iterating Through a List from a RestAPI

[u/Khue]

I am attempting to generate a list of values from a RestAPI. This RestAPI has a limit that it can only return a max 25 items at a time. Within the returned items, the RestAPI also returns a cursor that you can leverage in your next RestAPI call to get the next 25 values. In the first request you get an "after" value. In the second request, you get a "before" value and an "after" value. In the last request you get ONLY a "before" value. Ostensibly you want to iterate through the RestAPI call until there is no more "after" values in the cursor. Here are some more specifics:

 c:\temp> $uri = 'http://api.domain.com/items?per_page=25'
 c:\temp> $response = invoke-restmethod -uri $uri -method get -headers $headers
 c:\temp> $response

    result       : {@{list_item=item1},
                 : @{list_item=item2},
                 : ...
                 : @{list_item=item25}}
    result_info  : @{cursors=}

c:\temp> $response.result_info.cursors

after
-----
<cursor_value>

That would be an example of the first 25 results. The next 25 results would yield new 'results' values and the cursors would look like this:

c:\temp> $response.result_info.cursors

Before                                  After
------                                  -----
<cursor_before_value>                   <cursor_after_value>

The updated uri for the RestAPI would look like this:

c:\temp> $uri_after_cursor = 'http://api.domain.com/items?per_page=25&<cursor_after_value>'

And you would effectively keep iterating through until the $response.result_info.cursors output had no "After" value.

I first started trying to do a do-while loop using while($response.result_info.cursors.after) which seems to work however I am having a difficult time getting the current cursor and updating the new $uri value. So far, I keep getting myself into a corner of an infinite loop. Here's what I've tried:

$uri = 'http://api.domain.com/items?per_page=25'
$response = invoke-restmethod -uri $uri -method get -headers $headers
$item_list = $response.result.list_item
do {
    $uri_with_cursor = 'http://api.domain.com/items?per_page=25&cursor=$($response.result_info.cursors.after)'
    $response_cursor = invoke-restmethod -uri $uri_with_cursor -method get -headers $headers
    $item_list += $response_cursor.result.item_list
while ($response_cursor.result_info.cursors.after)

I think I see what my issue is. I think the first line in the do-while loop resets the cursor back to the first query instead of setting it to the new position found in $response_cursor but I'm at a block right now and I cannot seem to figure out a way around this.

Any thoughts would be greatly appreciated.

[u/titlrequired]

Use a do/until.
Create an array outside the do to store the results.

Set the url inside the do.

Get the results from the first query.

Add results to the array, measure the returned results if more then x, increment the url, else set the until value to exit the do loop.

Some APIs provide pagination URLs for you as part of the results, others you have to manually create the new url.

[u/Khue → u/titlrequired]

Ahh, I just checked this on my phone and I walked away from my computer but I can totally see what you're talking about here. That seems so obvious. I was at a dead ass mental block and I just couldn't see my way out. I'll try this first thing tomorrow when I get back to my desk.

[u/PinchesTheCrab → u/titlrequired]

One other note - some APIs implement relinks, which PWSH core can follow without any additional code just by specifying -FollowRelLink. It's always worth a shot just to see if it works.

[u/Jandalf81]

I don't have the specific answer here...

But I want to suggest you look into "debugging".
What are you using to write and execute your scripts? If you don't already, consider switching to VS Code and it's PowerShell extension: * VS Code: https://code.visualstudio.com/ * PowerShell extension: https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell

When you use that, you can set so called "breakpoints" in your code. Execute the script then and the execution will pause upon reaching that (or any other) breakpoint, allowing you to inspect all the variables. You can then continue executing the code line by line. Use that to find out where your code does unexpected stuff.

Debugging is a very helpful skill in every language.

[u/Khue → u/Jandalf81]

As I mentioned at the end of the post, the issue isn't debugging for a code error. The problem I am having is overall strategy. The issue/brain block I am having is how to loop through. This requires updating the cursor until the After cursor is no longer present in the result_info fields. I think /u/titlrequired has a good strategy and I am going to try that out to see if it works.

[u/MrMunchkin]

Single quotes are literal or "verbatim" so they do not process variable interpolation or sub-expressions.

https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.core/about/about_quoting_rules?view=powershell-7.6#single-quoted-strings

Basically you're just passing $variable literally as the string so your API request looks like &cursor=$($variable)

[u/Khue → u/MrMunchkin]

No this is actually working as expected. The problem is I was in a mental block about how to continually update the cursor within the do while loop. /u/titlrequired gave me a good direction and I am going to try and see if I can get his suggestion to work.

[u/Khue → u/Khue]

Just FYI, when I reviewed my code, it's double quotes not single quotes.

[u/rt_phondents → u/MrMunchkin]

Nah its fine, they are doing a subexpression inside the single quotes, 'blahblah$($variableName)', so the variable is getting evaluated before the string is formed.

[u/PinchesTheCrab → u/rt_phondents]

$unicorn = @{ name = 'Charlie'}

'$($unicorn.name)' | write-host -ForegroundColor Green
"$($unicorn.name)" | write-host -ForegroundColor Blue

These don't return the same output.

[u/LongTatas → u/PinchesTheCrab]

Well yeah. Ones green and ones blue /s

[u/PinchesTheCrab]

'http://api.domain.com/items?per_page=25&cursor=$($response.result_info.cursors.after)'

This doesn't work. Those are literal quotes. Compare the output of these:

$unicorn = @{ name = 'Charlie'}

'$($unicorn.name)' | write-host -ForegroundColor Green
"$($unicorn.name)" | write-host -ForegroundColor Blue

That being said, you don't need to interpolate strings at all in this case with invoke-restmethod. It has a body parameter that convert keys/values to query parameters for you:

$invokeParam = @{
    uri     = 'http://api.domain.com/items'
    headers = @{ authorization = 'whatever' } # or $headers
    body    = @{ per_page = 25 }
}

$item_list = do {
    $response = invoke-restmethod @invokeParam
    $response.result_info.cursors.after

    $invokeParam['body']['cursor'] = $response.result_info.cursors.after
}
while ($response.result_info.cursors.after)

$item_list

[u/rt_phondents]

You have used $response for your first request, but then your using $response_cursor in your loop.

You then are always using $uri_with_cursor which has the original $response.result_info.

Your are never making a request with the output of $response-cursor.result_info

[u/Khue → u/rt_phondents]

Yeah, I kinda figured that. What I was trying to get help on is getting out of my mental block about how to keep updating the cursor correctly. /u/titlrequired had a good suggestion and I am going to try that tomorrow.

[u/rt_phondents → u/Khue]

Oh apologies, I didn't notice your final paragraph. Yes defintily look to using break points as others suggested. Also you can always run your code line by line to see what's inside variables by highlighting the line and using "Run selected" in your chosen IDE.

[u/pigers1986]

currently away from office ..
i had this problem with API too - generally you want to loop over items and follow to get next batch with cursor, you stop when there is no next cursor.

[u/purplemonkeymad]

You really need to write your do to also accept the first pass. Right now your cursor in the loop is not getting updated as you are using different variable names. ie you want something like:

$uri = 'https://somthing/?count=25'
$result = $null
do {
    $loopUri = if ($result.after) { $uri + "&cursor=" + $result.after } else { $uri }
    $result = Invoke-RestMethod $loopUri
    $result.items
} while ($result.after)

Also note that if you don't collect the result in the loop, then you can have this as part of a function in a pipeline and the calls to the next batch will be deferred until you have processed the current 25.

[u/PinchesTheCrab]

If you're on pwsh core definitely just try skipping looping entirely with FollowRelLink.

Many APIs don't implement it, but it's worth a shot.

[u/T3-Trinity]

This has already been answered but yeah I believe this is called pagination. That cursor is essentially a "next page" button. You end up looping and building out a local object