How should I properly optimize batching, retries, or query shaping to make this process stable at scale in SharePoint Online?

To optimize batching, retries, and query shaping for your operations in SharePoint Online, especially when dealing with large lists like yours with around 25k items, consider the following best practices:

1. Batch Size: While you are currently using batches of 200-300 items, you may want to experiment with smaller batch sizes. Reducing the batch size can help mitigate the risk of hitting limits and encountering errors like 400 Bad Request or 500 Internal Server Error.
2. Error Handling and Retries: Implement robust error handling in your script. When you receive a 400 or 500 error, consider implementing a retry mechanism with exponential backoff. This means that after each failure, you wait a bit longer before trying again, which can help reduce the load on SharePoint and increase the chances of success on subsequent attempts.
3. Throttling Awareness: Be aware of SharePoint's throttling policies. If your application is making too many requests in a short period, it may be throttled. Use the Retry-After header in responses to determine how long to wait before retrying requests.
4. Query Optimization: Ensure that your queries are as efficient as possible. Limit the number of fields retrieved to only those necessary for your updates. This reduces the amount of data being processed and can improve performance.
5. Traffic Decoration: Make sure your requests are properly decorated with AppID/AppTitle and User-Agent strings. Well-decorated traffic is prioritized over undecorated traffic, which can help with stability and performance.
6. Regular Audits: Conduct regular audits of your queries to identify and eliminate redundant calls. This can help streamline your operations and reduce the load on SharePoint.
7. Use of Delta Queries: If applicable, consider using delta queries to only retrieve changes since the last query. This can significantly reduce the amount of data being processed and improve performance.

By implementing these strategies, you should be able to improve the stability and performance of your batched updates in SharePoint Online.

References:

• Avoid getting throttled or blocked in SharePoint Online
• Performance guidance for SharePoint Online portals

Hi Buchanan Christopher

Thank you for reaching out to Microsoft Q&A forum

When performing batched updates on a large SharePoint Online list (e.g., ~25,000 items) using the PnP Core SDK, errors such as 400 Bad Request or 500 Internal Server Error often occur. These failures typically happen because the combined payload of a batch exceeds SharePoint’s request size limit (~2 MB) or due to throttling when too many operations are executed in rapid succession. Smaller lists work fine because they don’t hit these scale-related constraints.

In practice, PnP Core SDK mitigates this by default paging (it retrieves items in batches of 100 by default for large lists), but if your query or CAML is not configured for paging or indexing, you might inadvertently hit this limit. So, kindly ensure your queries use indexed columns or are broken into pages to respect the 5k threshold, as stated in Working with list items

That said, in order to mitigate this, for the best practice, you should:

1. Limit each batch to 50–100 items and use paged queries or CAML row limits to avoid exceeding the 2 MB request size and the 5,000-item view threshold.
2. Select only required fields and rely on PnP Core SDK’s built-in retry for HTTP 429/503. Optionally, add short delays between batches for stability.
3. Process batches one at a time, log results, and retry failed items individually or in smaller batches since SharePoint batch operations are not transactional.

Link references:

https://dori-uw-1.kuma-moon.com/en-us/sharepoint/dev/sp-add-ins/make-batch-requests-with-the-rest-apis

https://dori-uw-1.kuma-moon.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online

Hope my answer will help you, for any further concern, kindly let me know in the comment section.

If the answer is helpful, please click "Accept Answer" and kindly upvote it. If you have extra questions about this answer, please click "Comment".     

Note: Please follow the steps in our documentation to enable e-mail notifications if you want to receive the related email notification for this thread.