Hi ​@kabilanrajasekar_c,

The PagerDuty API does have a hard cap of 10,000 results per query, even when using pagination. This is a built-in API limitation and can’t be bypassed directly.

The best way to fetch more than 10,000 incidents is to split your queries into smaller chunks. You can do this by breaking up your time range (for example, querying one week or one day at a time), or by filtering on other fields like service or team. This way, each individual query stays under the 10,000-record limit, but you can still collect all the data you need by combining the results.

I suggest using the official PagerDuty Python API library to make this process easier. It handles authentication, pagination, and API requests for you, so you can focus on your logic for splitting up the queries.

Let me know if this works as a solution for your case, and have a nice day!

Hello!

I just started working on unrelated feature of our Python library that will include some new helper methods for relative time intervals. They will make it easier to implement such a feature, i.e. to iterate over all historical records in a given time window, when the total number of records in that window is expected to be greater than 10,000, by breaking it down into smaller time intervals.

I’ll post updates here once I start working on it.

Edit: I ended up working on it in my existing pull request: https://github.com/PagerDuty/python-pagerduty/pull/50

Hello ​@kabilanrajasekar_c,

I have released python-pagerduty version 3.0.0, which includes a new method `RestApiV2Client.iter_history`. Documentation for its usage:

https://pagerduty.github.io/python-pagerduty/user_guide.html#retrieving-large-historical-datasets

https://pagerduty.github.io/python-pagerduty/module_reference.html#pagerduty.RestApiV2Client.iter_history

Let me know what you think, and if that helps!

Also, note: this won’t necessarily speed up the retrieval of incident records; it will proceed at roughly the same speed as if `iter_all` were used directly. If one were to use it to retrieve the whole history of incidents every time the automation runs, it will download the same data over and over again, and this will be time consuming. For better performance, one could memoize or store results using the same system that retrieves and processes results, and subsequently fetch only new results using the previous value of `until` as the new value of `since`.