Ideas for improvement:
paginate(url, head = FALSE, limit = 10000, verbose = NULL)
| url | API endpoint |
|---|---|
| head | Return the data ( |
| limit | Number of items per query |
| verbose | Whether to produce verbose output. |
- Paginated endpoints return a 'X-Total-Count' header record with the total number of records. This could be used to pre-allocate space and avoid the while() loop. First call to endpoint could have limit=0 to avoid retrieving any results. - Move the JSON processing outside of the loop. Just accumulate all of the JSON into a single document. Evaluate whether this is an improvement! - Use HEAD to find expected number of results. - Could avoid final API query by checking if the number of results returned was less than limit.