Ideas for improvement:

paginate(url, head = FALSE, limit = 10000, verbose = NULL)



API endpoint


Return the data (FALSE) or the number of records (TRUE)?


Number of items per query


Whether to produce verbose output.


- Paginated endpoints return a 'X-Total-Count' header record with the total number of records. This could be used to pre-allocate space and avoid the while() loop. First call to endpoint could have limit=0 to avoid retrieving any results. - Move the JSON processing outside of the loop. Just accumulate all of the JSON into a single document. Evaluate whether this is an improvement! - Use HEAD to find expected number of results. - Could avoid final API query by checking if the number of results returned was less than limit.