So I understand that faction data is limited to 100 items at once. But it's hard to know what 100 items is to request when you haven't pulled it before. So I am trying to ask for 1 day at a time, starting with the day our faction was formed back last September. But the data it gives me is for the last 24 hours. Here is the request string I am using (minus key and comment):
Okay, finally got a chance to fix that. It is behaving better, but it's still a real hassle to figure out the timestamps- if there is an hour between events, were there no events, or did I just hit a 100 event boundary and now it is time to request again?
If anyone has an elegant solution for this, I'd appreciate hearing it. Thanks.
So I had something of an epiphany last night- if I reverse the order and focus on loading oldest first, then I can just use a max+1 function to set the lower bound of the new pull and then run the upper bound as 24 hours later. Seems to be working much better...
And it worked better until it didn't. I got ~6000 good rows (85 or so days), then I started getting tons of duplicates. I've gotten up to 97 days of history now, but I'm getting more duplicates than new records at this point. I am noticing that Google is choking on the spreadsheet (taking a very long time to update the formulas) so it may not even be on the Torn side of the equation, so I may need to move it out into a separate file or something, but if anyone has any suggestions, I would appreciate them.
EDIT- Nevermind. My sheet had grown and my check to pull the last date wasn't getting all the way to the end.
By way of update, I finally have all of the faction data pulled (thank goodness it is less than a year- I can only imagine pulling everything for something with 100 players and many years of history). Once the job was reliable, I set it to pull another 100 rows every 5 minutes. Now that it is finished, I set it to once per hour.
So now to setup a functional dashboard and charts off of the data.