blog.sspboyd.ca

Twitter API: to cache or not to cache? When, why and how?

How do twitter-based apps access tweets going back several months? Do they keep data in dbs or do they re-search for tweets each time a visitor makes a query?

I’m asking because most of the playing around I’ve done with the twitter api has focused on the query functionatlity. This is a great way to pull in data but it is slow and I believe it would be much quicker if I had that data stored in a csv or db format. So the question I’m asking myself is whether I should be setting up a cron job that bakes out a csv or writes to a db once per day for the specified query strings. That would allow me to quickly pull in old tweets going back as far as I’d like. But this would mean that I would still need to write another set of methods to handle incoming queries not already in the daily cron list.

In looking at the Design Patterns section of the Twitter API FAQ, I came across this suggestion for caching:

“We recommend that you cache API responses in your application or on your site if you expect high-volume usage. For example, don’t try to call the Twitter API on every page load of your hugely popular website. Instead, call our API once a minute and save the response on your end, displaying your cached version on your site.”

Maybe this is the answer, but at what point does the traffic and response time warrant this approach? Lots of questions!

Edit:

Links

• http://www.smashingmagazine.com/2009/03/02/twitter-web-designer-and-developer-toolbox-api-and-tutorials/
• http://stackoverflow.com/questions/3040904/save-json-outputed-from-a-url-to-a-file