From The Dev Corner: How Our Feed Fetching Works
May 22nd, 2007
We see quite a few questions from people asking about why their feed hasn’t updated or is slow to update (we’re working on speeding it up) so we thought we’d give everyone a little heads up on how it is all working and how we are working to speed it up.
Step 1: Figure out what needs to be updated
The common case is that we will update anything more than 30 minutes old, we don’t have a per-feed setting for update frequency at the moment, though we’d like one.
Step 2: Queue ‘em up
We grab a certain number of the items that haven’t been updated the longest and add them to a queue for processing.
A couple services requested that we poll less frequently so we don’t query any feeds from their servers more than once every minute — if we run into one of them that has happening too fast it gets kicked back in the queue for a bit. Unfortunately, there are only 1440 minutes in a day so this obviously doesn’t scale, we are interested in finding ways to alleviate this so that users of these services can have a better experience on Jaiku.
Step 3: Process them
Download the streams, find out if there are updates, add them to the site. The tricky part is again related to scale, we can only process so many of these every minute and unfortunately (well, a bit fortunately for us) we have quite more than enough users who have feeds becoming due to be updated every minute than we can actually run through in a minute, this means it can take more than the desired 30 minutes to update the feed.
Conclusion
Solutions to the update frequency problems will be adding more servers and bandwidth to the mix, which we are actively doing, so it will become more timely as we try to keep up with new users and their feeds.
Hope that helped explain some things for people.
smperris Says:
Thanks for that. I found it interesting, and the different etiquettes that come with polling feeds is intriguing.
Also glad to see that you’ve ironed out the bugs in catching new feeds :>
Keep up the good work, lads.
May 22nd, 2007 at 3:08 pm
William Lanphear Says:
Understand you’re working on speeding up feed updates. That doesn’t explain why many items from my Twitter feed don’t get picked up.Frequent Twitter updates seem get missed even though they are in the current available RSS feed.
May 22nd, 2007 at 6:46 pm
Sebastiaan Says:
Thank you! I’d been googling for this info, the updating schedule seemed extremely flaky, now I know why.
The good news is I added a flickr feed earlier and it got updated within 5 minutes!
May 22nd, 2007 at 11:44 pm
Kevin Says:
Thanks for the post. It is great to see a company showing us how things are done. My expectations are more in line now, knowing how difficult it is for you guys to process the feeds! ;)
May 23rd, 2007 at 6:31 pm
Tateru Nino Says:
I’m not certain that the feed machine is respecting the iso8601 timezones when inserting into the timeline.
May 24th, 2007 at 10:29 am
Troed Says:
I’m missing the description field in feeds you fetch from Wordpress and del.icio.us. I was planning on only publishing my RSS feed from jaiku since it should include my blog and bookmarks - but the title is not enough. Both the blog entries and the notes for my bookmarks (description-fields) are critical for that to be useful.
June 4th, 2007 at 6:28 pm
Peter Saint-Andre Says:
Polling is so twentieth-century. Aren’t you guys on the FeedMesh? How about Atom-Over-XMPP? ;-)
June 10th, 2007 at 6:44 am
Paul Jacobson Says:
Ok, I have been grumbling about the slow updates. Has the service become too busy to update in under an hour? I am frustrated with the delays and it would help if I understood the reason why. I’d like to use Jaiku as the focal point of my online activities and that only really works if the service is both reliable and fast. Perhaps you could post an update on your upgrades?
July 16th, 2007 at 12:31 am
Hendry Lee Says:
My Jaiku page is not updated and out of sync with my Twitter page. That seems very strange.
Have I done anything wrong? I tried deleting and adding the feed again, but still no result.
December 19th, 2007 at 7:29 am
Breggol Says:
It looks like my Twitter feed is not being parsed by Jaiku. No updates visible since I initially added the feed. Is something wrong? is Twitter too much of competition? Who can tell me?
May 16th, 2008 at 5:46 pm
delade | From the Dev Corner: An Under-The-Hood Preview of Our New Engine Says:
[…] Just like on the inside the locomotive is composed of a drive train, transmission, and so on, Jaiku is composed of services for features like fetching Web feeds, receiving and sending SMS, the IM bot, and Jaiku Mobile. We’re reimplementing them all so that they work well with the App Engine. In some cases we’re expanding the App Engine platform itself. We’re taking the opportunity to learn from porting Jaiku to benefit both our users and make App Engine better. […]
August 19th, 2008 at 10:33 am
Jaiku ist tot, es lebe Jaiku :: RSS Blogger Says:
[…] Just like on the inside the locomotive is composed of a drive train, transmission, and so on, Jaiku is composed of services for features like fetching Web feeds, receiving and sending SMS, the IM bot, and Jaiku Mobile. We’re reimplementing them all so that they work well with the App Engine. In some cases we’re expanding the App Engine platform itself. We’re taking the opportunity to learn from porting Jaiku to benefit both our users and make App Engine better. […]
August 28th, 2008 at 1:44 pm