Jeetu's shared memory

When Facebook acquired FriendFeed last month, everyone knew it was getting some pretty impressive technology along with the obvious talent in the company. What people probably didn’t expect is that Facebook would open source a portion of it. But that’s what they’re doing today with the release of Tornado, a real-time web framework for Python, onto the web.

Another new Facebook addition, Dave Recordon, explains the open-sourcing today on Facebook’s Developers blog. That Recordon is the one doing this post isn’t all that surprising given his central role in the open source community. Here’s how he explains Tornado:

Tornado is a relatively simple, non-blocking Web server framework written in Python, designed to handle thousands of simultaneous connections, making it ideal for real-time Web services.
…
While Tornado is similar to existing Web-frameworks in Python (Django, Google’s webapp, web.py), it focuses on speed and handling large amounts of simultaneous traffic.

FriendFeed co-founder Bret Taylor has more on his own blog. He notes that in open sourcing Tornado, FriendFeed and Facebook hope that others will use it to build their own real-time web services. They have set up a demo of how it works at its most basic (commenting) here. As you can see, it looks a lot like the FriendFeed commenting system (pictured below).

Taylor lays out three key parts of Tornado:

All the basic site building blocks – Tornado comes with built-in support for a lot of the most difficult and tedious aspects of web development, including templates, signed cookies, user authentication, localization, aggressive static file caching, cross-site request forgery protection, and third party authentication like Facebook Connect. You only need to use the features you want, and it is easy to mix and match Tornado with other frameworks.

Real-time services – Tornado supports large numbers of concurrent connections. It is easy to write real-time services via long polling or HTTP streaming with Tornado. Every active user of FriendFeed maintains an open connection to FriendFeed’s servers.

High performance – Tornado is pretty fast relative to most Python web frameworks. We ran some simple load tests against some other popular Python frameworks, and Tornado’s baseline throughput was over four times higher than the other frameworks:

But there’s more. Buried in Recordon’s explanation is the following:

Tornado is a core piece of infrastructure that powers FriendFeed’s real-time functionality, which we plan to actively maintain.

I’ve bolded the key part there. That would seem to suggest that Facebook is now committing to activity maintaining FriendFeed. While the FriendFeed co-founders have more or less said that the service would live on despite the Facebook deal, Facebook has been pretty mum on the topic up until now. Of course, Facebook could simply be saying that it will maintain the technology (for its own uses), and not the service.

Next question: How long until Twitter starts using some components of Tornado?

You can download and find out more about Tornado here.

[photo is an altered version of the great Toronado bar logo]

Crunch Network: MobileCrunch Mobile Gadgets and Applications, Delivered Daily.

Once again, the Internet is shifting before our eyes. Information is increasingly being distributed and presented in real-time streams instead of dedicated Web pages. The shift is palpable, even if it is only in its early stages. Web companies large and small are embracing this stream. It is not just Twitter. It is Facebook and Friendfeed and AOL and Digg and Tweetdeck and Seesmic Desktop and Techmeme and Tweetmeme and Ustream and Qik and Kyte and blogs and Google Reader. The stream is winding its way throughout the Web and organizing it by nowness.

This real-time stream has been building for a while. It began with RSS, but is now so much stronger and swifter, encompassing not just periodic news and musings but constant communication, status updates, instantly shared thoughts, photos, and videos.

What does this mean for how we will come to consume information? John Borthwick from Betaworks has identified the real-time Web as a key investment opportunity (Betaworks portfolio companies include Twitter, bit.ly, Tweetdeck, Chartbeat, and Tumblr). He admits he and other investors are still feeling in the dark, but he describes the shift he is trying to capitalize on this way in a post titled “Distribution . . . now”:

First and foremost what emerges out of this is a new metaphor — think streams vs. pages.

In the initial design of the web reading and writing (editing) were given equal consideration – yet for fifteen years the primary metaphor of the web has been pages and reading. The metaphors we used to circumscribe this possibility set were mostly drawn from books and architecture (pages, browser, sites etc.). Most of these metaphors were static and one way. The steam metaphor is fundamentally different. It’s dynamic, it doesn’t live very well within a page and still very much evolving.

A stream. A real time, flowing, dynamic stream of information — that we as users and participants can dip in and out of and whether we participate in them or simply observe we are a part of this flow.

In a sense, he is trying to rationalize his investment strategy. But if he is correct, the shift from pages to ever-widening eddies of information will have a dramatic downstream impact on many Web businesses, especially media businesses. This rising stream has the potential to fundamentally change the contours of media distribution on the Web. Large destination sites like Yahoo and AOL, already weakened as distribution hubs by search and social networks, now face the prospect of becoming completely bypassed. No wonder AOL is sticking the stream in every part of its service, from its homepage to Bebo to AIM. (Yahoo is grappling with the emergence of the stream as well, but so far still thinks it can hold onto its place as a central traffic and distribution hub).

The stream does not replace Web pages or search, for that matter, but it has the potential to completely transform them. Already, we are seeing Web pages adopt the stream as a new user-interface. Web pages are increasingly being designed as places to present the most relevant streams of information. And with streams of data spreading everywhere, search actually becomes more important than ever as a navigation tool. As Borthwick points out:

Traffic isn’t distributed evenly in this new world. All of a sudden crowds can show up on your site.

Traffic occurs in bursts, depending on what people are paying attention to at that second across a variety of services. Someone might notice an obscure blog post on Twitter, where it starts spreading, then it moves to FriendFeed and Facebook and desktop stream readers such as Tweetdeck or Seesmic desktop and before you know it, a hundred thousand people are reading that article. The stream creates a different form of syndication which cannot be licensed and cannot be controlled.

The problem, more than ever before, becomes one of information overload. How do you keep from drowning in the deluge? Borthwick suggests letting go of teh notion that you can ever master the stream, even just your own personal data stream of friend’s Tweets, updates, blog posts, Flickr photos, YouTube video finds and so on:

This isn’t an inbox we have to empty, or a page we have to get to the bottom of — its a flow of data that we can dip into at will but we can’t attempt to gain an all encompassing view of it.

So jump into the stream and let it carry you away. Or you can stand timidly on the banks until everyone else around you has already taken the plunge.

(Photo credit: Flickr/Justin Lowery)

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Facebook has the second largest installation of Hadoop (a software platform that lets one easily write and run applications that process vast amounts of data), Yahoo being the first.

Learn how they do it and what are the challenges on DBMS2 blog, which is a blog for people who care about database and analytic technologies.

Facebook has just announced that applications on Facebook Platform can now be able to take advantage of the site’s built-in chat functionality, which launched last spring. Developers will now be able to present users with a list of their Facebook Chat buddies, tailoring the list to best suit their application (for example, they can choose to only present friends that already have the app installed).

Facebook users have been able to use Chat and their Facebook apps simultaneously since Chat launched (one of its biggest selling points is that it remains open at the bottom of the screen, no matter where on the site you go). But until now applications didn’t really have a way to tap into the power of Facebook Chat to help make their applications more social.

Aside from adding an enhanced social element to applications, the new feature could also help apps go viral much more quickly than they would using the standard Email invite system most Facebook apps employ. Developers can now present users with a list their friends who are online (even those that don’t necessarily have their apps installed), who they can then send invites via chat messages. Invites sent over chat have a greater sense of urgency and intimacy, so it’s likely that they’ll be more effective than invites sent through the site’s Email system.

Of course, integration with Chat gives apps on Facebook yet another way to try to spam you. In the dark ages of Platform, when every app seemed to spam users with reckless abandon, I might have been more concerned about this, but I suspect Facebook already has some measures in place to prevent abuse. And even if they don’t, you can always just sign out of Chat if things get bad.

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

Our latest strategy is taken from a great post by Paul Saab of Facebook, detailing how with changes Facebook has made to memcached they have:

…been able to scale memcached to handle 200,000 UDP requests per second with an average latency of 173 microseconds. The total throughput achieved is 300,000 UDP requests/s, but the latency at that request rate is too high to be useful in our system. This is an amazing increase from 50,000 UDP requests/s using the stock version of Problems are always specific. The understanding of the problem must be specific. The fix must be specific.

Burn profiling into your regression tests. Detect when and where performance tanks as a regular part of your build.

Use resources in proportion to what grows slowest. This requires multiplexing, but at least your resource usage is more predictable and bounded.

Batch work. When you have the CPU do all the work you possibly can in the quantum or the whole system grinds to a halt in processing overhead.

Do work and maintain resources per task. Otherwise locking for shared resources takes more and more time when there’s less and less time to do the work that needs to be done.

Change algorithms. Sometimes you simply need to do things differently. Tweaking will only get you so far.

You can find their changes on github, the hub that says “git.”

Facebook has launched Facebook Connect (first announced in May 2008) that allows its members to log onto other Web sites using their Facebook identification and see their friends’ activities on those sites.

FB has announced that in the next few weeks, few prominent sites (Digg/SFChronicle etc) will start rolling out it’s Connect service on their website.

So essentially, you can leave a comment on Digg using your FB id/password and the same will show up on your FB news feed.

What about OpenID/DataPortability?

FB Connect achieves the same functionality as OpenID  – just that it is built on proprietary code (which is not open at all) – a clear threat to OpenID.

Essentially, Facebook is trying to replace all logins with their own, and control the creation, distribution and application of the social graph using their proprietary platform.

The most scary part of this, is that while Facebook is quietly and methodically building out this vision with massive partners, the standards community is busy squabbling about naming the open alternative. – Chris Saad (co-founder of the Data Portability project).

FB’s policy on Connect:

When Facebook Connect is used to allow a user to “sign in” or authenticate with your website, the Facebook Connect option must be presented at least as prominently as the most prominent of any other sign in or authentication method on your site, and not as a secondary option. – Dev wiki

So, the social network overlord wants to own your users and in the process, build a better social graph for it’s own purpose.
Only time will tell whether OpenID gets past the internal struggle and catches up with Facebook.

If you are a socionet owner, will you participate in FB Connect?

Related posts:

1. Are we friends? Well, Google FriendConnect is live and we have enabled the…
2. Social networks and Human behavior – Why few social networks are successful? Ever wondered why only a few social network work, and…
3. Minekey mines Social Opinions – Revamps as Destination Site (for opinions) Minekey which started as a recommendation service for blogs, and…
4. Rediffmail integrates news feeds from Orkut and Facebook : The Fight for Starting Point Rediff is taking the threat from Orkut and Facebook very…
5. Orkut opening up Channel (or a social networking operating system?) Google plans to release the new set of Orkut APIs…
6. FaceBook to announce something big in India? Orkut apps and MySpace apps – both the big social…

Facebook is launching its “Verified Apps” program today. The program was first announced over the summer at their F8 Developer Conference – Third party applications will be segmented into “Great Apps” (currently reserved for iLike and Causes), “Verified Apps” (trusted apps, just not the best of the best), and everything else.

Facebook says they expect at least 10% of the 48,000 applications currently available to eventually become Verified Apps, although they won’t speculate beyond that. Apps that get the Verified designation will be given a special badge to place on the application, designation on the application directory, plus a few other bonuses like advertising credits and easier rules on how many notifications, emails and invitations they can send out to users.

The guidelines for acceptance are here, and consist primarily of proving that they are “trustworthy.” This is determined based on how secure, respectful and transparent the apps are:

Secure: Protects user data and honors privacy choices for everyone across the social graph. Facebook users are deliberate and specific about which data they choose to share, how they share it, and with whom. All applications must respect users’ choices and the choices of their friends by only accessing, using and sharing data users have explicitly allowed. Users put their trust in Facebook, our Platform and your applications. This trust enables us to provide with social information for your applications. So it is up to all of us to earn and maintain user trust.

Respectful: Values user attention and honors their intentions in communications and actions. Users trust that when they use your application, you will represent their intent and best interests, especially the messages you send about them or on their behalf. The more control you give them over how you represent them, the more likely they are to trust your application and want to use it more. Make sure to also value users’ time by employing proper communication channels and neither spamming users, nor encouraging them to become spammers.

Transparent: Explains how features will work and how they won’t work, especially in triggering user-to-user communications. Nothing is more frustrating than to click a button expecting one thing to happen and having something entirely different and confusing happen instead. Even worse is sending communications to a user’s friends that the user did not intend or want to send. This can undermine a user’s personal relationships and deters users from freely communicating on Facebook and through applications. The best applications are clear about their features and don’t try to deceive users.

Developers can apply now for the program, and will be slotted for a much longer application process at a later date. Once all apps are reviewed the program will go live, sometime early next year.

This Will Be A Serious Revenue Machine For Facebook.

There’s just one catch – developers must pay a $375 to “cover some of the operational costs of the program.” If every application applied, that would be $18 million in incremental revenue to Facebook. Our guess is half or more will apply. Certifications are good for one year, so this revenue is recurring.

Developers are given three chances to get approval (with feedback along the way). If they fail after the third attempt, they can re-apply 3-6 months down the road.

Crunch Network: CrunchGear drool over the sexiest new gadgets and hardware.

Tom Kincaid, a top platform developer and blogger mentioned in the Facebook Developer Forums last night that Beacon seems to be rearing its ugly head once again. (Update: And although Facebook’s Beacon platform was never actually removed from the service and the feature is not new, partners had backed off from it, which gave the perception to some that it was dead. Since then, Facebook has improved the service to make it more amenable to partners and users alike.)

According to Kincaid, he signed up for CBS Sportsline and got a Beacon-like pop-up, which he thinks may have used a Facebook cookie.

“I signed up on CBS Sportsline and joined fantasy football,” he wrote on the forum. “I got a pop-up on the bottom right. It looks like the old beacon stuff. I thought that didn’t work anymore, but it published a story to the homepage. I didn’t go through any kind of connect log in, it must have used the Facebook cookie somehow.”

I joined CBS Sportsline myself and added a Fantasy Football league to recreate Kincaid’s experience. Once I joined CBS Sportsline, I didn’t see the Beacon pop-up. But as soon as I created a team, the Beacon pop-up was displayed saying I created a fantasy football team, which gave me the option of “learning more,” saying it wasn’t me, or simply saying, “No Thanks.”

Much like Jesse Stay from the Staynalive blog, I found the same Beacon script (src=”http://www.facebook.com/beacon/beacon.js.php?source=10228841580″>) in the source code. I then clicked on the “Learn more” button and was brought to a Beacon information page.

After seeing the Beacon pop-up on CBS Sportsline, I went to other original Beacon partner sites to see if I could recreate the same experience on sites that Facebook may have been able to coax back into the fold, but I wasn’t able to. I signed up for TripAdvisor, but no Beacon pop-ups were displayed and I had another failing experience when I signed up for Zappos.

It now looks like Beacon gives you the option of posting Beacon updates in your timeline. If you click OK, it will be posted. But if you instead choose to remove it, you won’t find any mention of it in the timeline.

(Update: This is not new. Since late 2007, Beacon has been available on dozens of participating sites after the company made a series of improvements to ensure that users have control over what information is shared with their friends on Facebook.)

Crunch Network: CrunchBoard because it’s time for you to find a new Job2.0

This was on the cards but is now official. The Indian creators of the popular Facebook application Scrabulous have been sued by Hasbro, the US based company that owns copyright over Scrabble. Check out reports here, here & here. The suit has been filed against RJ Softwares, the Kolkata based software development company owned by Rajat Agarwalla & Jayant Agarwalla, who together created the game. Hasbro is apparently seeking damages from RJ and asking them to stop using the name ‘Scrabulous’ for their game. Not just that, they have also sent a copyright notice to Facebook under the DMCA (Digital Millennium Copyright Act) saying that it infringes the company’s intellectual property and asked them to remove Scrabulous from its site, which Facebook has refused.

Hasbro incidentally now has its own version of Scrabble on Facebook and that has 8K users compared to over two million for Scrabulous.

Its not going to be easy for the Indian team to fight the lawsuit since it is filed in New York, while they are based in India. Apparently some talks had happened between the two parties over a possible partnership (or acquisition) but that did not materialise. Facebook’s position is precarious as well. Under the Safe harbour provisions the DMCA, they are bound to take down the application failing which they could be dragged into a bigger litigation.

While I’m not an expert on this topic, I do have a working knowledge of DMCA. I think the use of the title “Scrabulous’ might have been a mistake by the Indian team and it might prove crucial in this case. The standard procedure adopted by internet companies in DMCA infringements is to either take down the disputed content straightaway if they think it violates IPR, or suspend it temporarily till a decision has been reached.

Question to Indian readers- without being jingoistic, whats your opinion on this issue?