Building Your First Yahoo! Pipe

If you have already used Yahoo! Pipes yourself, you may be wondering why you need a how to on building a pipe since it is actually quite simple. However, after speaking with some friends, I was forced to admit that the interface makes it *seem* complex and intimidating - that is until you actually get started. So, for those of you who have not tried it yet, here is a simple tutorial on how I built the ColdFusion aggregator feed I discussed yesterday.Step 1: Gather Your Sources
The first thing you need to do is add some data sources (ex. RSS or Atom feeds). Yahoo! has integrated some predefined sources like Yahoo! search and local as well as Flickr. In this case, however, we are going to start by adding the three Adobe aggregators using a "Fetch" component. So, start by dragging the "Fetch" component onto the stage and then add the three feed URLs into the form box (you can add additional feeds by clicking the plus button).

Step 2: Filter the Feed
These three aggregator feeds have quite a bit of overlap. This would render our combined feed difficult to use if we had a lot of duplicate items, therefore we are going to filter them based upon their titles. If you see my actual design you will notice a union in there before the filter; ignore this for the moment.

Click the "Operators" navigation item on the left menu and drag a "Unique" onto the stage. If it isn't already selected, select "title" from the "Filter non-unique items based on" drop down. To connect this to your "Fetch," click the blue dot on top of the "Unique" item and drag it to the bottom blue dot on the "Fetch" item. You can now click the top bar of the "Unique" item (it should turn orange) and run the debugger by clicking the "Refresh" link within the Debugger at the bottom of the stage.

Step 3: Sort the Data
Drag a "Sort" item from the Operators menu onto the stage. You can now add sorts by clikcing the plus button within the sort object. From the first drop down, select "pubDate" and set that to "Descending" order using the second drop down. Connect this Sort object to your Unique object using the method described above.

Step 5: Truncate the Feed
Our combined feeds could return a rather large number of results that would make the feed itself more difficult to use. Therefore, I decided to truncate the feed to 50 items. Doing so is simple, simple drag a "Truncate" object onto the stage (it is also located under "Operators") and set the number of items to 50 in the text box.

That's it!
Simply connect your "Truncate" object to your "Sort" object on one end and to your "Pipe Output" object on the other and you are done. You have just created a combined feed of 50 entries based upon three aggregator feeds that filters only unique items and sorts by date.

Getting More Advanced
Next I decided to get a little more advanced by looking for ColdFusion related items in TechMeme, Slashdot and dZone though they are rare indeed. To do so, I use the Fetch object to add the feeds, but I then use a "Content Analysis" operator to create a keyword analysis of the feed items. Next I add a filter to "[Permit] items that match [all] of the following" rules and add a rule whereby the [y:content_analysis] [contains] [coldfusion].

In order to combine this feed with my other Fetch I drag a "Union" object onto the screen and connect my "filter" to one entry point and my Adobe aggregator "Fetch" to another. I then continue to pipe the content through the "Unique", "Sort" and "Truncate" items discussed earlier.

Conclusion
This is still a pretty basic example of what Yahoo! Pipes can do. If you would like to get more advanced, you can even create feeds based upon user submitted data and such. I admit, I have only scratched the surface here, but, if perchance you were intimidated when you first looked at the Pipes IDE, hopefully this has assuaged our concern. Enjoy and feel free to share some ColdFusion or Adobe related feeds you might create with this tutorial in the comments.

Great tutorial! My only suggestion would be to add the del.icio.us feed for pages tagged with ColdFusion.

I quite like Pipes, but one small thing still bothers me: looking at all my input from various sources, it's often a little confusing trying to figure out the source of each, which gets in the way of quickly parsing the info. Do you know if there is a way to set up Pipes so that the title of a post reads, for example, like "[NAME OF BLOG/SOURCE] Title of post"?

Recently found out about Yahoo Pipe. But
If you haven't got anything good to say.......
Found the concept extremely worth my time to get to know
Almost exactly what i was looking for.

The rest of the story
Documentation EXTREMELY deficient.
Got one going with a Google module and an output.
In debugging couldn't copy the resulting URL's
Tried again but couldn't find the output moduloe or couldn't originate the connectors between two other modules

Can't believe I am the only one. Help