Yahoo Pipes Insider's Tips: Custom RSS Feeds

When Pasha Sadri talks about Yahoo Pipes, he’s like a proud parent talking about his suddenly famous child.

And no wonder. Sadri, a principal software engineer for Yahoo, is one of the masterminds behind Yahoo Pipes. The highly popular Pipes is an interactive tool that allows you to combine many RSS feeds into a single data stream. The Pipes web site has an intuitive visual programming page that lets you mix and filter feeds exactly as you choose.

When it recently launched, I posted a users’s guide geared for users wanting to build a basic pipe.

So Pasha, what are your insider’s tips for Pipes?

Reusable Components

“As a software engineer, one of things that is very valuable is this concept of reusing components,” Sadri says. Pipes allows you to create stand-alone components (or “modules”) that you can reuse again and again.

To do so, simply save a pipe component with a new name, which saves it for later use. There’s no reason to start from scratch when you build a new pipe.

“I would like to encourage this as much as possible,” he says.

In fact, take a look at the more complicated pipes you’ve built and separate out several components to save for later use.

Saving and reusing components makes a pipe more readable, he says. “Because you can look at one module and you know what it’s generally doing, and you don’t have to do a lot with the details.”

Flexible Parameters

When you’re creating a module you intend to reuse, it’s helpful to save its various parameters with generic, open-ended labels.

For example, Sadri says, in his Craigslist Apartments pipe, “I can hardcode a location like Palo Alto in the search [parameter], then my pipe would only be producing stuff from around Palo Alto. But if I wanted it to be more reusable, I could take the ‘Palo Alto’ portion out and make it a user input. And give it a label, something like ‘Your Location’ – just a descriptive label for that input.”

Saving a module with generic parameters allows you to easily plug that component into your next pipe. And, if other people use your modules, “People can now configure it without having to clone it and change the location, for example, for their own purposes.”

“Open Source” Pipes

An interesting aspect of Pipes is how it parallels the open source software community. Just as open source programmers share their code, users who create pipes can make their work available to others.

If you click the “publish” button on Pipes, your modules are available for the world to use and share. Pipes – which requires no programming chops – allows non-techies a glimpse of life in the open source world.

Sadri is well aware of Pipes’s similarity to the open source model, and encourages user sharing. “Once you have a component, if you publish it, it’s potentially useful to other people,” he says.

“Definitely in the future, as the number of developers and users increases, I think the likelihood that somebody’s already done the work is going to increase.” There’s no reason for everyone to start from scratch.

To encourage this sharing, Sadri says his team is working on improving the site’s search function. (Pipes is still very much in beta mode, a point he stresses.)

“We spend most of our development time on the engine and the editor,” he says. And the Web site didn’t get as much attention as it should have. We are going to address this in the near future.”

URL Builder Module: Parameters within an RSS Feed

Sadri recommends that users explore the URL Builder Module, which he says is one of Pipes’s more powerful modules. This module allows users to access data beyond the usual RSS feeds that many Pipes users access.

Typically, Sadri notes, Pipes users find an RSS feed and copy and paste it into a Pipes Fetch module to get that data. However, some RSS feeds accept parameters.

“For example, again, Craigslist. If you do a search in Craigslist, at the URL for the RSS feed, you’ll see that there is a parameter, like ‘Q = your search term.’ This offers a lot of possibilities. Because if you could build that URL dynamically you can get the remote service to build different types of data for you.”

In the case of Craigslist Apartments, the location parameter is a user input, and whatever value the user enters is used with the URL builder module to construct a URL dynamically. That value is then used to fetch something from Craigslist.

Many online services offer this type of search interface, Sadri says, such as databases of real estate or stocks, or blog search services.

Dynamic Date Manipulation

One pipe building technique that’s not yet very well known, Sadri says, is the relative date parameter. (A lot of people have asked for data manipulation, he notes.)

Pipes allows input of not just exact dates, like April 1, 2007, but also accepts relative dates. So you can specify ‘two weeks from now,’ or ‘two days ago.’ This relative date parameter translates to an actual date at the time the pipe is executed. Using this technique, you can construct queries that change over time.

“Let’s say you want to look for events that are happening within the next few days,” Sadri says. “You have a source of events but you want to filter it, based on dates so that it’s always two or three days in the future. If you had used a hardcoded date you’d always have to come back and update your pipes. But with this dynamic date input, you can say ‘three days in the future’ and it will compute three days in the future.”

User Votes Make a Difference

Pipes has a Suggestion Board that Sadri says the Yahoo developers pay a lot of attention to.

(Note that you can view the Suggestion Board by “Most Recent” post or “Most Rated” post – this latter option shows you the updates that users are clamoring for.)

“We are actually paying attention to the number of votes that people have assigned to different features, and using that to prioritize the sequence of how we work on new features,” Sadri says. “It’s not a guarantee, because some of them are harder than others. So it’s a mixture of how easy it is to roll out and how much demand there is for it.”

He encourages people to post to the Board if they want a new module or an enhancement to an existing one. “And we’ll try to address it as soon as possible.”