Yahoo Pipes Insider's Tips: Custom RSS Feeds

When Pasha Sadri talks about Yahoo Pipes, he’s like a proud parent talking about his suddenly famous child.

And no wonder. Sadri, a principal software engineer for Yahoo, is one of the masterminds behind Yahoo Pipes. The highly popular Pipes is an interactive tool that allows you to combine many RSS feeds into a single data stream. The Pipes web site has an intuitive visual programming page that lets you mix and filter feeds exactly as you choose.

When it recently launched, I posted a users’s guide geared for users wanting to build a basic pipe.

But to provide more advanced instruction, I spoke with Pasha Sadri about his personal “insider’s tips.” As one its lead developers, Sadri is the ultimate Pipes expert. (In fact his own personal pipes are the most popular of any on the site.)

So Pasha, what are your insider’s tips for Pipes?

Reusable Components

“As a software engineer, one of things that is very valuable is this concept of reusing components,” Sadri says. Pipes allows you to create stand-alone components (or “modules”) that you can reuse again and again.

To do so, simply save a pipe component with a new name, which saves it for later use. There’s no reason to start from scratch when you build a new pipe.

“I would like to encourage this as much as possible,” he says.

In fact, take a look at the more complicated pipes you’ve built and separate out several components to save for later use.

Saving and reusing components makes a pipe more readable, he says. “Because you can look at one module and you know what it’s generally doing, and you don’t have to do a lot with the details.”

Flexible Parameters

When you’re creating a module you intend to reuse, it’s helpful to save its various parameters with generic, open-ended labels.

For example, Sadri says, in his Craigslist Apartments pipe, “I can hardcode a location like Palo Alto in the search [parameter], then my pipe would only be producing stuff from around Palo Alto. But if I wanted it to be more reusable, I could take the ‘Palo Alto’ portion out and make it a user input. And give it a label, something like ‘Your Location’ – just a descriptive label for that input.”

Saving a module with generic parameters allows you to easily plug that component into your next pipe. And, if other people use your modules, “People can now configure it without having to clone it and change the location, for example, for their own purposes.”

“Open Source” Pipes

An interesting aspect of Pipes is how it parallels the open source software community. Just as open source programmers share their code, users who create pipes can make their work available to others.

If you click the “publish” button on Pipes, your modules are available for the world to use and share. Pipes – which requires no programming chops – allows non-techies a glimpse of life in the open source world.

Sadri is well aware of Pipes’s similarity to the open source model, and encourages user sharing. “Once you have a component, if you publish it, it’s potentially useful to other people,” he says.

“Definitely in the future, as the number of developers and users increases, I think the likelihood that somebody’s already done the work is going to increase.” There’s no reason for everyone to start from scratch.

To encourage this sharing, Sadri says his team is working on improving the site’s search function. (Pipes is still very much in beta mode, a point he stresses.)

“We spend most of our development time on the engine and the editor,” he says. And the Web site didn’t get as much attention as it should have. We are going to address this in the near future.”