Download the authoritative guide: Cloud Computing 2018: Using the Cloud to Transform Your Business
When Pasha Sadri talks about Yahoo Pipes, hes like a proud parent talking about his suddenly famous child.
And no wonder. Sadri, a principal software engineer for Yahoo, is one of the masterminds behind Yahoo Pipes. The highly popular Pipes is an interactive tool that allows you to combine many RSS feeds into a single data stream. The Pipes web site has an intuitive visual programming page that lets you mix and filter feeds exactly as you choose.
When it recently launched, I posted a userss guide geared for users wanting to build a basic pipe.
So Pasha, what are your insiders tips for Pipes?
As a software engineer, one of things that is very valuable is this concept of reusing components, Sadri says. Pipes allows you to create stand-alone components (or modules) that you can reuse again and again.
To do so, simply save a pipe component with a new name, which saves it for later use. Theres no reason to start from scratch when you build a new pipe.
I would like to encourage this as much as possible, he says.
In fact, take a look at the more complicated pipes youve built and separate out several components to save for later use.
Saving and reusing components makes a pipe more readable, he says. Because you can look at one module and you know what its generally doing, and you dont have to do a lot with the details.
When youre creating a module you intend to reuse, its helpful to save its various parameters with generic, open-ended labels.
For example, Sadri says, in his Craigslist Apartments pipe, I can hardcode a location like Palo Alto in the search [parameter], then my pipe would only be producing stuff from around Palo Alto. But if I wanted it to be more reusable, I could take the Palo Alto portion out and make it a user input. And give it a label, something like Your Location just a descriptive label for that input.
Saving a module with generic parameters allows you to easily plug that component into your next pipe. And, if other people use your modules, People can now configure it without having to clone it and change the location, for example, for their own purposes.
Open Source Pipes
An interesting aspect of Pipes is how it parallels the open source software community. Just as open source programmers share their code, users who create pipes can make their work available to others.
If you click the publish button on Pipes, your modules are available for the world to use and share. Pipes which requires no programming chops allows non-techies a glimpse of life in the open source world.
Sadri is well aware of Pipess similarity to the open source model, and encourages user sharing. Once you have a component, if you publish it, its potentially useful to other people, he says.
Definitely in the future, as the number of developers and users increases, I think the likelihood that somebodys already done the work is going to increase. Theres no reason for everyone to start from scratch.
To encourage this sharing, Sadri says his team is working on improving the sites search function. (Pipes is still very much in beta mode, a point he stresses.)
We spend most of our development time on the engine and the editor, he says. And the Web site didnt get as much attention as it should have. We are going to address this in the near future.
URL Builder Module: Parameters within an RSS Feed
Sadri recommends that users explore the URL Builder Module, which he says is one of Pipess more powerful modules. This module allows users to access data beyond the usual RSS feeds that many Pipes users access.
Typically, Sadri notes, Pipes users find an RSS feed and copy and paste it into a Pipes Fetch module to get that data. However, some RSS feeds accept parameters.
For example, again, Craigslist. If you do a search in Craigslist, at the URL for the RSS feed, youll see that there is a parameter, like Q = your search term. This offers a lot of possibilities. Because if you could build that URL dynamically you can get the remote service to build different types of data for you.
In the case of Craigslist Apartments, the location parameter is a user input, and whatever value the user enters is used with the URL builder module to construct a URL dynamically. That value is then used to fetch something from Craigslist.
Many online services offer this type of search interface, Sadri says, such as databases of real estate or stocks, or blog search services.
Dynamic Date Manipulation
One pipe building technique thats not yet very well known, Sadri says, is the relative date parameter. (A lot of people have asked for data manipulation, he notes.)
Pipes allows input of not just exact dates, like April 1, 2007, but also accepts relative dates. So you can specify two weeks from now, or two days ago. This relative date parameter translates to an actual date at the time the pipe is executed. Using this technique, you can construct queries that change over time.
Lets say you want to look for events that are happening within the next few days, Sadri says. You have a source of events but you want to filter it, based on dates so that its always two or three days in the future. If you had used a hardcoded date youd always have to come back and update your pipes. But with this dynamic date input, you can say three days in the future and it will compute three days in the future.
User Votes Make a Difference
Pipes has a Suggestion Board that Sadri says the Yahoo developers pay a lot of attention to.
(Note that you can view the Suggestion Board by Most Recent post or Most Rated post this latter option shows you the updates that users are clamoring for.)
We are actually paying attention to the number of votes that people have assigned to different features, and using that to prioritize the sequence of how we work on new features, Sadri says. Its not a guarantee, because some of them are harder than others. So its a mixture of how easy it is to roll out and how much demand there is for it.
He encourages people to post to the Board if they want a new module or an enhancement to an existing one. And well try to address it as soon as possible.