Data Munging for Non-Programming Biologists
by Amir Karger, Eitan Rubin
|
Pages: 1, 2, 3
Filling the Niche: The Scriptome and Other Solutions
One of my greatest concerns in talking to people about biologists' data munging is that people don't even realize that there's a problem, or they think it's already been solved. Biologists--who happily pipette things over and over and over again--don't realize that computers could save them lots of time. Too many programmers figure that anyone who needs to can just read Learning Perl. I'm all for that, of course, but experimental biologists need to spend much more of their time getting data (dissecting bee brains, say) than analyzing it, so they can't afford the time it takes to become programmers. They shouldn't have to. Does the average biologist need multiple inheritance, getprotobyname(), and negative look-behind regexes? There's a large body of problems out there that are too diverse for simple, inflexible tools to handle, but are too simple to need full-fledged programming.
How about teaching a three-hour course with just enough Perl to munge simple data? At minimum, it should teach variables, arrays, hashes, regular expressions, and control structures--and then there's syntax. "Wait, what's the difference between @{$a[$a]} and @a{$a[$a]} again?" "Oh, my, look at the time." As Damian Conway writes in "Seven Deadly Sins of Introductory Programming Language Design" (PDF link), syntax oddities often distract newbies from learning basic programming concepts. How much can you teach in three hours, and how much will they remember after a month without practicing?
Another route would be building a graphical program that can do everything a biologist would want, where pipelines are developed by dragging and dropping icons and connectors. Unfortunately, a comprehensive graphical environment requires a major programming effort to build, and to keep current. Not only that, but the interface for such a full-featured, graphical program will necessarily be complex, raising the learning barrier.
In building the Scriptome, we purposely narrowed our scope, to maximize learnability and memorability for occasional users. While teaching programming and graphical tools are effective solutions for some, I believe the Scriptome fills an empty niche in the data munging ecosphere (the greposphere?).
Creation Is Not Easy
How much progress have we made in addressing the problem space between tool use and programming? Our early reviews have been mostly positive, or at least constructive. Suzy, our first power user, started out skeptical, saying she'd probably have to learn Perl because any tools we gave her wouldn't be flexible enough. I encouraged her to use the Scriptome in parallel with learning Perl. She ended up a self-described "Scriptome monster," tweaking tool code and creating a 16-step protocol that did real bioinformatics. Still, one good review won't get you any Webby awards. Our first priority at this point is to build a user base and to get feedback on the learnability, memorability, and effectiveness of the website, with its 50 or so tools.
It will take more than just feedback to implement the myriad ideas we have for improving the Scriptome, which is why I'm here to make a bald-faced plea for your help. The project needs lots of new tools, new protocols, and possibly new interfaces. You, the Perl.com reader, can certainly write code for new tools; the real question is whether you (unlike certain, unnamed CPAN contributors) can also write good documentation and examples, or find bugs in early versions of tools. We would also love to get relevant protocol ideas. Check out the Scriptome project page and send something to me or the scriptome-users mailing list.
Here's a little challenge. I really did have a client who renamed 768 files by hand before I could Perl it for him. Can you write a generic renaming atom that a NPB could use? (Hint: "Tell the user to learn regular expressions" is not a valid solution.) The winner will receive a commemorative plaque (<bgcolor="gold">) on the Scriptome website.
Speaking of new interfaces, one common concern we hear from programmers is that NPBs won't be able or willing to handle the command-line paradigm shift and the few commands needed (cd, more, dir/ls) to use the Scriptome. In case our users do tell us it's a problem, we're exploring a few different ways to wrap the Scriptome tools, such as:
- A Firefox plugin that gives you a command line in a toolbar and displays your output file in the browser. (Currently being developed by Rob Miller and his group at MIT.)
- An Excel VBA that lets you put command lines into a column, and creates a shell script out of it.
- Wrapping the command-line tools in Pise (web forms around shell commands) or GenePattern (a more general GUI bio tool).
We'll probably try several of these avenues, because they allow us to keep using the command-line interface if desired.
As for the future, well, who says that only biologists are interested in munging tabular data? Certainly, chemists and astronomers could get into this. I set my sights even higher. How about a Scriptome for a business manager wanting to munge reports? An Apache Scriptome to munge your website's access logs? An iTunes Scriptome to manage your music? Let's give users the power to do what they want with their data.
Sorry, GUI Neanderthalis, but you can't adapt to today's data munging needs. Make room for Homo Scriptiens!
You must be logged in to the O'Reilly Network to post a talkback.
Showing messages 1 through 10 of 10.
- Try this one on for size
2005-11-02 07:06:10 msporleder [Reply]
perl -MFile::Copy -e '$ostr = "^"; $nstr = "prepend"; map { $o = $_; s/$ostr/$nstr/; move $o, $_} @ARGV' file*
This way, ostr = any pattern in the file (including ^ for prepending to files and $ to postpending), nstr = the replacement. This particular call will prepend "prepend" to your file's.
perl -MFile::Copy -e '$ostr = "file"; $nstr = "otherstuff"; map { $o = $_; s/$ostr/$nstr/; move $o, $_} @ARGV' Prefile*
This call will replace the word "file" with the word "otherstuff".
- good stuff
2005-10-29 11:14:53 tclimb [Reply]
thanks for sharing it! i can see opportunity to apply the same idea to marketers and marketing data.
- EMBOSS
2005-10-27 07:16:32 danny_klose [Reply]
What about packages like EMBOSS, does the scriptome repeat functions available in EMBOSS?- EMBOSS
2005-10-27 20:13:24 TheAmirOfKarger [Reply]
There might be a bit of overlap with the simplest commands, such as taking a substring from a sequence. But mostly, EMBOSS does "real bioinformatics".
A few sample EMBOSS commands:
- Draw circular maps of DNA constructs
- Create a consensus from multiple alignments
The general model of the Scriptome, on the other hand, is that you use "real bioinformatics" tools to get as far as you can, then use the Scriptome to convert that data into tabular data and play with it (before possibly using the munged data as input for other bioinfo apps).
A couple examples of Scriptome things I've been doing lately with users:
- filter tabular BLAST output to get certain hits, get those sequences out of a big file, run BLAST on those sequences
- rearrange and filter tabular output that's too big for Excel
In both cases, there are other apps (BLAST, a camera-recording robot, a Perl script I wrote) on either side. In both cases, much of the work could be done with sh, or Excel, or some other bioinfo apps, but the Scriptome is a quick and dirty (and NPB-usable) way to move the data between the "real apps". Perl Hackers might call this "bioinformatics glue".
-Amir
- EMBOSS
- prepend and postpend
2005-10-21 11:17:49 msporleder [Reply]
perl -MFile::Copy -e 'for ($x = 0; $x < $#ARGV; $x++) { move $ARGV[$x], $ARGV[$#ARGV].$ARGV[$x]; }' `ls files*` prependstring
perl -MFile::Copy -e 'for ($x = 0; $x < $#ARGV; $x++) { move $ARGV[$x], $ARGV[$x].$ARGV[$#ARGV]; }' `ls files*` postpendstring- prepend and postpend
2005-10-24 05:24:22 TheAmirOfKarger [Reply]
msporleder wrote:
> perl -MFile::Copy -e 'for ($x = 0; $x < $#ARGV; $x++) { move $ARGV[$x], $ARGV[$#ARGV].$ARGV[$x]; }' `ls files*` prependstring
Hm. The Scriptome puts the parameters into the Perl code. And why do `ls files*` instead of just files*? Then a more idiomatic foreach lets you do:
perl -e '$pre="Prepend"; for (@ARGV) {rename $_, $pre . $_}' files*
Now add in a warn to say how many files we renamed, and you're set.
This isn't particularly generic. What about people who want to add a number to each file, or REMOVE things from filenames, etc? Then again, maybe it's a good idea to have a bunch of different rename tools, so that each one can be pretty simple. And we could always post the easy tools and worry about the complicated ones later. 80/20 rule and all that.
By the way, is File::Copy::move() better than rename()?
-Amir Karger
- prepend and postpend
2005-11-02 06:30:02 msporleder [Reply]
I generally use File::Copy rather than rename() because of the caveats mentioned in programming perl about rename(). Just habit, mainly.
- prepend and postpend
- prepend and postpend
- Bulk file renaming
2005-10-21 05:45:25 mysjkin [Reply]
Have a look at how the frontend is done in krename You can get some ideas at their screenshot page: (http://www.krename.net/Screenshots.11.0.html)
- Have you seen Sprog ?
2005-10-20 18:22:45 MattMan [Reply]
The concepts you're talking about here seem very similar to the ones that are behind the Sprog tool.
http://www.perl.com/pub/a/2005/06/23/sprog.html
http://sprog.sourceforge.net/
Unfortunately, it doesn't run on Windows, but the ideas are interesting.
There's also something called Taverna which seems to be targeted at bioinformatics, but which uses Java.
http://taverna.sourceforge.net/
- Have you seen Sprog ?
2005-10-21 11:17:37 TheAmirOfKarger [Reply]
MattMan said:
> The concepts you're talking about here seem very similar to the ones that are behind the Sprog tool.
> Unfortunately, it doesn't run on Windows, but the ideas are interesting.
I agree. There may be good stuff to steal there. Of course, it's not bio-centric, but actually our stuff isn't very bio-centric yet either. I appreciate that Sprog isn't overdoing the GUI. If we do build a GUI, it'll probably be similarly simple.
The lack of Windows support is a showstopper since most biologists here are on Windows.
> There's also something called Taverna which seems to be targeted at bioinformatics, but which uses Java.
I've talked to a couple Taverna folks. We could probably wrap our tools to make Taverna tools. Taverna has, I think, much larger goals - but that makes it necessarily have a slightly more complex interface.
-Amir
- Have you seen Sprog ?
