SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
PacBio ZMW Loading Productivity 2 HeidiJTP Pacific Biosciences 12 07-27-2015 05:57 AM
5500 Chemical Waste Disposal JPC SOLiD 5 06-12-2012 01:09 AM
what wrong with boost? feng Bioinformatics 3 10-16-2011 10:50 AM
Second-gen sequencing and lab productivity brjordan General 0 11-07-2009 06:53 AM

Reply
 
Thread Tools
Old 04-19-2013, 03:07 AM   #1
Azazel
Member
 
Location: Japan

Join Date: Oct 2010
Posts: 52
Default scientific workflow systems: waste of time or boost of productivity?

I read (online) and hear (conferences) a lot about scientific workflow systems, in the sense of http://en.wikipedia.org/wiki/Scientific_workflow_system.

There are many such systems under development, for example Kepler, Taverna, VisTtrails, LoniPipeline, Apache Airavata.

Scientific workflow systems *sound* very interesting when I read or hear about them; they promise reproducibility, grid and cluster support, use R without programming etc etc.

But when digging a bit deeper it looks like a tool where I can string together boxes to coordinate different programs. Nothing I can't do with make, bash and awk a dozen times faster. Also it's great that they have grid support but I don't have access to a grid. I have access to one of the Top 500 supercomputers but this system is so secured that not in a million years, any of these workflow systems can connect to it.

I work in a very big genomics research insitute in the bioinformatics department and *noone* is using scientific workflow systems, either.

So I am just wondering. Are scientific workflow systems (Kepler, Taverna, VisTtrails, LoniPipeline, Apache Airavata...) useful for anything? Are they just a waste of time or do they boost productivity? Do *you* use them for anything? And if yes, do you use them for ad-hoc type of data analysis, or for byuilding analysis services?

Looking forward to your opinions.
Azazel is offline   Reply With Quote
Old 04-19-2013, 09:54 AM   #2
Bukowski
Senior Member
 
Location: Aberdeen, Scotland

Join Date: Jan 2010
Posts: 373
Default

This is a very interesting question. I have worked with and been part of efforts to create scientific workflow systems. Do I use one in my day to day job? No. If I set out to create a workflow tomorrow would I use one? Almost certainly not.

These are largely over-engineered, poorly supported solutions to a problem which doesn't really exist. They do not deliver 'building block' approaches to workflow building. Many of them will require you some new arbitrary language to sequence workflows, and the brittleness of most bioinformatics pipelines (changing input and output streams from various tools) and fast pace of underlying component change means they are even harder to keep updated and debugged than a more traditional approach.

A lot of these grew up around 'Grid' computing, I don't even hear the term 'Grid computing' anymore, or 'e-Science'. Very little in my bioinformatics life cannot be solved by a few instances on EC2 and a bit of patience. I'd rather take the time to learn a cluster based scheduling system, which generally already have all you need to sequence workflows in place and apply that over a supercomputing infrastructure (I too have access to a number of supercomputers and still can't see a compelling reason to stop doing what I'm doing).

I think essentially these systems get in the way of bioinformatics which has to be largely agile. Adding in additional layers of complexity is not worth it. These systems were designed against the computing paradigms we imagined we might have 10 years ago. Well now we have cheap disks, cheap cores, cheap memory - I don't think that was considered at the time.

My personal opinions, YMMV
Bukowski is offline   Reply With Quote
Old 04-19-2013, 09:56 AM   #3
Richard Finney
Senior Member
 
Location: bethesda

Join Date: Feb 2009
Posts: 699
Default

Cloud is the new Grid.
Richard Finney is offline   Reply With Quote
Old 04-19-2013, 10:05 AM   #4
Azazel
Member
 
Location: Japan

Join Date: Oct 2010
Posts: 52
Default

Thanks for your input Bukowski! I have very similar impressions about these workflow systems.
Azazel is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 09:37 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2019, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO