Data Pipeline Automation, what are you using?

rdeborja · 06-26-2013, 04:27 AM

I'm curious to know what type of pipeline workflow automation tool people are using these days. At the start of my career, I started cobbling together various shell scripts which quickly turned into an administrative nightmare. Since then, our institute has implemented SeqWare to perform FASTQ file generation, data QC, alignment and some post-processing.

It would be nice to know what others are using and some of their experiences.

RickBioinf · 06-26-2013, 04:45 AM

I prefer to make my own pipeline workflow, so I know exactly what the code is and I can change everything. But when it's really necessary I like to use Galaxy.

rhinoceros · 06-26-2013, 04:52 AM

Lots of pipes in shell scripts.

maasha · 06-26-2013, 05:23 AM

Biopieces -> www.biopieces.org

rhinoceros · 06-26-2013, 05:44 AM

Quote:

Originally Posted by maasha

Biopieces -> www.biopieces.org

That's in essence how I construct my pipelines, except that I have individually added all the algorithms to my path. So, for example, instead of 'align_pair_seq' I'd use 'muscle' or 'clustal' or whatever does IMO the best job for the given task..

dpryan · 06-26-2013, 07:50 AM

Bash scripts and the queuing system on our cluster.

asiangg · 06-27-2013, 11:09 AM

Does anyone use GNU makefiles?

sdriscoll · 06-27-2013, 05:44 PM

bash, perl, python and C.

rdeborja · 06-27-2013, 05:51 PM

Quote:

Originally Posted by asiangg

Does anyone use GNU makefiles?

I'm a huge fan of makefiles as a very basic workflow system, especially since it works well with sge's qmake.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
RNA-Seq data analysis pipeline	preet	Introductions	14	02-14-2014 12:07 PM
Pipeline for Solid data	unique379	RNA Sequencing	0	03-04-2013 06:53 AM
Data Processing Pipeline question	mboursnell	Bioinformatics	10	02-28-2013 07:03 PM
pipeline for ion torrent data	aituka	Bioinformatics	11	05-21-2012 05:08 PM
The meaning of B in Illumina 1.5 pipeline data?	maasha	General	1	06-01-2011 06:04 AM

06-26-2013, 04:27 AM	#1
rdeborja Member Location: Toronto Join Date: Aug 2008 Posts: 42	Data Pipeline Automation, what are you using? I'm curious to know what type of pipeline workflow automation tool people are using these days. At the start of my career, I started cobbling together various shell scripts which quickly turned into an administrative nightmare. Since then, our institute has implemented SeqWare to perform FASTQ file generation, data QC, alignment and some post-processing. It would be nice to know what others are using and some of their experiences.

06-27-2013, 05:44 PM	#8
sdriscoll I like code Location: San Diego, CA, USA Join Date: Sep 2009 Posts: 438	bash, perl, python and C. __________________ /* Shawn Driscoll, Gene Expression Laboratory, Pfaff Salk Institute for Biological Studies, La Jolla, CA, USA */

06-26-2013, 04:45 AM	#2
RickBioinf Member Location: Leiden, The Netherlands Join Date: Sep 2012 Posts: 28	I prefer to make my own pipeline workflow, so I know exactly what the code is and I can change everything. But when it's really necessary I like to use Galaxy.

06-26-2013, 04:52 AM	#3
rhinoceros Senior Member Location: sub-surface moon base Join Date: Apr 2013 Posts: 372	Lots of pipes in shell scripts.

06-26-2013, 05:23 AM	#4
maasha Senior Member Location: Denmark Join Date: Apr 2009 Posts: 153	Biopieces -> www.biopieces.org

06-26-2013, 07:50 AM	#6
dpryan Devon Ryan Location: Freiburg, Germany Join Date: Jul 2011 Posts: 3,480	Bash scripts and the queuing system on our cluster.

06-27-2013, 11:09 AM	#7
asiangg Member Location: New York Join Date: Dec 2008 Posts: 44	Does anyone use GNU makefiles?