SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
Galaxy workflow for GATK pipeline [Work in progress] Carlos Borroto Bioinformatics 3 11-30-2011 07:41 AM
repeat sequences/large files in galaxy Giles Bioinformatics 2 06-27-2011 11:08 AM
Avoiding primer dimers lpalacios 454 Pyrosequencing 9 05-05-2011 01:24 AM
RNA-seq Galaxy workflow for PE barcoded samples? jjw14 Bioinformatics 0 04-19-2011 12:58 PM
Galaxy and gz files Giles Bioinformatics 4 03-31-2010 06:52 AM

Reply
 
Thread Tools
Old 11-03-2010, 07:33 AM   #1
tooony13
Junior Member
 
Location: Lyon, France

Join Date: Sep 2010
Posts: 2
Default Galaxy workflow management system customization - avoiding duplication of files

Hi all,

My team is trying to set up an instance of galaxy workflow management system (http://galaxy.psu.edu/) that will launch jobs on our local cluster. We are involved in projects dealing with high-throughput sequencing. We then have to manage a LOT of large files (several Gb each).

When uploading files into galaxy, these files are automatically copied to a folder named "database/files" and they are also sequentially renamed (dataset_1, dataset_2, dataset_3 ...etc..). This name convention is independent of the fact that a file is an input/intermediary/output file.

Copying and renaming files this way is too much time consuming and makes us lose our file structure.

Is there a way to avoid this behavior and that galaxy just remember the file path instead of possessing his own copy ?

If someone here has experience with this tool, any help or useful link would be appreciated.

Cheers,

tony
tooony13 is offline   Reply With Quote
Old 11-03-2010, 10:31 PM   #2
dawe
Senior Member
 
Location: 4530'25.22"N / 915'53.00"E

Join Date: Apr 2009
Posts: 258
Default

We solved this using a filesystem with deduplication capabilities (ZFS). To do this you have to switch to an operating system that support it (IllumOS, Nexenta, Solaris).
dawe is offline   Reply With Quote
Old 11-16-2010, 09:57 AM   #3
jgoecks
Member
 
Location: Washington, D.C.

Join Date: Jan 2010
Posts: 28
Default

Hi Tony,

I recommend you post your question to (and subscribe to) the galaxy-dev mailing list: http://lists.bx.psu.edu/listinfo/galaxy-dev The Galaxy team reads and responds to email on this list, and it serves a forum for discussion about the technical aspects of Galaxy as well.

FYI, what you want to do is achievable by modifying Galaxy's universe file, but I'm not exactly sure how as it's not my area of expertise.

Thanks,
J.

Emory University
Galaxy Team

Quote:
Originally Posted by tooony13 View Post
Hi all,

My team is trying to set up an instance of galaxy workflow management system (http://galaxy.psu.edu/) that will launch jobs on our local cluster. We are involved in projects dealing with high-throughput sequencing. We then have to manage a LOT of large files (several Gb each).

When uploading files into galaxy, these files are automatically copied to a folder named "database/files" and they are also sequentially renamed (dataset_1, dataset_2, dataset_3 ...etc..). This name convention is independent of the fact that a file is an input/intermediary/output file.

Copying and renaming files this way is too much time consuming and makes us lose our file structure.

Is there a way to avoid this behavior and that galaxy just remember the file path instead of possessing his own copy ?

If someone here has experience with this tool, any help or useful link would be appreciated.

Cheers,

tony
jgoecks is offline   Reply With Quote
Reply

Tags
computer science, file, ngs, workflow

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 07:22 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO