Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Galaxy workflow management system customization - avoiding duplication of files

    Hi all,

    My team is trying to set up an instance of galaxy workflow management system (http://galaxy.psu.edu/) that will launch jobs on our local cluster. We are involved in projects dealing with high-throughput sequencing. We then have to manage a LOT of large files (several Gb each).

    When uploading files into galaxy, these files are automatically copied to a folder named "database/files" and they are also sequentially renamed (dataset_1, dataset_2, dataset_3 ...etc..). This name convention is independent of the fact that a file is an input/intermediary/output file.

    Copying and renaming files this way is too much time consuming and makes us lose our file structure.

    Is there a way to avoid this behavior and that galaxy just remember the file path instead of possessing his own copy ?

    If someone here has experience with this tool, any help or useful link would be appreciated.

    Cheers,

    tony

  • #2
    We solved this using a filesystem with deduplication capabilities (ZFS). To do this you have to switch to an operating system that support it (IllumOS, Nexenta, Solaris).

    Comment


    • #3
      Hi Tony,

      I recommend you post your question to (and subscribe to) the galaxy-dev mailing list: http://lists.bx.psu.edu/listinfo/galaxy-dev The Galaxy team reads and responds to email on this list, and it serves a forum for discussion about the technical aspects of Galaxy as well.

      FYI, what you want to do is achievable by modifying Galaxy's universe file, but I'm not exactly sure how as it's not my area of expertise.

      Thanks,
      J.

      Emory University
      Galaxy Team

      Originally posted by tooony13 View Post
      Hi all,

      My team is trying to set up an instance of galaxy workflow management system (http://galaxy.psu.edu/) that will launch jobs on our local cluster. We are involved in projects dealing with high-throughput sequencing. We then have to manage a LOT of large files (several Gb each).

      When uploading files into galaxy, these files are automatically copied to a folder named "database/files" and they are also sequentially renamed (dataset_1, dataset_2, dataset_3 ...etc..). This name convention is independent of the fact that a file is an input/intermediary/output file.

      Copying and renaming files this way is too much time consuming and makes us lose our file structure.

      Is there a way to avoid this behavior and that galaxy just remember the file path instead of possessing his own copy ?

      If someone here has experience with this tool, any help or useful link would be appreciated.

      Cheers,

      tony

      Comment

      Latest Articles

      Collapse

      • seqadmin
        Strategies for Sequencing Challenging Samples
        by seqadmin


        Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
        03-22-2024, 06:39 AM
      • seqadmin
        Techniques and Challenges in Conservation Genomics
        by seqadmin



        The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

        Avian Conservation
        Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
        03-08-2024, 10:41 AM

      ad_right_rmr

      Collapse

      News

      Collapse

      Topics Statistics Last Post
      Started by seqadmin, Yesterday, 06:37 PM
      0 responses
      11 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, Yesterday, 06:07 PM
      0 responses
      10 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-22-2024, 10:03 AM
      0 responses
      51 views
      0 likes
      Last Post seqadmin  
      Started by seqadmin, 03-21-2024, 07:32 AM
      0 responses
      67 views
      0 likes
      Last Post seqadmin  
      Working...
      X