Seqanswers Leaderboard Ad

Collapse

Announcement

Collapse
No announcement yet.
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • FTP file upload to Galaxy via SFTP?

    Hi,

    Short question: Can I use FTP to "put" a file onto Galaxy rather than using the Upload File interface? See below for my specific problem.

    I've found the Galaxy web interface to be a great resource for analyzing Illumina sequence files. However, upload of these files (>4 GB) from my desktop to Galaxy's servers is painfully slow. Using FTP from our main campus bioinformatics computers would be much faster and take the load off of my desktop and the Galaxy webpage states, "TIP: For NGS and other large files (>~2GB), uploading via HTTP/FTP URL (below) is the most reliable method".

    However, our campus computing policies states, "FTP connections from remote systems are not allowed. Remote file transfer connections must be initiated using a secure FTP (SFTP) client program that authenticates to the server using SSH version 2".

    Rather than having Galaxy GET the file via FTP, can I use the PUT command to upload the file onto Galaxy's servers?

    Thanks in advance,
    jjw

  • #2
    Hello jjw14,

    Did you ever find a solution to your problem?

    Regards,
    Johnathon

    Comment


    • #3
      Jonathan,

      Yes and no. I found a solution, but it's not free. I purchased an account from a web hosting service that allows SFTP transfers for $10 a month. Peanuts to some, I know. In the end, it is worth the cost for reduced aggravation.

      So my file transfer path is:

      University Server ---(SFTP)--> Web Host ---(HTTP)--> Galaxy / UCSC Server

      It's a two step process, but it keeps the files off of my personal hard drive. You can set up an HTTP server from your desktop (Mac/Linux/PC), but for me, the file transfers were way too slow.

      Technically, the hosting service isn't for file storage, just web page hosting. If you search, there are plenty available. Hope this helps a bit.

      jjw

      Comment


      • #4
        Hello jjw,

        Thank you for the reply. I was actually just getting ready to set up an HTTP server from my desktop due to the fact that for this set of samples I have to manipulate the files on my desktop prior to uploading them to Galaxy (barcode separation and splitting). If you don't mind me asking, which program did you use that was giving you the slow results; I was thinking of trying to use Apache...

        Thanks again for the post. Its nice to have options.

        Regards,
        Johnathon

        Comment


        • #5
          Hi Jonathan,

          I hope the option of serving the files from your desktop works. My desktop computer is a Mac, so I was using the Apache web server in OS X. The slow transfer wasn't a software problem, though. The data transfer rate in our department was simply too low, even for compressed files. I would have to wait hours for the upload to Galaxy, and several times get timeout errors with files >1.5GB.

          You might want to test your transfer with a few small files first to verify that the upload goes smoothly.

          Good luck!
          jjw

          Comment


          • #6
            Hello jjw14,

            Wow, I wish I would have paid heed to your advice and tried to use smaller files to test the connection. I uploaded a Solexa 40SR file and it took almost 24 hours. In the meantime I did the analysis with programs in the command prompt, the old fashioned way. It seems ashame that there is no work-around for this issue... Perhaps in the future.

            Thanks again for your advice. I think in the future I may have to alter my file transfer setup for Galaxy as you mentioned or maybe stick to the command prompt.

            Cheers,
            Johnathon

            Comment


            • #7
              Galaxy now has integrated FTP server

              Hi all,

              With Galaxy's most recent update, you can now FTP your files directly to Galaxy rather than uploading via a web browser. While this isn't perfect, it's definitely better than most other options.

              Best,
              J.

              Emory University
              Galaxy Team

              Comment


              • #8
                Very much appreciated!
                JJW

                Comment


                • #9
                  I tried to use the FTP Filezilla to upload some NGS data to the galaxy. I registered as [email protected]. I input as follows:
                  Host: main.g2.bx.psu.edu
                  Userame: [email protected]
                  Password: …………

                  Then I click the quickconnect,

                  Status: Waiting to retry...
                  Status: Connecting to main.g2.bx.psu.edu...
                  Response: fzSftp started
                  Command: open "[email protected]@main.g2.bx.psu.edu" 22
                  Error: Connection timed out
                  Error: Could not connect to server

                  Does anybody have any idea what is wrong with my input or I just do not have an account in the main serve? Thank you for your time.

                  Comment


                  • #10
                    Easy way to sftp file for Galaxy upload?

                    Does anyone know of an easy way to upload fastq files via sftp to the main Galaxy at Penn State or to a local instance of Galaxy?

                    We are in the process of setting up a local instance of Galaxy here at my university and I have access to an sftp server. Any advice would be appreciated.

                    Thanks!

                    Comment


                    • #11
                      @ASL:

                      *I do not think sFTP to usegalaxy.org is implemented yet, but it is coming; for now you should use FTP
                      *instructions for configuring FTP on local instance are here: http://wiki.galaxyproject.org/Admin/...ad%20via%20FTP

                      Please post additional questions to the galaxy-dev mailing list. You'll reach the wider Galaxy community and likely get more detailed responses. Also, if you search the list, there may be discussion of including SSL support in local FTP.



                      Best,
                      J.

                      Comment

                      Latest Articles

                      Collapse

                      • seqadmin
                        Strategies for Sequencing Challenging Samples
                        by seqadmin


                        Despite advancements in sequencing platforms and related sample preparation technologies, certain sample types continue to present significant challenges that can compromise sequencing results. Pedro Echave, Senior Manager of the Global Business Segment at Revvity, explained that the success of a sequencing experiment ultimately depends on the amount and integrity of the nucleic acid template (RNA or DNA) obtained from a sample. “The better the quality of the nucleic acid isolated...
                        03-22-2024, 06:39 AM
                      • seqadmin
                        Techniques and Challenges in Conservation Genomics
                        by seqadmin



                        The field of conservation genomics centers on applying genomics technologies in support of conservation efforts and the preservation of biodiversity. This article features interviews with two researchers who showcase their innovative work and highlight the current state and future of conservation genomics.

                        Avian Conservation
                        Matthew DeSaix, a recent doctoral graduate from Kristen Ruegg’s lab at The University of Colorado, shared that most of his research...
                        03-08-2024, 10:41 AM

                      ad_right_rmr

                      Collapse

                      News

                      Collapse

                      Topics Statistics Last Post
                      Started by seqadmin, Yesterday, 06:37 PM
                      0 responses
                      8 views
                      0 likes
                      Last Post seqadmin  
                      Started by seqadmin, Yesterday, 06:07 PM
                      0 responses
                      8 views
                      0 likes
                      Last Post seqadmin  
                      Started by seqadmin, 03-22-2024, 10:03 AM
                      0 responses
                      49 views
                      0 likes
                      Last Post seqadmin  
                      Started by seqadmin, 03-21-2024, 07:32 AM
                      0 responses
                      67 views
                      0 likes
                      Last Post seqadmin  
                      Working...
                      X