SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
upload > 30 GB file on Galaxy ericguo Bioinformatics 6 03-20-2012 02:55 PM
Upload .bam file to Galaxy, display on UCSC biznatch Bioinformatics 3 12-20-2011 03:12 PM
upload error in local instance of galaxy Ami Bioinformatics 2 10-19-2011 08:53 AM
FTP upload in galaxy neetu Bioinformatics 1 10-18-2011 05:17 AM
Upload genome index to Galaxy for Bowtie alignment? jjw14 Bioinformatics 0 06-08-2010 08:22 AM

Reply
 
Thread Tools
Old 06-01-2010, 05:23 PM   #1
jjw14
Member
 
Location: Missouri

Join Date: Apr 2010
Posts: 39
Question FTP file upload to Galaxy via SFTP?

Hi,

Short question: Can I use FTP to "put" a file onto Galaxy rather than using the Upload File interface? See below for my specific problem.

I've found the Galaxy web interface to be a great resource for analyzing Illumina sequence files. However, upload of these files (>4 GB) from my desktop to Galaxy's servers is painfully slow. Using FTP from our main campus bioinformatics computers would be much faster and take the load off of my desktop and the Galaxy webpage states, "TIP: For NGS and other large files (>~2GB), uploading via HTTP/FTP URL (below) is the most reliable method".

However, our campus computing policies states, "FTP connections from remote systems are not allowed. Remote file transfer connections must be initiated using a secure FTP (SFTP) client program that authenticates to the server using SSH version 2".

Rather than having Galaxy GET the file via FTP, can I use the PUT command to upload the file onto Galaxy's servers?

Thanks in advance,
jjw
jjw14 is offline   Reply With Quote
Old 10-04-2010, 08:48 PM   #2
jdanderson
Member
 
Location: Davis, CA

Join Date: Sep 2010
Posts: 45
Default

Hello jjw14,

Did you ever find a solution to your problem?

Regards,
Johnathon
jdanderson is offline   Reply With Quote
Old 10-05-2010, 10:20 AM   #3
jjw14
Member
 
Location: Missouri

Join Date: Apr 2010
Posts: 39
Default

Jonathan,

Yes and no. I found a solution, but it's not free. I purchased an account from a web hosting service that allows SFTP transfers for $10 a month. Peanuts to some, I know. In the end, it is worth the cost for reduced aggravation.

So my file transfer path is:

University Server ---(SFTP)--> Web Host ---(HTTP)--> Galaxy / UCSC Server

It's a two step process, but it keeps the files off of my personal hard drive. You can set up an HTTP server from your desktop (Mac/Linux/PC), but for me, the file transfers were way too slow.

Technically, the hosting service isn't for file storage, just web page hosting. If you search, there are plenty available. Hope this helps a bit.

jjw
jjw14 is offline   Reply With Quote
Old 10-05-2010, 12:56 PM   #4
jdanderson
Member
 
Location: Davis, CA

Join Date: Sep 2010
Posts: 45
Default

Hello jjw,

Thank you for the reply. I was actually just getting ready to set up an HTTP server from my desktop due to the fact that for this set of samples I have to manipulate the files on my desktop prior to uploading them to Galaxy (barcode separation and splitting). If you don't mind me asking, which program did you use that was giving you the slow results; I was thinking of trying to use Apache...

Thanks again for the post. Its nice to have options.

Regards,
Johnathon
jdanderson is offline   Reply With Quote
Old 10-05-2010, 01:43 PM   #5
jjw14
Member
 
Location: Missouri

Join Date: Apr 2010
Posts: 39
Default

Hi Jonathan,

I hope the option of serving the files from your desktop works. My desktop computer is a Mac, so I was using the Apache web server in OS X. The slow transfer wasn't a software problem, though. The data transfer rate in our department was simply too low, even for compressed files. I would have to wait hours for the upload to Galaxy, and several times get timeout errors with files >1.5GB.

You might want to test your transfer with a few small files first to verify that the upload goes smoothly.

Good luck!
jjw
jjw14 is offline   Reply With Quote
Old 10-06-2010, 07:24 PM   #6
jdanderson
Member
 
Location: Davis, CA

Join Date: Sep 2010
Posts: 45
Default

Hello jjw14,

Wow, I wish I would have paid heed to your advice and tried to use smaller files to test the connection. I uploaded a Solexa 40SR file and it took almost 24 hours. In the meantime I did the analysis with programs in the command prompt, the old fashioned way. It seems ashame that there is no work-around for this issue... Perhaps in the future.

Thanks again for your advice. I think in the future I may have to alter my file transfer setup for Galaxy as you mentioned or maybe stick to the command prompt.

Cheers,
Johnathon
jdanderson is offline   Reply With Quote
Old 11-16-2010, 12:52 PM   #7
jgoecks
Member
 
Location: Washington, D.C.

Join Date: Jan 2010
Posts: 28
Default Galaxy now has integrated FTP server

Hi all,

With Galaxy's most recent update, you can now FTP your files directly to Galaxy rather than uploading via a web browser. While this isn't perfect, it's definitely better than most other options.

Best,
J.

Emory University
Galaxy Team
jgoecks is offline   Reply With Quote
Old 11-16-2010, 02:48 PM   #8
jjw14
Member
 
Location: Missouri

Join Date: Apr 2010
Posts: 39
Default

Very much appreciated!
JJW
jjw14 is offline   Reply With Quote
Old 08-27-2012, 08:59 AM   #9
chaomeizhang
Junior Member
 
Location: Tampa, Florida

Join Date: Aug 2011
Posts: 5
Question

I tried to use the FTP Filezilla to upload some NGS data to the galaxy. I registered as [email protected]. I input as follows:
Host: main.g2.bx.psu.edu
Userame: [email protected]
Password:

Then I click the quickconnect,

Status: Waiting to retry...
Status: Connecting to main.g2.bx.psu.edu...
Response: fzSftp started
Command: open "[email protected]@main.g2.bx.psu.edu" 22
Error: Connection timed out
Error: Could not connect to server

Does anybody have any idea what is wrong with my input or I just do not have an account in the main serve? Thank you for your time.
chaomeizhang is offline   Reply With Quote
Old 10-26-2013, 02:02 PM   #10
ASL
Junior Member
 
Location: Virginia, USA

Join Date: Apr 2013
Posts: 5
Unhappy Easy way to sftp file for Galaxy upload?

Does anyone know of an easy way to upload fastq files via sftp to the main Galaxy at Penn State or to a local instance of Galaxy?

We are in the process of setting up a local instance of Galaxy here at my university and I have access to an sftp server. Any advice would be appreciated.

Thanks!
ASL is offline   Reply With Quote
Old 10-27-2013, 06:26 AM   #11
jgoecks
Member
 
Location: Washington, D.C.

Join Date: Jan 2010
Posts: 28
Default

@ASL:

*I do not think sFTP to usegalaxy.org is implemented yet, but it is coming; for now you should use FTP
*instructions for configuring FTP on local instance are here: http://wiki.galaxyproject.org/Admin/...ad%20via%20FTP

Please post additional questions to the galaxy-dev mailing list. You'll reach the wider Galaxy community and likely get more detailed responses. Also, if you search the list, there may be discussion of including SSL support in local FTP.

http://dev.list.galaxyproject.org/

Best,
J.
jgoecks is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 08:39 AM.


Powered by vBulletin® Version 3.8.6
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.