FTP file upload to Galaxy via SFTP?

[jjw14]

Hi,

Short question: Can I use FTP to "put" a file onto Galaxy rather than using the Upload File interface? See below for my specific problem.

I've found the Galaxy web interface to be a great resource for analyzing Illumina sequence files. However, upload of these files (>4 GB) from my desktop to Galaxy's servers is painfully slow. Using FTP from our main campus bioinformatics computers would be much faster and take the load off of my desktop and the Galaxy webpage states, "TIP: For NGS and other large files (>~2GB), uploading via HTTP/FTP URL (below) is the most reliable method".

However, our campus computing policies states, "FTP connections from remote systems are not allowed. Remote file transfer connections must be initiated using a secure FTP (SFTP) client program that authenticates to the server using SSH version 2".

Rather than having Galaxy GET the file via FTP, can I use the PUT command to upload the file onto Galaxy's servers?

Thanks in advance,
jjw

[jdanderson]

Hello jjw14,

Did you ever find a solution to your problem?

Regards,
Johnathon

[jjw14]

Jonathan,

Yes and no. I found a solution, but it's not free. I purchased an account from a web hosting service that allows SFTP transfers for $10 a month. Peanuts to some, I know. In the end, it is worth the cost for reduced aggravation.

So my file transfer path is:

University Server ---(SFTP)--> Web Host ---(HTTP)--> Galaxy / UCSC Server

It's a two step process, but it keeps the files off of my personal hard drive. You can set up an HTTP server from your desktop (Mac/Linux/PC), but for me, the file transfers were way too slow.

Technically, the hosting service isn't for file storage, just web page hosting. If you search, there are plenty available. Hope this helps a bit.

jjw

[jdanderson]

Hello jjw,

Thank you for the reply. I was actually just getting ready to set up an HTTP server from my desktop due to the fact that for this set of samples I have to manipulate the files on my desktop prior to uploading them to Galaxy (barcode separation and splitting). If you don't mind me asking, which program did you use that was giving you the slow results; I was thinking of trying to use Apache...

Thanks again for the post. Its nice to have options.

Regards,
Johnathon

[jjw14]

Hi Jonathan,

I hope the option of serving the files from your desktop works. My desktop computer is a Mac, so I was using the Apache web server in OS X. The slow transfer wasn't a software problem, though. The data transfer rate in our department was simply too low, even for compressed files. I would have to wait hours for the upload to Galaxy, and several times get timeout errors with files >1.5GB.

You might want to test your transfer with a few small files first to verify that the upload goes smoothly.

Good luck!
jjw

[jdanderson]

Hello jjw14,

Wow, I wish I would have paid heed to your advice and tried to use smaller files to test the connection. I uploaded a Solexa 40SR file and it took almost 24 hours. In the meantime I did the analysis with programs in the command prompt, the old fashioned way. It seems ashame that there is no work-around for this issue... Perhaps in the future.

Thanks again for your advice. I think in the future I may have to alter my file transfer setup for Galaxy as you mentioned or maybe stick to the command prompt.

Cheers,
Johnathon

[jgoecks]

Galaxy now has integrated FTP server

Hi all,

With Galaxy's most recent update, you can now FTP your files directly to Galaxy rather than uploading via a web browser. While this isn't perfect, it's definitely better than most other options.

Best,
J.

Emory University
Galaxy Team

[jjw14]

Very much appreciated!
JJW

[chaomeizhang]

I tried to use the FTP Filezilla to upload some NGS data to the galaxy. I registered as [email protected]. I input as follows:
Host: main.g2.bx.psu.edu
Userame: [email protected]
Password: …………

Then I click the quickconnect,

Status: Waiting to retry...
Status: Connecting to main.g2.bx.psu.edu...
Response: fzSftp started
Command: open "[email protected]@main.g2.bx.psu.edu" 22
Error: Connection timed out
Error: Could not connect to server

Does anybody have any idea what is wrong with my input or I just do not have an account in the main serve? Thank you for your time.

[ASL]

Easy way to sftp file for Galaxy upload?

Does anyone know of an easy way to upload fastq files via sftp to the main Galaxy at Penn State or to a local instance of Galaxy?

We are in the process of setting up a local instance of Galaxy here at my university and I have access to an sftp server. Any advice would be appreciated.

Thanks!

[jgoecks]

@ASL:

*I do not think sFTP to usegalaxy.org is implemented yet, but it is coming; for now you should use FTP
*instructions for configuring FTP on local instance are here: http://wiki.galaxyproject.org/Admin/...ad%20via%20FTP

Please post additional questions to the galaxy-dev mailing list. You'll reach the wider Galaxy community and likely get more detailed responses. Also, if you search the list, there may be discussion of including SSL support in local FTP.

http://dev.list.galaxyproject.org/

Best,
J.