SEQanswers

Go Back   SEQanswers > General



Similar Threads
Thread Thread Starter Forum Replies Last Post
help with ftp files from dbSNP cinta Bioinformatics 2 07-11-2014 05:54 AM
How to transfer files off of computer harddrive onto hpc account? noobie Bioinformatics 2 06-19-2012 04:34 PM
Corrupt BAM file ashkot Bioinformatics 2 12-19-2011 12:42 PM
mpileup of BAM files through ftp vplagnol Bioinformatics 6 03-23-2011 07:51 AM
Looking for FTP/HTTP accessible BAM files krobison Bioinformatics 1 07-15-2010 10:24 AM

Reply
 
Thread Tools
Old 08-17-2015, 05:17 AM   #1
bio_informatics
Senior Member
 
Location: USA

Join Date: Nov 2013
Posts: 182
Default FTP transfer - corrupt files at other end

Hello members,

I've 218 genome (E. coli) files - 436 fastq.gz files.

I created their symbolik links in one folder (as they are scattered over multiple sequencing folders and different depths of directory), and sent them using bash script via FTP utility.

I do not get any error, warning or whatsoever during transfer. I can see their size at FTP web interface as in on command line interface on my end.

But the data on the other end is corrupted.
I'm unable to understand what might be causing this, and how should I fix this?

Please guide.

Last edited by bio_informatics; 08-17-2015 at 05:33 AM.
bio_informatics is offline   Reply With Quote
Old 08-17-2015, 05:59 AM   #2
Richard Finney
Senior Member
 
Location: bethesda

Join Date: Feb 2009
Posts: 699
Default

Did you set "binary on" ?

You have to set "binary on" when using FTP.
Richard Finney is offline   Reply With Quote
Old 08-17-2015, 06:07 AM   #3
bio_informatics
Senior Member
 
Location: USA

Join Date: Nov 2013
Posts: 182
Default

Quote:
Originally Posted by Richard Finney View Post
Did you set "binary on" ?

You have to set "binary on" when using FTP.
Hi Richard,
Thanks for your reply.
It by default says:

Uploading in BINARY file XTGEN_20121130_SN180_IL100028322_TTGGTATG-FMC_UTI_01875_L004_R1_trimmed.fastq.gz
bio_informatics is offline   Reply With Quote
Old 08-17-2015, 06:11 AM   #4
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 6,800
Default

What FTP program are you using? Some FTP programs can't follow soft-links correctly.
GenoMax is offline   Reply With Quote
Old 08-17-2015, 06:14 AM   #5
bio_informatics
Senior Member
 
Location: USA

Join Date: Nov 2013
Posts: 182
Default

Quote:
Originally Posted by GenoMax View Post
What FTP program are you using? Some FTP programs can't follow soft-links correctly.
Hello genomax,
FTP linux utility.
bio_informatics is offline   Reply With Quote
Old 08-17-2015, 07:13 AM   #6
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 6,800
Default

Have you tried sftp instead of plain ftp?
GenoMax is offline   Reply With Quote
Old 08-17-2015, 07:16 AM   #7
bio_informatics
Senior Member
 
Location: USA

Join Date: Nov 2013
Posts: 182
Default

Quote:
Originally Posted by GenoMax View Post
Have you tried sftp instead of plain ftp?
Hi,

I cannot use CURL, stfp as the user name I've has already an @ in it, and thus, many command line transfer utilities refuse to accept it.

something like:

[email protected]_name

bio_informatics is offline   Reply With Quote
Old 08-17-2015, 07:22 AM   #8
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 6,800
Default

Can you clarify what you are trying to do?

Is this transfer between servers i.e. you have created a symlinked folder of files on server A that you are trying to get at from server B (via the web?).
GenoMax is offline   Reply With Quote
Old 08-17-2015, 07:26 AM   #9
bio_informatics
Senior Member
 
Location: USA

Join Date: Nov 2013
Posts: 182
Default

Quote:
Originally Posted by GenoMax View Post
Can you clarify what you are trying to do?

Is this transfer between servers i.e. you have created a symlinked folder of files on server A that you are trying to get at from server B (via the web?).
Hi Genomax,
I created symlinks at remote server A.
Wrote bash script at Server A, with FTP which uses - password, username of Server B. This bash script loops over .fastq.gz files (which are symlinks here).

Folder:
-->has 436 symlinks..

From Server A fastq.gz -->FTP Server B

bash script.sh inside the symlinks folder.
bio_informatics is offline   Reply With Quote
Old 08-17-2015, 07:34 AM   #10
GenoMax
Senior Member
 
Location: East Coast USA

Join Date: Feb 2008
Posts: 6,800
Default

Quote:
Originally Posted by bio_informatics View Post
Hi,

I cannot use CURL, stfp as the user name I've has already an @ in it, and thus, many command line transfer utilities refuse to accept it.

something like:

[email protected]_name

Try the options included here to handle the @ for sftp: http://unix.stackexchange.com/questi...ontains-symbol
GenoMax is offline   Reply With Quote
Reply

Tags
ftp

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 05:08 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2018, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO