SEQanswers

Go Back   SEQanswers > Bioinformatics > Bioinformatics



Similar Threads
Thread Thread Starter Forum Replies Last Post
multiplex few variants in hundreds of samples fadista Bioinformatics 2 12-02-2011 12:10 AM
IQ run eoh001 SOLiD 1 07-05-2011 06:07 AM
pileup output different using Maq and Samtools commands mr_boourns Bioinformatics 1 03-31-2011 03:05 AM
25 bp/frag run vs 50 bp/frag run regarding N2S ratio and cost efficiency smallcreek SOLiD 1 12-02-2009 04:47 AM
Error-correcting barcoded primers for pyrosequencing hundreds of samples in multiplex ECO Literature Watch 0 02-12-2008 10:30 PM

Reply
 
Thread Tools
Old 06-14-2011, 03:02 PM   #1
CNVboy
Member
 
Location: boston

Join Date: Jun 2011
Posts: 27
Default Run hundreds of BWA commands without waiting

Hi guys
I'm analyzing some high-coverage trio data. So Need to run BWA for hundreds of fastq.gz files. Obviously I should write some script to finish such task without waiting and typing in hundreds of commands one by one. But as a beginner without coding experience, I don't know how to do.

For example, I just put
bwa aln -t 24 index file1>1.sam
bwa aln -t 24 index file2>2.sam
bwa aln -t 24 index file3>3.sam
...............
..............

into the script, and run it..........and it doesn't work at all.
I know I must miss sth., say, the pathway for fastq files.

anyone can give any pattern about such script of executing multiple jobs? thx
CNVboy is offline   Reply With Quote
Old 06-14-2011, 04:14 PM   #2
fpepin
Member
 
Location: Berkeley

Join Date: Feb 2011
Posts: 30
Default

How about giving us the error message that you're getting? What if you just run one of them in isolation? It's hard to guess what the problem is as we have no idea what your environment and files are.
fpepin is offline   Reply With Quote
Old 06-14-2011, 04:16 PM   #3
Kennels
Senior Member
 
Location: Sydney

Join Date: Feb 2011
Posts: 149
Default

Without having to learn perl or other languauge, the simplest but probably very clunky way of doing it is to use a shell script.

Create a text file (e.g. with gedit would be simplest), and name it xxx.sh (where .sh stands for shell).

In the file at the top, put in:

Code:
#!/bin/bash

echo `bwa aln -t 24 index file1>1.sam`
echo `bwa aln -t 24 index file2>2.sam`
echo `bwa aln -t 24 index file3>3.sam`
...

exit

This will execute each line as step by step commands. Make sure the quotes is the one generated by the tilde ('~') key at the top left of your keyboard. Of course typing all the lines would be a chore, so you might want to use a spreadsheet to fill in the increments.

One last thing. After you've created this file, you need to give it permission to be executable. In the terminal, cd to where this file is created, and type:

Code:
chmod +x xxx.sh
Then, to execute:

Code:
nohup ./xxx.sh > log.txt
You could also just execute ./xxx.sh, but 'nohup' sends all output of this script to log.txt, so if you run into any errors you can trace it back in the log file.

I understand your position so I've tried my best to help out here (not sure what the general rule here about writing code is for other people, but...). Try googling about shell scripts, and I'm sure you'll pick it up quickly and improve the above too.

Last edited by Kennels; 06-14-2011 at 04:19 PM.
Kennels is offline   Reply With Quote
Old 06-14-2011, 04:40 PM   #4
CNVboy
Member
 
Location: boston

Join Date: Jun 2011
Posts: 27
Default

Many many thanks Kennels!!!

Yeah, I just made the spreadsheet, actually with 800 fastq files. And the shell script works quite well!
btw, can you give any clue about writing Perl to finish such task? I think I can read and understand some Perl script, but just cannot write by myself.

thx a lot!
CNVboy is offline   Reply With Quote
Old 06-14-2011, 05:13 PM   #5
Kennels
Senior Member
 
Location: Sydney

Join Date: Feb 2011
Posts: 149
Default

In perl, you can probably do it with 'for' or 'while' conditional loops using counters, and use 'filehandles' for input/output files, but I'm still a novice at best.
As there is no simple answer to your request other than to sit down and learn it from scratch, I would recommend buying 'Beginning Perl for Bioinformatics' by James Tisdall (O'Reilly books).

Even reading the first few chapters is enough to get you through many common tasks. It sure did for me, and I'm not even finished with the book.
Good luck!
Kennels is offline   Reply With Quote
Old 06-14-2011, 05:40 PM   #6
fpepin
Member
 
Location: Berkeley

Join Date: Feb 2011
Posts: 30
Default

I hadn't understood the question was about the script, not bwa.

I haven't read that book but I've heard good things about it.

You can do it in the shell fairly easily too:
Code:
for i in file*
do
bwa aln -t 24 index $i>$i.sam
done
Another way that I like is:
Code:
ls file* | xargs -n1 -P24 -I{} bwa aln index {}>{}.sam
xargs is nice in that it can run many jobs in parallel. For example, the example above would run 24 jobs with 1 thread each as opposed to running each sequentially with 24 threads. It makes little difference here, but it comes in real handy when working with tools that are only single-threaded.

Both of the examples above have a longer name for the sam file. You if you prefer your original naming scheme, you can use something like:
Code:
for i in file*.sam
do
echo mv $i `echo $i | sed -e 's/file//'`
done
That being said, learning a bit of perl will come in handy.
fpepin is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off




All times are GMT -8. The time now is 09:14 AM.


Powered by vBulletin® Version 3.8.9
Copyright ©2000 - 2020, vBulletin Solutions, Inc.
Single Sign On provided by vBSSO