Got it with your suggestions...thanks Chandra!
Seqanswers Leaderboard Ad
Collapse
Announcement
Collapse
No announcement yet.
X
-
Preprocessed_Reads.com
Hi Chandra,
May I run a couple more questions by you?
1) during ./QualityFilter.com run, javac command not found. I guess I have the user rather developer version of java but the fq1 files were complete and the error did not stop the program. Before installing it thought i would ask if I need javac for the quality filter?
Quality filter processing
Read 1: /media/f64f8101-5d47-4e9b-b138-977bf462383d/RNAseq_november/111102_TENNISON_0143_BC028AACXX_L2_CGATGT_1_pf.fastq
Running the Quality filtering with default paramters
2
Conversion from Fastq to FQ one
javac: Command not found.
175655756 reads in this dataset. Amount of time: 1368.345
Here is the Quality reads /media/f64f8101-5d47-4e9b-b138-977bf462383d/RNAseq_november/111102_TENNISON_0143_BC028AACXX_L2_CGATGT_1_pf.fastq_111102_TENNISON_0143_BC028AACXX_L2_CGATGT_2_pf.fastq_Qualityreads.fq1
javac: Command not found.
Read length: 101bp
175655756 reads in this dataset.
175655756 reads have passed the quality filter.
Amount of time: 1870.882
2) Do you recognize this error?
shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ sudo ./Preprocessed_Reads.com 111102_TENNISON_0143_BC028AACXX_L2_CGATGT_1_pf.fastq_111102_TENNISON_0143_BC028AACXX_L2_CGATGT_2_pf.fastq_Qualityreads.fq1
[sudo] password for shared:
Newline in variable name.
shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$
I checked to make sure jobs.config looked good...and it seems to
# Name of the Job (Sample: testjob)
NAMEJOB=testjobs
# S3 Bucket name for Reference Genomes in the Cloud (Sample: ami-ref)
REFBUCKET=ami-refs
# Local directory to download the reference genomes (Sample: test)
LOCALREFDIR=test_refs
# S3 Bucket name for Illumina sequencing reads in the Cloud (Sample: ami-testreads)
READSBUCKET=ami-testreads
My S3 account has ami-pathseqimage1000, ami-refs and ami-refs-tmp
Thank you for helping us!
Comment
-
Chanda,
I figured out question one from above. Reran w/ javac and no apparent difference.
I still cannot figure out the the "Newline in variable name" error.
Seems like no matter what I w/ Preprocessed_Reads.com I get that message.
I checked the Preprocessed_Reads.com and see no reference to that.
Would really appreciate that any suggestions you may have
I am trying to read up on it http://tinyurl.com/7lyaqpm
shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ sudo ./Preprocessed_Reads.com /media/f64f8101-5d47-4e9b-b138-977bf462383d/RNAseq_november/111102_TENNISON_0143_BC028AACXX_L2_CGATGT_1_pf.fastq_111102_TENNISON_0143_BC028AACXX_L2_CGATGT_2_pf.fastq_Qualityreads.fq1
Newline in variable name.
shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ sudo ./Preprocessed_Reads.com
Newline in variable name.
shared@redwoodpath:~/PathSeq/Pathseq_cloud_V5.2$ ./Preprocessed_Reads.com
Newline in variable name.
Comment
-
wondering about changing n_para=$# at top of script to n_para=$status ...as it seems several references mention changing variable=$? to variable=$status works to resolve the problem is some linux environoments. I am using ubuntu 11.04 ...hesitant to change the script without running it by you
http://shell.deru.com/~gdt/unix2/lectures/13.shtml (bottom)
Comment
-
Accessing log file to monitor Pathseq_Launch.com progress
Hi all,
In this forum there are numerous posts that show the log file for the Pathseq_Launch.com run on the cluster.
Here is how you access the log file:
go into ~/PathseqDirectory/bin
./hadoop-ec2 login test-cluster (test-cluster is what you named the cluster in jobs.config)
Then you will see a log file in root directory.
You can view the log,txt. w/ vim log.txt
Comment
-
Hello fellow Pathseq users,
First, thanks to everyone who posted so far, this thread has been a lifesaver!
Does anyone know if Pathseq will analyze paired-end data in pairs? In other words, when the pipeline has a pile of unmapped non-human reads, does it align each read by itself, or does it do paired-end alignment with both ends? I'm thinking it does not do paired-end alignment, because as far as I can tell BLAST doesn't have that capability. However I'm new to this and a lot of what this pipeline is doing behind the scenes is a mystery to me.
Thanks!
Comment
-
Hi Chandra,
I've had success with Pathseq so far, but I just hit a snag on a sample. I let it run over 40 hours, however it never progressed beyond Job 2. I'm wondering if my log file can provide any insight on what happened.
Thank you!
Code:rmr: cannot remove config: No such file or directory. rmr: cannot remove s3config: No such file or directory. rmr: cannot remove load: No such file or directory. Master data_loader 12/03/10 20:56:46 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. packageJobJar: [/root/mapper_data_compsub.py, /mnt/hadoop/hadoop-unjar449601472672210086/] [] /tmp/streamjob6443148539614277804.jar tmpDir=null 12/03/10 20:56:47 INFO mapred.FileInputFormat: Total input paths to process : 20 12/03/10 20:56:47 INFO streaming.StreamJob: getLocalDirs(): [/mnt/hadoop/mapred/local] 12/03/10 20:56:47 INFO streaming.StreamJob: Running job: job_201203102044_0001 12/03/10 20:56:47 INFO streaming.StreamJob: To kill this job, run: 12/03/10 20:56:47 INFO streaming.StreamJob: /usr/local/hadoop-0.19.0/bin/../bin/hadoop job -Dmapred.job.tracker=hdfs://ip-10-34-46-200.ec2.internal:50002 -kill job_201203102044_0001 12/03/10 20:56:48 INFO streaming.StreamJob: Tracking URL: [url]http://ip-10-34-46-200.ec2.internal:50030/jobdetails.jsp?jobid=job_201203102044_0001[/url] 12/03/10 20:56:49 INFO streaming.StreamJob: map 0% reduce 0% 12/03/10 20:57:02 INFO streaming.StreamJob: map 10% reduce 0% 12/03/10 20:57:03 INFO streaming.StreamJob: map 30% reduce 0% 12/03/10 20:57:04 INFO streaming.StreamJob: map 45% reduce 0% 12/03/10 20:57:06 INFO streaming.StreamJob: map 55% reduce 0% 12/03/10 20:57:07 INFO streaming.StreamJob: map 100% reduce 0% 12/03/10 22:05:24 INFO streaming.StreamJob: Job complete: job_201203102044_0001 12/03/10 22:05:24 INFO streaming.StreamJob: Output: load real 68m38.462s user 0m3.772s sys 0m0.738s Master loader completed ERROR: Bucket 'ami-kippsample03job-stat' does not exist Bucket 's3://ami-kippsample03job-stat/' removed Bucket 's3://ami-kippsample03job-stat/' created ERROR: Bucket 'ami-kippsample03job-output' does not exist Bucket 's3://ami-kippsample03job-output/' removed Bucket 's3://ami-kippsample03job-output/' created File s3://ami-kippsample03reads/input1.local saved as '/usr/local/hadoop-0.19.0/input1.local' (97 bytes in 0.1 seconds, 1678.37 B/s) File s3://ami-kippsample03reads/input10.local saved as '/usr/local/hadoop-0.19.0/input10.local' (98 bytes in 0.1 seconds, 1848.92 B/s) File s3://ami-kippsample03reads/input11.local saved as '/usr/local/hadoop-0.19.0/input11.local' (98 bytes in 0.0 seconds, 2.34 kB/s) File s3://ami-kippsample03reads/input12.local saved as '/usr/local/hadoop-0.19.0/input12.local' (98 bytes in 0.1 seconds, 1576.68 B/s) File s3://ami-kippsample03reads/input13.local saved as '/usr/local/hadoop-0.19.0/input13.local' (98 bytes in 0.1 seconds, 741.02 B/s) File s3://ami-kippsample03reads/input14.local saved as '/usr/local/hadoop-0.19.0/input14.local' (98 bytes in 0.1 seconds, 1012.16 B/s) File s3://ami-kippsample03reads/input15.local saved as '/usr/local/hadoop-0.19.0/input15.local' (98 bytes in 0.1 seconds, 1903.09 B/s) File s3://ami-kippsample03reads/input16.local saved as '/usr/local/hadoop-0.19.0/input16.local' (98 bytes in 0.0 seconds, 1989.20 B/s) File s3://ami-kippsample03reads/input17.local saved as '/usr/local/hadoop-0.19.0/input17.local' (98 bytes in 0.0 seconds, 1997.18 B/s) File s3://ami-kippsample03reads/input18.local saved as '/usr/local/hadoop-0.19.0/input18.local' (98 bytes in 0.0 seconds, 2.15 kB/s) File s3://ami-kippsample03reads/input19.local saved as '/usr/local/hadoop-0.19.0/input19.local' (98 bytes in 0.0 seconds, 2.40 kB/s) File s3://ami-kippsample03reads/input2.local saved as '/usr/local/hadoop-0.19.0/input2.local' (97 bytes in 0.1 seconds, 1263.51 B/s) File s3://ami-kippsample03reads/input20.local saved as '/usr/local/hadoop-0.19.0/input20.local' (98 bytes in 0.1 seconds, 1340.78 B/s) File s3://ami-kippsample03reads/input21.local saved as '/usr/local/hadoop-0.19.0/input21.local' (98 bytes in 0.1 seconds, 1857.47 B/s) File s3://ami-kippsample03reads/input22.local saved as '/usr/local/hadoop-0.19.0/input22.local' (98 bytes in 0.1 seconds, 1100.16 B/s) File s3://ami-kippsample03reads/input23.local saved as '/usr/local/hadoop-0.19.0/input23.local' (98 bytes in 0.1 seconds, 1780.13 B/s) File s3://ami-kippsample03reads/input24.local saved as '/usr/local/hadoop-0.19.0/input24.local' (98 bytes in 0.1 seconds, 1927.05 B/s) File s3://ami-kippsample03reads/input25.local saved as '/usr/local/hadoop-0.19.0/input25.local' (98 bytes in 0.1 seconds, 1430.26 B/s) File s3://ami-kippsample03reads/input26.local saved as '/usr/local/hadoop-0.19.0/input26.local' (98 bytes in 0.1 seconds, 1714.27 B/s) File s3://ami-kippsample03reads/input27.local saved as '/usr/local/hadoop-0.19.0/input27.local' (98 bytes in 0.0 seconds, 2.18 kB/s) File s3://ami-kippsample03reads/input28.local saved as '/usr/local/hadoop-0.19.0/input28.local' (98 bytes in 0.0 seconds, 2.59 kB/s) File s3://ami-kippsample03reads/input29.local saved as '/usr/local/hadoop-0.19.0/input29.local' (98 bytes in 0.0 seconds, 2.21 kB/s) File s3://ami-kippsample03reads/input3.local saved as '/usr/local/hadoop-0.19.0/input3.local' (97 bytes in 0.0 seconds, 2.20 kB/s) File s3://ami-kippsample03reads/input30.local saved as '/usr/local/hadoop-0.19.0/input30.local' (98 bytes in 0.1 seconds, 1836.96 B/s) File s3://ami-kippsample03reads/input31.local saved as '/usr/local/hadoop-0.19.0/input31.local' (98 bytes in 0.1 seconds, 1788.36 B/s) File s3://ami-kippsample03reads/input32.local saved as '/usr/local/hadoop-0.19.0/input32.local' (98 bytes in 0.0 seconds, 2.07 kB/s) File s3://ami-kippsample03reads/input33.local saved as '/usr/local/hadoop-0.19.0/input33.local' (98 bytes in 0.1 seconds, 793.01 B/s) File s3://ami-kippsample03reads/input34.local saved as '/usr/local/hadoop-0.19.0/input34.local' (98 bytes in 0.0 seconds, 2.05 kB/s) File s3://ami-kippsample03reads/input35.local saved as '/usr/local/hadoop-0.19.0/input35.local' (98 bytes in 0.0 seconds, 2.04 kB/s) File s3://ami-kippsample03reads/input36.local saved as '/usr/local/hadoop-0.19.0/input36.local' (98 bytes in 0.0 seconds, 2004.88 B/s) File s3://ami-kippsample03reads/input37.local saved as '/usr/local/hadoop-0.19.0/input37.local' (98 bytes in 0.1 seconds, 1686.98 B/s) File s3://ami-kippsample03reads/input38.local saved as '/usr/local/hadoop-0.19.0/input38.local' (98 bytes in 0.1 seconds, 1425.54 B/s) File s3://ami-kippsample03reads/input39.local saved as '/usr/local/hadoop-0.19.0/input39.local' (98 bytes in 0.0 seconds, 3.35 kB/s) File s3://ami-kippsample03reads/input4.local saved as '/usr/local/hadoop-0.19.0/input4.local' (97 bytes in 0.0 seconds, 2.06 kB/s) File s3://ami-kippsample03reads/input40.local saved as '/usr/local/hadoop-0.19.0/input40.local' (98 bytes in 0.0 seconds, 2.07 kB/s) File s3://ami-kippsample03reads/input41.local saved as '/usr/local/hadoop-0.19.0/input41.local' (98 bytes in 0.0 seconds, 1960.43 B/s) File s3://ami-kippsample03reads/input42.local saved as '/usr/local/hadoop-0.19.0/input42.local' (98 bytes in 0.0 seconds, 2.09 kB/s) File s3://ami-kippsample03reads/input43.local saved as '/usr/local/hadoop-0.19.0/input43.local' (98 bytes in 0.1 seconds, 1370.40 B/s) File s3://ami-kippsample03reads/input44.local saved as '/usr/local/hadoop-0.19.0/input44.local' (98 bytes in 0.3 seconds, 358.54 B/s) File s3://ami-kippsample03reads/input45.local saved as '/usr/local/hadoop-0.19.0/input45.local' (98 bytes in 0.0 seconds, 2.85 kB/s) File s3://ami-kippsample03reads/input46.local saved as '/usr/local/hadoop-0.19.0/input46.local' (98 bytes in 0.0 seconds, 2013.97 B/s) File s3://ami-kippsample03reads/input47.local saved as '/usr/local/hadoop-0.19.0/input47.local' (98 bytes in 0.0 seconds, 3.13 kB/s) File s3://ami-kippsample03reads/input48.local saved as '/usr/local/hadoop-0.19.0/input48.local' (98 bytes in 0.1 seconds, 1585.68 B/s) File s3://ami-kippsample03reads/input49.local saved as '/usr/local/hadoop-0.19.0/input49.local' (98 bytes in 0.1 seconds, 1626.69 B/s) File s3://ami-kippsample03reads/input5.local saved as '/usr/local/hadoop-0.19.0/input5.local' (97 bytes in 0.0 seconds, 2.61 kB/s) File s3://ami-kippsample03reads/input50.local saved as '/usr/local/hadoop-0.19.0/input50.local' (98 bytes in 0.0 seconds, 2.09 kB/s) File s3://ami-kippsample03reads/input51.local saved as '/usr/local/hadoop-0.19.0/input51.local' (98 bytes in 0.1 seconds, 1158.50 B/s) File s3://ami-kippsample03reads/input52.local saved as '/usr/local/hadoop-0.19.0/input52.local' (98 bytes in 0.0 seconds, 2.41 kB/s) File s3://ami-kippsample03reads/input53.local saved as '/usr/local/hadoop-0.19.0/input53.local' (98 bytes in 0.0 seconds, 2018.21 B/s) File s3://ami-kippsample03reads/input54.local saved as '/usr/local/hadoop-0.19.0/input54.local' (98 bytes in 0.1 seconds, 1402.40 B/s) File s3://ami-kippsample03reads/input55.local saved as '/usr/local/hadoop-0.19.0/input55.local' (98 bytes in 0.0 seconds, 2.10 kB/s) File s3://ami-kippsample03reads/input56.local saved as '/usr/local/hadoop-0.19.0/input56.local' (98 bytes in 0.0 seconds, 2.37 kB/s) File s3://ami-kippsample03reads/input57.local saved as '/usr/local/hadoop-0.19.0/input57.local' (98 bytes in 0.0 seconds, 2.38 kB/s) File s3://ami-kippsample03reads/input58.local saved as '/usr/local/hadoop-0.19.0/input58.local' (98 bytes in 0.1 seconds, 1810.96 B/s) File s3://ami-kippsample03reads/input59.local saved as '/usr/local/hadoop-0.19.0/input59.local' (98 bytes in 0.0 seconds, 2.03 kB/s) File s3://ami-kippsample03reads/input6.local saved as '/usr/local/hadoop-0.19.0/input6.local' (97 bytes in 0.0 seconds, 2.09 kB/s) File s3://ami-kippsample03reads/input60.local saved as '/usr/local/hadoop-0.19.0/input60.local' (98 bytes in 0.0 seconds, 2.50 kB/s) File s3://ami-kippsample03reads/input7.local saved as '/usr/local/hadoop-0.19.0/input7.local' (97 bytes in 0.0 seconds, 2.15 kB/s) File s3://ami-kippsample03reads/input8.local saved as '/usr/local/hadoop-0.19.0/input8.local' (97 bytes in 0.1 seconds, 1397.15 B/s) File s3://ami-kippsample03reads/input9.local saved as '/usr/local/hadoop-0.19.0/input9.local' (97 bytes in 0.0 seconds, 2.47 kB/s) rmr: cannot remove test: No such file or directory. rmr: cannot remove maq: No such file or directory. Maq alignments + Duplicate remover 12/03/10 22:05:39 WARN streaming.StreamJob: -jobconf option is deprecated, please use -D instead. packageJobJar: [/root/mapper_maqalignment.py, /root/Sam2Fastq.java, /root/FQone2Fastq.java, /root/Fastq2FQone.java, /root/removeduplicates_new.java, /root/MAQunmapped2FQone.java, /root/MAQunmapped2fastq.java, /mnt/hadoop/hadoop-unjar7647762000067869068/] [] /tmp/streamjob8505619080343769434.jar tmpDir=null 12/03/10 22:05:40 INFO mapred.FileInputFormat: Total input paths to process : 60 12/03/10 22:05:40 INFO streaming.StreamJob: getLocalDirs(): [/mnt/hadoop/mapred/local] 12/03/10 22:05:40 INFO streaming.StreamJob: Running job: job_201203102044_0002 12/03/10 22:05:40 INFO streaming.StreamJob: To kill this job, run: 12/03/10 22:05:40 INFO streaming.StreamJob: /usr/local/hadoop-0.19.0/bin/../bin/hadoop job -Dmapred.job.tracker=hdfs://ip-10-34-46-200.ec2.internal:50002 -kill job_201203102044_0002 12/03/10 22:05:40 INFO streaming.StreamJob: Tracking URL: [url]http://ip-10-34-46-200.ec2.internal:50030/jobdetails.jsp?jobid=job_201203102044_0002[/url] 12/03/10 22:05:41 INFO streaming.StreamJob: map 0% reduce 0% 12/03/10 22:06:00 INFO streaming.StreamJob: map 3% reduce 0% 12/03/10 22:06:01 INFO streaming.StreamJob: map 17% reduce 0% 12/03/10 22:06:02 INFO streaming.StreamJob: map 25% reduce 0% 12/03/10 22:06:04 INFO streaming.StreamJob: map 28% reduce 0% 12/03/10 22:06:05 INFO streaming.StreamJob: map 33% reduce 0% 12/03/10 22:06:06 INFO streaming.StreamJob: map 47% reduce 0% 12/03/10 22:06:07 INFO streaming.StreamJob: map 55% reduce 0% 12/03/10 22:06:08 INFO streaming.StreamJob: map 62% reduce 0% 12/03/10 22:06:09 INFO streaming.StreamJob: map 65% reduce 0% 12/03/10 22:06:10 INFO streaming.StreamJob: map 70% reduce 0% 12/03/10 22:06:11 INFO streaming.StreamJob: map 83% reduce 0% 12/03/10 22:06:12 INFO streaming.StreamJob: map 92% reduce 0% 12/03/10 22:06:14 INFO streaming.StreamJob: map 95% reduce 0% 12/03/10 22:06:15 INFO streaming.StreamJob: map 98% reduce 0% 12/03/10 22:06:16 INFO streaming.StreamJob: map 100% reduce 0%
Comment
-
Hi Chandra,
Thanks for making Pathseq. The sample source is human, RNA-seq data. 155million reads originally, with 55 million passing quality filters. I was expecting the vast majority of reads to be human, and I've had success previously with similar datasets, always <30 hr run times.
My plan is to try a different dataset in the next few days, if that works I'll suspect it was something in that particular data set or a one time glitch.
Thanks!
Comment
Latest Articles
Collapse
-
by seqadmin
The field of epigenetics has traditionally concentrated more on DNA and how changes like methylation and phosphorylation of histones impact gene expression and regulation. However, our increased understanding of RNA modifications and their importance in cellular processes has led to a rise in epitranscriptomics research. “Epitranscriptomics brings together the concepts of epigenetics and gene expression,” explained Adrien Leger, PhD, Principal Research Scientist on Modified Bases...-
Channel: Articles
Yesterday, 07:01 AM -
-
by seqadmin
Proteins are often described as the workhorses of the cell, and identifying their sequences is key to understanding their role in biological processes and disease. Currently, the most common technique used to determine protein sequences is mass spectrometry. While still a valuable tool, mass spectrometry faces several limitations and requires a highly experienced scientist familiar with the equipment to operate it. Additionally, other proteomic methods, like affinity assays, are constrained...-
Channel: Articles
04-04-2024, 04:25 PM -
ad_right_rmr
Collapse
News
Collapse
Topics | Statistics | Last Post | ||
---|---|---|---|---|
Started by seqadmin, 04-11-2024, 12:08 PM
|
0 responses
39 views
0 likes
|
Last Post
by seqadmin
04-11-2024, 12:08 PM
|
||
Started by seqadmin, 04-10-2024, 10:19 PM
|
0 responses
41 views
0 likes
|
Last Post
by seqadmin
04-10-2024, 10:19 PM
|
||
Started by seqadmin, 04-10-2024, 09:21 AM
|
0 responses
35 views
0 likes
|
Last Post
by seqadmin
04-10-2024, 09:21 AM
|
||
Started by seqadmin, 04-04-2024, 09:00 AM
|
0 responses
55 views
0 likes
|
Last Post
by seqadmin
04-04-2024, 09:00 AM
|
Comment