Can't find Demultiplex_Stats.htm file

[hetappi]

Dear all,

After demultiplexing an Illumina run I can not find the
Statistics file (Demultiplex_Stats.htm), supposed to be in the directory Basecall_Stats...

I use the following command to do demultiplexing:

configureBclToFastq.pl --input-dir \
/opt/gc/illumina-data/110610_SN863_0054_AB08V9ABXX/Data/Intensities/BaseCalls/ \
--output-dir /opt/gc/projects/test/test1 --sample-sheet SampleSheet.csv \
--mismatches 1
cd /opt/gc/projects/test/test1
nohup make -j 30

any help appreciated

hetappi

[GenoMax]

Can you provide some additional info?

What kind of a server/OS is this? How long did it take for the job to "finish"? Are there any files in the Project_flowcellID\Sample_* directories? Did you get anything in the terminal screen while this job was running?

[hetappi]

Run ended normally otherwise...

Thanks for your reply,

We run it on a UNIX server (8 processor machine):

uname -a
Linux blakey 2.6.32-5-amd64 #1 SMP Mon Oct 3 03:59:20 UTC 2011 x86_64 GNU/Linux

The run did what is was supposed to do. Nice fastq files are
produced. No obvious error messages from the splitting:

Sample_361:
total 430584
-rw-r--r-- 1 hetappi su_gc 400958523 Dec 19 15:28 361_GATCAG_L003_R1_001.fastq.gz
-rw-r--r-- 1 hetappi su_gc 39947778 Dec 19 15:41 361_GATCAG_L003_R1_002.fastq.gz
-rw-r--r-- 1 hetappi su_gc 167 Dec 19 14:11 SampleSheet.csv

etc.

Only in the (existing) Directory Basecall_Stats_D094WACXX
no Demultiplex_Stats.htm file is found which has been seen in other
runs (sometimes) We currently run the most recent
version of CASAVA_v1.8.2

hetappi

[GenoMax]

Did you look in the "nohup.out" file (should be in the same dir from where you launched the job) for error messages?

Alternatively, you may want to capture the standard error output to a file (that would mean re-running the analysis though) to see what may have been the problem.

[sgw]

Try running at -j 8. We saw the same thing when we tried to run at more than 8 threads. Somewhere in the CASAVA literature it says it only handles 8 threads.

[sklages]

We usually run 'make' with '-j 48' without the problem mentioned above (there are others though).

I had a similar issue when running demultiplexing on data where read access was restricted to just a few (not all) lanes. AFAIR it was prior 1.8.x, but it may be worth checking.

But as GenoMax already mentioned, have look at make's output.

Sven

[hetappi]

Took al look in nohup.out

Thanks for your replies so far.

Actually I run the demultiplexing with make -j 3 in this case...

As suggested i took a look in the file nohup.out

Since this file is 4.6 MB I cite only the last two lines here:

[2011-12-19 17:54:34] [blakey] [Temp/L001_R1_demux_summary.xml] Processing BCL files for lane 1, tile 2308, read 1
[2011-12-19 17:55:33] [blakey] [Temp/L001_R1_demux_summary.xml] opening /opt/gc/results/illum_run006/run1/Undetermined_indices/Sample_lane1/lane1_Undetermined_L001_R1_021.fastq.gz

Another case where I correctly got the Demultplex_Stats.htm the
file nohup.out end with these lines:

[2011-12-13 00:42:00] [gc-node-2] [all] make Reason: clean_intermediate.done
[2011-12-13 00:42:00] [gc-node-2] [all] make Prereqs: clean_intermediate.done
[2011-12-13 00:42:00] [gc-node-2] [all] make Cmd: [[ 2 == 0 ]] || 1>&2 echo -e "INFO:" all completed successfully.
[2011-12-13 00:42:00] [gc-node-2] [all] INFO: all completed successfully.

So my guess is that it stops sometime early for unknown reasons and
without error message.

hetappi

[pmiguel]

All,
This might be unrelated, but we recently upgraded our instrument software to HCS v1.5/RTA v1.13 in the hopes of being able to demultiplex indexes on the fly, the off instrument part of our processing pipeline started to "act up". In some cases it failed to demultiplex a lane at all. It also failed to produce the final demultiplexing report.

Rick downloaded the newest version of the downstream processing software and the issue went away.

--
Phillip

[GenoMax]

Can you "grep" for "error"/"Error" in your nohup.out file?

I have found that the actual error is not always found at the end of the log file so the problem step may be further up in the nohup.out file.

Thanks for your replies so far.

Actually I run the demultiplexing with make -j 3 in this case...

As suggested i took a look in the file nohup.out

Since this file is 4.6 MB I cite only the last two lines here:

[2011-12-19 17:54:34] [blakey] [Temp/L001_R1_demux_summary.xml] Processing BCL files for lane 1, tile 2308, read 1
[2011-12-19 17:55:33] [blakey] [Temp/L001_R1_demux_summary.xml] opening /opt/gc/results/illum_run006/run1/Undetermined_indices/Sample_lane1/lane1_Undetermined_L001_R1_021.fastq.gz

Another case where I correctly got the Demultplex_Stats.htm the
file nohup.out end with these lines:

[2011-12-13 00:42:00] [gc-node-2] [all] make Reason: clean_intermediate.done
[2011-12-13 00:42:00] [gc-node-2] [all] make Prereqs: clean_intermediate.done
[2011-12-13 00:42:00] [gc-node-2] [all] make Cmd: [[ 2 == 0 ]] || 1>&2 echo -e "INFO:" all completed successfully.
[2011-12-13 00:42:00] [gc-node-2] [all] INFO: all completed successfully.

So my guess is that it stops sometime early for unknown reasons and
without error message.

hetappi

[hetappi]

Solution found

Dear all,

Finally the problem seems to be insufficient RAM in the computer I
used for demultiplexing (about 16 GByte). After switching the
program to a computer with 512 GByte RAM the program run to the
end and produced correctly all files including the final
Demultiplex_Stats.htm

Thanks for all help

hetappi

[GenoMax]

Did you ever figure out if there was a specific error being generated as a part of the failure.

512GB seems like an overkill

We have used servers with 32GB of RAM for a long while and have not had any problems.

Dear all,

Finally the problem seems to be insufficient RAM in the computer I
used for demultiplexing (about 16 GByte). After switching the
program to a computer with 512 GByte RAM the program run to the
end and produced correctly all files including the final
Demultiplex_Stats.htm

Thanks for all help

hetappi

[sklages]

AFAIR casava needed around 2G per thread. If this is a memory problem, then using 'make -j6' shouldn't fail (just as some kind of control).

Sven

[hetappi]

programs missing

Dear all,

Well, GBytes seem only part of the answer. Looking for errors in nohup.out
showed that on the smaller machine the programs "gnuplot" and "convert"
are missing:


[2011-12-22 15:54:29] [blakey] [Basecall_Stats_C04VCACXX/l8_all_pngs.txt] sh: gnuplot: not found
sh: convert: not found
[2011-12-22 15:54:29] [blakey] [Basecall_Stats_C04VCACXX/l8_all_pngs.txt] [plotIntensity_tiles.pl] ERROR:In gnuplot Basecall_Stats_C04VCACXX/Temp/tmp-plotIntensity-10101-22905.txt | convert ps:- Basecall_Stats_C04VCACXX/Plots/s_8_1101_all.png 2>&1 - See errors above


hetappi

[sklages]

hä, hä, .. nice trap, .. with an "illuminating" error ... ;-)

[PabloMarin-Garcia]

not being able to find 'Demultiplex_Stats.htm file' is a sign of something being wrong.

Looking for the tail of nohup.out is no hope because the make -j writes the STDOUT of all processes at the same time to the file giving a interlaced lines of garbage.

In my own case, no Demultiplex_Stats.htm was found and grepping for 'error' or 'failed' I found that I have one

'make: *** [Project_control/Sample_PhiX/PhiX_0_L008_R1_001.fastq.gz] Error 1'

but I needed to demultiplex again with 'make -j 1' to be able to see the other lines of the error properly and discover that casava was defaulting to .clocs instead my *_pos.txt for the positions_format, and demultiplexBcls was giving --barcode=0

'/usr/local/CASAVA-1.8.2/libexec/CASAVA-1.8.2/demultiplexBcls --need-separate-controls --positions-format=.clocs .....--sample PhiX --sample lane8 --barcode 0...'

in disagreement with the barcode-cycles

And then the definitive error:

[2012-01-20 11:51:03] [pipeline] [ct_control/Sample_PhiX/PhiX_0_L008_R1_001.fastq.gz] Failed to parse the options:
[2012-01-20 11:51:03] [pipeline] [ct_control/Sample_PhiX/PhiX_0_L008_R1_001.fastq.gz] *** sample-barcode not valid: length of barcode must match the number of barcode-cycles ***
[2012-01-20 11:51:03] [pipeline] [ct_control/Sample_PhiX/PhiX_0_L008_R1_001.fastq.gz] Incorrect barcode: 0