[caddymob]

I made a lot of people angry with Ray today....

Hi seb567,

So, I had Ray 1.2.0 installed on our cluster, complied with the same with Intel Compiler. The job starts to run, then is crashing.... Apparently today I took down 32 compute nodes according to the angry email I received from IT...

They sent me the following information about the job, I am told the problem is the huge amount of swap space I was using -- nearly 1TB!

Req[0]  TaskCount: 256  Partition: anon
Utilized Resources Per Task:  PROCS: 120.18  MEM: 2596M  SWAP: 881G
Avg Util Resources Per Task:  PROCS: 120.18
Max Util Resources Per Task:  PROCS: 237.11  MEM: 2596M  SWAP: 881G
Average Utilized Memory: 1641.30 MB
Average Utilized Procs: 48473.59
NodeSet=ONEOF:FEATURE:awesometown
NodeAccess: SINGLEJOB
NodeCount:  32

Here is the tail of my outfile:

Rank 230 is adding ingoing edges (reverse complement) 300001/1095111
[[18226,1],230][btl_openib_component.c:3224:handle_wc] from s54-5.local to: s54-15 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 246144128 opcode 0  vendor error 129 qp_idx 2
[s56-6.local:00534] 33 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
[s56-6.local:00534] Set MCA parameter "orte_base_help_aggregate" to 0 to see all help / error messages
[[18226,1],232][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13189504 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],173][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 16690688 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],237][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 121666816 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],234][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s56-3 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13992832 opcode 32767  vendor error 244 qp_idx 0
[[18226,1],175][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 113500032 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],172][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 117999232 opcode 32767  vendor error 129 qp_idx 2
[[18226,1],170][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-13 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 59806336 opcode 1  vendor error 129 qp_idx 2
[s56-6.local:00534] 2 more processes have sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],238][btl_openib_component.c:3224:handle_wc] from s54-4.local to: s55-16 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 17854848 opcode 128  vendor error 244 qp_idx 0
[[18226,1],215][btl_openib_component.c:3224:handle_wc] from s54-7.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 115086208 opcode 0  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],197][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 102530048 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],199][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-12 error polling LP CQ with status RETRY EXCEEDED ERROR status number 12 for wr_id 61565696 opcode 32767  vendor error 129 qp_idx 2
[s56-6.local:00534] 1 more process has sent help message help-mpi-btl-openib.txt / pp retry exceeded
[[18226,1],171][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s54-15 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 9502080 opcode 128  vendor error 244 qp_idx 0
[[18226,1],196][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s54-2 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 13076352 opcode 128  vendor error 244 qp_idx 0
[[18226,1],169][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 21503744 opcode 128  vendor error 244 qp_idx 0
[[18226,1],174][btl_openib_component.c:3224:handle_wc] from s54-12.local to: s56-6 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 11039872 opcode 128  vendor error 244 qp_idx 0
[[18226,1],194][btl_openib_component.c:3224:handle_wc] from s54-9.local to: s55-8 error polling HP CQ with status WORK REQUEST FLUSHED ERROR status number 5 for wr_id 12326400 opcode 32767  vendor error 244 qp_idx 0
=>> PBS: job killed: node 26 (s54-7) requested job terminate, 'EOF' (code 1099) - internal or network failure attempting to communicate with sister MOM's
mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

15 total processes killed (some possibly by mpirun during cleanup)

So, now IT is really angry with me, I'm kinda proud of myself, but I still have a genome to try to assemble. What do you suggest?

[seb567]

Hi !



error polling HP CQ with status WORK REQUEST FLUSHED ERROR

Seems to be an Open-MPI issue.



SWAP: 881G


Also, it is a bad idea to have 1 terabyte of swap memory, really.





How many MPI ranks do you have ?




How much physical memory is available on your cluster for your job ?




How many files / reads do you have ?



What is the Ray command you used ?



I wrote a post on my blog about Ray and large datasets:

I scale Ray on 512 cores and 1536 gigabytes of distributed memory on a human genome. Still doing some tests though before Christmas.

http://dskernel.blogspot.com/2010/12/ray-and-large-datasets.html

[pallo]

Hi,

Ray 1.2.0 crashes here, and I mean segfaults:

ray.err:

Loaded openMPI 1.4.2, compiled with intel11.1 (found in /opt/openmpi/1.4.2intel11.1/)
[q241:26410] *** Process received signal ***
[q241:26410] Signal: Segmentation fault (11)
[q241:26410] Signal code: Address not mapped (1)
[q241:26410] Failing at address: 0x9
[q241:26410] [ 0] /lib64/libpthread.so.0 [0x2b51e6679b10]
[q241:26410] [ 1] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN6Vertex15addOutgoingEdgeEmiP1\
1MyAllocator+
[q241:26410] [ 2] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor23call_TAG_\
OUT_EDGES_DAT
[q241:26410] [ 3] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN16MessageProcessor14processMe\
ssageEP7Messa
[q241:26410] [ 4] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(_ZN7Machine5startEv+0x1464) [0x4\
31c04]
[q241:26410] [ 5] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(main+0x89) [0x45a739]
[q241:26410] [ 6] /lib64/libc.so.6(__libc_start_main+0xf4) [0x2b51e68a3994]
[q241:26410] [ 7] /bubo/home/h12/pallol/glob/sandbox/Ray-1.2.0/code/Ray(__gxx_personality_v0+0xe9) [0x41\
7289]
[q241:26410] *** End of error message ***
--------------------------------------------------------------------------
mpirun noticed that process rank 144 with PID 26410 on node q241 exited on signal 11 (Segmentation fault\
).
--------------------------------------------------------------------------
slurmd[q75]: *** JOB 220623 CANCELLED AT 2010-12-16T01:07:13 DUE TO NODE FAILURE ***
mpirun: abort is already in progress...hit ctrl-c again to forcibly terminate

I have 16 lanes of illumina data (about 500M 90bp PE reads).
I call Ray simply with:

mpirun -np 160 ~/sandbox/Ray-1.2.0/code/Ray -p l_1_1.fastq l_1_2.fastq -p l_2_1.fastq...

cheers
pallo

[seb567]

Hi pallo,

Can you provide additional information (see my questions above) ?

(You can send me an email too.)


Otherwise it is difficult for me to rule out the problem.


Thank you !

[pallo]

Hi seb567

Good news, looks like it works with gcc 4.4 + openmpi 1.4.2 (still ran out of memory though, but I will try a bigger set of nodes next time).

I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.

Which questions are you referring to? The ones you posted to caddymob?

Most of this is in my post: the data is ca 500M reads of illumina PE data on 16 lanes plus 3 runs of 454 (total 16x2 + 3 = 35 files). The nodes are 20x8core so Im running mpirun 160. Each node has 24G mem + 24G swap.

cheers

[seb567]

@pallo: sorry, I mixed caddymob with you.

I will do tests with the Intel compiler myself.

[drio]

I dont think we need to debug the Intel compiled version, but if you want debug data, Ill provide as much as I can.

Have you compiled other biotools with the intel compilar and compared performance against gcc?

[pallo]

Have you compiled other biotools with the intel compilar and compared performance against gcc?

hi drio,

Nope, I dont have any benchmarks...
Its just SOP here to try to compile with the "native" intel and then gcc if intel binaries fail.

[caddymob]

next steps...

error polling HP CQ with status WORK REQUEST FLUSHED ERROR

Seems to be an Open-MPI issue.

Maybe .. our system was under high load and the time and the infinband links on the LUSTRE file system may have been saturated, compounding the problem. Nevertheless I have been warned not to try again until I fix the swap issue or risk losing my cluster account. IT is really mad at me for this

SWAP: 881G Also, it is a bad idea to have 1 terabyte of swap memory, really.

Agreed, but this is something Ray did, I do not, to my knowledge have a way to specify how much swap to use. What can I do to prevent this?

How many MPI ranks do you have ?
What is the Ray command you used ?
How much physical memory is available on your cluster for your job ?

I was using 256 cores -- our system has 8-core nodes, each with 24GB of RAM.

My command:

###Parameterized PBS Script ####
#PBS -S /bin/bash
#PBS -N LUNDE.Ray4
#PBS -l nodes=256
#PBS -l walltime=25:00:00
#PBS -q normal
#PBS -j oe
#PBS -o LUNDE.Ray4.o
#PBS -M ---redacted---
#PBS -m abe

wd=/scratch/myfiles/LUNDE_ASSEMBLE/
cd $wd

use intel-openmpi-1.4.2
use Ray-1.2.0

mpirun -np 256 Ray -p $wd\Lunde_1.fastq $wd\Lunde_2.fastq -o Lunde-contigs

How many files / reads do you have ?

I have 2 FASTQ files, one for the forward read, one for the reverse. I have 140,174,250 paired reads, 105mers. Would it be better to split these up into smaller chunks or does that matter?

Thanks for your help!

[seb567]

Dear caddymob,

I did some investigative work for you this morning.



You have 140 174 250 paired reads, each having a length of 105.

Would it be better to split these up into smaller chunks or does that matter?

Splitting the files would change nothing since Ray loads sequences in a lazy manner.

http://en.wikipedia.org/wiki/Lazy_loading

You use 256 MPI ranks, which are mapped on 8-core nodes, each having 16 GiB of memory.
You use 32 such nodes.

You have 512 GiB of distributed physical memory.



In modern operating systems, a program's addresses are virtual. If the computer has 2 GiB of random-access memory (RAM) and 100 GiB of swap memory,
then a program can allocate more than 2 GiB for its use. However, if the heap size (utilized memory) exceeds the random-access memory size, page faults occur.

"A page fault is a trap to the software raised by the hardware when a program accesses a page that is mapped in the virtual address space, but not loaded in physical memory."

http://en.wikipedia.org/wiki/Page_fault

It is followed by paging, 'one of the memory-management schemes by which a computer can store and retrieve data from secondary storage for use in main memory.'

http://en.wikipedia.org/wiki/Paging

Basically, if you exceed the physical memory, the job will just take forever because a lot of instructions will be only to swap pages between physical memory and swap memory.

Agreed, but this is something Ray did, I do not, to my knowledge have a way to specify how much swap to use. What can I do to prevent this?

Ray, presumably, requested memory, and page faults occured because physical memory was exausted.

You can't specify not to use swap because the programmer, and the program executing are unaware of which addresses are resident in physical memory and which are not.


Luckily, it seems you can limit the physical and virtual memory available to any process using a properly configured PBS scheduler: http://wiki.hpc.ufl.edu/index.php/PBS_Directives#Memory



You use PBS to launch the job.

According to the post numbered 69, your script is the following:

http://seqanswers.com/forums/showpost.php?p=31722&postcount=69

###Parameterized PBS Script ####
#PBS -S /bin/bash
#PBS -N LUNDE.Ray4
#PBS -l nodes=256
#PBS -l walltime=25:00:00
#PBS -q normal
#PBS -j oe
#PBS -o LUNDE.Ray4.o
#PBS -M ---redacted---
#PBS -m abe

wd=/scratch/myfiles/LUNDE_ASSEMBLE/
cd $wd

use intel-openmpi-1.4.2
use Ray-1.2.0

mpirun -np 256 Ray -p $wd\Lunde_1.fastq $wd\Lunde_2.fastq -o Lunde-contigs

To properly understand the parameters of the PBS scheduler, I have read http://wiki.hpc.ufl.edu/index.php/PBS_Directives
Your setup seems OK to me although I never worked with PBS.



In the post numbered 61 http://seqanswers.com/forums/showpos...8&postcount=61,

you wrote this:

Req[0] TaskCount: 256 Partition: anon
Utilized Resources Per Task: PROCS: 120.18 MEM: 2596M SWAP: 881G
Avg Util Resources Per Task: PROCS: 120.18
Max Util Resources Per Task: PROCS: 237.11 MEM: 2596M SWAP: 881G
Average Utilized Memory: 1641.30 MB
Average Utilized Procs: 48473.59
NodeSet=ONEOF:FEATURE:awesometown
NodeAccess: SINGLEJOB
NodeCount: 32

As I understand it, you have 256 tasks, and a task utilizes 120 processors, 2596 MiB of physical memory, and 881 GiB of swap space.

For all that matters, the node set is named 'awesometown'.

That does not make sense.

Too, 'Average Utilized Procs: 48473.59' does not make sense neither.



Your PBS script might be wrongly written or PBS might be misconfigured.

To ascertain the wholly truth, get support from your compute department.



As I promised in the post numbered 66 http://seqanswers.com/forums/showpos...4&postcount=66, I tested Ray with the Intel compiler.

I compiled Ray 1.2.1 with Open-MPI 1.4.3 and the Intel compiler, version 11.1.059.

I then launched a job, using mpirun (from Open-MPI 1.4.3) compiled with the Intel compiler. The script for Sun Grid Engine follows.

#!/bin/bash
#$ -N Ray1.2.1
#$ -P nne-790-aa
#$ -l h_rt=0:20:00
#$ -pe node 64
#$ -M sebastien.boisvert.3@<removed>
#$ -R y
#$ -m bea
module load compilers/intel/11.1.059 mpi/openmpi/1.4.3_intel
/software/MPI/openmpi-1.4.3_gcc/bin/mpirun /home/sboisver12/Ray/trunk/code/Ray  \
-p /home/sboisver12/nne-790-aa/SRA001125/SRR001665_1.fastq /home/sboisver12/nne-790-aa/SRA001125/SRR001665_2.fastq \
-p /home/sboisver12/nne-790-aa/SRA001125/SRR001666_1.fastq /home/sboisver12/nne-790-aa/SRA001125/SRR001666_2.fastq \
-o  Intel

The output in stdout is http://pastebin.com/yprcXwDe

Each MPI rank needed, on average, 284 MiB. The utilized distributed memory was 17 GiB.

I then used MUMmer (via the script named print-latex.sh in the scripts directory of the Ray distribution) to assess the quality of the assembly.

[sboisver12@colosse1 ~]$ print-latex.sh nne-790-aa/nuccore/Ecoli-k12-mg1655.fasta Intel.fasta  Ray-1.2.1
        %  & numberOfContigs & bases & meanSize  & n50  & max   & coverage   & misassembled & mismatches & indels
 Ray-1.2.1 & 123 & 4616336 & 37531 & 72499 &  176360 &  0.9819 & 0 & 2 & 4 \\

Everything's OK with the Intel compiler.


Cheers.

-seb

[seb567]

Ray 1.2.1 'stringray'

Dear all,

Ray 1.2.1 is now available.

Source:

http://sourceforge.net/projects/denovoassembler/files/Ray-1.2.1.tar.bz2

This version fixes 2 critical flaws, can assemble polymorphic positions by
forcing bubble traversal, and adds an experimental feature: memory usage
reduction during the construction of the distributed graph.

A more detailed catalog of changes follows.

• SplayTreeIterator now iterates in preorder instead of inorder.
  
• Now the SplayTreeIterator is utilized to iterate over the vertices nstead ofstoring them in an array (takes too much memory).
  
• The forest of splay trees now contains 16384 trees instead of 4096. otethat as usual, each MPI rank has its own forest. Furthermore, the forest freezes when vertices distribution is properly finished -- which means no more splaying in the splay trees is to occur. The process ensures that vertices with low redundancy remain at leaves.
  
• Bubble traversal ensures no misassemblies as polymorphic positions (substitution, and indels) are assembled !!!
• Works with 454 data too (454 homopolymers error are interpreted as polymorphic positions).
• Added numeric indicator for files. (example: [1/9])
• Corrected a bug in library lengths messaging that leaded to hanging and/or Bus Error.
• Fixed a segmentation fault when the -a (output AMOS) is provided. Thanks to Daniel Brami from J. Craig Venter Institute (La Jolla, CA) for the timely report.
• Preliminary version of an algorithm to preemptively reduce the memory usage while building the distributed graph.
• Under GNU/Linux platforms, Ray outputs the virtual memory (VmData from /proc, that is the heap) utilized before exiting.

Thank you.

ps.

I am currently working on two human genome data sets: SRA000271
(African) and SRA010766 (Jay Flatley).

[talioto]

dnGASP

Seb,

If you haven't seen the posting already, I'd like to bring to your attention the de novo Genome Assembly Assessment Project (dnGASP). All details can be found at cnag.bsc.es, but briefly it's a project that solicits submissions of assemblies of a synthetic 1.8Gb diploid genome, which will be followed up with a workshop in April in Barcelona. Please take a look and see if you may be interested. We at the CNAG are looking at Ray as an option for assembling human genomes and other genomes that we have already sequenced here (but have been unable to assemble thus far). The project and associated workshop may be a good opportunity to gain exposure and to participate in a forum where issues regarding assembly and sequence data can be discussed. If you are interested, please sign up for the dngasp mailing list and register your team (Ray) on the cnag.bsc.es site as soon as possible in order to download the reads and submit assemblies.

Regards,
Tyler Alioto

[seb567]

Sounds interesting !

[gstitan]

bug in Ray1.2.1?

Hey,

I have uploaded the last version of Ray to can obtain an amos output format. It ran during ten hours but I obtain errors as:
MPI_Isend(145): MPI_Isend(buf=0x2b196b013ad0, count=1, MPI_UNSIGNED_LONG_LONG, dest=23612, tag=89, MPI_COMM_WORLD, request=0x7fff25927d24) failed
MPI_Isend(95).: Invalid rank has value 23612 but must be nonnegative and less than 16

Do you konw this problem (and have you a solution)?

Best regards

PS: on the same data with a default output format, it was ok...!

[corthay]

Sanger reads

Hi seb567,

I have two questions.

1) Can Ray handle sanger reads such as WGS, Fosmid end ?

2) Does Ray do scaffolding ? The results did not contain N even though
I used paired end as following.

Ray \
-s \
solexa/xxx.2.2.41.single.fasta \
-p \
solexa/xxx.2.2.41.1.fasta \
solexa/xxx.2.2.41.2.fasta \
-o \
ray/ray.kmer31.contig \
-k \
31

The header line of fasta file is modified by SOAPdenovo correction tool.
Is this problem ?

Thanks,
Corthay