Me again -- new crash with Ray. I'm running it at Amazon EC2 on a High-Memory Quadruple Extra Large instance (26 compute units; 68.4Gb of memory). I'm wondering if (a) I somehow ran out of memory and (b) whether I should use a smaller "-np" parameter (current 26, the number of processors) to try to fix this.
The run was initiated with the following command. The sequences are Illumina 100bp paired end sequences pre-processed with FLASH; because of the way the libraries were constructed most sequences in the single read file (wgc339.extendedFrags.fastq)
The message in the file from STDERR is
The bottom of the STDOUT redirect is:
the last messages on Rank 3 from STDOUT were
Does mpirun leave some informative log files I should be checking?
thanks in advance for any guidance
The run was initiated with the following command. The sequences are Illumina 100bp paired end sequences pre-processed with FLASH; because of the way the libraries were constructed most sequences in the single read file (wgc339.extendedFrags.fastq)
Code:
mpirun -np 24 /home/ec2-user/Ray/Ray -p wgc339.notCombined_1.fastq wgc339.notCombined_2.fastq -s wgc339.extendedFrags.fastq -o wgc339-ray 1> ray.wgc339.out 2> ray.wgc339.err
Code:
-------------------------------------------------------------------------- mpirun noticed that process rank 3 with PID 30808 on node ip-10-136-61-52 exited on signal 9 (Killed). --------------------------------------------------------------------------
Code:
Rank 9 is computing vertices & edges [3370001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 51 units/second Estimated remaining time for this step: 9 hours, 22 minutes, 2 seconds Rank 3 is computing vertices & edges [3360001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 41 units/second Estimated remaining time for this step: 11 hours, 43 minutes, 11 seconds Rank 18 is computing vertices & edges [3360001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 44 units/second Estimated remaining time for this step: 10 hours, 55 minutes, 14 seconds Rank 6 is computing vertices & edges [3350001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 51 units/second Estimated remaining time for this step: 9 hours, 28 minutes, 34 seconds Rank 14 is computing vertices & edges [3350001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 40 units/second Estimated remaining time for this step: 12 hours, 4 minutes, 56 seconds Rank 17 is computing vertices & edges [3360001/5089855] Speed RAY_SLAVE_MODE_EXTRACT_VERTICES 48 units/second Estimated remaining time for this step: 10 hours, 38 seconds Rank 6 has 60400000 vertices Rank 6: assembler memory usage: 2643428 KiB
Code:
Rank 3 has 60400000 vertices Rank 3: assembler memory usage: 2643424 KiB Rank 3 is computing vertices & edges [3350001/5089855] Rank 3 is computing vertices & edges [3360001/5089855]
thanks in advance for any guidance
Comment