Core dump in hmmufotu-build

[jshleap]

Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignments:

MSA loaded
MSA pruned
MSA database created for 9742 X 242 aligned sequences
CSFM index built
hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed.
Aborted (core dumped)

Any ideas on how I can overcome it?

Best,

Sergio

[e00011027]

Hi Sergio, This looks like a out of RAM error, if you are running it under bsub or ibash on a cluster. Try to allocate my RAM and SWAP before running it. If it is not, can you send me your input files to take a look? Best Qi

…

On Tue, Sep 3, 2019 at 8:27 PM Jose Sergio Hleap ***@***.***> wrote: Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignment: MSA loaded MSA pruned MSA database created for 9742 X 242 aligned sequences CSFM index built hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped) Any ideas on how I can overcome it? Best, Sergio — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABXZ44OBCVWAGSDKY7RHO6LQH36HBA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJDXY6A>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXZ44JPDT2ZW3U7DP6YVJ3QH36HBANCNFSM4ITMBYYQ> .

[jshleap]

Hi Qi, It is still not working, even with 128Gb of RAM... Do you have an idea of the required RAM based on the size of the file? None of the files I am trying to run are bigger than 200M. Where can I send you the data? Do you still have the zhengqi@pennmedicine.upenn.edu?

[jshleap]

Hey! I tried with 502GB of ram and had the same error:

MSA loaded
MSA pruned
MSA database created for 16657 X 313 aligned sequences
CSFM index built
hmmufotu-build: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/eigen/3.3.5/include/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed.
/var/spool/slurmd/job21271023/slurm_script: line 8: 27484 Aborted                 (core dumped) 

I used the following command: hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.tree --fmt 'fasta' -V -v -p 32

MIS_realized.reference fasta en tree being the fasta-formated sequence file and the newick-formatted tree, respectively.

[e00011027]

For now this looks like an error caused by all zero branch lengths of some internal node of your newick tree. Try to parse/check your customized tree to make sure at least one non-zero branch exists for every node, otherwise the matrix operation calculating tree likelihood will fail due to numerical error. It is even better if ALL branch lengths are greater than zero. Let me know whether it works. Best Qi

…

On Tue, Oct 29, 2019, 6:07 PM Jose Sergio Hleap ***@***.***> wrote: Hey! I tried with 502GB of ram and had the same error: MSA loaded MSA pruned MSA database created for 16657 X 313 aligned sequences CSFM index built hmmufotu-build: /cvmfs/soft.computecanada.ca/easybuild/software/2017/Core/eigen/3.3.5/include/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. /var/spool/slurmd/job21271023/slurm_script: line 8: 27484 Aborted (core dumped) I used the following command: hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.tree --fmt 'fasta' -V -v -p 32 MIS_realized.reference fasta en tree being the fasta-formated sequence file and the newick-formatted tree, respectively. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABXZ44N7YCOXVVS6AEOJPBLQRCX2NA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECSH35Q#issuecomment-547651062>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXZ44MJV2KWXWSRCHBWHJ3QRCX2NANCNFSM4ITMBYYQ> .

[jshleap]

Hi Qi! It does not work. I added 1E-10 to every branch length in the tree and it does not work. Please let me know how to send you the alignment and the tree (too big for github). Here is the command launched and its error:

hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.hmmufotu.tree --fmt 'fasta' -p 20

hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed.
Aborted (core dumped)

[e00011027]

Hi, you can try to share me a link for the files via Google Drive, Drop Box or something similar. Best Qi

…

On Wed, Oct 30, 2019 at 2:42 PM Jose Sergio Hleap ***@***.***> wrote: Hi Qi! It does not work. I added 1E-10 to every branch length in the tree and it does not work. Please let me know how to send you the alignment and the tree (too big for github). Here is the command launched and its error: hmmufotu-build MIS_realized.reference.fasta MIS_realized.reference.hmmufotu.tree --fmt 'fasta' -p 20 hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped) — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABXZ44JEZZIGKN3V2LNYRSLQRHIRNA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOECVLFIY#issuecomment-548057763>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABXZ44MJSFS6WGBBKZAVHNTQRHIRNANCNFSM4ITMBYYQ> .

[jshleap]

Hi Qi, Here is the link: https://drive.google.com/file/d/1sSPjjQX3PKUA28BY3hXMxwq8nQRgCGtM/view?usp=sharing
Best

[e00011027]

Hi Sergio, After testing we found it is caused by a bug for certain MSA profiles with all sites consensus. The GitHub source code has been updated and you should pull the latest to remake HmmUFOtu. However, after fixing the bug your input still won't build into a database, because the "Unmatched MSA and Tree. Found 16657 leaf sequences from MSA but expecting 27664 leaves in the Phylogenetic Tree". In another word, your tree file contains more leaf nodes than the sequences listed in the .fasta file. If it is just because you truncated the input for easier transfer than ignore it. Otherwise you need to re-build your tree only based on these aligned sequences. Please let us know if you have any questions. Best Qi

…

On Tue, Sep 3, 2019 at 8:27 PM Jose Sergio Hleap ***@***.***> wrote: Hello, I am trying to use hmmufotu in big reference datasets. I have created big alignments with MAFFT and created trees with fasttree. However, I am getting the following core dumped error with some (but not all) of my alignment: MSA loaded MSA pruned MSA database created for 9742 X 242 aligned sequences CSFM index built hmmufotu-build: /usr/include/eigen3/Eigen/src/Core/Block.h:122: Eigen::Block<XprType, BlockRows, BlockCols, InnerPanel>::Block(XprType&, Eigen::Index) [with XprType = const Eigen::Matrix<double, -1, -1>; int BlockRows = -1; int BlockCols = 1; bool InnerPanel = true; Eigen::Index = long int]: Assertion `(i>=0) && ( ((BlockRows==1) && (BlockCols==XprType::ColsAtCompileTime) && i<xpr.rows()) ||((BlockRows==XprType::RowsAtCompileTime) && (BlockCols==1) && i<xpr.cols()))' failed. Aborted (core dumped) Any ideas on how I can overcome it? Best, Sergio — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#6?email_source=notifications&email_token=ABXZ44OBCVWAGSDKY7RHO6LQH36HBA5CNFSM4ITMBYY2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HJDXY6A>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABXZ44JPDT2ZW3U7DP6YVJ3QH36HBANCNFSM4ITMBYYQ> .