Hmsc Error: 8 nodes produced errors; first error: the leading minor of order 4 is not positive definite about hmsc

[jarioksa]

Can you prepare a reproducible example? If not, can you reproduce this in non-parallel run?

Non-parallel runs allow better tracing: there are now 60 calls to chol in 12 functions that can give that very same error message, and we have hard time guessing which of them is involved here. Now clusterApplyLB is the last command we have written and tracing ends there from our point of view.

Yes, this can happen, and issue #45 indeed has similar features.

[wwhbeach]

I tried running a non-parallel run and got this error:

Computing chain 1
Hmsc::computeInitialParameter - initializing fixed effects with SSDM estimates
Error in chol.default(V) :
the leading minor of order 4 is not positive definite
In addition: Warning messages:
1: glm.fit: fitted probabilities numerically 0 or 1 occurred
2: glm.fit: fitted probabilities numerically 0 or 1 occurred
3: glm.fit: fitted probabilities numerically 0 or 1 occurred
4: glm.fit: fitted probabilities numerically 0 or 1 occurred
5: glm.fit: fitted probabilities numerically 0 or 1 occurred
6: glm.fit: fitted probabilities numerically 0 or 1 occurred
7: glm.fit: fitted probabilities numerically 0 or 1 occurred
8: glm.fit: algorithm did not converge
9: glm.fit: fitted probabilities numerically 0 or 1 occurred
10: glm.fit: fitted probabilities numerically 0 or 1 occurred

[jarioksa]

Seems to be the "unexplained variance in species 'niches' (linear coefficients)" that cannot be inverted. It could be that failures in glm fit are a part of the problem. @ovaskain?

[ovaskain]

Hi, I had a look at the unfitted model (thanks for emailing it). The problem does not seem to be with Hmsc itself, but with using ML fixed effects for initializing. When I used sampleMcmc(m,samples = 500) it run in a couple of seconds without problems. When I used sampleMcmc(m,samples=500, initPar = "fixed effects"), it gave the error message. So simply let Hmsc randomize the initial state rather than using the ML solution for that. Best, O2 From: Jari Oksanen ***@***.***> Sent: keskiviikko 27. joulukuuta 2023 23:15 To: hmsc-r/HMSC ***@***.***> Cc: Ovaskainen, Otso T ***@***.***>; Mention ***@***.***> Subject: Re: [hmsc-r/HMSC] Hmsc Error: 8 nodes produced errors; first error: the leading minor of order 4 is not positive definite (Issue #174) Seems to be the "unexplained variance in species 'niches' (linear coefficients)" that cannot be inverted. It could be that failures in glm fit are a part of the problem. @ovaskain<https://github.com/ovaskain>? — Reply to this email directly, view it on GitHub<#174 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEIYMZSDUM5Y535CIXREO6LYLSFUZAVCNFSM6AAAAABBERPPZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZQGYZTCMJYGQ>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[jarioksa]

@ovaskain We have similar reports when initPar="fixed effects" was used (issue #45). Now we also have warnings from glm. These seem to be cases with complete separation or other extreme cases where binary models tend to either 0 or 1, or Poisson model to 0 which cannot be represented with the link functions, and the linear predictor tends towards minus or plus infinity. It won't go that far, but the initial values of linear predictors will be extreme, and as initial values the MCMC chains can wander off to impossible territories. I don't know if we should try to handle these cases that give a warning glm.fit: fitted probabilities numerically 0 or 1 occurred warning?

[ovaskain]

I think in such cases it is better not to use initPar=”fixed effects” (which in general should not be used except as a last resort to solve mixing issues). This is a problem of glm rather than a problem of Hmsc… From: Jari Oksanen ***@***.***> Sent: tiistai 2. tammikuuta 2024 21:35 To: hmsc-r/HMSC ***@***.***> Cc: Ovaskainen, Otso T ***@***.***>; Mention ***@***.***> Subject: Re: [hmsc-r/HMSC] Hmsc Error: 8 nodes produced errors; first error: the leading minor of order 4 is not positive definite (Issue #174) @ovaskain<https://github.com/ovaskain> We have similar reports when initPar="fixed effects" was used (issue #45<#45>). Now we also have warnings from glm. These seem to be cases with complete separation or other extreme cases where binary models tend to either 0 or 1, or Poisson model to 0 which cannot be represented with the link functions, and the linear predictor tends towards minus or plus infinity. It won't go that far, but the initial values of linear predictors will be extreme, and as initial values the MCMC chains can wander off to impossible territories. I don't know if we should try to handle these cases that give a warning glm.fit: fitted probabilities numerically 0 or 1 occurred warning? — Reply to this email directly, view it on GitHub<#174 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEIYMZRWOY5VJT2OSPQISP3YMROOXAVCNFSM6AAAAABBERPPZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZUGQ2TSMBUG4>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>

[wwhbeach]

Hi @jarioksa and @ovaskain,

Thank you for your help! This seems to have fixed the issue.

[wwhbeach]

Hi again,

I've since rerun a few models that did not have any error when initPar= "fixed effects" was set and have run them without this. I've noticed that in many cases the Tjur R2 values, AUC, and WAIC results have changed quite a lot (For example, one model that originally had a mean Tjur R2 of .40 and was initially my best fitting model overall dropped to 0.32).
Any idea why this may have occurred?

[ovaskain]

If those values change, it sounds like the MCMC runs were not converged (if they converge, does not matter if you start from fixed effect or not). O2 From: wwhbeach ***@***.***> Sent: keskiviikko 3. tammikuuta 2024 22:57 To: hmsc-r/HMSC ***@***.***> Cc: Ovaskainen, Otso T ***@***.***>; Mention ***@***.***> Subject: Re: [hmsc-r/HMSC] Hmsc Error: 8 nodes produced errors; first error: the leading minor of order 4 is not positive definite (Issue #174) Hi again, I've since rerun a few models that did not have any error when initPar= "fixed effects" was set and have run them without this. I've noticed that in many cases the Tjur R2 values, AUC, and WAIC results have changed quite a lot (For example, one model that originally had a mean Tjur R2 of .40 and was initially my best fitting model overall dropped to 0.32). Any idea why this may have occurred? — Reply to this email directly, view it on GitHub<#174 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEIYMZQGXI5B7BNXCZ5SIMDYMXAYVAVCNFSM6AAAAABBERPPZ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZVHE3DCMRYGY>. You are receiving this because you were mentioned.Message ID: ***@***.******@***.***>>