No bounds update and fixed number of calls each iteration about dynesty

[segasai]

Hi,

Thanks for the report. I'm currently trying to refactor/streamline/fix the bounding update code in #427 #425 #428.

I think there is certainly an issue there, I'm just trying to nail the proper fix.
If you can send me privately the code I could run myself, that'd be helpful.

Thanks

[ajw278]

Hi Sergey, Thanks very much for getting back to me. I've attached a tarred version of the code. Apologies that it is a mess, it is very much in development. Unfortunately it's not easy to show you this without everything, since you need the underlying data, It should just run in the main directory: python dmapper.py -m ns -N 3 -p 16 -d -R 1000

…

-m is mode (ns for nested sampling) -N is number of Gaussians (not important) -p is number of processes (I set to 16) -d means dynamic -R is the (initial) number of live points The main file that is relevant is 'dsolver_ns.py', where the call is made to execute_ns, which is where the sampling is run. There are some dependencies (astropy etc.) that should be easy enough, but let me know if anything doesn't work. As a side note, I would really like to use 'cubes' for the bounds, but had a similar problem to someone else online that the KDE call through a complex number error. I think it's sensible to focus on one issue at a time, but I thought I'd mention it in case it came up. Best, Andrew

On Thu, 16 Mar 2023 at 16:19, Sergey Koposov ***@***.***> wrote: Hi, Thanks for the report. I'm currently trying to refactor/streamline/fix the bounding update code in #427 <#427> #425 <#425> #428 <#428>. I think there is certainly an issue there, I'm just trying to nail the proper fix. If you can send me privately the code I could run myself, that'd be helpful. Thanks — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITS44I5Y4JZAVPM5DNC3W4MVORANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[segasai]

I don't think I received the attach. Presumably github strips them.
But in the meantime, could you try this branch https://github.com/joshspeagle/dynesty/tree/ellipsoid_fix
I'm wondering if that fixes your problem

[ajw278]

Ah, I see -- I've made this drive link public for now: https://drive.google.com/file/d/10uoa3gfAR9qWfwij_7S0dTseoUxPikBt/view?usp=sharing So you should be able to download it there. Although I will try the branch you mentioned, thanks!

…

On Thu, 16 Mar 2023 at 23:42, Sergey Koposov ***@***.***> wrote: I don't think I received the attach. Presumably github strips them. But in the meantime, could you try this branch https://github.com/joshspeagle/dynesty/tree/ellipsoid_fix I'm wondering if that fixes your problem — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITSYQFHJSAJTLZOVIGLLW4OJLXANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[ajw278]

Hi, So it seems like the problem hasn't gone away, but changed. Now when the efficiency goes below 25 percent (min ncall=5000), the bounds are not updated at all: 2092it [26:56, 1.27s/it, batch: 0 | bound: 0 | nc: 14 | ncall: 7342 | eff(%): 25.078 | loglstar: -inf < -1017.622 < inf | logz: -1025.676 +/- nan | 2093it [26:58, 1.30s/it, batch: 0 | bound: 0 | nc: 14 | ncall: 7356 | eff(%): 25.048 | loglstar: -inf < -1017.424 < inf | logz: -1025.318 +/- nan | 2094it [27:01, 1.62s/it, batch: 0 | bound: 0 | nc: 12 | ncall: 7368 | eff(%): 25.024 | loglstar: -inf < -1017.402 < inf | logz: -1025.030 +/- nan | 2095it [27:10, 3.10s/it, batch: 0 | bound: 0 | nc: 34 | ncall: 7402 | eff(%): 24.935 | loglstar: -inf < -1017.381 < inf | logz: -1024.803 +/- nan | 2096it [27:13, 3.07s/it, batch: 0 | bound: 0 | nc: 8 | ncall: 7410 | eff(%): 24.923 | loglstar: -inf < -1017.102 < inf | logz: -1024.589 +/- nan | 2098it [27:19, 2.97s/it, batch: 0 | bound: 0 | nc: 9 | ncall: 7425 | eff(%): 24.902 | loglstar: -inf < -1016.605 < inf | logz: -1024.122 +/- nan | 2100it [27:24, 2.90s/it, batch: 0 | bound: 0 | nc: 31 | ncall: 7458 | eff(%): 24.829 | loglstar: -inf < -1016.524 < inf | logz: -1023.739 +/- nan | 2101it [27:28, 3.00s/it, batch: 0 | bound: 0 | nc: 16 | ncall: 7474 | eff(%): 24.793 | loglstar: -inf < -1016.497 < inf | logz: -1023.583 +/- nan | 104it [27:30, 2.01s/it, batch: 0 | bound: 0 | nc: 3 | ncall: 7489 | eff(%): 24.785 | loglstar: -inf < -1015.960 < inf | logz: -1023.170 +/- nan |

…

On Thu, 16 Mar 2023 at 23:42, Sergey Koposov ***@***.***> wrote: I don't think I received the attach. Presumably github strips them. But in the meantime, could you try this branch https://github.com/joshspeagle/dynesty/tree/ellipsoid_fix I'm wondering if that fixes your problem — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITSYQFHJSAJTLZOVIGLLW4OJLXANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[segasai]

Thanks for the code and the test, now https://github.com/joshspeagle/dynesty/tree/ellipsoid_fix branch should fix it.
There was definitely a bug when update_interval was incorrectly set for the dynamic sampling.
As the branch contains some refactoring as well, it's not yet final.

[ajw278]

Thanks - it seems with that version that the bounds are now updating, but the updated bounds makes sampling less efficient somehow, and also then settles into the same behaviour as before (i.e. a fixed number of calls each iteration): 2081it [28:02, 1.43s/it, batch: 0 | bound: 0 | nc: 10 | ncall: 7418 | eff(%): 25.018 | loglstar: -inf < -990.726 < inf | logz: -998.397 +/- nan | 2083it [28:04, 1.27s/it, batch: 0 | bound: 0 | nc: 4 | ncall: 7425 | eff(%): 25.021 | loglstar: -inf < -990.299 < inf | logz: -997.895 +/- nan | 2086it [28:08, 1.33s/it, batch: 0 | bound: 0 | nc: 16 | ncall: 7452 | eff(%): 24.976 | loglstar: -inf < -989.168 < inf | logz: -997.154 +/- nan | 2088it [28:09, 1.18s/it, batch: 0 | bound: 1 | nc: 38 | ncall: 7492 | eff(%): 24.881 | loglstar: -inf < -988.763 < inf | logz: -996.544 +/- nan | 2104it [28:11, 2.64it/s, batch: 0 | bound: 2 | nc: 36 | ncall: 8068 | eff(%): 23.461 | loglstar: -inf < -984.964 < inf | logz: -992.766 +/- nan | 2120it [28:13, 4.10it/s, batch: 0 | bound: 3 | nc: 36 | ncall: 8644 | eff(%): 22.213 | loglstar: -inf < -980.443 < inf | logz: -987.999 +/- nan | 2136it [28:14, 5.47it/s, batch: 0 | bound: 4 | nc: 36 | ncall: 9220 | eff(%): 21.107 | loglstar: -inf < -976.901 < inf | logz: -984.507 +/- nan | 2152it [28:16, 6.72it/s, batch: 0 | bound: 5 | nc: 36 | ncall: 9796 | eff(%): 20.120 | loglstar: -inf < -972.974 < inf | logz: -980.418 +/- nan | 2168it [28:18, 7.31it/s, batch: 0 | bound: 6 | nc: 36 | ncall: 10372 | eff(%): 19.233 | loglstar: -inf < -968.066 < inf | logz: -975.588 +/- nan | Perhaps this is an issue in dealing with my specific problem, but it would be good to understand it if possible. All help very much appreciated.

…

On Sat, 18 Mar 2023 at 01:02, Sergey Koposov ***@***.***> wrote: Thanks for the code and the test, now https://github.com/joshspeagle/dynesty/tree/ellipsoid_fix branch should fix it. There was definitely a bug when update_interval was incorrectly set for the dynamic sampling. As the branch contains some refactoring as well, it's not yet final. — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITS6DP4AWVYUGV3CQ6J3W4T3QBANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[segasai]

It is not surprising that as you progress in the posterior, the efficiency goes down as the ellipsoidal approximation becomes less accurate. Keep in mind that the efficiency is a 'cumulative' efficiency. I.e. from the very start.
So I'm not seeing evidence of an issue here.
Regarding same nc, It is not obvious why it is the case, but not impossible, so again, unless there is evidence of an issue, I'm not sure there is much to do.

[ajw278]

I understand the decrease in efficiency (although shouldn't be the case after 1 bounds call, because this implies the area being searched is larger than the unit-cube that is originally searched). However, surely it is impossible for the nc to remain fixed forever? This behaviour continues regardless of the number of bounds calls or iterations, which I do not understand. There should at least be some random scatter from sampling.

…

On Sat, 18 Mar 2023 at 15:00, Sergey Koposov ***@***.***> wrote: It is not surprising that as you progress in the posterior, the efficiency goes down as the ellipsoidal approximation becomes less accurate. Keep in mind that the efficiency is a 'cumulative' efficiency. I.e. from the very start. So I'm not seeing evidence of an issue here. Regarding same nc, It is not obvious why it is the case, but not impossible, so again, unless there is evidence of an issue, I'm not sure there is much to do. — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITS3UXDCGSUCTTXOPXZ3W4W5YJANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[segasai]

Okay, I was confused - I thought you're using the uniform sampler, but you actually use the automatic sampler which means for this number of dimensions it'll be rwalk . The way rwalk is implemented is that it walks for a fixed number of steps. By default that number is determined by the dimension 20 + ndim. Hence 36.

[ajw278]

Ah, I see - thanks, I missed this in the manual sorry. Thanks again for your help! Best, Andrew

…

On Sun, 19 Mar 2023 at 02:33, Sergey Koposov ***@***.***> wrote: Okay, I was confused - I thought you're using the uniform sampler, but you actually use the automatic sampler which means for this number of dimensions it'll be rwalk . The way rwalk is implemented is that it walks for a fixed number of steps. By default that number is determined by the dimension 20 + ndim. Hence 36. — Reply to this email directly, view it on GitHub <#431 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADXITS57BDUHAGQLUSJOBV3W4ZO5PANCNFSM6AAAAAAV377Y6Y> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[segasai]

Thanks for the original report Andrew, I will close the issue in the meantime.