slow editing for large org files about poly-org

[vspinu]

a large org-mode file (276K)

How big is this in lines? several secs after typing one letter?

I am investigating various performance issues atm. There is polymode/polymode#227 which I am working on at the moment but the fix is non-trivial I am afraid. Your issue might be unrelated. I will check.

[matthuszagh]

5704 lines. yes, each character-insertion keypress has somewhere in the neighborhood of 1-5sec delay. When I'm not actually inserting or deleting characters (i.e. moving point) I don't experience the same lag.

I can't say I've noticed the same problems mentioned in the issue you linked, but I haven't tested it thoroughly.

[vspinu]

Is there anything specific about the document? I am testing with docs containing 10k lines and typing behaves okeish. Can you share or put together a reproducible document?

[matthuszagh]

sure.
example.zip

[vspinu]

I don't see any latency. Which section in particular?

My bet is that you are loading old polymode somehow. Could you git pull poly-org and do make start from within and test on your file from the plain emacs which was open?

[matthuszagh]

you're right, that works fine. There seems an issue between polymode and another package/setting in my emacs config. When i disable polymode everything works fine, but poly/org on their own also work correctly. I've tried profiling but not seeing anything obvious yet. I'll keep looking; maybe there's a compatibility issue with another package that would actually be useful for someone else with a similar config. My plan was to keep running the profiler, but if you have any other strategies for how you would go about this I'd be happy to hear 'em!

[vspinu]

No good ideas here. Profiler might be useful. In which place does it show the bottle-neck?

A full proof strategy is to bisect your config.

[matthuszagh]

I've looked a bit and the issue seems to be related to the combination of org-ref and polymode. If I disable either org-ref or polymode, editing speed is fine but using both together creates lag. This issue seems to be the problem on the org-ref side. If I use this suggestion, editing works but then I have to call font-lock-fontify-block manually. This other suggestion helps, but editing is still too slow to be usable.

[vspinu]

I have just pushed a major speed improving change to the core. Chances that it would help with this issue are small, but you could still try it out.

AFAIS the issue is on the org-ref side with the buffer parsing in font-lock. The interaction between org and polymode is most likely due to the fact that jit-lock does interrupt the fontification on user-input but poly-lock does not. I avoided that complication as fontification is lazy anyhow and is done by small chunks (2000 chars). Crazy parsing of the entire buffer during the font-lock is not the right way to go for any mode.

I haven't met other use-cases of very slow font-lock keywords so far. So I really hope this could be made right on the org-ref side.

[matthuszagh]

@vspinu I've just tested the changes, and I actually do notice a significant difference. Without the org-ref-colorize-links change suggested by @jkitchin, there is little to no lag now. With his suggestion, there is little to no lag anyway as reported on that issue.

[vspinu]

Thanks for reporting back. The change in the core concerns syntax-propertize which is now 20x faster than before and was most likely triggered by org-ref on the entire buffer before.

Can we close this then?

[matthuszagh]

Closing, thanks @vspinu!