"Apparent recursion within a with_prototype action: 25000 context sanity limit hit" about ecmascript-sublime

[blake-regalia]

@bathos I figured this out. Oddly enough, it appears that ST 3176 may or may not (depending on your OS it seems) load the Ecmascript syntax as the default scope:source.js, such as when the base HTML syntax embeds javascript. However, if I change our root scope from source.js source.es to just source.js, then indeed the Markdown and HTML base syntaxes will start using Ecmascript syntax for embedded js if they weren't already. This inevitably causes the errors @acconrad is reporting here.

I'm going to disable the affected nested syntaxes for now: DOT and HTML. Unfortunately, it seems that we cannot nest any syntax that ultimately embeds or pushes a source:js to the stack, (e.g., we nest DOT, DOT pushes HTML, HTML pushes JS, stack blows up). Really a poor oversight by the ST3 team if you ask me :/

As a remedy, I'm going to look into generating a compatible HTML, DOT, and Markdown syntax directly from the base syntaxes. That way we can just ship our own "safe" nested syntaxes.

[blake-regalia]

This is most certainly caused by https://github.com/sublimehq/Packages/blob/master/Markdown/Markdown.sublime-syntax#L993 which loads contexts from Ecmascript, which then includes contexts from Markdown, when then loads contexts from Ecmascript, which then includes contexts from Markdown, .... at least, that is my suspicion: sublimehq/sublime_text#2252

I will see if switching to embed will fix this. Btw, can our releases target specific builds of Sublime?

[bathos]

That example is tags vs branch. I didn’t see any with tags unprefixed + tags prefixed, but I didn’t search for very long so I dunno. (I’m surprised that I couldn’t find any of this documented — I might just be missing it?)

[bathos]

Thanks for going deep on this blake, it’s very appreciated. I haven’t been able to repro either unfortunately.

[blake-regalia]

I'm not getting this on my end. Does this happen no matter the file contents or only when using nested syntaxes (i presume that's why it's complaining about with_prototype)? If so, could you post a snippet that is causing this? Which version of Sublime?

[bathos]

I‘m getting it in Dev Build 3166. It happens with any file, even completely empty; it seems as if it may occur during a phase prior to the point of application, like it‘s validating/processing/compiling and that pops up.

[blake-regalia]

I don't have a license so I can't access the dev build channel, but this could have to do with the new embed feature making nesting syntaxes with include/push/set obsolete. I've definitely encountered this error before during development with this and other syntaxes, but not sure what's causing it now.

I would think that disabling the rule associated with the interpolation match in the syntax_AFTER_OPEN context would prevent ST from thinking there's any recursion, this one:

Ecmascript-Sublime/ecmascript.sublime-syntax

Line 5528 in e203d3a

- match: '((\$\{))'

[rafapaulin]

Same problem here... using latest sublime build..

Tried commenting the line @blake-regalia mentioned, but it dit no fix =(

[blake-regalia]

@rafapaulin Did you only comment out that line or the whole rule? I should have highlighted the range, try commenting out these 5 lines:

Ecmascript-Sublime/ecmascript.sublime-syntax

Lines 5528 to 5532 in e203d3a

	- match: '((\$\{))'
	captures:
	1: entity.quasi.element.js punctuation.quasi.element.begin.js # ^BS
	2: punctuation.definition.string.interpolated.element.begin.es
	push: [ syntax_ELEMENT, expression ]

[blake-regalia]

I have another hunch. It's possible that the Sublime devs thought of making push/set native syntax defs automatically use the new embed feature, which would be problematic for the nested JavaScript rule because it would recursively load the current default JavaScript syntax, which would be our Ecmascript syntax def. This would certainly be a Sublime bug and not a feature. If this is the case and if commenting out these lines solves the issue, it would be a good indication of whats causing it:

Ecmascript-Sublime/ecmascript.sublime-syntax

Lines 5687 to 5698 in e203d3a

	- match: '`'
	scope: string.interpolated.es punctuation.definition.string.interpolated.begin.es
	set: Packages/JavaScript/JavaScript.sublime-syntax
	with_prototype:
	# We must also compete with the 'tagged-template' rule in the default
	# JavaScript.sublime-syntax so that we can pop out
	- match: '([_$\p{L}\p{Nl}][_$\p{L}\p{Nl}\p{Mn}\p{Mc}\p{Nd}\p{Pc}\x{200C}\x{200D}]*)(?=\s*`)'
	captures:
	1: variable.other.readwrite.es
	2: string.interpolated.es punctuation.definition.string.interpolated.end.es
	pop: true
	- include: syntax_AFTER_OPEN

[rafapaulin]

@blake-regalia tried to comment out both ranges you suggested, appart from each other and together... still did not fix =(

[rafapaulin]

I may share my sublime license with you if you need

[bathos]

I don’t know much about the new nesting stuff — am I understanding correctly that it is a breaking change in Sublime? Like, is it even possible to have the nested syntaxes working in both current Sublime 3 and the dev build?

[blake-regalia]

@bathos it should not be a breaking change, and could have just been a bug in 3166 -- looking at the current dev build changelogs seems like there were some crash issues w 3166, does this still happen in the most recent build?

[bathos]

3167 and 3168 still exhibit the same behavior (sorry I haven’t been able to experiment with this properly myself yet, just been busy w/ work)

[bathos]

3170 is now stable and released, so I believe this now affects all users, womp. Gonna look into it tonight.

[bathos]

Confirmed that it’s related to nested syntax by fully disabling them — gonna try to re-enable bit-by-bit to see when it freaks out.

[bathos]

...it’s the markdown one :o

[bathos]

I published the easy fix (disabling nested markdown) till we can figure out why this happens. There’s nothing obvious to me; it seems like it’s got to be something unique about the Markdown syntax definition triggering it.

[bathos]

Btw, can our releases target specific builds of Sublime?

I think/thought this is possible, but I’m not sure what the correct way to do it is. The registry repo is Package Control Channel, where you’ll see our entry complete with the typo ES2105 haha:

{
  "name": "Ecmascript Syntax",
  "details": "https://github.com/bathos/Ecmascript-Sublime",
  "labels": [ "Javascript", "ES6", "ES2105" ],
  "releases": [
    {
      "sublime_text": ">=3084",
      "tags": true
    }
  ]
},

In the releases array, sublime_text can associate an entry with a sublime version range. In this case it’s indicating that it should be available only for 3084 (the version that introduced sublime-syntax) and later, and tags: true means that Package Control should look for semver git tags and treat them as releases.

If you look around at other entries, you’ll see that many specify a branch instead of tags. This seems like what we’d be looking for, but there’s a catch:

Please note that branch-based releases have been deprecated and no new packages will be accepted that utilize that feature. Existing packages are encouraged to update to using tags.

So we can’t use the branch option, which leaves me uncertain what one is supposed to do to have different releases associated with sublime version ranges. I don’t see any mention of this in the Package Control docs so far.

[bathos]

I think I found an example that does it with tags — apparently it can specify a prefix:

{
  "name": "ChineseLocalizations",
  "previous_names": ["ChineseLocalization"],
  "details": "https://github.com/rexdf/ChineseLocalization",
  "labels": ["localization", "chinese", "中文", "japanese", "日本語"],
  "releases": [
    {
      "sublime_text": "<3000",
      "tags": "st2-"
    },
    {
      "sublime_text": ">=3000",
      "tags": "st3-"
    }
  ]
},

Their tags list looks like this.

[blake-regalia]

Interesting, it looks like you can also use "tags": true for the default/latest build and optionally use a prefix for others, for example:

"releases": [
	{
		"sublime_text": "<3000",
		"branch": "st2"
	},
	{
		"sublime_text": ">=3000",
		"tags": true
	}
]

Although they are using 'branch' for the alternate instead of tags. The better question is, why doesn't Package Control have this documented somewhere?

[blake-regalia]

Found it:

// If your package is only compatible with specific builds of
// Sublime Text, this will cause the package to be hidden from
// users with incompatible versions.
//
// The "tags" key can be true for all valid semantic version tags, or
// can be a prefix string. Only tags in the form
// {prefix}{semantic_version} will be selected. In the example below,
// the entry with "sublime_text": "<3000" will match tags like:
//
// "st2-1.0.0"
// "st2-1.1.0"
//
// The release with "sublime_text": ">=3000" will match tags like:
//
// "st3-1.0.0"
// "st3-1.1.0"
{
	"details": "https://github.com/wbond/sublime_alignment",
	"releases": [
		{
			"sublime_text": "<3000",
			"tags": "st2-"
		},
		{
			"sublime_text": ">=3000",
			"tags": "st3-"
		}
	]
},

It's in their example-repository.json.

[blake-regalia]

New changes don't use embed and are backwards-compatible so no need for tags yet.

[bathos]

I’m told this has begun again. My guess is that it was triggered indirectly by changes in one of the delegated syntaxes, though I won’t be able to check which till later :/

[blake-regalia]

Can anyone share a snippet that causes this? Also what version of Sublime? I'm running build 3176 and am not able to get any errors.

[bathos]

I’ll try to get some more info. Two people messaged me directly about it today. My impression was that, as before, no file even needs to be open — it just happens from the extension existing at all.

[blake-regalia]

Hmm... just looked through all the latest base syntax package commits that might affect this and found: CSS, SQL, and ShellScript syntaxes changed in last few weeks, the only suspicious commit being for bash at this line here: sublimehq/Packages@4cfa7b9#diff-fe682d41416db7dad26aa8c60e0dc02eR300 so I cloned it and tested, but no problems. I'll wait for more info.

[acconrad]

Hi there! So I was one of those two people to message about this. I have the Ecmascript syntax installed on ST3.1.1 build 3176.

Steps to reproduce:

1. Open SublimeText. It will blow up only one
2. Open a repo with both .md and .js files where the default syntax for .md is Markdown and the default syntax for .js is Ecmascript
3. Open a readme or javascript file
4. The following popup is shown:

Another example I caught while opening a JS file:

[blake-regalia]

Thanks @acconrad , can you please confirm which version of the "Ecmascript Syntax" package you have installed? Ctrl+Shift+P > "Package Control: List Packages" > "Ecmascript Syntax".

EDIT: I suspect you have an outdated version since we recently removed nested Markdown support, mainly for this very reason.

[acconrad]

Looks like I'm on the same version as you

[blake-regalia]

This is a tricky one -- not sure what's causing it but I went ahead and published a release that might fix it. Please restart Sublime, check that it updates Ecmascript Syntax to v1.6.7 and then see if the problem still exists.

[acconrad]

@blake-regalia unfortunately no change, and I verified I now have 1.6.7 :(

[blake-regalia]

Hmm - I'm still unable to reproduce this no matter what I try, but I have some ideas about diagnosis, @acconrad . This should help us narrow it down:

1. Remove the Ecmascript Syntax package via Package Control.
2. Open the user preferences directory, Sublime Text > Preferences > Browse Packages ... "User". Do you have an "ecmascript.sublime-settings" file here? What about a "Markdown.sublime-settings" or any other relevant files? Please let me know if you do. And if you do, move them away from this directory temporarily.
3. Restart Sublime.
4. Open a Markdown .md file. There should be no errors at this point, if there are -- something else is up.
5. Copy this code-fenced section into the markdown file:

```js
call();
```

6. Put your caret on the call text and Ctrl+Shift+P (oops, last time I said this I meant Cmd+Shift+P, this time i really mean Ctrl). Screenshot/copy the scopes that list under the caret and post it here.
7. With the Markdown file still open, change the syntax association by going View > Syntax > Open all with current extension as ... > Plain Text.
8. Install the Ecmascript Syntax via Package Control.
9. Open a JavaScript .js file. What happens?
10. If you get to this step and you haven't encountered any errors yet, try changing the Markdown association back by opening an .md file and opening all with current extension as Markdown>Markdown.

[acconrad]

@blake-regalia It did blow up on Step 4. I followed it to the tee, but I only had a ecmascript.sublime-settings file in that User/ folder but not a Markdown.sublime-settings, so what's next?

[blake-regalia]

Interesting... What are the errors you got this time? Could you post a screenshot?

[blake-regalia]

Btw @acconrad for our own sanity, and to get you coding again in the meantime, I created a branch without nested syntax support. Here's how you can install it:

cd ~/Library/Application\ Support/Sublime\ Text\ 3/Packages/User/
git clone -b safe-mode https://github.com/bathos/Ecmascript-Sublime ecmascript

Then, View > Syntax > Open all with current extension as... > User > Ecmascript

[acconrad]

@blake-regalia you are awesome, thank you for diving into this!

[blake-regalia]

@bathos I drafted a release for 1.7.0, but want to make sure it doesn't break anything else incidentally. Could you update your local repo and check that everything looks good?

[blake-regalia]

Released 1.7.0

[bathos]

Sorry I missed that last request @blake-regalia ... but wow! This is a crazy impressive update (and everything seems to be fine).

[bathos]

May have spoken too soon. I’ve seen no highlighting issues, but it seems that shortcuts for commenting out lines (both single and multi) have stopped working. If you can confirm (not 100% sure ES Sublime is the cause yet) I’ll open a new issue.

[blake-regalia]

Hmm I can't seem to reproduce this yet. I just noticed some old scopes on a few .tmPreferences files and patched them real quick, see if that did anything?

[bathos]

This was fixed by the new release! Thanks again. 🙇