Hitting TS Limits: "Max Call Stack Size Exceeded" about typescript

[DanielRosenwasser]

While I think someone can take a look after the weekend, can you post any of the specific call stacks that you saw?

[texttechne]

Hi @ahejlsberg,

so sorry to have messed up the example like this.

Generation works now, but you don't need to generate the code. I've committed it directly under folder build.
So you can run npm run test-casePacer directly to see the mentioned error. The other two examples do work and are there for comparison.

[texttechne]

Stacktraces

The example from the repo:

RangeError: Maximum call stack size exceeded
    at String.replace (<anonymous>)
    at Object.toFileNameLowerCase [as getCanonicalFileName] (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:913:46)
    at getCanonicalFileName (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:119641:17)
    at toPath (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:5559:10)
    at toPath3 (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:118057:12)
    at getResolvedProjectReferenceToRedirect (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:119430:78)
    at getRedirectReferenceForResolution (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:118012:22)
    at Object.getModeForUsageLocation2 [as getModeForUsageLocation] (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:120537:36)
    at resolveExternalModule (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:46951:83)
    at resolveExternalModuleNameWorker (C:\Users\h\project\playground\max-call-stack-exceeded\node_modules\typescript\lib\tsc.js:46928:61)

From a previous odata2ts version (contained some bugs that have been resolved now). Also, this is directly from ts-morph trying to output js + dts

RangeError: Maximum call stack size exceeded
    at isTypeReferenceWithGenericArguments (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63953:49)
    at getRelationKey (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63992:14)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62324:20)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isRelatedToWorker2 (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63623:18)
    at compareSignaturesRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61254:16)
    at signatureRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63612:16)
    at signaturesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63514:29)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62982:26)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at isPropertySymbolTypeRelated (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63178:16)
    at propertyRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63233:25)
    at propertiesRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:63450:31)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62971:23)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)
    at isRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:61862:124)
    at eachTypeRelatedToType (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62198:30)
    at unionOrIntersectionRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62040:176)
    at structuredTypeRelatedToWorker (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62597:25)
    at structuredTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62439:23)
    at recursiveTypeRelatedTo (/home/user/www/node/cp/react-cp/node_modules/.pnpm/@[email protected]/node_modules/@ts-morph/common/dist/typescript.js:62409:21)

Using a private repo example

RangeError: Maximum call stack size exceeded
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59646:25)
    at isPropertySymbolTypeRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61084:14)
    at propertyRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61139:23)
    at propertiesRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61356:29)
    at structuredTypeRelatedToWorker (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60877:21)
    at structuredTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60325:21)
    at recursiveTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60295:19)
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59743:122)
    at eachTypeRelatedToType (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60084:28)
    at unionOrIntersectionRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59921:174)

Using yet another private repo example

RangeError: Maximum call stack size exceeded
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59646:25)
    at isRelatedToWorker2 (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61529:16)
    at compareSignaturesRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59128:14)
    at signatureRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61518:14)
    at signaturesRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61420:27)
    at structuredTypeRelatedToWorker (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60888:24)
    at structuredTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60325:21)
    at recursiveTypeRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:60295:19)
    at isRelatedTo (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:59743:122)
    at isPropertySymbolTypeRelated (C:\Users\h\project\odata2ts\node_modules\typescript\lib\tsc.js:61084:14)

[fatcerberus]

All those stacks are different which indicates they are (probably) seperate bugs.

[ahejlsberg]

I tried to reproduce using the repo at https://github.com/texttechne/max-call-stack-size-exceeded, but the npm run generate command fails with the following:

> max-call-stack-exceeded@1.0.0 generate
> odata2ts

Loaded config file:  odata2ts.config.ts
---------------------------
Starting generation process. Service name "Dynamics"
Didn't find metadata file at:  odata/dynamics.xml
Input source [odata/dynamics.xml] doesn't exist!

[ahejlsberg]

Ok, I can reproduce the issue. This looks to be a case of very deep type dependencies for which the compiler is attempting to compute variance information. For example, from top to bottom, each type below depends on the next type in the list, and it keeps going for >150 levels:

FirmCustomFieldService<ClientType>
CustomFieldTypeService<ClientType>
FirmService<ClientType>
PartyService<ClientType>
DocumentTemplateService<ClientType>
PartyTypeService<ClientType>
LawsuitPartyTypeService<ClientType>
DepositionService<ClientType>
LawsuitPartyService<ClientType>
LawsuitService<ClientType>
AppointmentService<ClientType>
ADRService<ClientType>
ADRTypeService<ClientType>
LitigationService<ClientType>
LawsuitPhaseTypeService<ClientType>
AnswerEnlargementService<ClientType>
LawsuitPhaseDateService<ClientType>
ComplaintService<ClientType>
SubPhaseTypeService<ClientType>
DefendantLitigationAssnService<ClientType>
DefendantLawsuitPartyService<ClientType>
DepositionLitigationAssnService<ClientType>
LawsuitTypeService<ClientType>
TicklerTemplateService<ClientType>
AccountingSyncGLService<ClientType>
RoleTemplateService<ClientType>
CasePacerUserService<ClientType>
CaseMenuItemService<ClientType>
CaseMenuGroupService<ClientType>
CaseMenuTabService<ClientType>
CPDirectCaseProgressService<ClientType>
RecordFileService<ClientType>
RecordFolderService<ClientType>
RecordSubtypeService<ClientType>
IntakeClientLawsuitService<ClientType>
...

Since variance computation is a recursive process, the deep dependencies eventually cause a stack overflow. I was going to suggest working around the problem using explicit variance annotations, but unfortunately the ClientType type parameter in each of these types ends up being bivariant and we lack syntax to express that in an annotation.

I'm not sure how important it is for these types to be generic. If the ClientType type parameter was eliminated, there likely wouldn't be an issue.

[texttechne]

Thanks so much! Good to know what the actual problem is. I was definitely lost there...

So I've tried out your suggestion to dispense with the ClientType type parameter and tada... it works. It works even with really large metadata files (20 MB). Takes it's time, but that's to be expected.

Background For the Problematic Type Parameter

The generated OData client relies on an HTTP client which is configurable: Axios, Fetch, JQuery, ....roll your own. Each HTTP client comes with its very own configuration options. And each request should be configurable. Using the generic type here allows me to offer the correct typing for the configuration options:

export abstract class EntitySetServiceV4<
  ClientType extends ODataHttpClient,
  T,
  EditableT,
  Q extends QueryObject,
  EIdType
> {
  public async query<ReturnType extends Partial<T> = T>(
    queryFn?: (builder: ODataQueryBuilderV4<Q>, qObject: Q) => void,
    requestConfig?: ODataHttpClientConfig<ClientType> // here the correct config object is inferred from the HTTPClient
  ): ODataResponse<ODataCollectionResponseV4<ReturnType>> {
    // ....
  }

}

See source code

I don't know how to define this differently...

Is this a Bug?

Let me ask naively: Do you think that's bad / imperformant code that I'm using here? Should I have known better?

To ask the other way around: Do you accept this a TS bug?

I think one can argue for both positions. However, I would argue for the latter one: That TS uses recursion to evaluate variance is an implementation detail that I, as end user, am not aware of & probably shouldn't be. Code that works perfectly fine in smaller examples, breaks for bigger code bases. But just my 2 cents...

[ahejlsberg]

I think this can reasonably be called a design limitation. Technically, there's nothing wrong with your code, but the same can be said for code that, for example, nests if statements to multiple thousand levels, or some other very deep construct that naturally doesn't occur in code authored by humans. All such code has the potential to overflow the call stack. We do have special handling for a few cases, such as our "trampoline" implementation of binary expression type checking, motivated by allowing concatenation of 1000s of string literals in a single expression. But externalizing the call stack through a trampoline comes with performance and complexity penalties, and it would be particularly hard to implement in this case because we'd have to completely revise our type relationship logic. Since the issue you're experiencing is rare (it's the first report I know of) and likely only occurs in machine generated code, a fix won't be a high priority.

You may want to experiment with adding in or out variance annotations for the ClientType type parameter. It will likely lead to errors, but you might be able to work around them. It is unfortunate that we don't have the ability to annotate bi-variance. If anything, that would be something to consider.

[texttechne]

Ok, I was afraid you would argue that way, unfortunately (for me) reasonable enough 😄

Before closing this issue: Do you see any problem with adding // @ts-nocheck to the beginning of each file and leaving the code as is?

[texttechne]

Finally found the time to toy around with variance annotations. After changing a certain implementation detail, I was able to annotate with out as well as in out and it works. The recursive variance computation is not triggered and the dreaded max call stack size exceeded error won't occur. Not as performant as leaving out the generic type parameter altogether, so the compilation will eventually run into the much more comfortable out of memory exception, but only for the 20MB metadata example and this is easily solved by giving the process more memory.

This is the best result as I'm now able to compile even those large examples which in turn brings any bugs with my generator to light.

But to be honest, that's more like trial and error for me than anything else. I think I got some hold of the variance topic in the meantime, but I fail to make the connection to my code. In the end I chose to annotate invariance...
However, not your responsibility to teach me stuff.

@ahejlsberg thank you so much for your analysis and all the right pointers! I was really lost there and needed help, which you provided brilliantly!