Dry run doesn't output tasks that changed for modified root workspace inputs about turborepo

[chris-olszewski]

I realize we have an error in our documentation.

The filter you want is --filter="{./packages/\*}[HEAD^1]" as this will run only tasks in packages that are in the packages/ directory that have changed since HEAD^1.

For all except package negation (--filter=!bar), multiple --filters are treated as a union. So --filter='./packages/* will produces foo and bar and --filter=[HEAD^1] will produce foo since root.txt changed. {foo, bar} U {foo} = {foo, bar}.

[psychobolt]

While that clarifies the usage, my use case still isn't working even with yarn turbo run build --filter="[HEAD^1]" --dry when a .root file is modified. It will work only if I modify a file from foo or bar packages. Considering two commits HEAD~1: .root/foo and HEAD~2: packages/foo/test . I expect both to return the same result:

➜  workspace git:(master) yarn turbo run build --filter="[HEAD~1]" --dry

Packages in Scope
Name Path 

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
➜  workspace git:(master) 
➜  workspace git:(master) yarn turbo run build --filter="[HEAD~2]" --dry

Packages in Scope
Name Path 
foo  packages/foo

Global Hash Inputs
  Global Files                          = 0
  External Dependencies Hash            = cab80e071f9bd92a
  Global Cache Key                      = I can’t see ya, but I know you’re here
  Global Env Vars                       = 
  Global Env Vars Values                = 
  Inferred Global Env Vars Values       = 
  Global Passed Through Env Vars        = 
  Global Passed Through Env Vars Values = 
  Engines Values                        = 

Tasks to Run
bar#build
  Task                           = build
  Package                        = bar
  Hash                           = 3565ae58cc7f544e
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/bar
  Command                        = echo 'build bar'
  Outputs                        = 
  Log File                       = packages/bar/.turbo/turbo-build.log
  Dependencies                   = 
  Dependents                     = foo#build
  Inputs Files Considered        = 3
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":[],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 
foo#build
  Task                           = build
  Package                        = foo
  Hash                           = c4f79d52425be1b6
  Cached (Local)                 = false
  Cached (Remote)                = false
  Directory                      = packages/foo
  Command                        = echo 'build foo'
  Outputs                        = 
  Log File                       = packages/foo/.turbo/turbo-build.log
  Dependencies                   = bar#build
  Dependents                     = 
  Inputs Files Considered        = 4
  Env Vars                       = 
  Env Vars Values                = 
  Inferred Env Vars Values       = 
  Passed Through Env Vars        = 
  Passed Through Env Vars Values = 
  Resolved Task Definition       = {"outputs":[],"cache":true,"dependsOn":["^build"],"inputs":["$TURBO_DEFAULT$","../../.root/*"],"outputLogs":"full","persistent":false,"env":[],"passThroughEnv":null,"interactive":false}
  Framework                      = 

However, they don't show same result. I know .root/* is being accounted from under Inputs Files Considered = 4

[mehulkar]

Can you try --dry=json and look at the expandedInputs key to see if the root file is there?

[psychobolt]

@mehulkar I ran three runs using --dry=json, I do not see expandedInputs:

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD]"
{
  "id": "2jeQm18Xk07gBZFkx3eoUVNKTPP",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output expected, with no changes on HEAD.

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD~1]"
{
  "id": "2jeQn5fEZ21pJul9ko7X65qUFeE",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [
    "//"
  ],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output not expected, only outputs root package that has changed. But foo package should detect changes from root.

➜  workspace git:(master) yarn turbo run build --dry=json --filter="[HEAD~2]"
{
  "id": "2jeQndZbeFl8TAxy8tKxkLFLUuO",
  "version": "1",
  "turboVersion": "2.0.7-canary.1",
  "monorepo": true,
  "globalCacheInputs": {
    "rootKey": "I can’t see ya, but I know you’re here",
    "files": {},
    "hashOfExternalDependencies": "cab80e071f9bd92a",
    "hashOfInternalDependencies": "",
    "environmentVariables": {
      "specified": {
        "env": [],
        "passThroughEnv": null
      },
      "configured": [],
      "inferred": [],
      "passthrough": null
    },
    "engines": null
  },
  "packages": [
    "//",
    "foo"
  ],
  "envMode": "strict",
  "frameworkInference": true,
  "tasks": [
    {
      "taskId": "bar#build",
      "task": "build",
      "package": "bar",
      "hash": "3565ae58cc7f544e",
      "inputs": {
        "bar": "6332e25f7949f16f81f6c45edcc296cd78efd9b4",
        "package.json": "cf49a1974a1e6505e4d3edfb74cc0c317386464d",
        "turbo.json": "95960709b10d9e4fa91f251ffafe0819e1a4e681"
      },
      "hashOfExternalDependencies": "7d7cdc66c0570bf8",
      "cache": {
        "local": false,
        "remote": false,
        "status": "MISS",
        "timeSaved": 0
      },
      "command": "echo 'build bar'",
      "cliArguments": [],
      "outputs": null,
      "excludedOutputs": null,
      "logFile": "packages/bar/.turbo/turbo-build.log",
      "directory": "packages/bar",
      "dependencies": [],
      "dependents": [
        "foo#build"
      ],
      "resolvedTaskDefinition": {
        "outputs": [],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [],
        "outputLogs": "full",
        "persistent": false,
        "env": [],
        "passThroughEnv": null,
        "interactive": false
      },
      "expandedOutputs": [],
      "framework": "",
      "envMode": "strict",
      "environmentVariables": {
        "specified": {
          "env": [],
          "passThroughEnv": null
        },
        "configured": [],
        "inferred": [],
        "passthrough": null
      }
    },
    {
      "taskId": "foo#build",
      "task": "build",
      "package": "foo",
      "hash": "c4f79d52425be1b6",
      "inputs": {
        "../../.root/foo": "53ed2f08cfcc441eeae915cdebc5225d64c1ad83",
        "package.json": "f9f7da0a9a715a67de735decd64899af44c62f2b",
        "test": "e69de29bb2d1d6434b8b29ae775ad8c2e48c5391",
        "turbo.json": "293cba7f07f40acf360f85dbe4b38c18b5d614d8"
      },
      "hashOfExternalDependencies": "7d7cdc66c0570bf8",
      "cache": {
        "local": false,
        "remote": false,
        "status": "MISS",
        "timeSaved": 0
      },
      "command": "echo 'build foo'",
      "cliArguments": [],
      "outputs": null,
      "excludedOutputs": null,
      "logFile": "packages/foo/.turbo/turbo-build.log",
      "directory": "packages/foo",
      "dependencies": [
        "bar#build"
      ],
      "dependents": [],
      "resolvedTaskDefinition": {
        "outputs": [],
        "cache": true,
        "dependsOn": [
          "^build"
        ],
        "inputs": [
          "$TURBO_DEFAULT$",
          "../../.root/*"
        ],
        "outputLogs": "full",
        "persistent": false,
        "env": [],
        "passThroughEnv": null,
        "interactive": false
      },
      "expandedOutputs": [],
      "framework": "",
      "envMode": "strict",
      "environmentVariables": {
        "specified": {
          "env": [],
          "passThroughEnv": null
        },
        "configured": [],
        "inferred": [],
        "passthrough": null
      }
    }
  ],
  "user": "",
  "scm": {
    "type": "git",
    "sha": "5c9071ab44e2ec2f1266f81beb80c559dca9fd06",
    "branch": "master"
  }
}

Above output is expected since I've made a change in foo package's test file. You can verify inputs consist of "../../.root/foo": "53ed2f08cfcc441eeae915cdebc5225d64c1ad83", from root input. I would expect my prior run to have the same output as this, but it's not.

[psychobolt]

Just nothing one work around at the moment is adding .root/foo to global dependencies. However that seems to affect all tasks. The inputs option supposedly is used to affect one task for a particular package, which is my use case. So to circumvent that issue, my CI had a job to copy one or more root files to a package to cache bust. This is particular to setups that rely on root meta files such as .yarn/versions/*.xml to trigger specific CI tests or package releases.

[chris-olszewski]

@psychobolt Apologies for misunderstanding the issue initially. There are two ways to possibly go about working around this shortcoming:

• Add .root/foo to global dependencies as you have already done. This will now factor into every task so anytime .root/foo changes, every task will be invalidated.
• Do not use a source control filter in your setup and instead rely on caching: e.g. --filter='./packages/*'. If .root/foo does not change, then the task that declares on .root/foo will be a cache hit.

The crux of the issue is that our --filter syntax targets packages and not tasks. This leaves us in a weird place when declaring inputs for a task that reside outside of a package: the inputs for foo#build changed, but the contents of foo did not. I realize this isn't a satisfactory answer, but I want to make it clear that offering a proper solution to this will probably require significant changes to either how --filter works or what is considered to be part of a package.