microsoft / fluentui-token-pipeline

The Fluent UI token pipeline generates source code for Fluent UI libraries.

Fluent UI token pipeline

The Fluent UI token pipeline transforms JSON files describing design tokens into source code for eventual use in the Fluent UI libraries.

This tool was originally designed to process files in Microsoft's proprietary design token format. Our design token files are now in the W3C Design Token Community Group draft standard format, so this tool is no longer required to work with them.

Do not add new dependencies on this tool. Over time, it is expected that Fluent UI teams will transition from using this tool to using standard open source solutions for consuming design tokens. But while this tool is deprecated, it is still fully functional and will remain so for the foreseeable future. And since this tool can also read DTCG format JSON, existing code written when continues to work even though we no longer use this proprietary format.

• Documentation site
• Source code on GitHub
• @fluentui/token-pipeline on NPM
• Fluent UI design tokens on GitHub 🔒 with links to packages of transformed tokens

---

The legal stuff

The actual token values in the src/demo folder are just examples, and they do not represent an evolution of Microsoft's design systems.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

fluentui-token-pipeline's Issues

Fluent UI React export

Still deciding on what this will look like, but to start with:

• Produce ES Module of global tokens

Investigate Adobe DSP

Adobe Design System Packages (DSP) might be an interesting long-term tie-in for this project. Our token hierarchy is massively larger and more component-focused than other projects that use Style Dictionary though, such as the Adobe Spectrum DSP which has fewer than 250 tokens, and with our more cross-platform plans, it might not make a lot of sense.

Turn the readme into a docs site

We've got a wiki set up! The readme is excessively long, so technical reference content should move into a docs folder and get built as a GitHub Pages site, and other details should move to the wiki.

Set-alias token naming in global JSON

We've continued receiving feedback from folks that Set is confusing - let's examine this & maybe change this naming?

WinUI export: alternate token names

WinUI already has a variety of tokens that they refer to as lightweight styling resources, and most of these match 1:1 with a token defined in the master tokens JSON.

For example, WinUI's ButtonBackgroundPointerOver is equivalent to a FluentUI token Button.Root.Fill.Color.Hover, and when exported for WinUI, the pipeline creates a SolidColorBrush resource named ButtonRootFillColorHover. WinUI can then define ButtonBackgroundPointerOver in its own code to be equal to ButtonRootFillColorHover, but performance could be somewhat better if that extra layer weren't present at all. To achieve that, the WinUI exporter could just export the token with the WinUI-specific name. That could be managed in one of two ways:

1. In the token JSON, add it as an additional attribute:

{ "Button": { "Root": { "Fill": { "Color": { "Hover": { "value": "#c0c0c0", "winUIKey": "ButtonBackgroundPointerOver" }}}}}}

2. In a separate mapping file only used by the WinUI exporter:

{ "Button.Root.Fill.Color.Hover": "ButtonBackgroundPointerOver" }

I think that the second option makes more sense: it keeps lots of platform-specific properties out of the mostly-cross-platform JSON file. But more importantly than that, Button.Root.Fill.Color.Hover would typically be defined through a set assigned to Button.Root.Fill.Color so there wouldn't even be a Hover node in the JSON to add a property to. Would we then have to assign a winUIKeys property to Color with one mapping for Hover, one for Pressed, and so on? The main upsides of the first option are avoiding a second file, and reducing the possibility of typos and the mapping file getting out-of-date.

Letter spacing tokens

JSON format value: -0.02

• W3C: -0.02rem (add rem)
• Figma Tokens: -2% (multiply by 100 and add %)
• CSS: -0.02em (add em for letter-spacing)
• Xaml: -20 (multiply by 1000 and round to an int for CharacterSpacing)

Swift uses fixed pt values for letter spacing, but all other platforms use em/%, and we don't know the corresponding font size for any letter spacing token, so it isn't possible to convert the values. For now, though, we don't actually require being able to use the same letter spacing token on other platforms, so we can just conveniently ignore that problem for now. When we switch to the W3C format it won't matter because that format supports both px and rem values anyway (though it should really be em, not rem...).

Set up a basic CI workflow

Finalize where JSON lives & its governance

We want to maintain the JSON as a single source of truth for platform-agnostic tokens. Right now this is the fluentui.json file - finalize where the JSON lives and ensure wherever it is, it can take PRs to update it as design system maintainers will need to do this as design language evolves either through the Figma Theme Designer Plugin or directly in this repo.

HTML export: set naming

The HTML export feature does a decent job at picking useful names for sets, but since sets aren't a distinct thing in the JSON, it could do better in some cases. For example, if there's a set of tokens that all follow the format Set-Neutral-Fill-Color-_____, the HTML export will call it Set-Neutral rather than Set-Neutral-Fill-Color. When it identifies a set, it should identify the longest string that still matches all defined tokens in the set.

In fluentui-html.js, look for the fluentui/html/reference formatter:

header = (header || "") + `<h2>Set-${thisProp.path[1]}</h2>\n\n`

Does the project take into consideration Focus States?

fluentui-token-pipeline/src/pipeline/utils.ts

Lines 11 to 19 in aac26f7

	const orderOfInteractionStates =
	{
	"Rest": 1,
	"Hover": 2,
	"Pressed": 3,
	"Disabled": 4,
	length: 4,
	}

The typical Interaction States are included here, but Focus is not included.

Perhaps it is treated differently than the other states, especially for Text and Form fields, but Windows in particular has a Focus Visual that is applied to controls when tabbing.

React Web and React Native Win32: Export a simple flat JSON

React Web and React Native Win32 would like to start with a simple flat JSON structure, and then will provide comments based on that. That should be easy enough to produce.

Rename "Root" in control tokens to "Base"

Update JSON to change Root to Backplate

Shadow exports

The latest plans for shadows suggest that they'll be defined algorithmically from an elevation value and a foreground luminance value. This may not be possible for the pipeline to export at all. If shadow values were stored that way in the JSON, what would we do with them for, for example, CSS?

• Export them as raw values such as --myshadow-elevation: 3; and then access the value of that CSS variable in JavaScript with something like document.documentElement.getPropertyValue("--myshadow-elevation")?
• Export shadow values not as CSS variables, but in a separate JSON exported alongside the CSS variables that the website also imports? { "myshadow.elevation": 3 }
• Simplify our shadow story so that we can deterministically export a single string value that can be assigned to box-shadow?
• Leave shadows out of the token structure entirely?

The latter two are clearly much simpler than the others, but limits what can be done with algorithmic shadows.

ThemeShadows in WinUI are already defined simply as an elevation value in Z-pixels, not specific offsets and colors and blur radii like in CSS.

Themes and dark mode

We don't currently have a solution designed for how we'll support different themes and dark mode.

Each platform export will also need to handle this differently: for example, CSS might use a single file with @media queries for each theme, or top-level classes that redefine each of the CSS variables coming from tokens. But iOS might include per-theme values all the way on the leaf nodes: for example, they might want ButtonRootFillColorHover to be exported as ButtonRootFillColorHover.Dark, ButtonRootFillColorHover.Light, and so on.

Entire themes might also be platform-specific as well.

RGBA hex in gradients doesn't work

Reported:
Skipping from target: Set.Control.Fill.Color.Rest, because we're not sure what "#00000033" is

WinUI export: acronyms

The WinUI exporter produces ugly names for tokens with two-letter acronyms. For example, HCOutline in the original token name is converted into HcOutline in the WinUI export.

The code is getNameForWinUI in fluentui-winui.js.

WinUI lightweight styling names

I'm going to be getting an Excel spreadsheet mapping the WinUI lightweight styling resource names to our standardized token names. I'll need to build a small console app that reads that and writes out token JSON with some platform overrides that can be merged with other token JSON from the plugin:

{
  "Meta": { "FluentUITokensVersion": 0 },
  "platform" : { "winui": {
    "Button": { "Base": { "Fill": { "Color": { "fullName": "ButtonBackground" }}}},
    "etc": "etc"
  }}
}

W3C export

The W3C has a working group producing the Design Tokens Format Module. It's not stable yet and has a lot of open questions so we shouldn't shift our tooling over to that format anytime soon, BUT it should be trivial to add "W3C" as another export from the pipeline. Then someday in the future we could rip out things that aren't supported by the W3C system into a preprocessing step (hopefully by using the extensions node in their format), the same way that this codebase heavily preprocesses tokens before sending them through Style Dictionary, and then we could use the standard W3C form as our format.

Token pipeline output "ERROR: Unable to determine data type based on token name "Global.Size.20"

Output from running npx transform-tokens --platform reactnative --in src/global.ios.json --in src/typography.ios.json --in src/light.ios.json --in src/shadow.json --out tokens/reactnative/ios/light:

lynns-macbook-pro:fluentui-design-tokens lyzhan$ npx transform-tokens --platform reactnative --in src/global.ios.json --in src/typography.ios.json --in src/light.ios.json --in src/shadow.json --out tokens/reactnative/ios/light

reactnative
ERROR: Unable to determine data type based on token name "Global.Size.20".
ERROR: Unable to determine data type based on token name "Global.Size.40".
ERROR: Unable to determine data type based on token name "Global.Size.60".
ERROR: Unable to determine data type based on token name "Global.Size.80".
ERROR: Unable to determine data type based on token name "Global.Size.100".
ERROR: Unable to determine data type based on token name "Global.Size.120".
ERROR: Unable to determine data type based on token name "Global.Size.160".
ERROR: Unable to determine data type based on token name "Global.Size.200".
ERROR: Unable to determine data type based on token name "Global.Size.240".
ERROR: Unable to determine data type based on token name "Global.Size.280".
ERROR: Unable to determine data type based on token name "Global.Size.320".
ERROR: Unable to determine data type based on token name "Global.Size.360".
ERROR: Unable to determine data type based on token name "Global.Size.400".
ERROR: Unable to determine data type based on token name "Global.Size.480".
ERROR: Unable to determine data type based on token name "Global.Size.560".
ERROR: Unable to determine data type based on token name "Global.Size.640".
ERROR: Unable to determine data type based on token name "Global.Size.720".
ERROR: Unable to determine data type based on token name "Global.Size.800".
ERROR: Unable to determine data type based on token name "Global.Size.1200".
ERROR: Unable to determine data type based on token name "Global.Size.None".
✔︎  tokens/reactnative/ios/light/tokens-global.json
✔︎  tokens/reactnative/ios/light/tokens-shadow.json
✔︎  tokens/reactnative/ios/light/tokens-aliases.json
✔︎  tokens/reactnative/ios/light/tokens-controls.json

This is on the latest token package version 0.37.0, but issue was introduced by https://github.com/microsoft/fluentui-design-tokens/pull/36 when we renamed "Spacing" to "Size". Seems like some of the error handling needs to be updated

Generate Penpot file

Generate a Penpot file for use with Penpot.

These files are contributed to the penpot-files repository:
https://github.com/penpot/penpot-files

Issue:
penpot/penpot-files#13

Explicit data types

The group is leaning toward relaxing some of the restrictions that are in place about how tokens are named. It's a requirement that the pipeline have some way of deterministically knowing what data type a token is—for example, if one token has the value "transparent", it needs to know that it refers to a color and not a font family, so that it produces the correct output. Today, the code in fluentui-shared.js handles this by analyzing the full name of the token, and our system's rules make it straightforward. That's also the way that a base Style Dictionary installation does it (but they have their own naming scheme that we don't use, CTI).

But, if the pipeline were to support other, potentially wildly different, design systems, that would no longer be true, so we'd need an alternate way of specifying that information. I'd recommend just adding a type property right next to value:

"Global": {
  "Color": {
    "Red": { "value": "#ff0000", "type": "color" }
  }
}

It would add a lot of extra information to the JSON, but it's straightforward, and would be simple enough to use. The existing code in fluentui-shared.js would just need to check that type property instead of looking through the full name. The rest of the formatters and export code would need no changes, or at most minimal changes.

Update on WinUI 2/3 token availability

Hi @TravisSpomer, can you provide an update on the availability of WinUI 2/3 tokens for desktop development? I was doing some preliminary planning to translate between their styles and these intermediate tokens but don't want to duplicate existing efforts if any.

Shorter names

On platforms like the web where names of tokens can affect download size, shorter token names are ideal. There are a few opportunities for shortening things.

• Remove Set from exported token names—the Set heading is an organizational tool, but doesn't need to be present in the exported token name.

• Remove Root from exported token names—the Root node under control tokens specifies properties for the control's top-level background element. For example, MyControl.Root.Stroke.Color might be used like this in a React component:

  const MyControl = () => <div style={{ borderColor: "var(--mycontrol-root-stroke-color)" }} />

The CSS variable name would still be unambiguous if it were named --mycontrol-stroke-color, since other stroke colors within the same control would have an extra word in them.

Fail the build on errors

Most errors in the token file are effectively just presented as warnings right now. Now that we're closer to putting this into production, we should make any errors fail the build so that packages aren't published without warnings that no one sees.

Add NPM packaging to the standalone JSON repo

Fluent UI React Web wants to consume tokens as an NPM package. I should add NPM packaging to my standalone JSON repo to show how this could be done.

Unlike a typical JavaScript library, that NPM package would just contain the build folder of fully-built files (CSS, JSON, etc.) and not the source code.

Fluent UI React Native export

System colors

I expect to need to add support for referencing system colors: at least the high contrast color set on Windows, and a couple on iOS. No hard requirements yet though.

Color ramp generation

I'm going to explore some different options for generating color ramps from a single base color.

Watch for token changes and reprocess

Back when the token JSON was in the same project as the pipeline itself, there was a watch mode that would reprocess the tokens whenever the JSON changed. That doesn't exist anymore in the CLI world, but it would still be useful for people "playing" with tokens and iterating.

I should add a new CLI flag --watch that would work like --watch flags in other CLIs, staying alive and reprocessing the file whenever it changes.

(The alternative would be for every token repo that uses the CLI to just set up its own watch task.)

Add Font Alignment to fluentui-shared.ts

When defining the alignment for type, being able to define the following:

"Alignment": {
  "Start": { "value": "left"},
  "Center": { "value": "center"},
  "End": { "value": "end"},
  "Justify": { "value": "justify"}
}

High contrast mode / system colors

In addition to general theming support (#9), high contrast mode will probably require a few global tokens that are system-defined high contrast colors rather than specific RGB values.

Stroke alignment

Tokens for stroke alignment:

Inner

• The most common alignment, for solid and transparent borders
• The stroke is drawn inside the edge of the control on top of the control's fill
• Figma: Inner stroke alignment
• XAML: BackgroundSizing="OuterBorderEdge"
• CSS: background-clip: border-box (default)

Outer

• The stroke is drawn outside the control's fill on top of the content behind the control
• Figma: Outer stroke alignment
• XAML: BackgroundSizing="InnerBorderEdge" (default)
• CSS: background-clip: padding-box (default)

(Neither property actually affects sizing of the element, either in layout or just visually—that would be border-box in CSS, with no WinUI equivalent.)

Something like:

ButtonPrimary-Box-Stroke-Alignment = Outer or Inner (default)

Fluent UI Web Components export

Fluent UI iOS export

Get onto NPM

Now that this is a standalone project with a CLI, I should get it onto NPM.

Tokens with sub-tokens

It's been proposed that we should support tokens that also have sub-tokens. Today, if you have Global.Color.Red and also specify Global.Color.Red.100, the latter is ignored. We could likely support and remove some of the assumptions in the existing code. It does, however, add complexity when aliasing. We would probably then declare that any token that has sub-tokens is itself not a set. Using { "aliasOf": "Global.Color.Red" } would always refer to the single Global.Color.Red token, and not any of the tokens beneath it.

There is not currently any scenario that requires this, but the plugin developers seem to want it; they may know of a scenario.

Token JSON schema

For people who are manually writing token JSON, it would be helpful to have a schema so that they get assistance from VS Code. (Or, some other way of making the process of editing tokens easier for people who don't want to use the Figma plugin.)

This is almost entirely already built out in src/pipeline/types.ts and would mostly just need to be converted to a JSON schema format.

JSON export: alias tokens only

Someone on the team needs token data for just the alias layer in a JSON format. I'll add that as a new export.

Gradient exports

• Support gradients with an arbitrary number of stops in any direction
  • Support the special case where the gradient stops are fixed pixel amounts instead of a percentage
  • Support colors being defined as token references instead of fixed values
• Export gradients in CSS
  • Fixed stops: linear-gradient() natively supports 4px as a stop definition
• Export gradients in WinUI
  • Fixed stops: LinearGradientBrush needs MappingMode="Absolute" and an appropriate RotateTransform
• Add documentation to the site

Platform-specific overrides

We need a way to have tokens that are only exported for specific platforms, and for some tokens to have different values on each platform. We'd want to use this sparingly, but it's going to be unavoidable in some cases. e.g.:

• There are platform-specific concepts like the Apple SegmentedControl (basically, their tabstrip) that we might want to restyle on Apple platforms, but not even export for Windows and CSS, since that control doesn't exist elsewhere.
• At the lower end of the corner radii spectrum we're consistent across platforms, but we want to slightly modify the larger radii on some platforms to fit better with the platform's visual style.

This would probably work just by adding some kind of platform node to certain tokens that would signify that all platforms other than the specified ones should ignore that node and its children. There could also be a way to specify the reverse, such as exceptPlatform. For example:

"SegmentedControl" : {
  "platform": [ "mac", "ios" ],
  "Root": {
    "Fill": { /* ... */ }
  }
  /* ... */
}

Is "Focus" or "Focused" treated as a State for either WinUI, HTML or other platforms which use Tab focusing?

fluentui-token-pipeline/src/pipeline/utils.ts

Lines 11 to 19 in aac26f7

	const orderOfInteractionStates =
	{
	"Rest": 1,
	"Hover": 2,
	"Pressed": 3,
	"Disabled": 4,
	length: 4,
	}

The typical Interaction States are included here, but Focus is not included.

Perhaps it is treated differently than the other states, especially for Text and Form fields, but Windows in particular has a Focus Visual that is applied to controls when tabbing.