Syntax Highlighting

Highlighting syntax is one of the key features expected of any editor. Editors typically have a few different solutions to choose from. Below is our view on how we expect editors to highlight OpenTofu code while using this language server.

Static Grammar

Highlighting OpenTofu language syntax via static grammar (such as TextMate) accurately may be challenging but brings more immediate value to the end user, since starting language server may take time. Also not all language clients may implement semantic token based highlighting.

HashiCorp maintains a set of grammars in https://github.com/hashicorp/syntax and we encourage you to use the available Terraform grammar as the primary way of highlighting the OpenTofu language. Once OpenTofu and Terraform syntaxes diverge to a point where we cannot circumvent the problem another way, we can consider forking or embedding the TextMate grammar file into one of our repositories. (For more details - https://github.com/opentofu/vscode-opentofu/issues/14)

Semantic Tokens

LSP (Language Server Protocol) 3.16 introduced language server-driven highlighting. This language server is better equipped to provide more contextual and accurate highlighting as it can parse the whole AST, unlike a TextMate grammar operating on a regex-basis.

LSP 3.17 does support use cases where semantic highlighting is the only way to highlight a file (through augmentsSyntaxTokens client capability). However in the context of the OpenTofu language we recommend semantic highlighting to be used as in addition to a static grammar - i.e. this server does not support augmentsSyntaxTokens: false mode and is not expected to be used in isolation to highlight configuration.

There are two main use cases we're targeting with semantic tokens.

Improving Accuracy

Regex-based grammars (like TextMate) operate on line-basis, which makes it difficult to accurately highlight certain parts of the syntax, for example nested blocks occurring in the OpenTofu language (as below).

terraform {
	required_providers {

	}
}

The OpenTofu language server can use the AST and other important context (such as OpenTofu version or provider schema) to fully understand the whole configuration and provide more accurate highlighting.

Custom Theme Support

Many default IDE themes are intended as general-purpose themes, highlighting token types, modifiers and scopes mappable to most languages. We recognize that theme authors would benefit from token types & modifiers which more accurately reflect the OpenTofu language.

LSP spec doesn't explicitly encourage defining custom token types or modifiers, however the default token types and modifiers which are part of the spec are not well suited to express all the different constructs of a DSL (Domain Specific Language), such as OpenTofu language. With that in mind we use the LSP client/server capability negotiation mechanism to provide the following custom token types & modifiers with fallback to the predefined ones.

Token Types

Primary token types are preferred if deemed supported by client per SemanticTokensClientCapabilities.TokenTypes, fallbacks are also only reported if client claim support (using the same capability).

Fallback types are chosen based on meaningful semantic mapping and default themes in VSCode.

Primary	Fallback
hcl-blockType	type
hcl-blockLabel	enumMember
hcl-attrName	property
hcl-bool	keyword
hcl-number	number
hcl-string	string
hcl-objectKey	parameter
hcl-mapKey	parameter
hcl-keyword	variable
hcl-referenceStep	variable
hcl-typeComplex	function
hcl-typePrimitive	keyword
hcl-functionName	function

Token Modifiers

Modifiers which do not have fallback are not reported at all if not received within SemanticTokensClientCapabilities.TokenModifiers (just like fallback modifier that isn't supported).

Primary	Fallback
hcl-dependent	defaultLibrary
opentofu-data
opentofu-locals
opentofu-module
opentofu-output
opentofu-provider
opentofu-resource
opentofu-provisioner
opentofu-connection
opentofu-variable
opentofu-terraform
opentofu-backend
opentofu-name
opentofu-type
opentofu-requiredProviders
opentofu-encryption