Lexical Structure

May 30, 2017 · View on GitHub

Scripts

A script is an ordered sequence of characters. Typically, a script has a one-to-one correspondence with a file in a file system, but this correspondence is not required.

Conceptually, a script is translated using the following steps:

  1. Transformation, which converts a script from a particular character repertoire and encoding scheme into a sequence of 8-bit characters.

  2. Lexical analysis, which translates a stream of input characters into a stream of tokens. 

  3. Syntactic analysis, which translates the stream of tokens into executable code.

Conforming implementations must accept scripts encoded with the UTF-8 encoding form (as defined by the Unicode standard), and transform them into a sequence of characters. Implementations can choose to accept and transform additional character encoding schemes.

Grammars

This specification shows the syntax of the Hack programming language using two grammars. The lexical grammar defines how source characters are combined to form white space, comments, and tokens. The syntactic grammar defines how the resulting tokens are combined to form Hack programs.

The grammars are presented using grammar productions, with each one defining a non-terminal symbol and the possible expansions of that non-terminal symbol into sequences of non-terminal or terminal symbols. In productions, non-terminal symbols are shown in slanted type like this, and terminal symbols are shown in a fixed-width font like this

The first line of a grammar production is the name of the non-terminal symbol being defined, followed by one colon for a syntactic grammar production, and two colons for a lexical grammar production. Each successive indented line contains a possible expansion of the non-terminal given as a sequence of non-terminal or terminal symbols. For example, the production:

  single-line-comment::
    // input-charactersopt
    #  input-charactersopt

defines the lexical grammar production single-line-comment as being the terminals // or #, followed by an optional input-characters. Each expansion is listed on a separate line.

Although alternatives are usually listed on separate lines, when there is a large number, the shorthand phrase “one of” may precede a list of expansions given on a single line. For example,

  hexadecimal-digit:: one of
    0   1   2   3   4   5   6   7   8   9
    a   b   c   d   e   f
    A   B   C   D   E   F

Grammar ambiguities

In general, lexical processing results in the creation of the longest possible lexical element. However, in certain cases involving generic type specifiers (and array specifiers, which use generic type notation), this is not the case. For example, in type specifiers like X<Y<int>> and array<array<int>>, the source characters > and > are treated as two separate tokens rather than as the right-shift operator >>.

Lexical Analysis

General

The production input-file is the root of the lexical structure for a script. Each script must conform to this production.

Syntax

  input-file::
    input-element
    input-file   input-element
  input-element::
    comment
    white-space
    token

Defined elsewhere

Semantics:

The basic elements of a script are comments, white space, and tokens.

The lexical processing of a script involves the reduction of that script into a sequence of tokens that becomes the input to the syntactic analysis. Tokens can be separated by white space and delimited comments.

Apart from the exception noted in §§, lexical processing always results in the creation of the longest possible lexical element. (For example, $a+++++$b must be parsed as $a++ ++ +$b, which syntactically is invalid.)

Comments

Two forms of comments are supported: delimited comments and single-line comments

Syntax

  comment::
    single-line-comment
    delimited-comment

  single-line-comment::
    //   input-charactersopt
    #    input-charactersopt

  input-characters::
    input-character
    input-characters   input-character

  input-character::
    Any source character except new-line

  new-line::
    Carriage-return character (U+000D)
    Line-feed character (U+000A)
    Carriage-return character (U+000D) followed by line-feed character (U+000A)

  delimited-comment::
    /*   No characters or any source character sequence except /*   */

Semantics

Except within a string literal or a comment, the characters /* start a delimited comment, which ends with the characters */. Except within a string literal or a comment, the characters // or # start a single-line comment, which ends with a new line. That new line is not part of the comment. However, if the single-line comment is the last source element in an embedded script, the trailing new line can be omitted. (Note: this allows for uses like <?hh ... // ... ?>.)

A delimited comment can occur in any place in a script in which white space can occur. (For example; /*...*/$c/*...*/=/*...*/567/*...*/;/*...*/ is parsed as $c=567;, and $k = $i+++/*...*/++$j; is parsed as $k = $i+++ ++$j;.)

A number of single-line, so-called special comments are recognized by a conforming implementation; they are:

where horizontal white space is permitted between the // and the start of the comment text.

Implementation Notes

During tokenizing, an implementation can treat a delimited comment as though it was white space.

White Space

White space consists of an arbitrary combination of one or more new-line, space, and horizontal tab characters.

Syntax

  white-space::
    white-space-character
    white-space   white-space-character

  white-space-character::
    new-line
    Space character (U+0020)
    Horizontal-tab character (U+0009)

Defined elsewhere

Semantics

The space and horizontal tab characters are considered horizontal white-space characters.

Tokens

General

There are several kinds of source tokens:

Syntax

  token::
    variable-name
    name
    keyword
    literal
    operator-or-punctuator

Defined elsewhere

Names

Syntax

  variable-name::
    $   name

  name::
    name-nondigit
    name   name-nondigit
    name   digit

  name-nondigit::
    nondigit
    one of the characters U+007f–U+00ff

  nondigit:: one of
    _
    a   b   c   d   e   f   g   h   i   j   k   l   m
    n   o   p   q   r   s   t   u   v   w   x   y   z
    A   B   C   D   E   F   G   H   I   J   K   L   M
    N   O   P   Q   R   S   T   U   V   W   X   Y   Z

Defined elsewhere

Semantics:

Names are used to identify the following: constants, variables, labels, enumerated types, functions, classes, class members, interfaces, traits, type aliases, namespaces, names in heredoc comments and nowdoc comments; and attributes.

A name begins with an underscore (_), name-nondigit, or extended name character in the range U+007f–U+00ff. Subsequent characters can also include digits. A variable name is a name with a leading dollar ($).

The name $_, referred to as the placeholder variable, is reserved for use in the (list intrinsic) and the (foreach statement). This name does not actually designate any storage; instead, it’s an indication that the value that would otherwise be stored in a variable in that context, is ignored.

Unless stated otherwise (§§, §§, §§, §§), names are case-sensitive, and every character in a name is significant.

Function and method names beginning with two underscores (__) are reserved by the Hack language.

Examples

const int MAX_VALUE = 100;
public function getData(): array<string> { ... }
class Point { ... }
interface ICollection { ... }

Implementation Notes

An implementation is discouraged from placing arbitrary restrictions on name length or length of significance.

Keywords

A keyword is a name-like sequence of characters that is reserved, and cannot be used as a name.

Syntax

  keyword:: one of
    abstract   arraykey   as   async   await   break   case   catch   class   classname   clone   const   continue   default   do
    echo   else   elseif   enum   extends   final   finally   for   foreach   function   if   implements
    instanceof   insteadof   interface   mixed   namespace   new   newtype   noreturn   num   parent   private
    protected   public   require   require_once   return   self   shape   static   switch   throw   trait   try
    tuple   type   use   while   yield

Semantics

Keywords are case-sensitive.

Note: Strictly speaking, false, null, and true are not keywords; however, they do have predefined meanings, and can be thought of as keywords. Likewise for the names of the intrinsics.

Literals

General

The source code representation of a value is called a literal.

Syntax

  literal::
    boolean-literal
    integer-literal
    floating-literal
    string-literal
    null-literal

Defined elsewhere

Boolean Literals

Syntax

  boolean-literal::
    true
    false

Semantics

The type of a boolean-literal is bool. The values true and false represent the Boolean values True and False, respectively.

Examples

$done = false;
computeValues($table, true);
Integer Literals

Syntax

  integer-literal::
    decimal-literal
    octal-literal
    hexadecimal-literal
    binary-literal

    decimal-literal::
      nonzero-digit
      decimal-literal   digit

    octal-literal::
      0
      octal-literal   octal-digit

    hexadecimal-literal::
      hexadecimal-prefix   hexadecimal-digit
      hexadecimal-literal   hexadecimal-digit

    hexadecimal-prefix:: one of
      0x  0X

    binary-literal::
      binary-prefix   binary-digit
      binary-literal   binary-digit

    binary-prefix:: one of
      0b  0B

    digit:: one of
      0  1  2  3  4  5  6  7  8  9

    nonzero-digit:: one of
      1  2  3  4  5  6  7  8  9

    octal-digit:: one of
      0  1  2  3  4  5  6  7

    hexadecimal-digit:: one of
      0  1  2  3  4  5  6  7  8  9
            a  b  c  d  e  f
            A  B  C  D  E  F

    binary-digit:: one of
        0  1

Constraints

The value of an integer literal must be representable by its type.

Semantics

The value of a decimal integer literal is computed using base 10; that of an octal integer literal, base 8; that of a hexadecimal integer literal, base 16; and that of a binary integer literal, base 2.

The type of an integer-literal is int.

Using a twos-complement system, can the smallest negative value (-9223372036854775808 for 64 bits) be represented as a decimal integer literal? No. Consider the expression -5. This is made up of two tokens: a unary minus followed by the integer literal 5. As such, there is no such thing as a negative-valued decimal integer literal in Hack. Instead, there is the non-negative value, which is then negated. Literals written using hexadecimal, octal, or binary notations are considered to have non-negative values.

Examples

$count = 10      // decimal 10

0b101010 >> 4    // binary 101010 and decimal 4

0XAF << 023      // hexadecimal AF and octal 23
Floating-Point Literals

Syntax

  floating-literal::
    fractional-literal   exponent-partopt
    digit-sequence   exponent-part

  fractional-literal::
    digit-sequenceopt . digit-sequence
    digit-sequence .

  exponent-part::
    e  signopt   digit-sequence
    E  signopt   digit-sequence

  sign:: one of
    +  -

  digit-sequence::
    digit
    digit-sequence   digit

Defined elsewhere

Constraints

The value of a floating-point literal must be representable by its type.

Semantics

The type of a floating-literal is float.

The constants INF and NAN provide access to the floating- point values for infinity and Not-a-Number, respectively.

Examples

$values = array(1.23, 3e12, 543.678E-23);
String Literals

Syntax

  string-literal::
    single-quoted-string-literal
    double-quoted-string-literal
    heredoc-string-literal
    nowdoc-string-literal

Defined elsewhere

Note: By conventional standards, calling heredoc-string-literals and nowdoc-string-literals) literals is a stretch, as each is hardly a single token. And given the variable substitution permitted in double-quote-string-literals, they are not really literals either.

Semantics

A string literal is a sequence of zero or more characters delimited in some fashion. The delimiters are not part of the literal's content.

The type of a string literal is string.

Single-Quoted String Literals

Syntax

  single-quoted-string-literal::
    ' sq-char-sequenceopt  '

  sq-char-sequence::
    sq-char
    sq-char-sequence   sq-char

  sq-char::
    sq-escape-sequence
    \opt   any member of the source character set except single-quote (') or backslash (\)

  sq-escape-sequence:: one of
    \'  \\

Semantics

A single-quoted string literal is a string literal delimited by single-quotes ('). The literal can contain any source character except single-quote (') and backslash (\), which can only be represented by their corresponding escape sequence.

A single-quoted string literal is a c-constant.

Examples

'This text is taken verbatim'

'Can embed a single quote (\') and a backslash (\\) like this'
Double-Quoted String Literals

Syntax

  double-quoted-string-literal::
    " dq-char-sequenceopt  "

  dq-char-sequence::
    dq-char
    dq-char-sequence   dq-char

  dq-char::
    dq-escape-sequence
    any member of the source character set except double-quote (") or backslash (\)
    \  any member of the source character set except "\$efnrtvxX or octal-digit

  dq-escape-sequence::
    dq-simple-escape-sequence
    dq-octal-escape-sequence
    dq-hexadecimal-escape-sequence
    dq-unicode-escape-sequence

  dq-simple-escape-sequence:: one of
    \"   \\   \$   \e   \f   \n   \r   \t   \v

  dq-octal-escape-sequence::
    \   octal-digit
    \   octal-digit   octal-digit
    \   octal-digit   octal-digit   octal-digit

  dq-hexadecimal-escape-sequence::
    \x  hexadecimal-digit   hexadecimal-digitopt
    \X  hexadecimal-digit   hexadecimal-digitopt

  dq-unicode-escape-sequence::
    \u{  codepoint-digits  }

  codepoint-digits::
     hexadecimal-digit
     hexadecimal-digit   codepoint-digits

Defined elsewhere

Semantics

A double-quoted string literal is a string literal delimited by double-quotes ("). The literal can contain any source character except double-quote (") and backslash (\), which can only be represented by their corresponding escape sequence. Certain other (and sometimes non-printable) characters can also be expressed as escape sequences.

An escape sequence represents a single-character encoding, as described in the table below:

Escape sequenceCharacter nameUnicode character
$Dollar signU+0024
"Double quoteU+0022
\BackslashU+005C
\eEscapeU+001B
\fForm feedU+000C
\nNew lineU+000A
\rCarriage ReturnU+000D
\tHorizontal TabU+0009
\vVertical TabU+000B
\ooo1–3-digit octal digit value ooo
\xhh or \Xhh1–2-digit hexadecimal digit value hhU+00hh
\u{xxxxxx}UTF-8 encoding of Unicode codepoint U+xxxxxxU+xxxxxx

Within a double-quoted string literal, except when recognized as the start of an escape sequence, a backslash (\) is retained verbatim.

Within a double-quoted string literal a dollar ($) character not escaped by a backslash (\) is handled using a variable substitution rules described below.

The \u{xxxxxx} escape sequence produces the UTF-8 encoding of the Unicode codepoint with the hexadecimal number specified within the curly braces. Implementations MUST NOT allow Unicode codepoints beyond U+10FFFF as this is outside the range UTF-8 can encode (see RFC 3629). If a codepoint larger than U+10FFFF is specified, implementations MUST error. Implementations MUST pass through \u verbatim and not interpret it as an escape sequence if it is not followed by an opening {, but if it is, implementations MUST produce an error if there is no terminating } or the contents are not a valid codepoint. Implementations MUST support leading zeroes, but MUST NOT support leading or trailing whitespace for the codepoint between the opening and terminating braces. Implementations MUST allow Unicode codepoints that are not Unicode scalar values, such as high and low surrogates.

A Unicode escape sequence cannot be created by variable substitution. For example, given $v = "41", "\u{$v}" results in "\u41", a string of length 4, while "\u{0$v}" and "\u{{$v}}" contain ill-formed Unicode escape sequences.

Variable substitution

The variable substitution accepts the following syntax:

    string-variable::
        variable-name   offset-or-propertyopt

    offset-or-property::
        offset-in-string
        property-in-string

    offset-in-string::
        [   name   ]
        [   variable-name   ]
        [   integer-literal   ]

    property-in-string::
        ->   name

Defined elsewhere

After the variable defined by the syntax above is evaluated, its value is converted to string according to the rules of string conversion and is substituted into the string in place of the variable substitution expression.

Subscript or property access defined by offset-in-string and property-in-string is resolved according to the rules of the subscript operator and member selection operator respectively. The exception is that name inside offset-in-string is interpreted as a string literal even if it is not quoted.

If the character sequence following the $ does not parse as name then the $ character is instead interpreted verbatim and no variable substitution is performed.

A double-quoted string literal is a c-constant if it does not contain any variable substitution.

Examples

$x = 123;
echo ">\$x.$x"."<"; // → >$x.123<
// -----------------------------------------
$colors = array("red", "white", "blue");
$index = 2;
echo "\$colors[$index] contains >$colors[$index]<\n";
  // → $colors[2] contains >blue<
// -----------------------------------------
class C {
    public $p1 = 2;
}
$myC = new C();
echo "\$myC->p1 = >$myC->p1<\n";  // → $myC->p1 = >2<
Heredoc String Literals

Syntax

  heredoc-string-literal::
    <<<  hd-start-identifier   new-line   hd-char-sequenceopt  new-line hd-end-identifier  ;opt   new-line

  hd-start-identifier::
    name

  hd-end-identifier::
    name

  hd-char-sequence::
    hd-char
    hd-char-sequence   hd-char

  hd-char::
    hd-escape-sequence
    any member of the source character set except backslash (\)
    \  any member of the source character set except \$efnrtvxX or
octal-digit

  hd-escape-sequence::
    hd-simple-escape-sequence
    dq-octal-escape-sequence
    dq-hexadecimal-escape-sequence
    dq-unicode-escape-sequence

  hd-simple-escape-sequence:: one of
    \\   \$   \e   \f   \n   \r   \t   \v

Defined elsewhere

Constraints

The start and end identifier must be the same. Only horizontal white space is permitted between <<< and the start identifier. No white space is permitted between the start identifier and the new-line that follows. No white space is permitted between the new-line and the end identifier that follows. Except for an optional semicolon (;), no characters—not even comments or white space—are permitted between the end identifier and the new-line that terminates that source line.

Semantics

A heredoc string literal is a string literal delimited by "<<< name" and "name". The literal can contain any source character. Certain other (and sometimes non-printable) characters can also be expressed as escape sequences.

A heredoc literal supports variable substitution as defined for double-quoted string literals.

A heredoc string literal is a c-constant if it does not contain any variable substitution.

Examples

$v = 123;
$s = <<<    ID
S'o'me "\"t e\txt; \$v = $v"
Some more text
ID;
echo ">$s<";
→ >S'o'me "\"t e  xt; $v = 123"
Some more text<
Nowdoc String Literals

Syntax

  nowdoc-string-literal::
    <<<  '  hd-start-identifier  '  new-line  hd-char-sequenceopt   new-line hd-end-identifier  ;opt   new-line

Defined elsewhere

Constraints

No white space is permitted between the start identifier and its enclosing single quotes ('). See also §§.

Semantics

A nowdoc string literal looks like a heredoc string literal except that in the former the start identifier name is enclosed in single quotes ('). The two forms of string literal have the same semantics and constraints except that a nowdoc string literal is not subject to variable substitution.

A nowdoc string literal is a c-constant.

Examples

$v = 123;
$s = <<<    'ID'
S'o'me "\"t e\txt; \$v = $v"
Some more text
ID;
echo ">$s<\n\n";
→ >S'o'me "\"t e\txt; \$v = $v"
Some more text<
The Null Literal
  null-literal::
    null

Semantics

There is one null-literal value, null.

A null-literal has the null type.

Operators and Punctuators

Syntax

  operator-or-punctuator:: one of
    [   ]    (   )   {    }   .   ->   ++   --   **   *   +   -   ~   !
    $   /   %   <<   >>   <   >   <=   >=   ==   ===   !=   !==   ^   |
    &   &&   ||   ?   ??   :   ; =   **=   *=   /=   %=   +=   -=   .=   <<=
    >>=   &=   ^=   |=   ,   @   ::   =>   ==>   ?->   \   ...    |>   $$

Semantics

Operators and punctuators are symbols that have independent syntactic and semantic significance. Operators are used in expressions to describe operations involving one or more operands, and that yield a resulting value, produce a side effect, or some combination thereof. Punctuators are used for grouping and separating.