Consolidate Lexical Scoping APIs

[wycats]

As we developed the <template>, we accumulated a number of APIs to allow Ember to communicate the JavaScript lexical scope to Glimmer.

We've ultimately landed on an approach in which the consumer of @glimmer/compiler provides a lexicalScope option (lexicalScope: (variable: string) => boolean). Glimmer uses this option to determine whether an unbound name used by a template refers to a lexical JavaScript value or, alternatively, is a keyword candidate.

Note

This approach ensures that in-scope variables always have the same behavior, regardless of whether they are JavaScript variables or Handlebars block params.

We should consolidate and remove remnants of these interim, out-of-date approaches.

@glimmer/compiler Public API

The Options passed in to precompile have gotten really hairy:

interface Options {

  /**
   * A function that generates a unique id for a template.
   *
   * The default is a function that generates a UUID, which ultimately gets attached to the
   * SerializedTemplate.
   *
   * ?? How is this used ??
   */
  id?: (src: string) => string | null;

  /**
   * Determines whether the template should be compiled in strict mode.
   *
   * Defaults to `false`, which compiles free variables into contextual
   * classic-mode resolution, which requires a runtime resolver.
   *
   * One small wrinkle: The Glimmer resolver exposes a `lookupBuiltinHelper`
   * hook which supports dynamic keywords provided by the embedding environment.
   *
   * This hook is used even in strict mode, and really has nothing to do with
   * the rest of the resolver, which allows templates to resolve application
   * code dynamically rather than statically at compile time.
   *
   * In my (Yehuda's) opinion, this is not the right way to model keywords, and
   * we should allow Ember to implement keywords via the compiler, but this is
   * a separate issue unrelated to strict mode.
   */
  strictMode?: boolean | undefined;

  /**
   * A list of variable names that exist in the JavaScript scope of the template.
   *
   * These locals are:
   *
   * - Together with `lexicalScope` (via `hasLexical`), used to determine whether
   *   the root scope "has" a variable (`hasBinding` on ASTv2's `BlockContext`)
   * - In classic mode, used to determine whether a variable should be resolved
   *   dynamically using the resolver. **I believe that this is a bug**, and
   *   that the code should use the same `hasBinding` path as other lookups.
   * - Exposed as the `blockParams` of the root Template node. **If we want to
   *   expose locals in this way, we will need to do so in a way that's
   *   compatible with `lexicalScope`.**
   */
  locals?: string[] | undefined;

  /**
   * A function that returns `true` if a variable is a lexical JavaScript
   * variable, and `false` otherwise.
   *
   * See `locals` above for more information. This should be the only API that
   * we need to expose to allow the consumer to communicate the JavaScript
   * lexical scope to Glimmer.
   *
   * In addition, the `locals` API could be implemented in terms of
   * `lexicalScope`, except for the way that `locals` are exposed as root
   * `blockParams`. We should decide what we want to do about that.
   */
  lexicalScope: (variable: string) => boolean;

 /**
  * Exposed as `moduleName` on the `SerializedBlock`. I don't believe that this
  * reliably works anymore. ??
  */
  meta?: {
    moduleName?: string | undefined;
  } | undefined;

  /**
   * If `emit.debugSymbols` is set to `true`, the name of lexical local variables
   * will be included in the wire format.
   *
   * This is used to allow debugging tools to display the component name as
   * originally written in the source code.
   */
  emit?:
    | {
        debugSymbols?: boolean;
      }
    | undefined;

  /**
   * A list of ASTv1 plugins to apply to the AST. This is still very much a public
   * API.
   *
   * The primary difference between Glimmer's view of plugin and Ember's is
   * that Ember extends this API to allow plugins to create bindings in the
   * compiled JavaScript module. We may want to make that explicit in some way
   * in the Glimmer API.
   */
  plugins?:
    | {
        ast?: ASTPluginBuilder[] | undefined;
      }
    | undefined;

  /**
   *
  customizeComponentName?: ((input: string) => string) | undefined;


  //// Handlebars Options ////

  /**
   * When passed, returns a source map with the source name of the template.
   *
   * Handlebars also exposes a `destName` option which is not documented in our
   * type signature.
   *
   * ?? Can this even be used ??
   */
  srcName?: string;

  /**
   * When passed, retains whitespace around blocks.
   *
   * It's possible that people use this. It notably won't affect Glimmer
   * components, and if we want to support this, we should probably decide on
   * and document a single, consolidated API for whitespace control.
   */
  ignoreStandalone?: boolean;
}

SerializedTemplateBlock in the Wire Format

Local variables are exposed in three separate ways:

• symbols: the names of Handlebars local variables. The wire format represents
  these as numbers, and indexing this array with a symbol number returns the
  name of the symbol as a string.
• upvars: the names of classic-mode free variables. These are variable names
  used in the template that are not block params, lexical variables or locals.
  Like symbols, these names are also stored in the wire format as numbers,
  and looked up by index. These were previously used in debugger, but that
  was superseded by debugSymbols.
• debugSymbols: the names of lexical variables, not used in production mode,
  but exposed to allow debugging tools to display the component name as
  originally written in the source code.

This setup is annoying for several reasons:

1. While the upvars strings are needed at runtime, the symbols strings are
not, except to support {{debugger}}.

2. There is no reason for symbols and debugSymbols to be separate, except
that debugSymbols was introduced recently (by me) and symbols has a long
and storied.

In addition, {{debugger}} is technically a runtime feature,
as the compiler API does not explicitly prevent the use of debugger in
production builds. In fact, the emit.debugSymbols option is the only way
to communicate that the template is being used in "debugging mode". It
would be a breaking change (technically) to make {{debugger}} sensitive to
this new flag.

For these reasons, I kept symbols and debugSymbols separate, and only made
debugSymbols sensitive to emit.debugSymbols. Now that {{debugger}} has
such a small footprint, I think we should consider making it sensitive to the
emit.debugSymbols flag and consolidating symbols and debugSymbols.

3. In classic mode, the upvar symbols are allocated at parse-time, but
some of the names are ultimately processed as keywords. These names are still
left in the wire format even though there are no symbol numbers in the wire
format that correspond to the offsets.

We may want to consider compacting the upvars before producing the wire format
to avoid including names that are never used.

Note

This does not occur in strict mode, since free variables are either:

1. Statically determined to be lexical variables (via the lexicalScope
   option), or
2. Assumed to be keywords at parse-time. If a free variable is never used as a
   keyword in strict mode, it's a compile-time error and no wire format is
   produced.

It also does not occur in the current symbols array, since all symbols
represent block params, and block params cannot be replaced by keywords.

If we consolidate symbols and debugSymbols and retain upvars as a
separate, classic-only array, we may want to leave well enough alone.

[ef4]

The primary difference between Glimmer's view of plugin and Ember's is
that Ember extends this API to allow plugins to create bindings in the
compiled JavaScript module. We may want to make that explicit in some way
in the Glimmer API.

Yeah and to be clear, the extension is done entirely by babel-plugin-ember-template-compilation, so even the ember repo knows nothing about it. There's no typescript-level integration. AST plugins find the extension as env.meta.jsutils. I was careful not to expose any babel concepts in the API, because I want it to be a general interface that AST transforms can rely on always being present, regardless of what tool is actually manipulating the outer JS scope.

There is also a missing feature still which I suspect people will need. Consider that in non-strict mode, if you want to write an AST transform to adjust the arguments passed to a given component at every place that component is invoked, your transform can do something like:

ElementNode(node) {
  if (node.tag === 'MyComponent') {
    // do stuff
  }
}

But in strict mode, that is no longer correct. What you want instead is an API to ask something like:

function referencesImport(
  where: WalkerPath, 
  name: string, 
  exportedName: string, 
  module: string
): boolean;

ElementNode(node, path) {
  if (referencesImport(path, node.tag, "default", "some-lib/components/my-component")) {
    // do stuff
  }
}