Improved type naming

[dmiller]

Proposal: a simplified type-naming standard for ClojureCLR

Background

ClojureCLR's current mechanism for specifying types is cumbersome. It follows the CLR conventions, which are not user-friendly, rather than something like you see in C# or F#.
For example,

|System.Collections.Generic.Dictionary`2[System.Int32, System.String]|

The CLR type naming syntax is quite complicated. It has to accommodate generic types, generic argument types, arrays, pointer designators, etc. Moreover, typenames can be fully assembly-qualfied, including aseembly name, assembly version, assembly culture, etc. Behold the lowly 64-bit integer:

"System.Int64, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e"

There is a fairly general mechanism for constructing types already in place in ClojureCLR. Check out ClrTypeSpec.
I stole took inspiration from the code for this in Mono. (This was back in the day before Microsoft open-sourced this kind of thing.) There are two parts to this code:

• a parser that takes a string representing a typename and generates a syntax tree representing its structure;
• a resolver that traverses the syntax tree and finds or generates (in the case of arrays, pointers, and generics) the types mentioned.

Now that Microsoft has open-sourced much of the .NET/CLR code, we can look at how Microsoft handles typename parsing. More below.

Goals

The primary goal:

• (G1) To define a syntax for expressing typenames that is closer in style to languages such as C# or F#.

The desired qualities:

• (G2) Must handle all the complexity of the type system: , arrays, pointers, by-refs, assembly qualification, generic type definitions, closed and open constructed generic types, etc.
• (G3) Does not require the \N` syntax for generic types
• (G4) Respects the use of type aliases, including the type aliases such as int and long defined by Clojure itself.

Some constraints to respect:

• (C1) Does not invalidate any existing application code.
• (C2) Does not use the ns construct or its dependencies, such as import. (The Clojure chiefs seem to react coldly to any suggestions of modifications here. Also, there are libraries that analyze ns declarations that would be affected.)
• (C3) Requires as little new code as possible.

We can do this.

The actual extensions are defined in goals (G3) and (G4). The former is a question of syntax; the latter is a question of typename resolution We consider each in turn.

Generic types

The goal is to get rid of the requirement of explicitly including backquote-numeral in generic type names, whether generic type definitions or constructed generic types. In other words, get rid of the '`2' in

|System.Collections.Generic.Dictionary`2[System.Int32, System.String]|

However, this causes a problem. We would not be able to tell if

T[]

is referring to a generic type T with one generic type argument or an array of type T (non-generic).

The simplest solution is to follow what C#, F#, and numerous other languages do: Use angle brackets for generics. Thus:

new syntax	old syntax equivalent	Meaning
T	T	non-generic type T
T[]	T[]	array of T
T[,,]	T[,,]	two-dimensional array of T
T<>	T`1	generic type definition T with one generic type argument
T<,>	T`2	generic type definition T with two generic type arguments
T<U,>	T `2[U,]	(open) constructed generic type with two arguments, one supplied (not supported by the current ClrTypeSpec mechanism
T<U,V>	T`2[U,V]	(closed) constructed generic type with two supplied arguments

Regarding our constraints:

• (C1) I certainly hope no one has been running around naming types with angle brackets in their names.
• (C2) ns certainly not involved
• (c3) This can be incorporated fairly easily into the parsing code. A left angle bracket can occur where a backquote can occur. We don't need the following number, but otherwise similar processing, with a left angle bracket serving to close instead of a right square bracket.

Type aliases

We'd like to be able to specify a type in this way:

|Dictionary[int, String]|

The current typename handler has no problem parsing this. The problem is in resolving the types named in it: Dictionary, int, String.
And they present three distinct problems.

Internal Clojure names

Names such as int, long, shorts, etc. are not part of the CLR specification. Languages such as Clojure, C#, and F# define standard type aliases. (And don't necessarily agree on the names. C# has double and float for 64- and 32-bit floating point types; F# has both double and float for 64-bit, and float32 and single for 32-bit. I have been caught out by this.)

Internal names can be incorporated fairly easily into the type resolver. We can consult an existing map from the pre-defined typename strings to the corresponding type.

Namespace-mapped types

You might have noted a discrepancy in how some type names are treated.
For example, if you are providing type hints, this works just fine:

^String

but

System.Collections.Generic.List`1[String]

will fail. The explanation is straightforward. The type resolver in ClojureCLR makes several checks on a symbol before passing it to ClrTypeSpec.
One check is to see if it is a special name such as int or long. A second check is look up the symbol in the current namespace's alias map.
The alias map maps symbols to types. A third check is to call System.Type.GeType(String). There are a few other special cases that don't concern us. After all of that, is we've not gotten a hit, the ClrTypeSpec parser/resolver is called.

The reason the first example succeeds is that String is entered in every namespace's alias map. When a namespace is created, it is set up with set of default mappings. The code runs through loaded assemblies and looks for all public classes/interfaces/value-types with a namespace of System (with a few other restrictions) and maps the basic type name (no namespace qualifier) to the type. Thus the symbol Int32 gets mapped to the type System.Int32; the symbol String gets mapped to the type System.String.

To make our second example work, we need to make the ClrTypeSpec resolver be namespace-aware. Resolving must be done with respect to supplied namespace. That namespace's alias map is consulted before trying other measures (but after checking for built-ins like long).

Other aliases

To get all the way to

|Dictionary[int, String]|

we need a way to supply meaning to Dictionary.

The easiest way is to type into the same namespace alias map referenced in the previous section. The standard way to add symbol-to-type entries to the map is via import. This can be done by direct call or by an :import clause in an ns declaration:

(ns my.namespace
  ; ...
  (:import (System.IO  FileInfo Path )))

maps symbol Path to type System.IO.Path. This won't work for our case. The symbol that import maps is the name of the type.

If we do

(import '|System.Collections.Generic.List`1[System.Int64]|)

we will get an mapping from symbol |List1| to the collection type.

user=> |List`1|
System.Collections.Generic.List`1[[System.Int64, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]

If we subsequently try to do

(import '|System.Collections.Generic.List`1[System.Int32]|)

we'll get

Execution error (InvalidOperationException) at user/eval4240 (NO_FILE:0).
List`1 already refers to: System.Collections.Generic.List`1[System.Int64] in namespace: user

Constraint (C1) forbids messing with import. It is easy enough to add our own macro:

(add-type-alias Dictionary |System.Collections.Generic.Dictionary<,>| )

After which, our example

|Dictionary<int, String>|

is now resolvable.

BTW, this would allow a sequence such as this:

(add-type-alias Dictionary |System.Collections.Generic.Dictionary<,>| )
(add-type-alias OSDict |Dictionary<Object,String>|)
(add-type-alias IIDict |Dictionary<int,int>|)

Then you could do something like:

(.Add ^OSDict mydict "help" "me")

Note: this is quite easy to implement. The macro can expand into a call to clojure.lang.Namepace.importClass which takes a symbol and a type to map the symbol to.

I am undecided on the actual name of the or whether it should be a macro or a regular function.

References

• ClrTypeSpec
  • the existing typename parser/resolver
• Specifying fully qualified type names
  • The MS doc defining the grammar for type names.
• System.Reflection.Metadata.TypeNameParser
  • Source code for the typename parser currently used in .NET/CL
  • We need to compare our ClrTypeSpec to this code for accuracy. Also, there are performance improvements, such as the use of ReadOnlySpan<char> -- spans were not around when the inspirational Mono code was written.

[dmiller]

Several months ago I posted an analysis of type lookup performance in ClojureCLR: Are you my type?. Certainly relevant in this context. Note that the proposal for a modified lookup mechanism at the end of that post would need to be modified somewhat if we were to adopt the aliasing mechanism proposed above.

[E-A-Griffin]

I like the macro solution via add-type-alias

[dmiller]

I'm thinking there should be some constraints on add-type-alias. A few ideas:
-- one shouldn't be able to redefine the meaning of int, longs, and other predesignated names.
-- Probably the alias should be a simple name. An arbitrary alias would be unparseable -- couldn't figure out where the token ends. Simple name would be something without any of the triggering characters for special processing -- <, [, etc. In other words, whatever characters already signal the end of a typename before array, generic, pointer, markers.

[dmiller]

If a macro, I'm thinking add-type-alias sounds too functional in nature. A name like def-type-alias or alias-type might be better.

[dmiller]

We need a clarificaion of what a "type alias" actually defines.

Two models come to mind:

• A type alias is a mapping from symbol to type.
• A type alias is purely a string (or symbol) substitution, used only during type resolution.

An example to show the distinction.

• (alias-type ListA |System.Collections.Generic.List`1|)
• (alias-type ListB System.Collections.Generic.List)

By our usual method of referring to types using CLR typename syntax, we would expect the associated value for ListA to be the indicated generic type definition.

There is no type System.Collections.Generic.List. The association for ListB would be to a symbol. Aha, you say, but we could have mapped that symbol to the same type as the ListA example.
Well, sometimes. Contemplate:

(alias-type MyFunc System.Func)

There are seventeen generic definitions of System.Func. Without context, you cannot tell. So it would have to be a symbol association.

One can think of ways to make a symbol association work.
When actually used, one typically would have a context:

ListB[int]
MyFunc[int,String]

However, I argue that we should implement symbol-to-type mapping instead of symbol-to-symbol mapping.

Namespaces already supply a mechanism for mapping symbols to types.

Namespaces are mappings from simple (unqualified) symbols to Vars and/or Classes.
Vars can be interned in a namespace, using def or any of its variants,
in which case they have a simple symbol for a name and a reference to their containing namespace,
and the namespace maps that symbol to the same var.
A namespace can also contain mappings from symbols to vars interned in other namespaces by using refer or use,
or from symbols to Class objects by using import. [ Emphasis mine. Reference: Namespaces ]

Use of the symbol-to-type map is embedded all over the Clojure interpreter/compiler code. I've written a little about this:

• C4: Symbolic of what?
• Are you my type?

But these don't give the full complexity. If you want to dig deeper, look at the code for

• HostExpr.MaybeType
• HostExpr.TagToType
• RT.classForName
• Compiler.AnalyzeSymbol
• Compiler.ResolveIn
• Compiler.maybeResolveIn

By building on the existing namespace mechanism for symbol-to-type mapping, we automatically participate in all
symbol-is-type decision-making in Clojure.

From a coding simplicity viewpoint, this approach also fits easily into the type resolution algorithm used in the parsing process.

[dmiller]

Rethinking the need for the <> addition to the syntax. It's more coding effort. And I can likely achieve most of the desired gain in expressivity using the existing [] syntax with minimal coding.

To me, there are several primary gains to achieve with this proposal:

• allow the use of pre-defined Clojure special types such as int or longs as generic arguments
• allow user-defined aliases, both top-level and as generic arguments
• reduce the use of backquote-arity (but as you will see, not eliminate entirely).

Without the <> addition, I can still manage to eliminate the need to put in backquote-arity except when specifying (including defining an alias) for a generic type definition. In other words, you could specify any of the following:

Construct	Comment
(alias-type GenList |System.Collections.GenericList`1|)	reference to generic type definition, requires backquote-arity
(alias-type IntList |GenList[int]|)	We know the arity from GenList itself
(alias-type StrList |System.Collections.GenericList[String]|)	we know the arity from the generic type arguments supplied.

I can live with having to specify the backquote-arity for generic type definition references.
We are not losing a lot. Particularly in cases like

(alias-type Func5  |Func`5|)

versus

(alias-type Func5 |Func<,,,,>|)  ;; did I get the right number of commas in there?