Skip to content

Commit e66974f

Browse files
Merge branch 'main' into ATSCALE-23628-database-and-schema-not-required-in-connection
2 parents 72c205d + 623c116 commit e66974f

File tree

3 files changed

+106
-31
lines changed

3 files changed

+106
-31
lines changed

README.md

Lines changed: 16 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,14 +1,21 @@
11
![logo](images/sml-logo-large.png)
22

3+
# SML version 1.1
4+
5+
This is documentation for SML spec version `1.1`. For earlier versions browse the repository tags. Examples:
6+
7+
- [SML version 1.0](https://github.com/semanticdatalayer/SML/tree/v1.0)
8+
39
# What is SML?
4-
Semantic Modeling Language, or SML for short, encompasses over a decade of hands-on development, solving use cases for hundreds of customers across industries such as finance, healthcare, retail, manufacturing, CPG, and more. SML covers more than just tabular use cases. At its core, it is a multidimensional semantic modeling language that supports metrics, dimensions, hierarchies, semi-additive measures, many-to-many relationships, cell-based expressions, and much more.
10+
11+
Semantic Modeling Language, or SML for short, encompasses over a decade of hands-on development, solving use cases for hundreds of customers across industries such as finance, healthcare, retail, manufacturing, CPG, and more. SML covers more than just tabular use cases. At its core, it is a multidimensional semantic modeling language that supports metrics, dimensions, hierarchies, semi-additive measures, many-to-many relationships, cell-based expressions, and much more.
512

613
SML delivers on the following requirements:
714

815
1. **Object-oriented**: SML is an object-oriented language that promotes composability and inheritance. This allows semantic objects to be shared within other semantic objects and across organizations, supporting easy and consistent model-building.
916
2. **Comprehensive**: SML is based on more than a decade of modeling experience across various industry verticals and use cases. SML handles multi-dimensional constructs and serves as a superset of all other existing semantic modeling languages.
1017
3. **Familiar**: SML is based on YAML, a widely adopted, human-readable, industry-standard syntax.
11-
4. **CI/CD Friendly**: SML is code, so it is compatible with Git and CI/CD practices for version control, automated deployment, and software lifecycle management.
18+
4. **CI/CD Friendly**: SML is code, so it is compatible with Git and CI/CD practices for version control, automated deployment, and software lifecycle management.
1219
5. **Extensible**: SML syntax can be enhanced to support additional properties and features.
1320
6. **Open**: SML is Apache open-sourced to support community innovation and is free to use in any application or use case.
1421

@@ -25,6 +32,7 @@ We are or will be open-sourcing the following:
2532
5. **Semantic Converters**: A CLI for translating other semantic modeling languages to and from SML, including Snowflake Cortex semantic models, Databricks UC Metrics, and Power BI semantic models.
2633

2734
## SML Example
35+
2836
The following is an example of an SML `model` object:
2937

3038
```
@@ -59,6 +67,7 @@ metrics:
5967
```
6068

6169
## SML Object Hierarchy
70+
6271
The following graphic illustrates the key SML objects and their relationships:
6372

6473
```mermaid
@@ -85,7 +94,7 @@ The following sections describe the different SML object types as well
8594
as the properties available for each:
8695

8796
- [Catalog](sml-reference/catalog.md) - Defines the control file for a SML repository. It contains all repository-level definitions.
88-
- [Package](sml-reference/package.md) - Defines additional Git repositories references whose objects can be used in the current repository.
97+
- [Package](sml-reference/package.md) - Defines additional Git repositories references whose objects can be used in the current repository.
8998
- [Model](sml-reference/model.md) - Defines the logical, business-friendly representation on top of the physical data.
9099
- [Dimension](sml-reference/dimension.md) - Defines the logical collection of attributes and hierarchies for supporting drill-down.
91100
- [Row Security](sml-reference/row-security.md) - Defines row-level data access rules for users and groups.
@@ -102,15 +111,18 @@ as the properties available for each:
102111
## Model Library
103112

104113
### Tutorial Models
114+
105115
1. [Internet Sales Model](https://github.com/semanticdatalayer/sml-models-tutorials-internet-sales) - a simple, single-fact model derived from the fictitious AdventureWorks retail dataset.
106116
2. [World Wide Importers Model](https://github.com/semanticdatalayer/sml-models-tutorials-ww-importers) - a more complex, multi-fact model representing a fictional wholesale and distribution company.
107117
3. [TPC-DS Model](https://github.com/semanticdatalayer/sml-models-tutorials-tpcds) - a complex, multi-fact model that encodes the [TPC-DS](https://www.tpc.org/tpcds/) benchmark model in SML.
108118
4. [TPC-H Model](https://github.com/semanticdatalayer/sml-models-tutorials-tpch) - a complex, multi-fact model that encodes the [TPC-H](https://www.tpc.org/tpch/) benchmark model in SML.
109-
5. [AdventureWorks2012 Model](https://github.com/semanticdatalayer/sml-models-tutorials-adventureworks2012) - the standard Microsoft SSAS tutorial in SML.
119+
5. [AdventureWorks2012 Model](https://github.com/semanticdatalayer/sml-models-tutorials-adventureworks2012) - the standard Microsoft SSAS tutorial in SML.
110120

111121
### Data Warehouse Usage/Cost Models
122+
112123
1. [Snowflake Usage Model](https://github.com/semanticdatalayer/sml-models-usage-snowflake) - a semantic model for analyzing Snowflake credit and data warehouse usage.
113124

114125
### Marketplace Models
126+
115127
1. [Snowplow Digital Analytics Model](https://github.com/semanticdatalayer/sml-models-snowplow) - Snowplow empowers organizations to create a scalable, first-party data foundation so marketing and data teams can effectively analyze and tackle Customer 360 use cases.
116128
2. [CRISP CPG Retail and Distributor Data Model](https://github.com/semanticdatalayer/sml-models-crisp-cpg-retail) - Crisp connects to over 40 leading U.S. retailers and distributors.

sml-reference/dimension.md

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -269,6 +269,7 @@ namespace Dimensions{
269269
Boolean is_aggregatable
270270
Boolean exclude_from_fact_agg
271271
String time_unit
272+
Int constraint_translation_rank
272273
Array~String~ allowed_calcs_for_dma
273274
CustomEmptyMember custom_empty_member
274275
String folder
@@ -308,12 +309,15 @@ namespace Dimensions{
308309
String label
309310
String description
310311
String folder
312+
Number precedence
311313
Array~CalculatedMembers~ calculated_members
314+
Boolean is_hidden
312315
}
313316
class CalculatedMembers{
314317
String unique_name
315318
String description
316319
String format
320+
Boolean is_hidden
317321
String expression
318322
Boolean use_input_metric_format
319323
String template
@@ -541,13 +545,36 @@ Defines the individual calculated members in the group.
541545

542546
A description of the calculation group.
543547

548+
## is_hidden
549+
550+
- **Type:** boolean
551+
- **Required:** N
552+
553+
Determines whether the calculation group is visible in BI tools.
554+
555+
Supported values:
556+
557+
- `false` (default)
558+
- `true`
559+
544560
## folder
545561

546562
- **Type:** string
547563
- **Required:** N
548564

549565
The name of the folder in which the calculation group is displayed in BI tools.
550566

567+
## precedence
568+
569+
- **Type:** number
570+
- **Required:** N
571+
572+
Update to "Precedence" explicitly defines the order of Calculation Group evaluation, making it consistent across BI tools.
573+
574+
Supported values:
575+
576+
- Integer and floating point numbers
577+
551578
# Calculated Members Properties
552579

553580
## unique_name
@@ -577,6 +604,18 @@ Supported templates:
577604

578605
If you do not want to use a built-in template, you can define a custom expression using the `expression` property (see below).
579606

607+
## is_hidden
608+
609+
- **Type:** boolean
610+
- **Required:** N
611+
612+
Determines whether the attribute is visible in BI tools.
613+
614+
Supported values:
615+
616+
- `false` (default)
617+
- `true`
618+
580619
## expression
581620

582621
- **Type:** string
@@ -1188,6 +1227,14 @@ If the key consists of one column, the values in that column must be
11881227
unique. If the key is a compound key, the columns together must provide
11891228
unique values.
11901229

1230+
## constraint_translation_rank
1231+
1232+
- **Type:** integer
1233+
- **Required:** N
1234+
- **Range:** should be a valid 32 bit integer
1235+
1236+
Defines the translation of dimension filter constraints into fact table partition column constraints. This can significantly improve query performance for cases where fact-based aggregates are not used.
1237+
11911238
## shared_degenerate_columns
11921239

11931240
- **Type:** array

sml-reference/model.md

Lines changed: 43 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,6 @@ label: Internet Sales
1919
visible: true
2020

2121
relationships:
22-
2322
- unique_name: factinternetsales_Date_Dimension_Order
2423
from:
2524
dataset: factinternetsales
@@ -116,7 +115,6 @@ dimensions:
116115
- Weight
117116

118117
metrics:
119-
120118
- unique_name: orderquantity
121119
folder: Sales Metrics
122120

@@ -127,19 +125,16 @@ perspectives:
127125
- unique_name: Internet Sales - No PII
128126
dimensions:
129127
- hierarchies:
130-
- levels:
131-
- Customer Name
128+
- level: Customer Name
132129
name: Customer Hierarchy
133130
name: Customer Dimension
134131
secondaryattributes:
135132
- d_firstname
136133
- d_lastname
137134

138135
drillthroughs:
139-
140136
- unique_name: Customer Details
141137
attributes:
142-
143138
- name: State
144139
dimension: Geography Dimension
145140

@@ -158,7 +153,6 @@ drillthroughs:
158153

159154
- unique_name: Shipping Details
160155
attributes:
161-
162156
- name: Size
163157
dimension: Size Dimension
164158

@@ -218,6 +212,7 @@ namespace Models{
218212
Object to
219213
String role_play
220214
String type
215+
ConstraintTranslation constraint_translation
221216
}
222217
class From{
223218
String dataset
@@ -228,6 +223,10 @@ namespace Models{
228223
String level
229224
String row_security
230225
}
226+
class ConstraintTranslation{
227+
String level
228+
Array~String~ from_columns
229+
}
231230
class Aggregate{
232231
String unique_name
233232
String label
@@ -279,6 +278,7 @@ namespace Models{
279278
}
280279
class PerspectiveHierarchy{
281280
String name
281+
String level
282282
Array~String~ levels
283283
}
284284
}
@@ -395,6 +395,20 @@ marks):
395395
For example, if you wanted to use the prefix **Order**, you would set
396396
`role_play` to `"Order {0}"`.
397397

398+
### constraint_translation
399+
400+
- **Type:** object
401+
- **Required:** N
402+
403+
Defines the translation of dimension filter constraints into fact table partition column constraints. This can significantly improve query performance for cases where fact-based aggregates are not used.
404+
405+
Supported properties:
406+
407+
- `level`: String, required. Indicates the dimension level to which the constraint translation applies.
408+
- `from_columns`: Array, required. Lists the column(s) in the dataset that should be used for the join.
409+
410+
If the `constraint_translation` property is defined, a corresponding `constraint_translation_rank` must be present in the associated level.
411+
398412
## metrics
399413

400414
- **Type:** array
@@ -440,7 +454,7 @@ analysts with the entire data model, you can make specific dimensions,
440454
hierarchies, levels, secondary attributes, measures, and calculated
441455
measures invisible to them.
442456

443-
**Note:** We recommend that you add perspectives *after* a model has
457+
**Note:** We recommend that you add perspectives _after_ a model has
444458
been fully tested. Although you can edit a model after adding
445459
perspectives, any changes might require you to update the perspectives
446460
to hide new objects that would otherwise be visible to all users.
@@ -476,9 +490,9 @@ perspective.
476490
A list of the specific dimensions and their hierarchies to be hidden in the
477491
perspective.
478492

479-
By default, all objects within a dimension are visible. The lowest granularity objects specified are
480-
hidden and the objects above it are not. Hiding a level in a hierarchy hides all levels below it.
481-
Hiding a hierarchy hides all levels in it. Hiding a dimension hides all objects within it including hierarchies
493+
By default, all objects within a dimension are visible. The lowest granularity objects specified are
494+
hidden and the objects above it are not. Hiding a level in a hierarchy hides all levels below it.
495+
Hiding a hierarchy hides all levels in it. Hiding a dimension hides all objects within it including hierarchies
482496
and secondary attributes. If a dimension is not hidden, secondary attributes can be hidden individually.
483497

484498
Supported properties:
@@ -488,8 +502,10 @@ Supported properties:
488502

489503
- `hierarchies`: Array, optional. A list of the specific hierarchies
490504
within the dimension to hide in the perspective. Supported properties:
491-
- `name`: String, required. The name of the hierarchy.
492-
- `levels`: Array, optional. Defines a single level in the hierarchy to be hidden in the perspective. All levels below the specified level will also be hidden. Only one level should be provided.
505+
506+
- `name`: String, required. The name of the hierarchy.
507+
- `level`: String, optional. Defines a single level in the hierarchy to be hidden in the perspective. All levels below the specified level will also be hidden.
508+
- `levels`: Array, optional. ⚠️ **DEPRECATED** use `level` instead.
493509

494510
- `secondary_attributes`: Array, optional. A list of the dimension's
495511
secondary attributes to hide in the perspective.
@@ -643,22 +659,22 @@ Supported properties:
643659
determines whether it should be defined on the key column, name
644660
column, or both. Supported values: `name`, `key`, `name+key`
645661

646-
When the engine builds an instance of this aggregate, it creates
647-
a partition for each combination of values in the dimensional
648-
attributes. The number of partitions depends on the
649-
left-to-right order of the attributes, as well as the number of
650-
values for each attribute.
662+
When the engine builds an instance of this aggregate, it creates
663+
a partition for each combination of values in the dimensional
664+
attributes. The number of partitions depends on the
665+
left-to-right order of the attributes, as well as the number of
666+
values for each attribute.
651667

652-
Essentially, the partitioning key functions as a `GROUP BY`
653-
column. Queries against the aggregate must use this dimensional
654-
attribute in a `WHERE` clause. A good candidate for a
655-
partitioning key is a set of dimensional attributes that
656-
together have a few hundred to under 1000 value combinations.
668+
Essentially, the partitioning key functions as a `GROUP BY`
669+
column. Queries against the aggregate must use this dimensional
670+
attribute in a `WHERE` clause. A good candidate for a
671+
partitioning key is a set of dimensional attributes that
672+
together have a few hundred to under 1000 value combinations.
657673

658674
- `distribution`: String, optional. The distribution keys to use when
659675
creating the aggregate table. If your aggregate data warehouse
660676
supports distribution keys, then the semantic engine uses the specified keys when
661-
creating the aggregate table.
677+
creating the aggregate table.
662678

663679
## partitions
664680

@@ -733,7 +749,7 @@ Supported properties:
733749
- `allow_peer_aggs`: Boolean, optional. Enables aggregation on data
734750
derived from datasets in data warehouses that are different from the
735751
source dataset.
736-
- `allow_preferred_aggs`: Boolean, optional. Allow aggregates to be built
752+
- `allow_preferred_aggs`: Boolean, optional. Allow aggregates to be built
737753
in preferred storage.
738754
- `create_hinted_aggregate`: Boolean, options. Enables the creation of
739755
hinted aggregates for the dataset.
@@ -771,8 +787,8 @@ Sample `overrides`:
771787

772788
```yaml
773789
overrides:
774-
salesamount:
790+
salesamount:
775791
query_name: deployed query name for metric
776792
Color Dimension:
777793
query_name: deployed query name for dimension
778-
```
794+
```

0 commit comments

Comments
 (0)