Statistics 12 fix unknown estimates

README.md

@@ -15,13 +15,13 @@ See the [User Manual](https://prestodb.io/docs/current/) for deployment instruct
 
 Presto is a standard Maven project. Simply run the following command from the project root directory:
 
-    mvn clean install
+    ./mvnw clean install
 
 On the first build, Maven will download all the dependencies from the internet and cache them in the local repository (`~/.m2/repository`), which can take a considerable amount of time. Subsequent builds will be faster.
 
 Presto has a comprehensive set of unit tests that can take several minutes to run. You can disable the tests when building:
 
-    mvn clean install -DskipTests
+    ./mvnw clean install -DskipTests
 
 ## Running Presto in your IDE
 

pom.xml

@@ -588,6 +588,12 @@
                 <version>42.0.0</version>
             </dependency>
 
+            <dependency>
+                <groupId>org.pcollections</groupId>
+                <artifactId>pcollections</artifactId>
+                <version>2.1.2</version>
+            </dependency>
+
             <dependency>
                 <groupId>org.antlr</groupId>
                 <artifactId>antlr4-runtime</artifactId>

presto-docs/src/main/sphinx/index.rst

@@ -15,6 +15,7 @@ Presto Documentation
     language
     sql
     migration
+    optimizer
     develop
     release
 

presto-docs/src/main/sphinx/optimizer.rst

@@ -0,0 +1,9 @@
+***************
+Query Optimizer
+***************
+
+.. toctree::
+    :maxdepth: 1
+
+    optimizer/statistics
+    optimizer/cost-in-explain

presto-docs/src/main/sphinx/optimizer/cost-in-explain.rst

@@ -0,0 +1,37 @@
+===============
+Cost in EXPLAIN
+===============
+
+During planning, the cost associated with each node of the plan is computed based on the root table statistics
+for the tables in the query. This calculated cost is printed as part of the output of an ``EXPLAIN`` statement.
+
+Cost information is displayed in the plan tree using the format ``{rows: XX, bytes: XX}``.  ``rows`` refers to the
+expected number of rows output by each plan node during execution.  ``bytes`` refers to the expected size of the
+data output by each plan node in bytes. If any of the values is not known, a ``?`` is printed.
+
+For example:
+
+.. code-block:: none
+
+    presto:default> EXPLAIN SELECT comment FROM nation_with_column_stats WHERE nationkey > 3
+
+    - Output[comment] => [comment:varchar(152)] {rows: ?, bytes: ?}
+      - RemoteExchange[GATHER] => comment:varchar(152) {rows: 12, bytes: ?}
+        - ScanFilterProject[table = hive:hive:default:nation_with_column_stats,
+                                    originalConstraint = (""nationkey"" > BIGINT '3'),
+                                    filterPredicate = (""nationkey"" > BIGINT '3')] => [comment:varchar(152)] {rows: 25, bytes: ?}/{rows: 12, bytes: ?}/{rows: 12, bytes: ?}
+                LAYOUT: hive
+                nationkey := HiveColumnHandle{clientId=hive, name=nationkey, hiveType=bigint, hiveColumnIndex=0, columnType=REGULAR}
+                comment := HiveColumnHandle{clientId=hive, name=comment, hiveType=varchar(152), hiveColumnIndex=3, columnType=REGULAR}
+
+Generally there is only one cost printed for each plan node.
+However, when a ``Scan`` operator is combined with a ``Filter`` and/or ``Project`` operator, then multiple cost structures will be printed,
+each corresponding to an individual logical part of the combined meta-operator.
+For example, for a ``ScanFilterProject`` operator three cost structures will be printed.
+
+ * the first will correspond to ``Scan`` part of operator
+ * the second will correspond to ``Filter`` part of opertor
+ * the third will corresponde to ``Project`` part of operator
+
+Estimated cost is also printed in ``EXPLAIN ANALYZE`` in addition to actual runtime statistics.
+

presto-docs/src/main/sphinx/optimizer/statistics.rst

@@ -0,0 +1,146 @@
+================
+Table Statistics
+================
+
+Presto supports statistics based optimizations for queries. For a query to take advantage of these optimizations,
+Presto must have statistical information for the tables in that query.
+
+Table statistics are provided to the query planner by connectors.
+Currently the only connector that supports statistics is the :doc:`/connector/hive`.
+
+Table Layouts
+-------------
+
+Statistics are exposed to the query planner by a table layout. A table layout represents a subset of a table's data
+and contains information about the organizational properties of that data (like sort order and bucketing).
+
+The number of table layouts available for a table and the details of those table layouts are specific to each connector.
+Using the Hive connector as an example:
+
+* Non-partitioned tables have just one table layout representing all data in the table
+* Partitioned tables have a family of table layouts. Each set of partitions to be scanned represents one table layout.
+  Presto will try to pick a table layout consisting of the smallest number of partitions based on filtering predicates
+  from the query.
+
+Available Statistics
+--------------------
+
+Currently, the following statistics are available in Presto:
+
+ * For the table:
+
+   * **row count**: the total number of rows for the table layout
+
+ * For each column in a table:
+
+   * **data size**: the data size that needs to be read
+   * **nulls fraction**: the fraction of null values
+   * **distinct value count**: the number of distinct values
+   * **low value**: the smallest value in the column
+   * **high value**: the largest value in the column
+
+
+The set of statistics available for a particular query depends on the connector being used and can also vary by table or
+even by table layout. For example, the Hive connector does not currently provide statistics on data size.
+
+Displaying Table Statistics
+---------------------------
+
+Table statistics can be displayed via the Presto SQL interface using the ``SHOW STATS`` command.
+There are two flavors of the command:
+
+ * ``SHOW STATS FOR <table_name>`` will show statistics for the table layout representing all data in the table
+ * ``SHOW STATS FOR (SELECT <column_list|*> FROM <table_name> WHERE <filtering_condition>)``
+   will show statistics for the table layout of table ``t`` representing a subset of data after applying the given filtering
+   condition. Both the column list and the filtering condition used in the ``WHERE`` clause can reference table columns.
+
+In both cases, the ``SHOW STATS`` command outputs two types of rows.
+For each column in the table there is a row with ``column_name`` equal to the name of that column.
+These rows expose column-related statistics for a table (data size, nulls count, distinct values count, min value, max value).
+Additionally there is one row with NULL as the ``column_name``. This row contains table-layout wide statistics - for now just the row count.
+
+For example:
+
+.. code-block:: none
+
+    presto:default> SHOW STATS FOR nation;
+
+     column_name | data_size | distinct_values_count | nulls_fraction | row_count |     low_value      |     high_value
+    -------------+-----------+-----------------------+----------------+-----------+--------------------+--------------------
+     regionkey   | NULL      |                   5.0 |            0.0 | NULL      | 0                  | 4
+     name        | NULL      |                  25.0 |            0.0 | NULL      | ALGERIA            | VIETNAM
+     comment     | NULL      |                  25.0 |            0.0 | NULL      |  haggle. carefu... | y final package...
+     nationkey   | NULL      |                  25.0 |            0.0 | NULL      | 0                  | 24
+     NULL        | NULL      | NULL                  | NULL           |      25.0 | NULL               | NULL
+     (5 rows)
+
+
+    presto:default> SHOW STATS FOR (SELECT * FROM nation WHERE nationkey > 10);
+
+     column_name | data_size | distinct_values_count | nulls_fraction | row_count |     low_value      |     high_value
+    -------------+-----------+-----------------------+----------------+-----------+--------------------+--------------------
+     regionkey   | NULL      |                   5.0 |            0.0 | NULL      | 0                  | 4
+     name        | NULL      |                   9.0 |            0.0 | NULL      | IRAN               | VIETNAM
+     comment     | NULL      |                  14.0 |            0.0 | NULL      |  pending excuse... | y final package...
+     nationkey   | NULL      |                   3.0 |            0.0 | NULL      | 10                 | 24
+     NULL        | NULL      | NULL                  | NULL           |      25.0 | NULL               | NULL
+     (5 rows)
+
+If provided ``SELECT`` will filter out all of the partitions (all table layouts),
+then the ``SHOW STATS`` will return no statistic which will be represented as in example below.
+
+.. code-block:: none
+
+    presto:default> SHOW STATS FOR (SELECT * FROM nation WHERE nationkey > 999);
+
+     column_name
+    -------------
+     NULL
+    (1 row)
+
+Note, that currently providing ``column_list`` instead of ``*`` in ``SELECT`` will not influence the output table.
+
+For example:
+
+.. code-block:: none
+
+    presto:default> SHOW STATS FOR (SELECT comment FROM nation WHERE nationkey > 10);
+
+     column_name | data_size | distinct_values_count | nulls_fraction | row_count |     low_value      |     high_value
+    -------------+-----------+-----------------------+----------------+-----------+--------------------+--------------------
+     regionkey   | NULL      |                   5.0 |            0.0 | NULL      | 0                  | 4
+     name        | NULL      |                   9.0 |            0.0 | NULL      | IRAN               | VIETNAM
+     comment     | NULL      |                  14.0 |            0.0 | NULL      |  pending excuse... | y final package...
+     nationkey   | NULL      |                   3.0 |            0.0 | NULL      | 10                 | 24
+     NULL        | NULL      | NULL                  | NULL           |      25.0 | NULL               | NULL
+     (5 rows)
+
+
+Updating Statistics For Hive Tables
+-----------------------------------
+
+For the Hive connector, Presto uses the statistics that are managed by Hive and exposed via the Hive metastore API.
+Depending on the Hive configuration, table statistics may not be updated automatically.
+
+If statistics are not updated automatically, the user needs to trigger a statistics update via the Hive CLI.
+
+The following command can be used in the Hive CLI to update table statistics for non-partitioned table ``t``::
+
+        hive> ANALYZE TABLE t COMPUTE STATISTICS FOR COLUMNS
+
+For partitioned tables, partitioning information must be specified in the command.
+Assuming table ``t`` has two partitioning keys ``a`` and ``b``, the following command would
+update the table statistics for all partitions::
+
+        hive> ANALYZE TABLE t PARTITION (a, b) COMPUTE STATISTICS FOR COLUMNS
+
+It is also possible to update statistics for just a subset of partitions.
+This command will update statistics for all partitions for which partitioning key ``a`` is equal to ``1``::
+
+        hive> ANALYZE TABLE t PARTITION (a=1, b) COMPUTE STATISTICS FOR COLUMNS
+
+And this command will update statistics for just one partition::
+
+        hive> ANALYZE TABLE t PARTITION (a=1, b=5) COMPUTE STATISTICS FOR COLUMNS
+
+For documentation on Hive's statistics mechanism see https://cwiki.apache.org/confluence/display/Hive/StatsDev

presto-hive/src/main/java/com/facebook/presto/hive/HiveMetadataFactory.java

@@ -169,6 +169,6 @@ public HiveMetadata create()
                 partitionUpdateCodec,
                 typeTranslator,
                 prestoVersion,
-                new MetastoreHiveStatisticsProvider(typeManager, metastore));
+                new MetastoreHiveStatisticsProvider(typeManager, metastore, timeZone));
     }
 }