Skip to content

Commit ed4ba37

Browse files
kr11kr11
authored
Merge pull request #14 from bytedance/kangrong/standalone_for_57
[Stats|Pipe|Perf] support MySQL 57 in standalone mode, support descending indexes
2 parents d8d440d + d1617f8 commit ed4ba37

36 files changed

+2642
-7784
lines changed

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,4 @@ videx.egg-info
2525
tpch_tiny.sql
2626
videx_metadata_tpch_tiny.json
2727
dist
28+
data

‎README.md‎

Lines changed: 62 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
<a href="https://hub.docker.com/repository/docker/kangrongme/videx">
1515
<img src="https://img.shields.io/docker/pulls/kangrongme/videx?style=for-the-badge&logo=docker" alt="Docker Pulls"/>
1616
</a>
17+
<img src="https://img.shields.io/badge/MySQL|Percona-8.0|_5.7-FF9800?style=for-the-badge&logo=mysql" alt="MySQL Support"/>
1718
</p>
1819

1920
**VIDEX**: The Disaggregated, Extensible **\[VI\]**rtual in**\[DEX\]** Engine for What-If Analysis in MySQL 🚀
@@ -40,8 +41,8 @@ By default, VIDEX includes implementations based on histograms and NDV collected
4041

4142
VIDEX offers two startup modes:
4243

43-
1. **Plugin to production database**: Install VIDEX as a plugin to the production database instance.
44-
2. **Individual instance**: This mode can completely avoid impacting the stability of online running instances, making it practical for industrial environments.
44+
1. **Plugin to production database** (Plugin-Mode): Install VIDEX as a plugin to the production database instance.
45+
2. **Individual instance** (Standalone-Mode): This mode can completely avoid impacting the stability of online running instances, making it practical for industrial environments.
4546

4647
Functionally, VIDEX supports creating and deleting indexes (single-column indexes, composite indexes, EXTENDED_KEYS indexes, [descending indexes](https://dev.mysql.com/doc/en/descending-indexes.html)).
4748
However, it currently does not support functional indexes, FULL-Text, and Spatial Indexes.
@@ -117,7 +118,7 @@ If you haven't installed Docker yet:
117118

118119
#### Launch VIDEX Container
119120
```bash
120-
docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:0.0.2
121+
docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:latest
121122
```
122123

123124
> **Alternative Deployment Options**
@@ -129,7 +130,7 @@ docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:0.0.2
129130
130131
## 3 Examples
131132

132-
### 3.1 TPCH-Tiny Example
133+
### 3.1 TPCH-Tiny 示例 (MySQL 8.0)
133134

134135
This example demonstrates the complete VIDEX workflow using the `TPC-H Tiny` dataset (1% random sample from TPC-H sf1).
135136

@@ -279,7 +280,63 @@ ALTER TABLE tpch_tiny.orders DROP INDEX idx_o_orderstatus;
279280
ALTER TABLE videx_tpch_tiny.orders DROP INDEX idx_o_orderstatus;
280281
```
281282

282-
### 3.2 TPCH sf1 (1g) Example
283+
### 3.2 TPCH-Tiny Example (MySQL 5.7)
284+
285+
VIDEX now supports high-precision simulation for MySQL 5.7 in the standalone mode.
286+
287+
#### Step 1: Import Test Data into MySQL 5.7 Instance
288+
289+
Import data into a MySQL 5.7 instance.
290+
291+
```bash
292+
mysql -h${HOST_MYSQL57} -P13308 -uvidex -ppassword -e "create database tpch_tiny_57;"
293+
mysql -h${HOST_MYSQL57} -P13308 -uvidex -ppassword -Dtpch_tiny_57 < tpch_tiny.sql
294+
```
295+
296+
#### Step 2: VIDEX Collection and Import of VIDEX Metadata
297+
298+
VIDEX employs a different data collection method for MySQL 5.7 compared to MySQL 8.0,
299+
while maintaining the same command parameters.
300+
301+
```bash
302+
cd $VIDEX_HOME
303+
python src/sub_platforms/sql_opt/videx/scripts/videx_build_env.py \
304+
--target ${HOST_MYSQL57}:13308:tpch_tiny_57:videx:password \
305+
--videx 127.0.0.1:13308:videx_tpch_tiny_57:videx:password
306+
```
307+
308+
#### Step 2.5: ✴️ Setting Parameters Adapted for MySQL 5.7
309+
310+
VIDEX can simulate MySQL 5.7 in standalone mode. Due to the differences between MySQL 5.7 and MySQL 8.0, we
311+
need to set the `optimizer-switch` variables and `server_cost` tables for VIDEX-optimizer.
312+
313+
✴️✴️ Note that, Since **setting the environment does not take effect in the current connection**, please run the following script first,
314+
then log into MySQL.
315+
316+
```bash
317+
mysql -h ${HOST_MYSQL57} -P13308 -uvidex -ppassword < src/sub_platforms/sql_opt/videx/scripts/setup_mysql57_env.sql
318+
```
319+
320+
#### Step 3: EXPLAIN SQL
321+
322+
We will use TPC-H Q21 as an example. The EXPLAIN result is as follows. We can see that the query plan for MySQL 5.7
323+
differs significantly from MySQL 8.0, yet VIDEX can still simulate it accurately:
324+
325+
![explain_tpch_tiny_table_for_mysql57.png](doc/explain_tpch_tiny_table_for_mysql57.png)
326+
327+
Below is a comparison of EXPLAIN cost details between MySQL 5.7 and VIDEX.
328+
329+
![explain_tpch_tiny_mysql57_compare.png](doc/explain_tpch_tiny_mysql57_compare.png)
330+
331+
#### Step 4: ✴️ Clear MySQL 5.7 Environment Variables
332+
333+
If you wish to revert the MySQL-optimizer to MySQL 8.0, please run the following script.
334+
335+
```bash
336+
mysql -h ${HOST_MYSQL57} -P13308 -uvidex -ppassword < src/sub_platforms/sql_opt/videx/scripts/clear_mysql57_env.sql
337+
```
338+
339+
### 3.3 TPCH sf1 (1g) Example (MySQL 8.0)
283340

284341
We provide metadata file for TPC-H sf1: `data/videx_metadata_tpch_sf1.json`, allowing direct import without collection.
285342

‎README_zh.md‎

Lines changed: 81 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,7 @@
1414
<a href="https://hub.docker.com/repository/docker/kangrongme/videx">
1515
<img src="https://img.shields.io/docker/pulls/kangrongme/videx?style=for-the-badge&logo=docker" alt="Docker Pulls"/>
1616
</a>
17+
<img src="https://img.shields.io/badge/MySQL|Percona-8.0|_5.7-FF9800?style=for-the-badge&logo=mysql" alt="MySQL Support"/>
1718
</p>
1819

1920
**VIDEX** 为 MySQL 提供了一个解耦的、可扩展的开源虚拟索引引擎 (**\[VI\]**rtual in**\[DEX\]**)。🚀
@@ -39,8 +40,8 @@
3940
默认情况下，VIDEX 可以以 `ANALYZE TABLE` 的方式收集统计信息，或者基于少量采样数据构建统计信息。
4041

4142
VIDEX 提供两种启动模式：
42-
1. **作为插件安装到生产数据库**：将 VIDEX 作为插件安装到生产数据库实例。
43-
2. **独立实例**：此模式可以完全避免影响在线运行实例的稳定性，在工业环境中很实用。
43+
1. **作为插件安装到生产数据库** (Plugin-Mode)：将 VIDEX 作为插件安装到生产数据库实例。
44+
2. **独立实例** (Standalone-Mode)：此模式可以完全避免影响在线运行实例的稳定性，在工业环境中很实用。
4445

4546
在功能方面，VIDEX 支持创建和删除索引（单列索引、复合索引、EXTENDED_KEYS 索引、[倒序索引](https://dev.mysql.com/doc/en/descending-indexes.html)）。
4647
目前暂不支持函数索引（`functional indexes`）、全文索引（`FULL-Text`）和空间索引（`Spatial Indexes`）。
@@ -115,7 +116,7 @@ python -m pip install -e . --use-pep517
115116

116117
#### 2.2.2 启动 VIDEX 容器
117118
```cmd
118-
docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:0.0.2
119+
docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:latest
119120
```
120121

121122
> **其他部署方式**
@@ -127,7 +128,7 @@ docker run -d -p 13308:13308 -p 5001:5001 --name videx kangrongme/videx:0.0.2
127128
128129
## 3. 示例
129130

130-
### 3.1 TPCH-Tiny 示例
131+
### 3.1 TPCH-Tiny 示例 (MySQL 8.0)
131132

132133
本示例使用 `TPC-H Tiny` 数据集(从 TPC-H sf1 随机采样 1% 数据)演示 VIDEX 的完整使用流程。
133134

@@ -279,7 +280,59 @@ ALTER TABLE tpch_tiny.orders DROP INDEX idx_o_orderstatus;
279280
ALTER TABLE videx_tpch_tiny.orders DROP INDEX idx_o_orderstatus;
280281
```
281282

282-
### 3.2 TPCH sf1 (1g) Example
283+
### 3.2 TPCH-Tiny 示例 (MySQL 5.7)
284+
285+
VIDEX 的独立实例模式现已支持高精度模拟 MySQL 5.7。
286+
287+
#### Step 1: 在 MySQL 5.7 实例中导入测试数据
288+
289+
在一台 MySQL 5.7 中导入数据。
290+
291+
```bash
292+
mysql -h${HOST_MYSQL57} -P13308 -uvidex -ppassword -e "create database tpch_tiny_57;"
293+
mysql -h${HOST_MYSQL57} -P13308 -uvidex -ppassword -Dtpch_tiny_57 < tpch_tiny.sql
294+
```
295+
296+
#### Step 2: VIDEX 采集并导入 VIDEX 元数据
297+
298+
VIDEX 对 MySQL5.7 会采取相适应的数据收集方式，但命令参数不变。
299+
300+
```bash
301+
cd $VIDEX_HOME
302+
python src/sub_platforms/sql_opt/videx/scripts/videx_build_env.py \
303+
--target ${HOST_MYSQL57}:13308:tpch_tiny_57:videx:password \
304+
--videx 127.0.0.1:13308:videx_tpch_tiny_57:videx:password
305+
```
306+
307+
#### Step 2.5: ✴️ 设置适配 MySQL5.7 的参数
308+
309+
VIDEX 能够以独立实例模式模拟 MySQL5.7。由于 MySQL5.7 与 MySQL8.0 的差异，我们需要设置 VIDEX-optimizer 的 `优化器参数`
310+
和 `代价常数表`。
311+
312+
✴️✴️ 请注意：由于**代价参数的变更无法在当前连接中直接生效**，因此，请首先运行如下脚本，再登入 MySQL。
313+
314+
```bash
315+
mysql -h ${HOST_MYSQL57} -P13308 -uvidex -ppassword < src/sub_platforms/sql_opt/videx/scripts/setup_mysql57_env.sql
316+
```
317+
318+
#### Step 3: EXPLAIN SQL
319+
320+
我们同样以 TPC-H Q21 作为示例。EXPLAIN 结果如下。可以看到，MySQL 5.7 的查询计划与 MySQL 8.0有显著不同，而 VIDEX 仍然可以准确模拟：
321+
322+
![explain_tpch_tiny_table_for_mysql57.png](doc/explain_tpch_tiny_table_for_mysql57.png)
323+
324+
下面是 MySQL5.7 和 VIDEX 的 EXPLAIN cost 细节对比。
325+
![explain_tpch_tiny_mysql57_compare.png](doc/explain_tpch_tiny_mysql57_compare.png)
326+
327+
#### Step 4: ✴️ 清除 MySQL5.7 的环境变量
328+
329+
如果想将 MySQL-optimizer 恢复为 8.0 模式，请执行如下脚本。
330+
331+
```bash
332+
mysql -h ${HOST_MYSQL57} -P13308 -uvidex -ppassword < src/sub_platforms/sql_opt/videx/scripts/clear_mysql57_env.sql
333+
```
334+
335+
### 3.3 TPCH sf1 (1g) 示例 (MySQL 8.0)
283336

284337
我们额外为 TPC-H sf1 准备了元数据文件：`data/videx_metadata_tpch_sf1.json`，无需采集，直接导入即可体验 VIDEX。
285338

@@ -409,7 +462,30 @@ VIDEX-Optimizer 将基于用户指定的地址，通过 `HTTP` 请求索引元
409462
}
410463
```
411464

465+
## 版本支持
466+
467+
### Plugin-Mode 支持列表
468+
469+
| 数据库系统 | 版本范围 | 支持状态 | 备注 |
470+
|---------|-----------|-----------|----------------------------|
471+
| Percona | 8.0.34-26 | ✅ 支持 | 在 全部 `TPC-H`、`JOB`场景下完成测试 |
472+
| MySQL | 8.0.42 | 🔄 即将支持 | 计划在下一版本提供支持 |
473+
| MariaDB | — | ⏳ 正在规划 | 与 MariaDB 社区持续讨论中 |
474+
| PG | - | 🔮 未来工作 | 期待与贡献者进行讨论 |
475+
476+
### Standalone-Mode 支持列表
477+
478+
| 数据库系统 | 版本范围 | 支持状态 | 备注 |
479+
|---------|------------|----------|----------------------------|
480+
| Percona | 8.0.34-26+ | ✅ 支持 | 在 全部 `TPC-H`、`JOB` 下完成测试 |
481+
| MySQL | 8.0.x | ✅ 支持 | 在 部分 `TPC-H` 下完成测试 |
482+
| MySQL | 5.7.x | ✅ 支持 | 在 部分 `TPC-H` 下完成测试 |
483+
| MariaDB | — | ⏳ 正在规划 | 与 MariaDB 社区持续讨论中 |
484+
| PG | - | 🔮 未来工作 | 期待与贡献者进行讨论 |
485+
486+
412487
## Authors
488+
413489
ByteBrain团队, 字节跳动
414490

415491
## Contact

0 commit comments

Comments
 (0)