CREATE DATABASE) CREATE DATABASE [IF NOT EXISTS] db_name [ENGINE=Cnch]
Parameter description:
IF NOT EXISTS: If the db_name database already exists, ByteHouse will not create a new database and:
Create database using the default engine.
CREATE DATABASE IF NOT EXISTS test;
Create database using an optional engine; currently, only Cnch is supported.
CREATE DATABASE IF NOT EXISTS test ENGINE=Cnch;
CREATE TABLE) CREATE TABLE [IF NOT EXISTS] [db.]table_name ( name1 [type1] [NULL|NOT NULL] [DEFAULT|ALIAS expr1 | MATERIALIZED expr1] [COMMENT 'string'][compression_codec][TTL expr1], name2 [type2] [NULL|NOT NULL] [DEFAULT|ALIAS expr2 | MATERIALIZED expr1] [COMMENT 'string'] [compression_codec] [TTL expr2], ... ) ENGINE = CnchMergeTree() ORDER BY expr [PARTITION BY expr] [CLUSTER BY (column, expression, ...) INTO value1 BUCKETS SPLIT_NUMBER value2 WITH_RANGE] [PRIMARY KEY expr] [UNIQUE KEY expr] [SAMPLE BY expr] [TTL expr] [SETTINGS name=value, ...] [COMMENT='COMMENT']
CREATE TABLE IF NOT EXISTS test.createTable( id UInt32, name String DEFAULT '', CONSTRAINT constraint1 CHECK id > 10 ) ENGINE=CnchMergeTree ORDER BY id
In column definitions, the NULL and NOT NULL modifiers after the data type allow or disallow the column to be nullable.
If the data type is not nullable and NULL is specified, the column will be treated as nullable; if NOT NULL is specified, it will not be.
For example, INT NULL is the same as Nullable(INT). If the data type is nullable and NULL or NOT NULL modifiers are specified, an exception will be thrown.
The column description can specify a default value expression in the form of DEFAULT expr, MATERIALIZED expr, or ALIAS expr. Example: URLDomain String DEFAULT domain(URL).
The expression expr is optional. If omitted, the column type must be explicitly specified. The default value for numeric columns is 0, for string columns is '' (empty string), for array columns is [] (empty array), for date columns is 1970-01-01, and for nullable columns is NULL.
The column type for default value columns can be omitted, in which case it will be inferred from the type of expr. For example, the column EventDate DEFAULT toDate(EventTime) will have the type date.
If both the data type and default value expression are specified, an implicit type conversion function will be inserted to convert the expression to the specified type. For example: Hits UInt32 DEFAULT 0 is internally represented as Hits UInt32 DEFAULT toUInt32(0).
The default value expression expr can reference any table column and constant. ByteHouse checks for changes to the table structure will not introduce cycles in expression calculation. For INSERT operations, it checks whether the expression is solvable—that is, all columns required for calculation have been provided.
Normal default value. If the value for such a column is not specified in the INSERT query, its value will be calculated according to expr.
CREATE TABLE [db.]test ( id UInt64, updated_at DateTime DEFAULT now(), updated_at_date Date DEFAULT toDate(updated_at) ) ENGINE = CnchMergeTree ORDER BY id; INSERT INTO test (id) Values (1); SELECT * FROM test; ┌─id─┬──────────updated_at─┬─updated_at_date─┐ │1 │ 2023-02-24 17:06:46 │2023-02-24 │ └────┴─────────────────────┴─────────────────┘
Materialized expression. The value of such columns will be automatically calculated based on the specified materialized expression when inserting rows. During the INSERT operation, you cannot explicitly specify the values for these columns.
Additionally, this type of default value column is not included in the results of SELECT *. This is to ensure the invariance that the results of SELECT * can always be inserted back into the table using INSERT. This behavior can be disabled by setting asterisk_include_materialized_columns.
CREATE TABLE [db.]test ( id UInt64, updated_at DateTime MATERIALIZED now(), updated_at_date Date MATERIALIZED toDate(updated_at) ) ENGINE = CnchMergeTree ORDER BY id; INSERT INTO test Values (1); SELECT * FROM test; ┌─id─┐ │1 │ └────┘ SELECT id, updated_at, updated_at_date FROM test; ┌─id─┬──────────updated_at─┬─updated_at_date─┐ │1 │ 2023-02-24 17:08:08 │2023-02-24 │ └────┴─────────────────────┴─────────────────┘ SELECT * FROM test SETTINGS asterisk_include_materialized_columns=1; ┌─id─┬──────────updated_at─┬─updated_at_date─┐ │1 │ 2023-02-24 17:08:08 │2023-02-24 │ └────┴─────────────────────┴─────────────────┘
Calculated columns (alias). This type of column is not stored in the table, nor can values be inserted into it.
When a SELECT query explicitly references this type of column, the value is calculated from expr at query time. By default, SELECT * excludes ALIAS columns. This behavior can be disabled by setting asterisk_include_alias_columns.
When adding new columns using an ALTER query, old data for these columns will not be written. Instead, when reading old data without new column values, the expression will be calculated on the fly by default. However, if running these expressions requires different columns not specified in the query, those columns will also be additionally read, but only for the relevant data blocks.
If a new column is added to the table and its default expression is later changed, the value used for old data will also change (for those not stored on disk). Caution: When running background merges, the missing column data will be written into these columns in the merged parts.
It is not possible to set default values for elements in nested data structures.
CREATE TABLE [db.]test ( id UInt64, size_bytes Int64, size String Alias formatReadableSize(size_bytes) ) ENGINE = CnchMergeTree ORDER BY id; INSERT INTO test Values (1, 4678899); SELECT id, size_bytes, size FROM test; ┌─id─┬─size_bytes─┬─size─────┐ │1 │4678899 │ 4.46 MiB │ └────┴────────────┴──────────┘ SELECT * FROM test SETTINGS asterisk_include_alias_columns=1; ┌─id─┬─size_bytes─┬─size─────┐ │1 │4678899 │ 4.46 MiB │ └────┴────────────┴──────────┘
The partition key defines partitions, which are logical data sets within a table divided according to specified rules. Partitions can be created according to any criteria, such as by date. To reduce the amount of data that needs to be operated on, each partition is stored separately. When querying, ByteHouse tries to use the smallest subset of these partitions. The PARTITION BY expr clause is specified when creating the table. The partition key can be any expression of columns in the table. For example, to partition by month, the expression is toYYYYMM(date); or by tuple expression, such as (toMonday(date), EventType) and others.
Caution, the value range calculated by the partition expression in the table should not be too large (recommended not to exceed 10,000). Too many partitions will consume significant memory and incur considerable IO and computational overhead.
Properly designing partition keys can greatly reduce the amount of data scanned during searches. Typically, columns that are most frequently used as conditions in queries and whose value range does not exceed 10,000 (such as Date and others) should be considered as partition keys.
Example 1: Partition key defined by columnExpr
CREATE TABLE test.partitionByClause ( VisitDate Date, Hour UInt8, ClientID UUID ) ENGINE = CnchMergeTree PARTITION BY VisitDate ORDER BY Hour;
Example 2: Partition key defined in a tuple of columnExpr
CREATE TABLE test.partitionByClause ( VisitDate Date, Hour UInt8, ClientID UUID ) ENGINE = CnchMergeTree PARTITION BY (VisitDate,Hour) ORDER BY Hour;
Order keys can be a tuple of columns or any expression. For example: ORDER BY (OrderID, Date).
If sorting is not required, you can use ORDER BY tuple(), and DataPart will store data in the order it is inserted.
Syntax
[columnExpr] [ASCENDING|ASC|DESCENDING|DESC] [NULLS [FIRST|LAST] [COLLATE STRING_LITERAL]
[ASCENDING|ASC|DESCENDING|DESC]: Determines the sorting direction. If the direction is not specified, the default is ASC.[NULLS [FIRST|LAST]: Determines the sorting order for NaN and NULL.
NULLS LAST modifier: values come first, followed by NaN, and finally NULL.NULLS FIRST modifier: NULL comes first, then NaN, and finally other values.[COLLATE STRING_LITERAL]: For sorting by String values, you can specify collation rules.
COLLATE, sorting is always case-insensitive.COLLATE for final sorting of a small number of rows, as sorting with COLLATE is less efficient than normal byte-wise sorting.Example 1: Sorting by multiple orderExpr
CREATE TABLE IF NOT EXISTS test.orderByClause (id UInt32, val UInt32) ENGINE=CnchMergeTree() ORDER BY id; INSERT INTO TABLE test.orderByClause VALUES (1,5),(1,15),(2,5),(2,15); SELECT * FROM test.orderByClause ORDER BY id ASC, val DESC;
Example 2: Sorting by NULL FIRST
CREATE TABLE IF NOT EXISTS test.orderByClause (id UInt32, val Nullable(UInt32)) ENGINE=CnchMergeTree() ORDER BY id; INSERT INTO test.orderByClause VALUES (1,2),(2,NULL),(3,2),(3,3),(3,NULL); SELECT * FROM test.orderByClause ORDER BY val DESC NULLS FIRST;
Example 3: Sorting by COLLATE
CREATE TABLE IF NOT EXISTS test.orderByClause (x UInt32, s Nullable(String)) ENGINE=CnchMergeTree ORDER BY x; INSERT INTO test.orderByClause VALUES (1,'bca'),(2,NULL),(3,'ABC'),(4,'123a'),(5,'abc'),(6,NULL),(7,'BCA'); SELECT * FROM test.orderByClause ORDER BY s ASC COLLATE 'ru';
By default, there is no need to explicitly specify a primary key. ByteHouse will use the order key as the primary key. When there are special scenarios where the primary key and order key are not consistent, the primary key must be the leftmost prefix of the order key. For example, if the order key is (OrderID, Date), the primary key must be OrderID and cannot be Date.
ByteHouse will build a sparse index on the primary key, with Granule as the unit (in contrast, a dense index creates index information for every row).
If the search condition matches the leftmost prefix of the primary key index, the primary key index can quickly filter out the data granules that may need to be read. Compared to scanning the entire DataPart, this is usually much more efficient.
Additionally, caution is required: PRIMARY KEY does not guarantee uniqueness, so it is possible to insert rows with duplicate primary keys.
Partition (PARTITION BY) and PRIMARY KEY are two different methods for accelerating data queries. When defining them, it is recommended to use different columns for each to cover more query scenarios. For example, the first column in ORDER BY should not be duplicated in PARTITION BY. The following are some considerations for selecting a primary key:
Example
CREATE TABLE IF NOT EXISTS test.sampleByClause ( id UInt32 ) ENGINE=CnchMergeTree ORDER BY id SAMPLE BY id;
PRIMARY KEY does not guarantee deduplication. If deduplication by unique key is required, a unique key index must be set when creating the table. After setting a unique key, ByteHouse provides upsert update write semantics, allowing efficient updates of data rows based on the unique key, or, during upsert, by setting the virtual column delete_flag=1, you can delete the specified key. Queries automatically return the latest value for each unique key. For details, refer to ByteHouse best practices for unique key tables.
A unique key can be a tuple of columns or any expression, such as UNIQUE KEY (product_id, sipHash64(city)).
When querying by unique key, the unique key index is used to filter data and accelerate queries. Therefore, the primary key can usually be set to columns different from the unique key to cover more query conditions. However, to use partial column update functionality, the unique key must be the leftmost prefix of the sort key.
Example 1: Single-column Unique Key
CREATE TABLE test.uniqueKeyClause ( `event_date` Date, `order_id` UInt64, `state` UInt32, `amount` UInt64 ) ENGINE = CnchMergeTree PARTITION BY event_date ORDER BY state UNIQUE KEY order_id; -- rows with the same key will replace previous rows INSERT INTO test.uniqueKeyClause VALUES ('2021-03-01', 1001, 1, 100), ('2021-03-01', 1002, 1, 200), ('2021-03-01', 1001, 2, 100), ('2021-03-02', 1001, 1, 400); SELECT * FROM test.uniqueKeyClause ORDER BY event_date, order_id; ┌─event_date─┬─order_id─┬─state─┬─amount─┐ │ 2021-03-01 │ 1001 │ 2 │100 │ │ 2021-03-01 │ 1002 │ 1 │200 │ │ 2021-03-02 │ 1001 │ 1 │400 │ └────────────┴──────────┴───────┴────────┘ INSERT INTO test.uniqueKeyClause VALUES ('2021-03-01', 1002, 2, 200), ('2021-03-02', 1001, 2, 400), ('2021-03-02', 1002, 1, 300); SELECT * FROM test.uniqueKeyClause ORDER BY event_date, order_id; ┌─event_date─┬─order_id─┬─state─┬─amount─┐ │ 2021-03-01 │ 1001 │ 2 │100 │ │ 2021-03-01 │ 1002 │ 2 │200 │ │ 2021-03-02 │ 1001 │ 2 │400 │ │ 2021-03-02 │ 1002 │ 1 │300 │ └────────────┴──────────┴───────┴────────┘
Example 2: Multi-column Unique Key
CREATE TABLE test.uniqueKeyClause ( `event_date` Date, `region` UInt64, `state` String, `amount` UInt64 ) ENGINE = CnchMergeTree PARTITION BY event_date ORDER BY state UNIQUE KEY (region, state);
Bytehouse supports CONSTRAINT syntactically, but except for NOT NULL, does not enforce these constraints.
Bucketing is commonly used in the following scenarios:
warning
Modifying an existing table to add buckets only affects new partitions and does not affect existing partitions.
Bucketing application example
Suppose a ByteHouse subscription user has enabled six compute nodes. Due to the large amount of data in a single partition, exceeding 200 million records, the application frequently performs aggregation and join operations based on the c1 and c2 fields. Therefore, it was decided to optimize using a bucketed table. The design options for the bucketed table are as follows:
c1 and c2 columns as the bucketing keys.-- Create table with bucketing create or replace table table_01 (c1 timestamp, c2 string, c3 number) cluster by (to_date(c1), c2) INTO 12 BUCKETS; -- Add bucket to existing data ALTER TABLE t CLUSTER BY (column, expression, ...) INTO 64 BUCKETS -- Add bucket to cluster by multiple columns ALTER TABLE t CLUSTER BY sipHash(a,b,c) INTO 64 BUCKETS -- Add or change the cluster properties ALTER TABLE t MODIFY CLUSTER BY (column, expression, ...) INTO 64 BUCKETS
The expression used for sampling. This configuration is optional.
To use a sampling expression, the primary key must contain this expression. For example: SAMPLE BY intHash32(UserID) ORDER BY (CounterID, EventDate, intHash32(UserID)).
A list of rules specifying the retention duration of rows and defining the logic for moving data fragments across disks and volumes. This configuration is optional. The expression must contain at least one column of Date or DateTime type, for example: TTL date + INTERVAL 1 DAY.
CREATE TABLE test.ttlClause ( VisitDate Date, Hour UInt8, ClientID UUID ) ENGINE = CnchMergeTree ORDER BY Hour PARTITION BY VisitDate TTL VisitDate + INTERVAL 1 DAY;
To implement TTL based on a string-type field, use the toDate function during table creation to first convert it to Date or DateTime type, and then set TTL using the converted date-time column. For usage of the toDate function, see toDate
CREATE TABLE test.ttlClause ( start_date STRING, -- String-type date field Hour UInt8, ClientID UUID ) ENGINE = CnchMergeTree ORDER BY Hour TTL toDate(start_date) + INTERVAL 1 DAY; -- Set TTL after converting the string to Date type
The compression_codec field can be used for codec configuration. This configuration is optional, and the default is LZ4.
Support project general-purpose encoding and specific encoding. General-purpose codecs are similar to default codecs (LZ4, ZTSD) and their modified versions. Specific codecs are designed to leverage particular characteristics of the data for more effective compression.
General encoding
Specific encoding algorithms
Multiple codecs
Reference example:
CREATE TABLE codec_example (date Date CODEC(Delta, ZSTD), ts DateTime CODEC(LZ4HC), float_value Float32 CODEC(NONE), double_value Float64 CODEC(LZ4HC(9)) ) ENGINE = CnchMergeTree PARTITION BY tuple() ORDER BY date
The table engine is the type of the table and determines:
The most commonly used table engine in the ByteHouse cloud data warehouse edition is CnchMergeTree, in addition to other special types of table engines including Hive external tables, Kafka tables, and others. This article focuses on the principles of the CnchMergeTree table engine.
CnchMergeTree is the most commonly used table engine. Its core concept is similar to LSM-Tree: data is partitioned by partition key (partition by), then stored in order by sort key (order by). Main features include:
If a partition key is specified, data is divided into different logical datasets (logical partitions, Partition) according to the partition key.
Each logical partition can contain zero or more DataParts. If the query conditions can prune partitions, this usually accelerates the query. If no partition key is specified, all data is stored in a single logical partition.
Data within a DataPart is sorted by the sort key. Each DataPart also has a min/max index to accelerate partition selection.
Each DataPart is logically divided into granules. The default granule is 8192 rows (determined by the table's index_granularity configuration). A granule is the smallest indivisible dataset for data query in ByteHouse. The first row of each granule is marked by the primary key value of that row. ByteHouse creates an index file for each DataPart to store these marks. For every column, whether or not it contains the primary key, ByteHouse stores similar marks. These marks allow you to directly locate data in the column file. Granule serves as the indexing goal for ByteHouse sparse indexes and is also the unit for data scanning in memory.
Background tasks periodically merge DataParts within the same partition and maintain order by the sort key. Background merge reduces the number of parts for more efficient storage and improves query performance.
warning
MySQL-compatible syntax is recommended for use under relevant dialects.
ByteHouse CDW 2.1 is compatible with most MySQL DDL and can directly execute MySQL create table statements in ByteHouse. Please refer to the following example:
MySQL
CREATE TABLE `table1` ( `uid` bigint NOT NULL AUTO_INCREMENT, `phone` varchar(255) DEFAULT NULL, `no1` bigint DEFAULT NULL, `appleid` varchar(255) DEFAULT NULL, PRIMARY KEY (`uid`) USING BTREE, UNIQUE KEY `no1` (`no1`) USING BTREE, KEY `idx_apple_id` (`appleid`) USING BTREE, KEY `phone` (`phone`) USING BTREE ) ENGINE=InnoDB AUTO_INCREMENT=60937910 DEFAULT CHARSET=utf8mb3;
ByteHouse
CREATE TABLE table1 ( `uid` Int64 NOT NULL, `phone` String NULL DEFAULT NULL, `no1` Int64 NULL DEFAULT NULL, `appleid` String NULL DEFAULT NULL ) ENGINE = CnchMergeTree ORDER BY tuple(uid) UNIQUE KEY tuple(uid) SETTINGS partition_level_unique_keys = 0, storage_policy = 'cnch_default_hdfs', allow_nullable_key = 1, storage_dialect_type = 'MYSQL', index_granularity = 8192
The following conversion logic can be observed
| | | | | \
| Detailed comparison items | MySQL | ByteHouse | Description | |
|---|---|---|---|---|
| \ | ||||
| Column attributes | Data type | Compatible | For details on MySQL data compatibility, please refer to basic data types. | |
| \ | ||||
| AUTO_INCREMENT | Compatible | ByteHouse currently supports self-increasing ID capability (generateSnowflakeID). When creating tables, the comparison between the two is as follows: | \ | |
| \ | ||||
| * ByteHouse | \ | |||
| ```SQL | \ | |||
| set dialect_type='CLICKHOUSE'; | \ | |||
| CREATE TABLE t1(id Int64 DEFAULT generateSnowflakeID(), val1 String) | \ | |||
| Engine=CnchMergeTree() Order by id; | \ | |||
| ``` | \ | |||
| \ | ||||
| * MySQL | \ | |||
| ```SQL | \ | |||
| set dialect_type='MYSQL'; | \ | |||
| create table t1(id Int64 auto_increment primary key, val1 String); | \ | |||
| ``` | \ | |||
| \ | ||||
| \ | ||||
| For more details about using ByteHouse self-increasing functions, please refer to self-increasing functions. | ||||
| \ | ||||
| NULL | Compatible | |||
| \ | ||||
| DEFAULT | Compatible | |||
| \ | ||||
| Other definitions | Compatible | ByteHouse is compatible with most commonly used field definitions | ||
| \ | ||||
| Table attributes | \ | |||
| Primary key: PRIMARY KEY | Unique key: UNIQUE KEY | In MySQL semantics, the primary key corresponds to the unique key in ByteHouse, not the primary key | ||
| \ | ||||
| Sort key: ORDER BY | Default sorting by MySQL primary key | |||
| \ | ||||
UNIQUE KEY no1 (no1) USING BTREE | Ignored | ByteHouse supports Constraint syntax, but ignores it functionally | \ | |
| \ | ||||
| Other definitions: | \ | |||
| ENGINE=InnoDB AUTO_INCREMENT=60937910 DEFAULT CHARSET=utf8mb3; | Ignored | Ignored | ||
| \ | ||||
| Settings | None | Settings | When creating tables in the MySQL dialect, ByteHouse will add the following settings by default | \ |
| \ | ||||
| 1. partition_level_unique_keys = 0 -> Partition deduplication is turned off to ensure deduplication across the entire table | \ | |||
| 2. storage_policy = 'cnch_default_hdfs', index_granularity = 8192 -> Default settings | \ | |||
| 3. allow_nullable_key = 1 -> Allows nullable key as unique key | \ | |||
| 4. storage_dialect_type = 'MYSQL' -> Storage dialect type |
tip
ByteHouse also provides a certain degree of compatibility for other database syntax. If you are interested, please contact us.
:::
Create a table whose structure is similar to the result of a SELECT query, use the specified engine, and populate it with data from the SELECT query.
CREATE TABLE [IF NOT EXISTS] [db.]table_name[(name1 [type1], name2 [type2], ...)] ENGINE = CnchMergeTree ORDER BY expr [PARTITION BY expr] AS SELECT ...
If the table already exists and the IF NOT EXISTS clause is specified, the query will not take any action. Other clauses may follow the ENGINE clause in the query.
Example
CREATETABLE DB2.unique_user_events ( user_id UInt32, event_time DateTime, event_type String, event_data String ) ENGINE = CnchMergeTree() ORDER BY user_id PRIMARY KEY user_id UNIQUE KEY event_time; INSERT into DB2.unique_user_events(user_id,event_time,event_type,event_data) values(10112,'2024-08-28 05:11:58','ERROR','this is a bug'); INSERT into DB2.unique_user_events(user_id,event_time,event_type) values(10112,'2024-07-28 05:11:58','ERROR'); INSERT into DB2.unique_user_events(user_id,event_time,event_data) values(10112,'2024-07-28 05:11:58','this is a bug'); CREATE TABLE IF NOT EXISTS DB2.table_name2(user_id UInt32,event_time DateTime,event_data String) ENGINE = CnchMergeTree ORDER BY user_id AS select user_id,event_time,event_data from DB2.unique_user_events limit 100; select * from DB2.table_name2;
A normal view (View) does not store any actual data; it is only used to save a predefined query statement. When reading view data, the predefined query will automatically be used as a subquery and embedded in the FROM clause of the final executed SQL statement.
In versions of ByteHouse Cloud Data Warehouse Engine prior to v2.3, the execution privilege of a normal view (View) depends on both the read privilege of the view itself and the read privilege of the underlying table. However, in some scenarios, users want to process only the view privilege without concern for the underlying table privilege. Starting from v2.3, ByteHouse Cloud Data Warehouse Edition introduces the view SQL Security function, adding the roles of DEFINER and INVOKER to control the execution behavior of views. This feature is suitable for scenarios where restricted data access privileges need to be provided to different users.DEFINER, INVOKER, and the differences compared to not using the SQL Security function are described as follows:
| | | \
| Roles | Role Description | |
|---|---|---|
| \ | ||
DEFINER | The DEFINER is the user specified when creating the view. When selecting users, ensure that the user has all privileges for the view and query privileges for the base table. | \ |
| When using this mode to create a view, subsequent users who initiate view queries only need query privilege for the view to successfully query the view, without requiring privilege for the base table. | ||
| \ | ||
INVOKER | The INVOKER is the user who initiates the view query. They must have query privilege for both the view and the base table in order to execute the view query. | |
| \ | ||
| No SQL Security settings | The view uses the INVOKER role by default, which means that the user who initiates the view query must have query privilege for both the view and the base table in order to execute the view query. |
View SQL Security only acts as a base table privilege proxy during view query execution by the ByteHouse engine. The actual privileges obtained by the user querying the view are not changed as a result. For views that have already been created, the ByteHouse engine uses the INVOKER role by default. You can refer to the section below on changing view SQL Security to adjust the SQL Security settings for the view.
This section introduces how to create and manage views, with a focus on using the SQL Security function for fine-grained access control.
warning
Currently, only engine versions v2.3 and above support the view SQL Security function.
:::
CREATE [OR REPLACE] VIEW [IF NOT EXISTS] [db.]view_name [DEFINER = { `user` | CURRENT_USER }] [SQL SECURITY { DEFINER | INVOKER }] AS SELECT ... [COMMENT 'comment']
parameter description
| | | | \
| Param | Required | parameter description | |
|---|---|---|---|
| \ | |||
| OR REPLACE | No | According to whether a view with the same Name currently exists in ByteHouse, select the corresponding Rules to create the view. The specific Rules are as follows: | \ |
| \ | |||
| * If there is no view with the same Name, ByteHouse will create a new view. | \ | ||
| * If a view with the same Name exists, ByteHouse will first Delete the original view with the same Name, then recreate it. | |||
| \ | |||
| IF NOT EXISTS | No | Determine whether the newly created view already exists. If the view view_name already exists, ByteHouse will not create a new Database and: | \ |
| \ | |||
| * If this clause is specified, no exception will be raised. | \ | ||
| * If this clause is not specified, an exception will be thrown. | |||
| \ | |||
| [db.]view_name | No | Defines the Database where the view is located and allows you to customize the view Name. The Database name is an optional parameter. If you have already run USE database_name, there is no need to specify the Database name here. | |
| \ | |||
| DEFINER | No | Saves information about the view's definers, which is used for privilege verification when running SELECT on the underlying table. This parameter must be used together with the SQL SECURITY DEFINER parameter to explicitly specify a User as the definer. The User specified by DEFINER must have all privileges on the underlying tables. For actions to grant Data privileges to a User, see Data privilege management. | \ |
| The default value is the User who initiates the current query, that is, CURRENT_USER. | \ | ||
| The parameter value supports setting as user or CURRENT_USER: | \ | ||
| \ | |||
* user: This parameter value is formed by concatenating the ByteHouse Account number ID and User ID, in the format {volcano_ID}.{bytehouse_ID}, and must be enclosed in backticks. Example: | \ | ||
| ```SQL | \ | ||
DEFINER = 21xxxxxx68.45xxxx79 | \ | ||
| ``` | \ | ||
| \ | |||
| The method for obtaining the ByteHouse Account number ID and User ID is as follows: | \ | ||
| * ByteHouse Account number ID: In the ByteHouse cloud Data warehouse Console, click the user's profile picture in the upper right corner to view and copy the Account number ID. | \ | ||
| * User ID: In the ByteHouse cloud Data warehouse Console, click the top access control, click User, in the User list, click User ID to copy the ID. | \ | ||
| * CURRENT_USER: Indicates the User who is currently running the statement. | |||
| \ | |||
| SQL SECURITY | No | Sets the privilege mode for running queries on the underlying tables. The parameter value supports setting as DEFINER or INVOKER, with the default value being INVOKER. | \ |
| \ | |||
* DEFINER: Indicates that view-related statements are run using the identity of the definer. When using this privilege mode, you must configure both the DEFINER parameter and the SQL SECURITY DEFINER parameter at the same time. | \ | ||
| In this privilege mode, when running SELECT statements to query view Data, the System only verifies whether the User executing the query has privilege to query the view; there is no need for underlying table privilege. | \ | ||
* INVOKER: Indicates that view-related statements are run using the identity of the invoker. When using this privilege mode, the DEFINER clause can be omitted. The engine will set the DEFINER clause to CURRENT_USER by default. | \ | ||
| Under this privilege mode, when subsequently executing a SELECT statement to query view data, the privileges used are those of the user who actually initiates the query. The system will verify whether the user has the following privileges: | \ | ||
| * Read privilege for the view; | \ | ||
| * Read privilege for the base table referenced by the view. | \ | ||
| If the user does not have privileges for the view or the base table, the system will report an error. | |||
| \ | |||
| AS SELECT | No | Use the SELECT statement to specify the view's data source. | |
| \ | |||
| COMMENT 'comment' | No | Add a descriptive comment for the materialized view. |
The syntax is the same as a normal query.
SELECT a, b, c FROM view_name
You can use the ALTER statement to adjust the privileges for using the view.
INVOKER to DEFINER, it is recommended to adjust both DEFINER and SQL SECURITY at the same time to obtain accurate information. This is because if only one item is adjusted, ByteHouse will automatically reset the other item, using the default value during the reset.DEFINER to INVOKER, you only need to set SQL SECURITY INVOKER.# Method 1 ALTER TABLE [db.]view_name MODIFY SQL SECURITY { DEFINER | INVOKER } [DEFINER = { user | CURRENT_USER }] # Method 2 ALTER TABLE [db.]view_name MODIFY [DEFINER = {user | CURRENT_USER }] SQL SECURITY { DEFINER | INVOKER }
You can use the GRANT statement to grant view-related privileges to users.
Privilege to create views
Creating a view requires the CREATE VIEW privilege. To use the SQL Security function when creating a view, if the user executing the create view (CREATE VIEW) operation is not the same as the specified DEFINER, you must grant the SET DEFINER privilege to the user executing the create view operation.
Example:
GRANT CREATE VIEW ON [db.]view_name TO user; GRANT SET DEFINER ON another_user TO user;
parameter description
| | | | \
| Param | Required | parameter description | |
|---|---|---|---|
| \ | |||
| CREATE VIEW | Yes | Privilege to create views. | |
| \ | |||
| SET DEFINER ON | Yes | Grant the privilege to specify DEFINER = user in the view schema when creating a view. Only users who acquire this privilege can specify DEFINER information when creating a view, and can only set it to a username that has acquired the SET DEFINER privilege. | \ |
| \ | |||
* If the user executing the create view (CREATE VIEW) operation is the same as the username specified in the DEFINER parameter in the create view statement, there is no need to explicitly grant the SET DEFINER privilege. | \ | ||
* If you need to specify DEFINER as another user when creating a view, you must explicitly grant the SET DEFINER privilege to that user. If this privilege is not granted, the user will not have permission to use DEFINER = user and SQL SECURITY when creating a view. | \ | ||
* Please ensure that the username authorized in GRANT SET DEFINER and the username specified in DEFINER = xxx when creating the view are identical. Otherwise, the user will not be able to obtain the corresponding DEFINER privileges. | |||
| \ | |||
| TO user | is | specifies the user to whom privileges are granted. This parameter value can be set to user or CURRENT_USER: | \ |
| \ | |||
* user: This parameter value is formed by concatenating the BytePlus account ID and user ID, in the format {volcano_ID}.{bytehouse_ID}, and must be enclosed in backticks (``). Example: 21xxxxxx68.45xxxx79. The BytePlus account ID and user ID can be obtained as follows: | \ | ||
| * BytePlus account ID: In the ByteHouse Console, click the user profile picture in the upper right corner to view and copy the account ID. | \ | ||
| * User ID: In the ByteHouse Console, click the top access control, click user, in the user list, click user ID to copy the ID. | \ | ||
| * CURRENT_USER: Refers to the user executing the current statement. |
Privileges for using views
Only the SELECT privilege on the view is granted to the user.
GRANT SELECT ON [db.]view_name TO user;
Privileges for deleting and modifying views
To delete a view, only the DROP VIEW privilege needs to be granted to the user.
GRANT DROP VIEW ON [db.]view_name TO user;
To modify a view, the user must be granted both ALTER VIEW MODIFY SQL SECURITY and SET DEFINER privileges. For granting the SET DEFINER privilege, refer to the above section "Privileges for creating views".
GRANT ALTER VIEW MODIFY SQL SECURITY ON [db.]view_name TO user; -- Alternatively, you can grant the ALTER VIEW privilege set to the user GRANT ALTER VIEW ON [db.]view_name TO user;
To facilitate demonstration of privileges for different users, the following examples use three users: administrator, ViewAdmin, and ViewUser. In actual use, replace the placeholders with the actual username (concatenation of BytePlus ID and ByteHouse ID). For details on how to obtain these, see the parameter description in the above create view section.
Procedure:
Create test users and base table.
-- Use the administrator account to create database and table -- Create test database CREATE DATABASE IF NOT EXISTS test_db; -- Create base table CREATE TABLE test_db.employee ( id INT, name STRING, salary INT ) ENGINE = MergeTree() ORDER BY id; -- Insert test data INSERT INTO test_db.employee VALUES (1, 'Alice', 5000), (2, 'Bob', 6000); -- Grant ViewAdmin user the base table access privileges and basic view management privileges GRANT SELECT ON test_db.employee TO ViewAdmin; GRANT CREATE VIEW ON test_db.* TO ViewAdmin; -- Optional: In this example, since the ViewAdmin user creates the view and the DEFINER is also ViewAdmin, there is no need to explicitly grant the SET DEFINER privilege GRANT SET DEFINER ON ViewAdmin TO ViewAdmin;
Create views with different privilege modes.
Create SQL SECURITY DEFINER view
-- Switch to ViewAdmin user to execute CREATE VIEW test_db.employee_view_definer DEFINER = ViewAdmin SQL SECURITY DEFINER AS SELECT id, name FROM test_db.employee; -- Only expose certain fields -- Grant view query privilege to ViewUser (no need to grant base table privilege) GRANT SELECT ON test_db.employee_view_definer TO ViewUser;
Verification result:
-- Use ViewUser to query the view; the query is successful SELECT * FROM test_db.employee_view_definer; -- Result: -- 1 Alice -- 2 Bob -- Use ViewUser to directly query the base table; due to lack of privilege, the query fails SELECT * FROM test_db.employee; -- Error: Permission Denied.
Create SQL SECURITY INVOKER view
-- Switch to ViewAdmin to execute CREATE VIEW test_db.employee_view_invoker SQL SECURITY INVOKER AS SELECT id, salary FROM test_db.employee; -- Only grant ViewUser view privilege (do not grant base table privilege) GRANT SELECT ON test_db.employee_view_invoker TO ViewUser;
Verification result:
-- Use ViewUser to query the view; due to lack of base table privilege, the query fails SELECT * FROM test_db.employee_view_invoker; -- After the Administrator grants ViewUser the base table privilege and queries again, the query is successful GRANT SELECT ON test_db.employee TO ViewUser; SELECT * FROM test_db.employee_view_invoker; -- Result: -- 1 5000 -- 2 6000
Create a view without using SQL Security
-- Switch to ViewAdmin to execute; the view by default uses SQL SECURITY INVOKER CREATE VIEW test_db.employee_view_normal AS SELECT id, name FROM test_db.employee; -- Only grant ViewUser view query privilege GRANT SELECT ON test_db.employee_view_normal TO ViewUser;
Verification result:
-- Use ViewUser to query the view; due to lack of base table privilege, the query fails SELECT * FROM test_db.employee_view_normal; -- Error: Permission Denied. -- Use ViewUser to directly query the base table; due to lack of base table privilege, the query fails SELECT * FROM test_db.employee; -- Error: Permission Denied.
At this point, if ViewUser is granted the base table query privilege, then ViewUser can query the view data.
-- Use the Administrator to grant ViewUser base table query privilege GRANT SELECT ON test_db.employee TO ViewUser; -- Use ViewUser to execute the view query again; the query is successful SELECT * FROM test_db.employee_view_normal; -- Result: -- 1 Alice -- 2 Bob
Change the view privilege mode.
-- Use the Administrator to grant ViewAdmin privilege to change the view GRANT ALTER VIEW MODIFY SQL SECURITY ON test_db.employee_view_definer TO ViewAdmin; -- Use ViewAdmin to change the DEFINER view to INVOKER mode ALTER TABLE test_db.employee_view_definer MODIFY SQL SECURITY INVOKER; -- Use ViewUser to query the view SELECT * FROM test_db.employee_view_definer; -- If ViewUser is not granted base table privilege, an error will occur at this time
Delete the view.
-- Use the Administrator to grant ViewAdmin delete privilege GRANT DROP VIEW ON test_db.employee_view_definer TO ViewAdmin; -- Use ViewAdmin to delete the view DROP VIEW test_db.employee_view_definer;
The following error examples use ViewCreator and ViewUser as examples to show common privilege exceptions.
Common error:
ViewCreator: Not enough privileges. To execute this query it's necessary to have grant SET DEFINER ON ViewAdmin
Reason:
Query user privileges and find that the user lacks the SET DEFINER privilege.
Solution:
Grant the affected user the SET DEFINER privilege, and the specified DEFINER username should match the DEFINER specified in the query.
Example:
GRANT SET DEFINER ON ViewCreator TO ViewCreator;
Common error:
ViewCreator: Not enough privileges. To execute this query it's necessary to have grant ALTER VIEW MODIFY SQL SECURITY ON testdb.test_view3
Reason:
Query user privileges and find that the user does not have privilege to change the view.
Solution:
Grant ALTER VIEW privilege to the user. Example:
GRANT ALTER VIEW ON db_name.view_name TO ViewCreator;
Common error:
ViewUser: Not enough privileges. To execute this query it's necessary to have grant SELECT(tag, value) ON testdb.test_table
Reason:
After checking the view schema, it is found that the execution privilege is INVOKER, and the view user ViewUser does not have base table privilege.
Solution:
Method 1: Change the view and set the execution privilege parameter value to DEFINER. Also, ensure that the user specified by DEFINER has base table privilege.
ALTER TABLE db_name.view_name MODIFY SQL SECURITY DEFINER DEFINER = `user`;
Method 2: Grant base table privilege to the query user.
GRANT SELECT ON db_name.table_name TO ViewUser;
For details, see the materialized view section.
ByteHouse Data dictionary is divided into built-in dictionary and external extension dictionary:
This section focuses on how to operate ByteHouse external extension dictionaries. For more Data dictionary function introduction, please refer to Data dictionary.
You can directly use DDL statements in the SQL workbench to create a new dictionary table.
CREATE DICTIONARY [IF NOT EXISTS] [db.]dictionary_name ( key1 type1[DEFAULT|EXPRESSION expr1] [HIERARCHICAL|INJECTIVE|IS_OBJECT_ID], key2 type2[DEFAULT|EXPRESSION expr2] [HIERARCHICAL|INJECTIVE|IS_OBJECT_ID], attr1 type2 [DEFAULT|EXPRESSION expr3], attr2 type2 [DEFAULT|EXPRESSION expr4] ) PRIMARY KEY key1, key2 SOURCE(SOURCE_NAME([param1 value1 ... paramN valueN])) LAYOUT(LAYOUT_NAME([param_name param_value])) LIFETIME({MIN min_val MAX max_val | max_val})
Search dictionary data
Get data from the Data dictionary by using dictGet / dictGetOrDefault statements and functions.
dictGet('db.dict_name', 'attr_name', id_expr) dictGetOrDefault('db.dict_name', 'attr_name', id_expr, default_value_expr)
tip
The first parameter of the dictGet function must be in the format db.dict_name, because the dictionary cache is indexed by this field.
Delete dictionary
ByteHouse will delete the record of this dictionary in the Catalog.
DROP DICTIONARY [IF EXISTS] [db.]dictionary_name;
Search dictionary
SHOW DICTIONARIES [FROM <db>];
If the FROM clause is not specified, the query will return the list of dictionaries in the current Database.
Search created by query dictionary statement
SHOW CREATE DICTIONARY [db.] dictionary_name
This query returns a string-type "statement" column, which contains a value—the CREATE query statement used to create the specified object.
ByteHouse supports dynamically setting the number of buckets at the partition level. You can set the bucketing key using an expression and create a dynamic partition table using the settings parameter. For specific examples, see Bucket Table best practices for dynamic scaling.
This function is currently in phased rollouts. If you encounter any problems during use, you can submit a ticket or contact the ByteHouse team.
CREATE TABLE t(name1 type1, name2 type2, ...) engine=CnchMergeTree() PARTITION BY ... CLUSTER BY EXPRESSION expr INTO 4 BUCKETS ORDER BY ... [UNIQUE KEY ...] SETTINGS enable_recluster_multiple_parts=1, enable_scale_bucket_optimization= 1
The parameter description is as follows:
| | | | \
| Param | Required | Description | |
|---|---|---|---|
| \ | |||
| CREATE TABLE t(name1 type1, name2 type2, ...) | Yes | Customize the table name and column information, including column names and data types. | |
| \ | |||
| engine=CnchMergeTree() | Yes | Define the engine to use, configured as CnchMergeTree. | |
| \ | |||
| PARTITION BY ... | Yes | Define the partition key for the partitioned table. | |
| \ | |||
| CLUSTER BY EXPRESSION expr INTO 4 BUCKETS | Yes | Use an expression to define the bucketing key. This expression will generate an unsigned integer (uint) output, and you can define this expression according to actual requirements. | \ |
To achieve better performance, it is recommended to use the cityHash64V2 function to wrap this expression. | |||
| \ | |||
| ORDER BY ... | Yes | Define the sorting key. | |
| \ | |||
| [UNIQUE KEY ...] | No | Define the unique key. | |
| \ | |||
| SETTINGS | Yes | Configuration parameters. Use the settings parameter to enable the dynamic scaling feature for the bucketing table. The currently supported settings parameters are as follows: | \ |
| \ | |||
* enable_recluster_multiple_parts: Table setting. When enabled, it can reduce the number of parts output after bucketing and improve query performance. The default value is 0 (Disabled), and it can be set to 1 (Enabled). | \ | ||
* enable_scale_bucket_optimization: Table setting. When enabled, supports using the new bucketing key expression for select queries. This feature also applies to scenarios where adjusting the number of buckets results in different total numbers of buckets per partition. If not enabled, queries will still use the old bucketing key and cannot use the new bucketing key expression. The default value is 0 (Disabled), and it can be set to 1 (Enabled). | \ | ||
* force_modify_unique_table_cluster_by: Query setting, used to modify the cluster by statement in unique key tables. The default value is 0 (Disabled), and it can be set to 1 (Enabled). | \ | ||
* enable_legacy_cluster_by: Query setting. When enabled, supports modifying the bucketing definition using both the new bucketing key expression (CLUSTER BY expression) and the old bucketing key syntax (CLUSTER BY <column_name>). The default value is 1 (Enabled). After setting this value to 0 (Disabled), only the new bucketing key expression is supported for creating bucketing tables and modifying the number of buckets. | \ | ||
| \ | |||
| The following are examples of adjusting the number of buckets using the new bucketing key expression: | \ | ||
| \ | |||
| ```SQL | \ | ||
| ALTER TABLE t MODIFY BUCKET NUMBER xx | \ | ||
| ``` | \ | ||
Related documentation