Lake AI Service
Lake AI Service
- Documentation
Lake AI ServiceMetadata managementDirectories, databases & tables & functionsData catalog management
Copy page
Download PDFCopy pageDownload PDFWhat is a data catalog (Catalog)
The metadata management of the AI Data Lake Service (LAS) adopts a three-layer entity structure: Catalog/Database/Table. Each tenant has one default Catalog (hive), and you can also create other Catalogs. Multiple databases (Database) can be created under a Catalog, and multiple tables (Table) can be created under each database.
Applicable scenarios
Mainly applicable to scenarios requiring metadata isolation, such as different clusters, business units, or organizational teams. By binding different Catalogs, metadata in different Catalogs is not visible to each other.
The following introduces the key limitations when using LAS Catalog.
Restrictions | Value or range | Recommendation |
|---|---|---|
Maximum number of partitions returned per batch | 200000 | Fetch the partition list in batches to avoid returning an excessively large result set in a single request. |
Table encryption file format | Only PARQUET is supported | Enable table encryption only after selecting the PARQUET format when creating a table. |
Maximum number of authorization/revocation requests per list | 100 items per request | The maximum total length of the request list in a single call is 100 items. If exceeded, please split and submit multiple times. |
Maximum number of principals in a single request | 20 principals/request | A single request can include up to 20 principals (users/roles). If there are many principals, split them into multiple requests by principal. |
Maximum permissions granted to a single principal at one time | 5000 permissions/principal/time | Up to 5000 permission points can be granted to the same principal at one time. For large permission sets, please execute in multiple batches. |
Query a data catalog (Catalog)
- Log in to LAS Console , click the left navigation Lake management > Metadata, and enter the metadata page.
- On the metadata page, click the Catalog name you want to query to enter the details page and view the details.
Edit a data catalog (Catalog)
tip
- Users can only edit data catalogs for which they have permission. Currently, modifying the data catalog name is not supported.
- Data storage paths only support TOS paths.
- After changing the storage path of the Catalog, if the storage path of the default db is empty or not a TOS path, the storage path of the default db will be updated to: "Catalog path/default.db".
- Log in to LAS Console , click the left navigation Lake management > Metadata, and enter the metadata page.
- Select the data catalog you want to edit and click Edit in the upper right corner.
- Modify the storage path or notes in the pop-up window.
Create a data catalog (Catalog)
- Log in to LAS Console , click the left navigation Lake management > Metadata, and enter the metadata page.
- Click “Create catalog” above the data catalog, fill in the catalog name, storage location, and notes in the Create catalogpop-up window, and after completing the information, click Confirm to complete the creation of the data catalog.
Delete a data catalog (Catalog)
- Log in to LAS Console , click the left navigation Lake management > Metadata, and enter the metadata page.
- Select the data catalog to be deleted, click Delete in the upper right corner to complete the deletion of the Catalog.
tip
- Users can only delete data catalogs for which they have permission.
- Deleting a Catalog follows the order of deleting child nodes first, then parent nodes. That is, delete the DB first, then delete the Catalog. Similarly, when deleting a DB, delete the Tables under the DB first, then delete the DB.
- Deleting a Table/DB/Catalog follows the order of deleting metadata first, then deleting the corresponding file directory at the storage location (location). The actions of deleting metadata and deleting files cannot guarantee transactional consistency. That is, metadata may be deleted successfully while the file directory still exists. In other words, Catalog/DB/Table may no longer be found, but the corresponding file directory remains.
- When deleting a data catalog, files will also be deleted. However, currently, file deletion does not support a recycle bin feature. Once deleted, files cannot be recovered. To recover accidentally deleted data, enable versioning in TOS. For detailed instructions, please refer to Version Control.
- When deleting a Catalog, DBs are deleted first. If an error occurs while deleting a DB, DBs that have already been deleted will not be rolled back. Users can refresh the page and continue to delete DBs that have not yet been deleted.
