Operations and maintenance (O&M) is essential for ensuring the stable operation of the platform. This document will guide O&M engineers in rapid problem analysis and resolution, as well as daily application O&M tasks. This document is divided into two main sections: application service O&M and core component O&M. It is primarily intended for on-premises deployments of Intelligent Data Insight.

O&M engineers can manage services via the Kubernetes dashboard or perform daily O&M tasks using kubectl commands.

2.1 Kubernetes Dashboard #

Users can access the Kubernetes page via the following URL: http://${datawind-domain}:32002/#!/overview?namespace=_all.

2.2 Dashboard interface operations #

The Dashboard interface provides a global O&M perspective for application services. The main objects for operation are clusters, workloads, configuration, and storage.

2.3 Cluster #

The cluster control panel allows viewing of current information and health status for Kubernetes cluster nodes.
Kubernetes node control panel:

2.3 Workloads #

Workloads are the primary entities carrying cluster services, including deployments and scheduled tasks. Through the workload operations panel, you can view current service information and perform corresponding actions (delete/scale/view and modify configuration).
Kubernetes deployment control panel:
Kubernetes scheduled task control panel:

2.4 Configuration and storage #

Configuration and storage provide the configuration and media for persistent storage of deployed services. The control panel allows deletion and editing operations.
Kubernetes configuration and storage control panel:

2.5 Kubectl command line #

The following are several commonly used commands.
View all containers:
./kubectl --kubeconfig=/opt/runtime/business/k8s/kube_config_cluster.yml get pods --all-namespaces
Terminate a specific Pod:
./kubectl --kubeconfig=/opt/runtime/business/k8s/kube_config_cluster.yml delete pod ${pod} -n ${namespace}
View logs:
./kubectl --kubeconfig=/opt/runtime/business/k8s/kube_config_cluster.yml logs ${pod} -n ${namespace}
Log in to a container:
./kubectl --kubeconfig=/opt/runtime/business/k8s/kube_config_cluster.yml exec -it ${pod} -n ${namespace} bash

To improve stability and availability, all core components of Intelligent Data Insight use systemd for daemon management, enabling automatic recovery after abnormal termination or server reboot. The following describes several core components.

3.1 Consul #

Component purpose: service discovery
Deployment path:
Server deployment path: /opt/consul/consul_deploy
Agent deployment path: /opt/tiger/consul_deploy
Log path:
Server logs: /opt/log/tiger/consul_server_agent.log, /opt/log/tiger/consul.server.access.log
Agent logs: /opt/log/tiger/consul_agent.log, /opt/log/tiger/consul.access.log
Start service: (run as root)
systemctl start consul-server # Start consul server
systemctl start consul # Start consul agent
Stop service: (run as root)
systemctl stop consul-server # Stop consul server
systemctl stop consul # Stop consul agent
View registered services:
curl -i localhost:2280/v1/catalog/services?pretty
View the nodes and ports where the services are located:
sd lookup {service-name}
Register service:
sd -h {host} up {service-name} {port}
Deregister service:
sd -h {host} down {service-name} {port}

3.2 Redis #

Component purpose: cache
Deployment path:
Server deployment path: /opt/redis/
Proxy deployment path: /opt/redis_proxy/
Log path:
Primary log: /opt/redis/master/master_runtime.log
Secondary log: /opt/redis/slave/slave_runtime.log
Proxy log: /opt/redis_proxy/redisproxy_runtime.log
Start service: (run as root)
systemctl start redis-master
systemctl start redis-slave
systemctl start redis-proxy
Stop service: (run as root)
systemctl stop redis-master
systemctl stop redis-slave
systemctl stop redis-proxy

3.3 MySQL #

Component purpose: metadata storage / business data storage
Deployment path:
Server deployment path: /opt/tiger/app/percona-5.7.23
Proxy deployment path: /opt/tiger/app/dbatman/
Log path:
Server log: /data00/mysql/log3406/
Read proxy log: /opt/tiger/app/dbatman/dbatman_3307/log/3307dbatman.log
Write proxy log: /opt/tiger/app/dbatman/dbatman_3306/log/3306dbatman.log
Start service: (run as root)
systemctl start mysqld
systemctl start mysql_proxy_read
systemctl start mysql_proxy_write
Stop service: (run as root)
systemctl stop mysqld
systemctl stop mysql_proxy_read
systemctl stop mysql_proxy_write

3.4 ZooKeeper #

Component purpose: distributed coordination, distributed lock, metadata storage
Deployment path: /home/zookeeper/software/zookeeper
Log path: /home/zookeeper/logs/zookeeper.log
Start service: (run as root)
systemctl start zookeeper
Stop service: (run as root)
systemctl stop zookeeper

3.5 Hadoop #

Component purpose: offline data storage, resource management
Deployment path:
Bookkeeper deployment path: /opt/tiger/bk_deploy/
Hadoop deployment path: /opt/tiger/yarn_deploy/hadoop
Log path: /data00/yarn/logs
Start service: (run as tiger)
systemctl --user start zkfc
systemctl --user start bookie
systemctl --user start namenode
systemctl --user start datanode
systemctl --user start resourcemanager
systemctl --user start nodemanager
Stop service: (run as tiger)
systemctl --user stop zkfc
systemctl --user stop bookie
systemctl --user stop namenode
systemctl --user stop datanode
systemctl --user stop resourcemanager
systemctl --user stop nodemanager
There are many Hadoop command-line operations. For details, refer to the official Apache Hadoop website. They are not described here.

3.6 Hive #

Component purpose: data warehouse
Deployment path: /opt/tiger/hive_deploy/
Log path: /opt/log/tiger/hive_noauth.log
Start service: (run as tiger)
systemctl --user start tob_hive_metastore
Stop service: (run as tiger)
systemctl --user stop tob_hive_metastore

3.7 ClickHouse #

Component purpose: OLAP engine
Deployment path: /opt/tiger/ch_debian_bin/
Log path: /data00/clickhouse/logs/
Start service: (run as root)
systemctl start clickhouse-server
Stop service: (run as root)
systemctl stop clickhouse-server
Enter ClickHouse client:
/opt/tiger/ch_debian_bin/usr/bin/clickhouse client