Docs

Torch Log Service

Query and analysis

Query and analyze Nginx access logs

You can use Nginx access logs to monitor website performance and quickly identify exception issues. This topic describes common scenarios and examples of analyzing Nginx access logs.

Background information

Nginx is a widely used web server, and its logs are an essential source of information for operations and maintenance. You can upload Nginx access logs to TLS and use real-time query and analysis features to gain deeper insights into website performance, quickly detect and resolve issues, and improve user experience and service quality. For example:

By analyzing client IP addresses, you can understand user distribution, identify where users are from, and optimize your site’s regional services.

Analyzing request status codes helps you understand the proportion of each code, know website health, and detect and resolve potential issues promptly.

By tracking page views (PV) and unique visitors (UV), you can assess website traffic and user activity, providing data to support marketing and operational decisions.

Analyze access sources to identify the channel with the highest traffic, enabling you to optimize your marketing strategy.

Analyze the response time of requests to quickly identify performance bottlenecks and optimize server performance.

Calculate the number of queries the system processes per second (QPS) to assess server load and adjust resources as needed.

Prerequisites

The log project and log topic are created. For details, see Create resources.

An index has been created based on the log fields. For details, see Configure indexes.

Log example

The table below lists key fields in Nginx access log examples.

Field	Description
__time__	Timestamp for log entries.
app_id	Unique identifier for the application.
host	Target host for the request.
hostname	Hostname of the server.
referer	The source page address of the request (HTTP Referer header).
remote_addr	Client IP address.
request_method	The HTTP request method (such as GET, POST, etc.).
request_time	Request processing time.
request_uri	The requested URI (Uniform Resource Identifier).
server_protocol	The protocol used by the server, such as HTTP/1.1 or HTTP/2.
status	HTTP response status code.
time_local	Local timestamp.
upstream	Information about the upstream server. It is typically used in reverse proxy scenarios.
upstream_response_time	The time taken for the upstream server to respond.
upstream_status	Upstream server response status code.
user_agent	The user agent string of the client. It is used to identify browser and operating system information.
x_forwarded_for	HTTP header. It is used to identify the client's IP address after passing through a proxy server.
x_real_ip	The client's real IP address. It is commonly used when passing through a proxy.

Index configuration example

This example creates the following index configuration based on the log example above. For specific operations and precautions related to index configuration, seeConfigure indexes.

Typical query and analysis scenarios

Typical analysis scenarios for Nginx access logs mainly cover business monitoring, performance tuning, monitoring, and alarm.

Business monitoring

Scenario 1 Count daily PV and UV

Scenario: Count website PV, UV, and trend changes every 5 minutes within the past day.

Query and analysis statement:

*| SELECT DATE_FORMAT(__time__ - __time__ % 300000, 'Y-M-d HH:mm') AS time, APPROX_DISTINCT(RemoteAddr) AS UV ,count(1) AS PV GROUP BY time ORDER BY time LIMIT 1000

Query and analysis results

Use a line chart to show the website's PV, UV, and trends every 5 minutes during the past day. For more information, see Line chart.

Scenario 2 Count QPS

Scenario: Track the changes in QPS over the past day

Query and analysis statement:

* | SELECT __time__-__time__%60000 AS time, COUNT(*)/60 AS qps GROUP BY time ORDER BY time LIMIT 10000

Query and analysis results

Display QPS trends over the past day using a line chart (enable time series mode) For more information, see Line chart.

Scenario 3 Analyze request method distribution

Scenario: Calculate the distribution of request methods

Query and analysis statement:

* | SELECT COUNT(*) AS count, "request_method" GROUP BY "request_method"

Query and analysis results

Display the distribution of request methods using a pie chart. For more information, see Pie chart.

Scenario 4 Calculate request failure rate

Scenario: Calculate the overall request failure rate per minute, 404 error rate, 4xx error rate, and 5xx error rate.

Query and analysis statement:

* | SELECT DATE_TRUNC('minute', __time__) AS time, ROUND(SUM(CASE WHEN status = 404 THEN 1.00 ELSE 0.00 END)/ CAST(COUNT(*) AS DOUBLE)*100,3) AS "404percentage（%）", ROUND(SUM(CASE WHEN status >= 500 THEN 1.00 ELSE 0.00 END)/cast(count(*) AS DOUBLE)*100,3) AS "5XXpercentage（%）", ROUND(SUM(CASE WHEN status >= 400 AND status < 500 THEN 1.00 ELSE 0.00 END)/CAST(COUNT(*) AS DOUBLE)*100,3) AS "4XXpercentage（%）", round(sum(CASE WHEN status >= 400 THEN 1.00 ELSE 0.00 END)/CAST(COUNT(*) AS DOUBLE)*100,3) AS "Total fail rate（%）" GROUP BY time ORDER BY time LIMIT 10000

Query and analysis results

Display the overall failure rate, the failure rates of various errors, and their trends using a line chart. For more information, see Line chart.

Scenario 5 Count the Top 100 requested applications

Scenario: Count the applications with the Top 100 request volumes and their corresponding request volumes, success rates, and QPS.

Query and analysis statement:

* | SELECT app_id, COUNT(*) AS PV, COUNT(DISTINCT(remote_addr)) AS UV, ROUND(SUM(CASE WHEN status < 400 THEN 1.00 ELSE 0.00 END)/CAST(COUNT(*) AS double)*100,3) AS "Req Success Rate", ROUND(COUNT(*)/60 , 2) AS QPS GROUP BY app_id ORDER BY PV DESC LIMIT 100

Query and analysis results

Scenario 6 Count the Top 10 request volumes

Scenario: Count the Top 10 request volumes and their corresponding request pages.

Query and analysis statement:

* | SELECT SPLIT_PART(request_uri,'?',1) AS path, COUNT(1) AS pv GROUP BY path ORDER BY pv DESC LIMIT 10

Query and analysis results

Performance optimization

Analyze Nginx access logs to know request latency and identify website bottlenecks, helping optimize website performance.

Scenario 1: Calculate average and maximum latency.

Scenario: Calculate the average and maximum request latency per hour over the past day.

Query and analysis statement:

* | SELECT DATE_TRUNC('hour', __time__) AS time, AvG(request_time) as avg_latency, MAX(request_time) AS max_latency GROUP BY time ORDER BY time

Query and analysis results

Use a line chart to show the hourly average and maximum request latency and trends for the past day. For more information, see Line chart.

Scenario 2: Calculate the maximum latency and its corresponding request page.

Scenario: Calculate the maximum request latency per hour over the past day and identify the corresponding request page.

Query and analysis statement:

* | SELECT DATE_TRUNC('hour', __time__) AS time, MAX(request_time) AS max_latency, MAX_BY(request_uri,request_time) AS uri GROUP BY time ORDER BY time

Query and analysis results

Scenario 3: Track the top 3 request times

Scenario: Count the three longest request times from the past day.

Query and analysis statement:

* | SELECT MAx(request_time,3) AS "TOP 3"

Query and analysis results

Scenario 4: Query average request latency, P50 latency, and P99 latency

Scenario: Query average request latency, P50 latency, and P99 latency.

Query and analysis statement:

* | SELECT DATE_TRUNC('minute', __time__) AS time, AVG(request_time) AS latency, APPROX_PERCENTILE(request_time, 0.5) as p50, APPROX_PERCENTILE(request_time, 0.99) as p99 GROUP BY time ORDER BY time

Query and analysis results

Display average latency, P50 latency, and P99 latency trends in a line chart. For more information, see Line chart.

Scenario 5: Histogram of request duration

Scenario: Generate a histogram of request duration statistics.

Query and analysis statement:

* |SELECT NUMERIC_HISTOGRAM(10,request_time)

Query and analysis results

Scenario 6: Page optimization

Scenario: Optimize page performance by calculating metrics such as page visits (healthz page) PV and UV, the number of requests by method, the number of requests by status, the number of requests by browser, average latency, and maximum latency.

Query and analysis statement:

request_uri:"healthz" | SELECT COUNT(1) AS PV,
APPROX_DISTINCT(remote_addr) AS UV,
HISTOGRAM(request_method) AS method_c,
HISTOGRAM(status) AS status_c,
HISTOGRAM(user_agent) AS user_agent_c,
AVG(request_time) AS avg_l,
MAX(request_time) AS max_l

Query and analysis results

Monitoring alarm

For request errors and other issues, you can monitor your website in real time through the alarm feature to quickly detect and handle exceptions.

Scenario 1: Request error alarm

Scenario: Track hourly error counts and trigger alerts when errors occur.

Query and analysis statement:

status >= 400 | SELECT COUNT(*) as count

Query and analysis results

Alarm policy configuration:

For specific steps to create an alarm policy, see Quickly configure log alarm.

Scenario 2: Request average latency alarm

Scenario: Create an alarm based on the average request latency of the healthz page and monitor the basic latency of healthz page requests.

Query and analysis statement:

request_uri:"healthz" | SELECT AVG(request_time) AS avg

Query and analysis results

Alarm policy configuration:

For instructions on creating an alarm policy, see Quickly configure log alarm.

Scenario 3: Request latency P99 alarm

Scenario: Using average latency does not effectively reflect requests with higher delays and can not show the website's true operating conditions. By creating alarms based on latency percentiles, such as P99 latency, you can quickly detect requests with relatively high delays.

Query and analysis statement:

request_uri:"healthz" | SELECT APPROX_PERCENTILE(request_time, 0.99) AS p99

Query and analysis results

Alarm policy configuration:

For instructions on creating an alarm policy, see Quickly configure log alarm.

Last updated: 2026.09.07 16:47:15