[{"data":1,"prerenderedAt":4},["ShallowReactive",2],{"post-content-mysql-slow-query-debugging-workflow":3},"\u003Cfigure>\n  \u003Cimg src=\"https:\u002F\u002Fimages.unsplash.com\u002Fphoto-1558494949-ef010cbdcc31?w=1200&q=80&auto=format&fit=crop\" alt=\"Rows of blinking servers in a data center rack, somewhere in there a query is scanning two million rows\" loading=\"lazy\" \u002F>\n\u003C\u002Ffigure>\n\n\u003Cp>Last month a support ticket landed on a Monday morning: the admin dashboard took eleven seconds to load. MySQL was sitting at 40% CPU, traffic looked normal, and nothing in the app logs pointed anywhere. A year ago I would have spent the afternoon guessing. Now I run a short workflow instead, and it found the culprit in about twenty minutes.\u003C\u002Fp>\n\n\u003Cp>The tools matter less than the sequence: capture the queries, rank them by how much total damage they do, and only then dig into what the database actually did with them. Here is the whole thing.\u003C\u002Fp>\n\n\u003Ch2>Start with the slow query log\u003C\u002Fh2>\n\n\u003Cp>MySQL's slow query log is off by default, and the default threshold is ten seconds. Ten seconds. A query taking nine and a half never gets logged, and your users have already left by then. So the first job is turning the threshold down to something a web app can live with:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SET GLOBAL slow_query_log = 'ON';\nSET GLOBAL long_query_time = 0.5;\nSET GLOBAL slow_query_log_file = '\u002Fvar\u002Flog\u002Fmysql\u002Fslow.log';\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>This takes effect immediately, no restart needed. One detail that bit me once: \u003Ccode>SET GLOBAL\u003C\u002Fcode> only applies to new connections. Existing pooled connections keep the old threshold until they reconnect, so if nothing shows up right away, that may be why.\u003C\u002Fp>\n\n\u003Cp>To survive a restart, put it in \u003Ccode>my.cnf\u003C\u002Fcode>:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-ini\">[mysqld]\nslow_query_log = 1\nlong_query_time = 0.5\nslow_query_log_file = \u002Fvar\u002Flog\u002Fmysql\u002Fslow.log\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>There is also \u003Ccode>log_queries_not_using_indexes\u003C\u002Fcode>. I leave it off on busy servers. It sounds useful, but on a database with lots of small lookup tables it will log half your traffic and fill the disk. If you do turn it on, pair it with \u003Ccode>min_examined_row_limit\u003C\u002Fcode> so tiny queries stay quiet.\u003C\u002Fp>\n\n\u003Ch2>Rank by total time, not by scariest single run\u003C\u002Fh2>\n\n\u003Cp>The raw log is unreadable, and reading it top to bottom is a trap. The slowest single query is rarely your biggest problem. A 300ms query that runs four hundred times an hour does far more damage than a 4-second report someone runs once a day. Sort by total time:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">mysqldumpslow -s t -t 10 \u002Fvar\u002Flog\u002Fmysql\u002Fslow.log\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>\u003Ccode>mysqldumpslow\u003C\u002Fcode> ships with MySQL and groups similar queries together. If you want richer output, \u003Ccode>pt-query-digest\u003C\u002Fcode> from Percona Toolkit is the upgrade. Honestly, either is fine. The point is the ranking, not the tool.\u003C\u002Fp>\n\n\u003Cp>Our Monday culprit had exactly this shape. Not the 4-second monster at the top of the file, but a 280ms query that appeared 1,900 times in one hour.\u003C\u002Fp>\n\n\u003Ch2>Then ask the database what it actually did\u003C\u002Fh2>\n\n\u003Cp>Plain \u003Ccode>EXPLAIN\u003C\u002Fcode> shows the plan the optimizer expects. Estimates. Useful, but I have been burned by estimates enough times that I don't trust them alone. Since MySQL 8.0.18 there is \u003Ccode>EXPLAIN ANALYZE\u003C\u002Fcode>, which runs the query and reports real timings and real row counts:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">EXPLAIN ANALYZE\nSELECT * FROM orders\nWHERE customer_id = 12345 AND status = 'pending'\nORDER BY created_at DESC LIMIT 20;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cpre>\u003Ccode class=\"language-text\">-> Limit: 20 row(s)  (actual time=1839.3..1839.4 rows=20 loops=1)\n    -> Sort: orders.created_at DESC, limit input to 20 row(s)  (actual time=1839.2..1839.3 rows=20 loops=1)\n        -> Filter: (orders.status = 'pending')  (actual time=0.11..1795.0 rows=64321 loops=1)\n            -> Index lookup on orders using idx_customer_id (customer_id=12345)  (actual time=0.08..1721.4 rows=198455 loops=1)\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Read the tree bottom up. The two things I check first: where actual time balloons, and where the actual row count is wildly different from the optimizer's estimate. In this output, the index lookup touches 198,000 rows to answer a question about one customer. That is the whole story of the query, right there.\u003C\u002Fp>\n\n\u003Cp>One warning worth shouting: \u003Ccode>EXPLAIN ANALYZE\u003C\u002Fcode> actually executes the statement. Fine for SELECTs. Do not point it at an UPDATE or DELETE in production, because your \"diagnostic\" will modify real data.\u003C\u002Fp>\n\n\u003Ch2>The fix is usually an index, then you prove it\u003C\u002Fh2>\n\n\u003Cp>For the query above, the existing index only covered \u003Ccode>customer_id\u003C\u002Fcode>, so MySQL fetched every order for that customer and then sorted and filtered in memory. A composite index that matches the WHERE clause and the ORDER BY fixed it:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">CREATE INDEX idx_customer_status_created\n  ON orders (customer_id, status, created_at);\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Same \u003Ccode>EXPLAIN ANALYZE\u003C\u002Fcode> afterward, and actual time dropped from 1.8 seconds to 0.4 milliseconds. The rule of thumb for column order: equality columns first, the sort column last. I wrote up the full indexing patterns in \u003Ca href=\"\u002Fposts\u002Fhow-i-cut-database-query-time-mysql-indexing\u002F\">my post on MySQL indexing\u003C\u002Fa>, so I won't repeat them here.\u003C\u002Fp>\n\n\u003Cp>Last thing. I keep the setup commands and the config flags I always forget in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa>, because at 9am on a bad Monday I don't want to be looking up syntax. The workflow itself is short enough to memorize: log, rank, analyze, fix, verify. It hasn't failed me yet.\u003C\u002Fp>\n",1789366676960]