[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-mysql-metadata-lock-blocks-every-select":3},{"content":4,"lastModified":5},"\u003Cp>On a Wednesday afternoon I ran this on a table with about 40 million rows:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">ALTER TABLE orders\n  ADD COLUMN fulfillment_ref VARCHAR(64) NULL,\n  ALGORITHM=INSTANT;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Column appended at the end, no index, no type change. The manual calls it instant, and on every other table it had finished before the tab stopped spinning. This one sat there. Two minutes in my phone started buzzing: checkout was timing out.\u003C\u002Fp>\n\n\u003Cp>Not because the ALTER was slow. Because of what it was waiting for, and what everything behind it was waiting for.\u003C\u002Fp>\n\n\u003Cimg src=\"https:\u002F\u002Fimages.pexels.com\u002Fphotos\u002F18530593\u002Fpexels-photo-18530593.jpeg?auto=compress&cs=tinysrgb&w=1600\" alt=\"A line of people queueing at terminal counters, everyone waiting on one point to move\" loading=\"lazy\" \u002F>\n\n\u003Ch2>The lock nobody thinks about\u003C\u002Fh2>\n\n\u003Cp>Every statement touching a table takes a metadata lock. Not an InnoDB row lock, the layer above: a lock on the table definition, so nobody can drop a column out from under a running query. A plain \u003Ccode>SELECT\u003C\u002Fcode> takes it shared. \u003Ccode>ALTER TABLE\u003C\u002Fcode> needs it exclusive.\u003C\u002Fp>\n\n\u003Cp>In autocommit mode that shared lock dies with the statement, which is why you can push thousands of reads a second for years and never learn this thing exists. Open a transaction and the lock lives until the transaction ends. Not until your \u003Ccode>SELECT\u003C\u002Fcode> ends. Until \u003Ccode>COMMIT\u003C\u002Fcode> or \u003Ccode>ROLLBACK\u003C\u002Fcode>, whichever comes later. Sometimes never.\u003C\u002Fp>\n\n\u003Cp>Something in our app held a shared MDL on \u003Ccode>orders\u003C\u002Fcode>. The ALTER asked for exclusive, got refused, and went to sleep. That part is recoverable. What came next is not.\u003C\u002Fp>\n\n\u003Ch2>Why one blocked ALTER stops every reader\u003C\u002Fh2>\n\n\u003Cp>Once that exclusive request sits in the MDL queue, MySQL will not let new readers jump ahead of it, or a busy table would starve the DDL forever. So every new \u003Ccode>SELECT\u003C\u002Fcode> on \u003Ccode>orders\u003C\u002Fcode> asks for its shared lock, sees an exclusive request pending, and waits. Then the next one waits behind that one.\u003C\u002Fp>\n\n\u003Cp>That is how one stuck session becomes an outage in seconds. In \u003Ccode>SHOW PROCESSLIST\u003C\u002Fcode> it looks like a hundred unrelated problems, because every connection is parked in the same state: \u003Ccode>Waiting for table metadata lock\u003C\u002Fcode>. Reads are not slow, they are not running at all: that is why the CPU graph sits flat while the site is down.\u003C\u002Fp>\n\n\u003Ch2>Find the session that holds it\u003C\u002Fh2>\n\n\u003Cp>Do not kill connections by age. Ask who owns the lock. \u003Ccode>sys\u003C\u002Fcode> has a view that hands you the kill statement as a column, so you are not writing one by hand at 3 a.m.:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT waiting_pid,\n       waiting_query_secs,\n       blocking_pid,\n       blocking_account,\n       blocking_lock_type,\n       sql_kill_blocking_query\nFROM sys.schema_table_lock_waits;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>One row came back. \u003Ccode>blocking_lock_type\u003C\u002Fcode> was \u003Ccode>SHARED_READ\u003C\u002Fcode>, held for 41 minutes, and the query text was a \u003Ccode>SELECT\u003C\u002Fcode> that had finished long before I opened my laptop. The connection was idle. The transaction was not.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT trx_id, trx_state, trx_started, trx_query\nFROM information_schema.innodb_trx\nWHERE trx_mysql_thread_id = 1187;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>\u003Ccode>trx_state\u003C\u002Fcode> said \u003Ccode>RUNNING\u003C\u002Fcode> while the connection's Command said \u003Ccode>Sleep\u003C\u002Fcode>. That is the signature of the same bug in every codebase where I have found it: something opened a transaction, read a row, then went off to do other work without committing. A report builder, here. I have also seen a worker throw between \u003Ccode>BEGIN\u003C\u002Fcode> and \u003Ccode>COMMIT\u003C\u002Fcode> and leave the transaction open.\u003C\u002Fp>\n\n\u003Cp>If \u003Ccode>sys\u003C\u002Fcode> comes back empty, \u003Ccode>performance_schema.metadata_locks\u003C\u002Fcode> shows the same dependency per object: \u003Ccode>PENDING\u003C\u002Fcode> rows are waiters, \u003Ccode>GRANTED\u003C\u002Fcode> rows are holders. The instrument is on by default in 8.x, so no restart needed:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT OBJECT_NAME, LOCK_TYPE, LOCK_STATUS, OWNER_THREAD_ID\nFROM performance_schema.metadata_locks\nWHERE OBJECT_TYPE = 'TABLE' AND OBJECT_NAME = 'orders';\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Ch2>The timeout is one year\u003C\u002Fh2>\n\n\u003Cp>\u003Ccode>innodb_lock_wait_timeout\u003C\u002Fcode> defaults to 50 seconds and everybody knows it. \u003Ccode>lock_wait_timeout\u003C\u002Fcode>, which governs metadata locks, defaults to 31536000 seconds. A year. Your migration does not fail and does not alert. It waits, and every reader waits with it.\u003C\u002Fp>\n\n\u003Cp>So DDL now goes out in a session that set its own budget first:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SET SESSION lock_wait_timeout = 5;\nALTER TABLE orders\n  ADD COLUMN fulfillment_ref VARCHAR(64) NULL,\n  ALGORITHM=INSTANT;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Five seconds, then \u003Ccode>ERROR 1205\u003C\u002Fcode>, and nothing else is affected. A migration you retry after lunch beats a successful one that took checkout down for six minutes. Pin the algorithm too. Without it, MySQL picks what it thinks is cheapest, and that belief was formed before your data volume had any say.\u003C\u002Fp>\n\n\u003Ch2>Instant does not mean the lock is cheap\u003C\u002Fh2>\n\n\u003Cp>This is what I had wrong, and why the incident happened on a column add rather than a scary type change. \u003Ccode>ALGORITHM=INSTANT\u003C\u002Fcode> makes the rebuild cheap, not the lock: an instant change still needs an exclusive MDL to flip the data dictionary entry. If a long transaction holds a shared lock, your instant ALTER waits and readers pile up behind it.\u003C\u002Fp>\n\n\u003Cp>Same for the online schema change tools. \u003Ccode>pt-online-schema-change\u003C\u002Fcode> and \u003Ccode>gh-ost\u003C\u002Fcode> copy rows into a shadow table without blocking anyone for most of their run, then need a short exclusive lock at cutover to swap names. Short means as soon as open transactions drain. If one never drains, the identical queue forms at the end of a process you were watching for twenty minutes.\u003C\u002Fp>\n\n\u003Ch2>What I changed\u003C\u002Fh2>\n\n\u003Cp>Two things in the app. The report builder runs with autocommit and a cap on transaction length, because a report holding a read lock for 41 minutes should have been a different query. The pool's connection init now runs \u003Ccode>SET SESSION lock_wait_timeout = 5\u003C\u002Fcode>, and an idle-in-transaction timeout keeps a mid-transaction connection out of the pool entirely. That one also guards against pool exhaustion, as \u003Ca href=\"\u002Fposts\u002Fmysql-too-many-connections-pool-math\u002F\">the connection math behind a full pool\u003C\u002Fa> explains.\u003C\u002Fp>\n\n\u003Cp>One habit: instant is no longer permission to run DDL mid-afternoon without checking what holds locks on that table. Six queries before, none after. Same lesson as \u003Ca href=\"\u002Fposts\u002Fmysql-slow-query-debugging-workflow\u002F\">the slow query workflow\u003C\u002Fa>, pointed at the lock queue instead of the planner.\u003C\u002Fp>\n\n\u003Cp>The pre-flight query and the kill statement live in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> now. If you run migrations against a live MySQL database, put them somewhere you will find them before the next deploy.\u003C\u002Fp>\n","2026-11-10",1794383305985]