Binary Sludge » MySQL

InnoDB disabled if ib_logfile files corrupted

Andy — Thu, 09 Feb 2012 19:07:20 +0000

I recently came across a dev VM running MySQL 5.0.77 (an old release, 28 January 2009) that didn’t have InnoDB available. skip-innodb wasn’t set, SHOW VARIABLES LIKE '%innodb%' looked as expected, but with one exception: the value of have-innodb was DISABLED.

I confirmed this with SHOW ENGINES:

(root@localhost) [(none)]> show engines;
+------------+----------+----------------------------------------------------------------+
| Engine     | Support  | Comment                                                        |
+------------+----------+----------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance         |
| MEMORY     | YES      | Hash based, stored in memory, useful for temporary tables      |
| InnoDB     | DISABLED | Supports transactions, row-level locking, and foreign keys     |
...

(and chuckled to myself over the comment about MyISAM’s performance)

/var/log/mysqld.log yielded the answer:

120127 17:17:51  mysqld started
120127 17:17:51 [Warning] /usr/libexec/mysqld: ignoring option '--engine-condition-pushdown' due to invalid value 'InnoDB'
InnoDB: Error: log file ./ib_logfile0 is of different size 0 20971520 bytes
InnoDB: than specified in the .cnf file 0 104857600 bytes!
120127 17:17:51 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.0.77-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3306  Source distribution

innodb_file_per_table was not set on this VM, and when the InnoDB log files had become corrupted and were not recognised, the engine couldn’t start. These ib_logfile files are the crash recovery logs:

The unsung heroes of InnoDB are the logfiles. They are what makes InnoDB automatic crash recovery possible.

Database administrators of other DBMS may be familiar with the concept of a “redo” log. When data is changed, affected data pages are changed in the innodb_buffer_pool. Then, the change is written to the redo log, which in MySQL is the InnoDB logfile (ib_logfile0 and ib_logfile1). The pages are marked as “dirty”, and eventually get flushed and written to disk.

If MySQL crashes, there may be data that is changed that has not been written to disk. Those data pages were marked as “dirty” in the innodb_buffer_pool, but after a MySQL crash the innodb_buffer_pool no longer exists. However, they were written to the redo log. On crash recovery, MySQL can read the redo log (InnoDB log files) and apply any changes that were not written to disk.

edit: As pointed out by James Day in the comments, the original solution listed could cause further problems and so is only appropriate to a development/staging environment. When using production data he has an alternative suggestion:

The correct solution to that error is to edit my.cnf/my.ini and set the size specified there to the size and number of log files on disk. InnoDB will then start normally.

Thanks James!

Of note: when this problem occurs, MySQL 5.0.77 erroneously returns this:

(root@some_host) [some_db]> SHOW ENGINE INNODB STATUS;
ERROR 1235 (42000): Cannot call SHOW INNODB STATUS because skip-innodb is defined

old solution for posterity:

The solution is to move the logs and allow InnoDB to recreate them. Don’t delete them – you may need them if your server has crashed or in case of data loss.

/etc/init.d/mysql stop
mv /var/lib/mysql/ib_logfile0 /var/lib/mysql/ib_logfile0.bak
mv /var/lib/mysql/ib_logfile1 /var/lib/mysql/ib_logfile1.bak
/etc/init.d/mysql start

With the files absent, InnoDB recreates them and the engine is loaded successfully.

Inode allocation on Amazon AWS RDS (Relation Database Service)

Andy — Mon, 28 Feb 2011 19:00:21 +0000

The attached storage used by Amazon’s managed Relational Database Service has a known issue where the bytes per inode ratio is very high (default on RHEL5 systems is 4096, to be found in /etc/mke2fs.conf). Amazon does not allow any administrative access to these instances so there is no way to reformat the filesystem to allocate more inodes, or attach storage the user can format with a different ratio.

This becomes problematic for databases that have many small tables (generally MyISAM tables, or InnoDB with the innodb_file_per_table setting) which quickly consume the available inodes. When the inode allocation is exhausted MySQL responds with

"ERROR 1030 (HY000): Got error 28 from storage engine"

The only current solution is to increase the size of attached storage, which increases the number of inodes (at the same fixed ratio). This is an expensive and inelegant fix; the RDS team are aware of the problem and working on a solution.

The difficulty of a solution in this context is RDS’s one-size-fits-all paradigm: many small tables are considered fringe usage and not accounted for, and allowing customers to set their own bytes to inode ratio will lead to wastefulness as admins over-estimate their total allocation.

My preferred solution would be to attach this metadata to an RDS instance (so it persists through disk resizing operations) but abstract it from the customer or only allow support to modify its value.

MySQL performance flow chart

Andy — Mon, 03 Jan 2011 16:00:09 +0000

Here’s an old-but-still-relevant (re)post from Major Hayen on improving MySQL performance. It’s a neat, concise reference guide for MySQL emergencies!

Original post

MySQL ERROR 1137 (HY000): Can’t reopen table: ‘tmp_journals’

Andy — Thu, 18 Mar 2010 15:00:49 +0000

When setting up a query using a temporary lookup table, I got this error:

ERROR 1137 (HY000): Can't reopen table: 'tmp_journals'

It transpires that since 4.1 the way MySQL handles temporary tables has changed. This affects joins, unions and subqueries. There is an obvious fix:

mysql> CREATE TEMPORARY TABLE tmp_journals_2 LIKE tmp_journals;
Query OK, 0 rows affected (0.00 sec)

mysql> INSERT INTO tmp_journals_2 SELECT * FROM tmp_journals;
Query OK, 3228659 rows affected (2.01 sec)
Records: 3228659  Duplicates: 0  Warnings: 0

Then the query is easy:

SELECT COUNT(1) cnt, journal_invoice_ref
FROM tmp_journals
GROUP BY journal_date
HAVING cnt > 10000

UNION SELECT COUNT(1) cnt, journal_invoice_ref FROM tmp_journals_2 GROUP BY journal_invoice_ref HAVING cnt < 10