Fear and Loathing on the Learning Curve: Observations on Life, Tech and Web Design from a Slightly Misanthropic Mind

MySQL Troubleshooting Tip of the Day

One of our EC2–hos­ted data­base serv­ers had to be rebooted last night, and when it came back up some of our database-driven sites were work­ing, oth­ers weren’t. Upon fur­ther invest­ig­a­tion I found the fol­low­ing mes­sage crop­ping up in mongrel_rails and phpMy­Ad­min logs:

Error No. 1033 Incorrect inform­a­tion in file: ‘<filename>’

A peek at phpMyAdmin’s data­base list showed that in many of the data­bases, some tables showed up fine while oth­ers were marked as “in use”. Attempting to view those tables pro­duced the above message.

After a bit of pok­ing around I real­ised that the my.cnf con­fig­ur­a­tion file had been over­writ­ten by some star­tup script, and now con­tained the line skip-innodb. This meant that the server couldn’t load any tables that used the InnoDB stor­age engine. Removing this line from my.cnf and restart­ing the mysqld ser­vice fixed the problem.

Incidentally, make sure you’re edit­ing the right my.cnf file — there were a few on our sys­tem and it took a couple of tries to fig­ure out which one MySQL was actu­ally using.

Thought I’d post this in case any other EC2 users run into this issue. I think the star­tup script in ques­tion belonged to WHM/cPanel, so this might be of interest to users of those pack­ages too. Phew.