Binary Sludge
The attached storage used by Amazon’s managed Relational Database Service has a known issue where the bytes per inode ratio is very high (default on RHEL5 systems is 4096, to be found in /etc/mke2fs.conf). Amazon does not allow any administrative access to these instances so there is no way to reformat the filesystem to allocate more inodes, or attach storage the user can format with a different ratio.
This becomes problematic for databases that have many small tables (generally MyISAM tables, or InnoDB with the innodb_file_per_table setting) which quickly consume the available inodes. When the inode allocation is exhausted MySQL responds with
"ERROR 1030 (HY000): Got error 28 from storage engine"
The only current solution is to increase the size of attached storage, which increases the number of inodes (at the same fixed ratio). This is an expensive and inelegant fix; the RDS team are aware of the problem and working on a solution.
The difficulty of a solution in this context is RDS’s one-size-fits-all paradigm: many small tables are considered fringe usage and not accounted for, and allowing customers to set their own bytes to inode ratio will lead to wastefulness as admins over-estimate their total allocation.
My preferred solution would be to attach this metadata to an RDS instance (so it persists through disk resizing operations) but abstract it from the customer or only allow support to modify its value.
If you enjoyed this article, please consider sharing it!
