两个月前客户的MySQL服务器(CentOS 7系统 + MySQL 社区版 5.6)启不动了,还好有数据备份
后在同一台机器上重新安装 CentOS7 和 MySQL 社区版 5.6,并导入数据
两个月不到,客户反映又启不动了,最后的日志文件内容如下:
后在同一台机器上重新安装 CentOS7 和 MySQL 社区版 5.6,并导入数据
两个月不到,客户反映又启不动了,最后的日志文件内容如下:
160420 16:09:38 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql 2016-04-20 16:09:38 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 2016-04-20 16:09:38 0 [Note] /usr/sbin/mysqld (mysqld 5.6.25-log) starting as process 15643 ... 2016-04-20 16:09:38 15643 [Warning] Buffered warning: Changed limits: max_open_files: 1024 (requested 5000) 2016-04-20 16:09:38 15643 [Warning] Buffered warning: Changed limits: table_open_cache: 431 (requested 2000) 2016-04-20 16:09:38 15643 [Note] Plugin "FEDERATED" is disabled. 2016-04-20 16:09:38 15643 [Note] InnoDB: Using atomics to ref count buffer pool pages 2016-04-20 16:09:38 15643 [Note] InnoDB: The InnoDB memory heap is disabled 2016-04-20 16:09:38 15643 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-04-20 16:09:38 15643 [Note] InnoDB: Memory barrier is not used 2016-04-20 16:09:38 15643 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-04-20 16:09:38 15643 [Note] InnoDB: Using Linux native AIO 2016-04-20 16:09:38 15643 [Note] InnoDB: Using CPU crc32 instructions 2016-04-20 16:09:38 15643 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2016-04-20 16:09:38 15643 [Note] InnoDB: Completed initialization of buffer pool 2016-04-20 16:09:38 15643 [Note] InnoDB: Highest supported file format is Barracuda. 2016-04-20 16:09:38 15643 [Note] InnoDB: The log sequence numbers 152286831 and 152286831 in ibdata files do not match the log sequence number 183310887 in the ib_logfiles! 2016-04-20 16:09:38 15643 [Note] InnoDB: Database was not shutdown normally! 2016-04-20 16:09:38 15643 [Note] InnoDB: Starting crash recovery. 2016-04-20 16:09:38 15643 [Note] InnoDB: Reading tablespace information from the .ibd files... 2016-04-20 16:09:38 15643 [Note] InnoDB: Restoring possible half-written data pages 2016-04-20 16:09:38 15643 [Note] InnoDB: from the doublewrite buffer... InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 4. InnoDB: You may have to recover from a backup. 2016-04-20 16:09:38 7f76981b4740 InnoDB: Page dump in ascii and hex (16384 bytes): InnoDB: End of page dump 2016-04-20 16:09:38 7f76981b4740 InnoDB: uncompressed page, stored checksum in field1 110335650, calculated checksums for field1: crc32 1390315341, innodb 1967879858, none 3735928559, stored checksum in field2 809028917, calculated checksums for field2: crc32 1390315341, innodb 809028917, none 3735928559, page LSN 0 178436591, low 4 bytes of LSN at page end 178436591, page number (if stored to page already) 4, space id (if created with >= MySQL-4.1.1 and stored already) 0 InnoDB: Page may be an update undo log page InnoDB: Page may be an index page where index id is 18446744069414584320 InnoDB: Database page corruption on disk or a failed InnoDB: file read of page 4. InnoDB: You may have to recover from a backup. InnoDB: It is also possible that your operating InnoDB: system has corrupted its own file cache InnoDB: and rebooting your computer removes the InnoDB: error. InnoDB: If the corrupt page is an index page InnoDB: you can also try to fix the corruption InnoDB: by dumping, dropping, and reimporting InnoDB: the corrupt table. You can use CHECK InnoDB: TABLE to scan your table for corruption. InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. InnoDB: Ending processing because of a corrupt database page. 2016-04-20 16:09:38 7f76981b4740 InnoDB: Assertion failure in thread 140147334793024 in file buf0buf.cc line 4248 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 08:09:38 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. key_buffer_size=8388608 read_buffer_size=131072 max_used_connections=0 max_threads=151 thread_count=0 connection_count=0 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68105 K bytes of memory Hope that"s ok; if not, decrease some variables in the equation. Thread pointer: 0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x3b)[0x8e3a6b] /usr/sbin/mysqld(handle_fatal_signal+0x491)[0x673f91] /lib64/libpthread.so.0(+0xf130)[0x7f7697d9f130] /lib64/libc.so.6(gsignal+0x39)[0x7f7696ba8989] /lib64/libc.so.6(abort+0x148)[0x7f7696baa098] /usr/sbin/mysqld[0xab4dc8] /usr/sbin/mysqld[0xac9038] /usr/sbin/mysqld[0xaafb50] /usr/sbin/mysqld[0x9bbafd] /usr/sbin/mysqld[0xaccc77] /usr/sbin/mysqld[0xa54b30] /usr/sbin/mysqld[0x9938c8] /usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x5b5c18] /usr/sbin/mysqld[0x6fb7e0] /usr/sbin/mysqld(_Z11plugin_initPiPPci+0x940)[0x701f20] /usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x8a3)[0x5af663] /lib64/libc.so.6(__libc_start_main+0xf5)[0x7f7696b94af5] /usr/sbin/mysqld[0x5a2d9d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 160420 16:09:38 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
解决方案
20
mysql 是异常终止的吗?
20
设置innodb_force_recovery=1后重启
对全部表进行check table xx 和 repair table xx 操作
对全部表进行check table xx 和 repair table xx 操作
20
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
这个是说你遇到了一个bug,可能是由于二进制或库,链接冲突,损坏了
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
这个是说你遇到了一个bug,可能是由于二进制或库,链接冲突,损坏了
20
看起来是客户的DB,停这么多天没问题吗?
是不是应该考虑做repair之前先做一个冷备保存现场;冻结已有备份不要定期删除;再看binlog和备份是不是足够恢复客户数据?
是不是应该考虑做repair之前先做一个冷备保存现场;冻结已有备份不要定期删除;再看binlog和备份是不是足够恢复客户数据?
20
假如没有合适的备份,那么在另一台机器上重新安装同版本MySQL,恢复库的冷备,尝试启动,至少可以排除OS、库等环境的损坏。