2017-06-12 191 views
0

我有一個基於MySQL Community版本的數據庫出現嚴重問題,該數據庫在數據庫分區調整大小後崩潰並且不會啓動備份。 db位於RHEL7.2 Linux lvm分區上,由於存儲問題,該分區必須減少。MySQL 5.7 Community Edition磁盤分區調整大小後崩潰

在調整大小期間,有inode問題導致數據庫文件和文件夾被移到lost + found中,我們恢復了這些文件並將它們移到了正確的位置。 重新啓動mysql服務是導致這些錯誤:

2017-06-12T09:38:12.510405Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details). 
2017-06-12T09:38:12.510568Z 0 [Warning] Insecure configuration for --secure-file-priv: Current value does not restrict location of generated files. Consider setting it to a valid, non-empty path. 
2017-06-12T09:38:12.510628Z 0 [Note] /usr/sbin/mysqld (mysqld 5.7.13) starting as process 40601 ... 
2017-06-12T09:38:12.515354Z 0 [Note] InnoDB: PUNCH HOLE support available 
2017-06-12T09:38:12.515396Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 
2017-06-12T09:38:12.515404Z 0 [Note] InnoDB: Uses event mutexes 
2017-06-12T09:38:12.515413Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 
2017-06-12T09:38:12.515421Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3 
2017-06-12T09:38:12.515428Z 0 [Note] InnoDB: Using Linux native AIO 
2017-06-12T09:38:12.515779Z 0 [Note] InnoDB: Number of pools: 1 
2017-06-12T09:38:12.515943Z 0 [Note] InnoDB: Using CPU crc32 instructions 
2017-06-12T09:38:12.518004Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 
2017-06-12T09:38:12.527531Z 0 [Note] InnoDB: Completed initialization of buffer pool 
2017-06-12T09:38:12.529516Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority(). 
2017-06-12T09:38:12.541493Z 0 [Note] InnoDB: Highest supported file format is Barracuda. 
2017-06-12T09:38:12.542896Z 0 [Note] InnoDB: The log sequence number 8204 in the system tablespace does not match the log sequence number 435564720566 in the ib_logfiles! 
2017-06-12T09:38:12.542919Z 0 [Note] InnoDB: Database was not shutdown normally! 
2017-06-12T09:38:12.542926Z 0 [Note] InnoDB: Starting crash recovery. 
2017-06-12 10:38:12 0x7f4e11608740 InnoDB: Assertion failure in thread 139973275715392 in file fut0lst.ic line 85 
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA 
InnoDB: We intentionally generate a memory trap. 
InnoDB: Submit a detailed bug report to http://bugs.mysql.com. 
InnoDB: If you get repeated assertion failures or crashes, even 
InnoDB: immediately after the mysqld startup, there may be 
InnoDB: corruption in the InnoDB tablespace. Please refer to 
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html 
InnoDB: about forcing recovery. 
09:38:12 UTC - mysqld got signal 6 ; 
This could be because you hit a bug. It is also possible that this binary 
or one of the libraries it was linked against is corrupt, improperly built, 
or misconfigured. This error can also be caused by malfunctioning hardware. 
Attempting to collect some information that could help diagnose the problem. 
As this is a crash and something is definitely wrong, the information 
collection process might fail. 

key_buffer_size=8388608 
read_buffer_size=131072 
max_used_connections=0 
max_threads=151 
thread_count=0 
connection_count=0 
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 68189 K bytes of memory 
Hope that's ok; if not, decrease some variables in the equation. 

Thread pointer: 0x0 
Attempting backtrace. You can use the following information to find out 
where mysqld died. If you see no messages after this, something went 
terribly wrong... 
stack_bottom = 0 thread_stack 0x40000 
/usr/sbin/mysqld(my_print_stacktrace+0x3b)[0xef0cfb] 
/usr/sbin/mysqld(handle_fatal_signal+0x461)[0x7af361] 
/lib64/libpthread.so.0(+0xf100)[0x7f4e111ef100] 
/lib64/libc.so.6(gsignal+0x37)[0x7f4e0fbe25f7] 
/lib64/libc.so.6(abort+0x148)[0x7f4e0fbe3ce8] 
/usr/sbin/mysqld[0x77f71c] 
/usr/sbin/mysqld(_Z22trx_undo_free_preparedP5trx_t+0x0)[0x77f4d4] 
/usr/sbin/mysqld(_Z19trx_undo_lists_initP10trx_rseg_t+0xdcc)[0x10bcacc] 
/usr/sbin/mysqld[0x10a11ec] 
/usr/sbin/mysqld[0x10a39bc] 
/usr/sbin/mysqld(_Z24trx_sys_init_at_db_startv+0x1883)[0x10aaa43] 
/usr/sbin/mysqld(_Z34innobase_start_or_create_for_mysqlv+0x4830)[0x10721a0] 
/usr/sbin/mysqld[0xf39451] 
/usr/sbin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x51)[0x7fa2b1] 
/usr/sbin/mysqld[0xce3925] 
/usr/sbin/mysqld(_Z11plugin_initPiPPci+0x610)[0xcea830] 
/usr/sbin/mysqld[0x7a82bd] 
/usr/sbin/mysqld(_Z11mysqld_mainiPPc+0x842)[0x7a97e2] 
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f4e0fbceb15] 
/usr/sbin/mysqld[0x79f5e5] 
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains 
information that should help you find out what is causing the crash. 

和:

2017-06-12T09:38:12.542919Z 0 [Note] InnoDB: Database was not shutdown normally! 
2017-06-12T09:38:12.542926Z 0 [Note] InnoDB: Starting crash recovery. 
2017-06-12 10:38:12 0x7f4e11608740 InnoDB: Assertion failure in thread 139973275715392 in file fut0lst.ic line 85 
InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset >= FIL_PAGE_DATA 

我試過多種innodb_force_recovery模式和的mysqld沒有啓動,直到我用innodb_force_recovery = 6。當啓動mysqld我不能備份數據庫,因爲這樣的多個錯誤:

2017-06-12T11:10:52.217568Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`plugin` in the cache. Attempting to load the tablespace with space id 2 
2017-06-12T11:10:52.277914Z 0 [ERROR] Function 'validate_password' already exists 
2017-06-12T11:10:52.277954Z 0 [Warning] Couldn't load plugin named 'validate_password' with soname 'validate_password.so'. 
2017-06-12T11:10:52.284445Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`gtid_executed` in the cache. Attempting to load the tablespace with space id 18 
2017-06-12T11:10:52.293339Z 0 [Note] Found ca.pem, server-cert.pem and server-key.pem in data directory. Trying to enable SSL support using them. 
2017-06-12T11:10:52.310829Z 0 [Warning] CA certificate ca.pem is self signed. 
2017-06-12T11:10:52.313911Z 0 [Note] Server hostname (bind-address): '*'; port: 3306 
2017-06-12T11:10:52.314120Z 0 [Note] IPv6 is available. 
2017-06-12T11:10:52.314139Z 0 [Note] - '::' resolves to '::'; 
2017-06-12T11:10:52.314179Z 0 [Note] Server socket created on IP: '::'. 
2017-06-12T11:10:52.433825Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`server_cost` in the cache. Attempting to load the tablespace with space id 19 
2017-06-12T11:10:52.449122Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`engine_cost` in the cache. Attempting to load the tablespace with space id 20 
2017-06-12T11:10:52.476635Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_leap_second` in the cache. Attempting to load the tablespace with space id 12 
2017-06-12T11:10:52.477431Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_name` in the cache. Attempting to load the tablespace with space id 8 
2017-06-12T11:10:52.490830Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone` in the cache. Attempting to load the tablespace with space id 9 
2017-06-12T11:10:52.507804Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition_type` in the cache. Attempting to load the tablespace with space id 11 
2017-06-12T11:10:52.508725Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`time_zone_transition` in the cache. Attempting to load the tablespace with space id 10 
2017-06-12T11:10:52.513876Z 0 [ERROR] InnoDB: Failed to find tablespace for table `mysql`.`servers` in the cache. Attempting to load the tablespace with space id 3 
2017-06-12T11:10:52.563565Z 0 [Note] Event Scheduler: Loaded 0 events 
2017-06-12T11:10:52.563876Z 0 [Note] /usr/sbin/mysqld: ready for connections. 
Version: '5.7.13' socket: '/dataspace/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) 

而當我試圖轉儲數據庫文件我看到這個:

mysqldump -uroot -pXXXXX --skip-lock-tables --skip-extended-insert --hex-blob office > office_dump.sql 
mysqldump: [Warning] Using a password on the command line interface can be insecure. 
mysqldump: Error 2013: Lost connection to `MySQL` server during query when dumping table `audit_log` at row: 334202 

請問我能做些什麼來恢復數據並使數據庫恢復在線狀態?

任何援助將認真讚賞。

回答

0

我做了什麼讓MySQL再次運行。

  1. 備份borked $ MYSQL_HOME目錄。
  2. 添加skip-grant-tables允許我以root身份登錄到mysql,而不需要密碼。
  3. 在/etc/my.cnf嘗試了不同的innodb_force_recovery模式,直到允許mysqld啓動。 (在我的情況下,它是innodb_force_recovery = 5)。
  4. 嘗試和失敗,轉儲所有數據庫:

    mysqldump -uroot -pXXXXX --skip-lock-tables --skip-extended-insert --hex-blob databasename > databasename_dump.sql mysqldump: [Warning] Using a password on the command line interface can be insecure. mysqldump: Error 2013: Lost connection to MySQL server during query when dumping table AUDIT_LOG at row: 334202

  5. 給自己買了一些Percona的支持,他們建議我遍歷在$ MYSQL_HOME每個數據庫目錄下的* .FRM文件,爲每一個創建單獨的mysqldump命令。 (在此之前,並不知道frm文件代表數據庫內的單個數據庫表)。他們還建議在每個命令後面輸入以避免Lost connection問題。 我寫了這樣一個腳本: for table in $(ls -1 *.frm); do table_name=$(echo $table | cut -d '.' -f 1); echo mysqldump --host=127.0.0.1 --user=root --skip-lock-tables databasename ${table_name} \> database.${table_name}.sql; echo sleep 5 >> commands.txt; done 它生成了我可以用來轉儲每個數據庫表的所有mysqldump命令。

  6. 我通過創建commands.txt中環文件轉儲所有可能仍然工作表。(sh -x commands.txt

  7. 我開始了一個新的MySQL實例,創建丟失的數據庫(一個或多個),和進口的所有.mysql文件轉儲到新的數據庫中。 for each in $(ls -1 $MYSQL_HOME/recoveries/*); do mysql -uroot -pXXXX databasename < $each; sleep 5; done

  8. 確信該數據庫已運行和固定與缺少表(mysqlfrm --diagnostic幫助類似的服務器)上的其他問題

  9. 當一切準備停當,服務器又回到網上,儘管有幾個月的關鍵數據丟失。

相關問題