1
我一直在使用PRM 2天前爲MySQL Master設置故障轉移，並且......每個問題都從這裏開始。MySQL與InnoDB表不斷崩潰
啓動後corosync，MySQL掛起，套接字文件丟失，我的第一個錯誤是做了kill -9。
無法通過innodb_force_recovery = 4恢復舊主，我宣傳奴隸成爲主人，但它也遇到了同樣的問題。
這是我的配置。
[mysqld]
# Settings user and group are ignored when systemd is used (fedora >= 15).
# If you need to run mysqld under different user or group,
# customize your systemd unit file for mysqld according to the
# instructions in http://fedoraproject.org/wiki/Systemd
user=mysql
datadir=/data2/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
## Files
back_log = 300
open-files-limit = 8192
open-files = 1024
skip-external-locking
skip-name-resolve
## Logging
relay-log = mysqld-relay-bin
relay-log-index = mysqld-relay-bin.index
#log = mysql-gen.log
log_warnings
log_bin = mysql-bin
#log_queries_not_using_indexes
max_binlog_size = 256M #max size for binlog before rolling
expire_logs_days = 4 #binlog files older than this will be purged
## Per-Thread Buffers * (max_connections) = total per-thread mem usage
thread_stack = 256K #default: 32bit: 192K, 64bit: 256K
sort_buffer_size = 1M #default: 2M, larger may cause perf issues
read_buffer_size = 1M #default: 128K, change in increments of 4K
read_rnd_buffer_size = 1M #default: 256K
join_buffer_size = 1M #default: 128K
binlog_cache_size = 64K #default: 32K, size of buffer to hold TX queries
## total per-thread buffer memory usage: 8832000K = 8.625GB
## Query Cache
query_cache_size = 32M #global buffer
query_cache_limit = 512K #max query result size to put in cache
## Connections
max_connections = 2000 #multiplier for memory usage via per-thread buffers
max_connect_errors = 100 #default: 10
concurrent_insert = 2 #default: 1, 2: enable insert for all instances
connect_timeout = 30 #default -5.1.22: 5, +5.1.22: 10
max_allowed_packet = 32M #max size of incoming data to allow
## Default Table Settings
sql_mode = NO_AUTO_CREATE_USER
## Table and TMP settings
max_heap_table_size = 1G #recommend same size as tmp_table_size
bulk_insert_buffer_size = 1G #recommend same size as tmp_table_size
tmp_table_size = 1G #recommend 1G min
#tmpdir = /data/mysql-tmp0:/data/mysql-tmp1 #Recommend using RAMDISK for tmpdir
## Table cache settings
table_cache = 512 #5.0.x <default: 64>
table_open_cache = 512 #5.1.x, 5.5.x <default: 64>
## Thread settings
thread_concurrency = 16 #recommend 2x CPU cores
thread_cache_size = 100 #recommend 5% of max_connections
## MyISAM Engine
key_buffer = 1M #global buffer
myisam_sort_buffer_size = 128M #index buffer size for creating/altering indexes
myisam_max_sort_file_size = 256M #max file size for tmp table when creating/alering indexes
myisam_repair_threads = 4 #thread quantity when running repairs
myisam_recover = BACKUP #repair mode, recommend BACKUP
## InnoDB Plugin Independent Settings
#innodb_data_home_dir = /data2/var/lib/mysql
#innodb_data_file_path = ibdata1:128M;ibdata2:10M:autoextend
#innodb_log_file_size = 512M #64G_RAM+ = 768, 24G_RAM+ = 512, 8G_RAM+ = 256, 2G_RAM+ = 128
#innodb_log_files_in_group = 4 #combined size of all logs <4GB. <2G_RAM = 2, >2G_RAM = 4
innodb_buffer_pool_size = 32G #global buffer
innodb_additional_mem_pool_size = 4M #global buffer
innodb_status_file #extra reporting
innodb_file_per_table #enable always
innodb_flush_log_at_trx_commit = 2 #2/0 = perf, 1 = ACID
innodb_table_locks = 0 #preserve table locks
innodb_log_buffer_size = 128M #global buffer
innodb_lock_wait_timeout = 60
innodb_thread_concurrency = 16 #recommend 2x core quantity
innodb_commit_concurrency = 16 #recommend 4x num disks
#innodb_flush_method = O_DIRECT #O_DIRECT = local/DAS, O_DSYNC = SAN/iSCSI
innodb_support_xa = 0 #recommend 0, disable xa to negate extra disk flush
skip-innodb-doublewrite
## Binlog sync settings
## XA transactions = 1, otherwise set to 0 for best performance
sync_binlog = 1
## TX Isolation
transaction-isolation = REPEATABLE-READ #REPEATABLE-READ req for ACID, SERIALIZABLE req XA
## Per-Thread Buffer memory utilization equation:
#(read_buffer_size + read_rnd_buffer_size + sort_buffer_size + thread_stack + join_buffer_size + binlog_cache_size) * max_connections
## Global Buffer memory utilization equation:
# innodb_buffer_pool_size + innodb_additional_mem_pool_size + innodb_log_buffer_size + key_buffer_size + query_cache_size
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0
# Semisynchronous Replication
# http://dev.mysql.com/doc/refman/5.5/en/replication-semisync.html
# uncomment next line on MASTER
;plugin-load=rpl_semi_sync_master=semisync_master.so
# uncomment next line on SLAVE
;plugin-load=rpl_semi_sync_slave=semisync_slave.so
# Others options for Semisynchronous Replication
;rpl_semi_sync_master_enabled=1
;rpl_semi_sync_master_timeout=10
;rpl_semi_sync_slave_enabled=1
# http://dev.mysql.com/doc/refman/5.5/en/performance-schema.html
;performance_schema
thread_cache_size = 4
query_cache_type = 1
slow-query-log
long_query_time = 10
slow_query_log_file = /var/log/mysql/mysql-slow.log
log-warnings = 2
# slave config
skip-slave-start
server-id = 22
#read_only = 1
skip-name-resolve
log-bin=/data2/var/log/mysql/mysql-bin
relay_log_purge=0
binlog-format=MIXED
#expire_logs_days = 31
log-bin-trust-function-creators = 1
slave-skip-errors = 1062,1146,1032
replicate-wild-ignore-table=%.norep%
#testing
table_definition_cache = 5000
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
此機器在其主人面臨此問題之前被提升爲主人（因爲我不知道爲什麼，我只是關閉它）。
我正在使用mysql-server-5.5.23-1.el5.remi版本。
這是MySQL日誌文件。在表（注意2012-08-20和2012-08-26在上面的查詢）
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Last MySQL binlog file position 0 795, file name /data/var/log/mysql/mysql-bin.000801
120719 10:25:47 InnoDB: Waiting for the background threads to start
120719 10:25:48 InnoDB: 1.1.8 started; log sequence number 8068861682758
120719 10:25:48 [Note] Recovering after a crash using /data/var/log/mysql/mysql-bin
120719 10:25:48 [Note] Starting crash recovery...
120719 10:25:48 [Note] Crash recovery finished.
120719 10:25:49 [Warning] 'proxies_priv' entry '@ [email protected]' ignored in --skip-name-resolve mode.
120719 10:25:49 [Note] Event Scheduler: Loaded 27 events
120719 10:25:49 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.5.23-log' socket: '/var/lib/mysql/mysql.sock' port: 3306 MySQL Community Server (GPL) by Remi
120719 10:25:56 [Warning] Aborted connection 23 to db: 'reportingdb' user: 'adproject' host: '192.168.3.87' (Got an error reading communication packets)
120719 10:25:57 [Warning] Aborted connection 44 to db: 'reportingdb' user: 'adproject' host: '192.168.3.87' (Got an error reading communication packets)
120719 10:25:57 [Warning] Aborted connection 1 to db: 'reportingdb' user: 'adproject' host: '192.168.5.192' (Got an error reading communication packets)
03:25:57 UTC - mysqld got signal 8 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=19
max_threads=300
thread_count=16
connection_count=16
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 664409 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
Thread pointer: 0x1286fcd0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 49a340b8 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x2e)[0x7ae02e]
/usr/libexec/mysqld(handle_fatal_signal+0x3e2)[0x679da2]
/lib64/libpthread.so.0[0x3154a0ebe0]
/usr/libexec/mysqld(_ZN12ha_partition21min_rows_for_estimateEv+0x4f)[0x9223af]
[0x49a2bf60]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (11a7ad38): SELECT * FROM (
SELECT B.bannerid, SUM(realclick)clicks, SUM(totalview)views,
ROUND((MAX(uv) + (SUM(uv) - MAX(uv))*0.1)*1.1)`uviews` ,
CAST(IF(SUM(realclick)=0 OR SUM(totalview)=0,'N/A',CONCAT(ROUND(SUM(realclick)*100/(SUM(totalview)),2))) as CHAR) `CTR`,
SUM(A.`money`) money
FROM `ox_banners` B
INNER JOIN `v3_ban_date_cpm7k` AS A ON B.`campaignid` = A.`campaignid` AND B.`bannerid` = A.`bannerid`
WHERE A.`bannerid` = NAME_CONST('_bannerid',144660) AND A.`campaignid` >0
AND A.`dt` BETWEEN NAME_CONST('_start',_latin1'2012-08-20' COLLATE 'latin1_swedish_ci') AND NAME_CONST('_end',_latin1'2012-08-26' COLLATE 'latin1_swedish_ci')) A
Connection ID (thread ID): 47
Status: NOT_KILLED
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
這聽起來像MySQL的崩潰每當有訪問的查詢不存在的分區。讓我感到意外的是，這並沒有發生。
我有足夠的RAM可用。
free -m
total used free shared buffers cached
Mem: 48289 31600 16688 0 666 23848
-/+ buffers/cache: 7085 41203
Swap: 8189 54 8135
數據目錄文件權限。
drwxr-xr-x 48 mysql mysql 118784 Jul 18 16:20 /data2/var/lib/mysql
和mysqld.log。
(-rw-r----- 1 mysql mysql 36M Jul 18 17:14 /var/log/mysqld.log)
那麼如何才能找出導致我的表崩潰以及如何解決它的原因？
Jul 19：更新日誌。
我一直在試圖調試遵循this指南，但它並不能幫助：
# resolve_stack_dump -s /tmp/mysqld.sym -n mysqld.stack | c++filt
0x7ae02e my_print_stacktrace + 46
0x679da2 handle_fatal_signal + 994
0x3fed00ebe0 _end + -335143280
0x9223af ha_partition::min_rows_for_estimate() + 79
0x417c2ff0 _end + 1082221216
# addr2line -fie /usr/libexec/mysqld 0x2e
??
??:0
# addr2line -fie /usr/libexec/mysqld 0x3e2
??
??:0
# addr2line -fie /usr/libexec/mysqld 0x4f
??
??:0