rocksdb本身提供了丰富的监控信息,myrocks通过information_schema下的表和show命令等将这些信息展示出来,下面主要以示例的形式来简单介绍下
先创建测试表
1. CREATE TABLE t1 (a INT, b CHAR(8), pk INT AUTO_INCREMENT ,PRIMARY KEY(pk) comment 'cf_1', key idx2(b) comment 'cf_2') engine=rocksdb;
SHOW STATUS
show status 也展示了部分rocksdb引擎的信息
1. show status like '%rock%';
2. +---------------------------------------+------------------------------------------+
3. | Variable_name | Value |
4. +---------------------------------------+------------------------------------------+
5. | rocksdb_rows_deleted | 0 |
6. | rocksdb_rows_inserted | 1048579 |
7. | rocksdb_rows_read | 3145755 |
8. | rocksdb_rows_updated | 7 |
9. | rocksdb_system_rows_deleted | 0 |
10. | rocksdb_system_rows_inserted | 0 |
11. | rocksdb_system_rows_read | 0 |
12. | rocksdb_system_rows_updated | 0 |
13. | rocksdb_block_cache_add | 16 |
14. | rocksdb_block_cache_data_hit | 76 |
15. | rocksdb_block_cache_data_miss | 6 |
16. | rocksdb_block_cache_filter_hit | 0 |
17. | rocksdb_block_cache_filter_miss | 6 |
18. | rocksdb_block_cache_hit | 76 |
19. | rocksdb_block_cache_index_hit | 0 |
20. | rocksdb_block_cache_index_miss | 6 |
21. | rocksdb_block_cache_miss | 18 |
22. | rocksdb_block_cachecompressed_hit | 0 |
23. | rocksdb_block_cachecompressed_miss | 0 |
24. | rocksdb_bloom_filter_prefix_checked | 0 |
25. | rocksdb_bloom_filter_prefix_useful | 0 |
26. | rocksdb_bloom_filter_useful | 0 |
27. | rocksdb_bytes_read | 13631762 |
28. | rocksdb_bytes_written | 108009584 |
29. | rocksdb_compact_read_bytes | 142 |
30. | rocksdb_compact_write_bytes | 0 |
31. | rocksdb_compaction_key_drop_new | 0 |
32. | rocksdb_compaction_key_drop_obsolete | 4 |
33. | rocksdb_compaction_key_drop_user | 4 |
34. | rocksdb_flush_write_bytes | 7211 |
35. | rocksdb_getupdatessince_calls | 0 |
36. | rocksdb_git_date | %cI |
37. | rocksdb_git_hash | bc5d7b70299b763127f3714055a63ebe7e04ad47 |
38. | rocksdb_l0_num_files_stall_micros | 0 |
39. | rocksdb_l0_slowdown_micros | 0 |
40. | rocksdb_memtable_compaction_micros | 0 |
41. | rocksdb_memtable_hit | 1048593 |
42. | rocksdb_memtable_miss | 1048609 |
43. | rocksdb_no_file_closes | 0 |
44. | rocksdb_no_file_errors | 0 |
45. | rocksdb_no_file_opens | 6 |
46. | rocksdb_num_iterators | 0 |
47. | rocksdb_number_block_not_compressed | 0 |
48. | rocksdb_number_deletes_filtered | 0 |
49. | rocksdb_number_keys_read | 2097202 |
50. | rocksdb_number_keys_updated | 0 |
51. | rocksdb_number_keys_written | 2097220 |
52. | rocksdb_number_merge_failures | 0 |
53. | rocksdb_number_multiget_bytes_read | 0 |
54. | rocksdb_number_multiget_get | 0 |
55. | rocksdb_number_multiget_keys_read | 0 |
56. | rocksdb_number_reseeks_iteration | 0 |
57. | rocksdb_number_sst_entry_delete | 12 |
58. | rocksdb_number_sst_entry_merge | 0 |
59. | rocksdb_number_sst_entry_other | 0 |
60. | rocksdb_number_sst_entry_put | 30 |
61. | rocksdb_number_sst_entry_singledelete | 0 |
62. | rocksdb_number_stat_computes | 0 |
63. | rocksdb_number_superversion_acquires | 21 |
64. | rocksdb_number_superversion_cleanups | 1 |
65. | rocksdb_number_superversion_releases | 1 |
66. | rocksdb_rate_limit_delay_millis | 0 |
67. | rocksdb_snapshot_conflict_errors | 0 |
68. | rocksdb_wal_bytes | 54006676 |
69. | rocksdb_wal_group_syncs | 0 |
70. | rocksdb_wal_synced | 13 |
71. | rocksdb_write_other | 0 |
72. | rocksdb_write_self | 58 |
73. | rocksdb_write_timedout | 0 |
74. | rocksdb_write_wal | 58 |
75. +---------------------------------------+------------------------------------------+
INFORMATION_SCHMEA
information_schema下rocksdb相关的表如下
1. select table_name from INFORMATION_SCHEMA.tables where table_name like '%rock%';
2. +-----------------------------+
3. | table_name |
4. +-----------------------------+
5. | ROCKSDB_PERF_CONTEXT |
6. | ROCKSDB_GLOBAL_INFO |
7. | ROCKSDB_COMPACTION_STATS |
8. | ROCKSDB_INDEX_FILE_MAP |
9. | ROCKSDB_CF_OPTIONS |
10. | ROCKSDB_PERF_CONTEXT_GLOBAL |
11. | ROCKSDB_CFSTATS |
12. | ROCKSDB_TRX |
13. | ROCKSDB_DBSTATS |
14. | ROCKSDB_DDL |
15. | ROCKSDB_LOCKS |
16. +-----------------------------+
- 数据字典相关
1. show create table INFORMATION_SCHEMA.ROCKSDB_INDEX_FILE_MAP\G
2. *************************** 1. row ***************************
3. Table: ROCKSDB_INDEX_FILE_MAP
4. Create Table: CREATE TEMPORARY TABLE `ROCKSDB_INDEX_FILE_MAP` (
5. `COLUMN_FAMILY` int(4) NOT NULL DEFAULT '0',
6. `INDEX_NUMBER` int(4) NOT NULL DEFAULT '0',
7. `SST_NAME` varchar(193) NOT NULL DEFAULT '',
8. `NUM_ROWS` bigint(8) NOT NULL DEFAULT '0',
9. `DATA_SIZE` bigint(8) NOT NULL DEFAULT '0',
10. `ENTRY_DELETES` bigint(8) NOT NULL DEFAULT '0',
11. `ENTRY_SINGLEDELETES` bigint(8) NOT NULL DEFAULT '0',
12. `ENTRY_MERGES` bigint(8) NOT NULL DEFAULT '0',
13. `ENTRY_OTHERS` bigint(8) NOT NULL DEFAULT '0'
14. ) ENGINE=MEMORY DEFAULT CHARSET=utf8
15. 1 row in set (0.00 sec)
17. mysql> show create table INFORMATION_SCHEMA.ROCKSDB_DDL\G
18. *************************** 1. row ***************************
19. Table: ROCKSDB_DDL
20. Create Table: CREATE TEMPORARY TABLE `ROCKSDB_DDL` (
21. `TABLE_SCHEMA` varchar(193) NOT NULL DEFAULT '',
22. `TABLE_NAME` varchar(193) NOT NULL DEFAULT '',
23. `PARTITION_NAME` varchar(193) DEFAULT NULL,
24. `INDEX_NAME` varchar(193) NOT NULL DEFAULT '',
25. `COLUMN_FAMILY` int(4) NOT NULL DEFAULT '0',
26. `INDEX_NUMBER` int(4) NOT NULL DEFAULT '0',
27. `INDEX_TYPE` smallint(2) NOT NULL DEFAULT '0',
28. `KV_FORMAT_VERSION` smallint(2) NOT NULL DEFAULT '0',
29. `CF` varchar(193) NOT NULL DEFAULT ''
30. ) ENGINE=MEMORY DEFAULT CHARSET=utf8
例如查询t1表的数据字典信息
1. select d.*,i.* from INFORMATION_SCHEMA.ROCKSDB_INDEX_FILE_MAP i,INFORMATION_SCHEMA.ROCKSDB_DDL d where i.INDEX_NUMBER=d.INDEX_NUMBER\G
2. *************************** 1. row ***************************
3. TABLE_SCHEMA: test
4. TABLE_NAME: t1
5. PARTITION_NAME: NULL
6. INDEX_NAME: PRIMARY
7. COLUMN_FAMILY: 2
8. INDEX_NUMBER: 263
9. INDEX_TYPE: 1
10. KV_FORMAT_VERSION: 11
11. CF: cf_1
12. COLUMN_FAMILY: 2
13. INDEX_NUMBER: 263
14. SST_NAME: 000039.sst
15. NUM_ROWS: 2
16. DATA_SIZE: 42
17. ENTRY_DELETES: 0
18. ENTRY_SINGLEDELETES: 0
19. ENTRY_MERGES: 0
20. ENTRY_OTHERS: 0
21. *************************** 2. row ***************************
22. TABLE_SCHEMA: test
23. TABLE_NAME: t1
24. PARTITION_NAME: NULL
25. INDEX_NAME: idx2
26. COLUMN_FAMILY: 3
27. INDEX_NUMBER: 264
28. INDEX_TYPE: 2
29. KV_FORMAT_VERSION: 11
30. CF: cf_2
31. COLUMN_FAMILY: 3
32. INDEX_NUMBER: 264
33. SST_NAME: 000040.sst
34. NUM_ROWS: 2
35. DATA_SIZE: 45
36. ENTRY_DELETES: 0
37. ENTRY_SINGLEDELETES: 0
38. ENTRY_MERGES: 0
39. ENTRY_OTHERS: 0
40. 2 rows in set (0.00 sec)
- 事务相关
1. begin;
2. select * from INFORMATION_SCHEMA.ROCKSDB_LOCKS;
3. INSERT INTO t1 (a,b) VALUES (1,'a');
4. select * from INFORMATION_SCHEMA.ROCKSDB_LOCKS;
5. +------------------+----------------+------------------+------+
6. | COLUMN_FAMILY_ID | TRANSACTION_ID | KEY | MODE |
7. +------------------+----------------+------------------+------+
8. | 2 | 14 | 0000010780000001 | X |
9. +------------------+----------------+------------------+------+
11. select * from INFORMATION_SCHEMA.ROCKSDB_TRX\G
12. *************************** 1. row ***************************
13. TRANSACTION_ID: 89
14. STATE: STARTED
15. NAME:
16. WRITE_COUNT: 2
17. LOCK_COUNT: 2
18. TIMEOUT_SEC: 2
19. WAITING_KEY:
20. WAITING_COLUMN_FAMILY_ID: 0
21. IS_REPLICATION: 0
22. SKIP_TRX_API: 0
23. READ_ONLY: 0
24. HAS_DEADLOCK_DETECTION: 0
25. NUM_ONGOING_BULKLOAD: 0
26. THREAD_ID: 13
27. QUERY: select * from INFORMATION_SCHEMA.ROCKSDB_TRX
其中KEY 0000010780100002表示indexnum:107(263)pk: 80000001(1) 表示(1,’a’,1)这条记录,具体参考myrocks记录格式分析
- 统计信息相关
1. select * from INFORMATION_SCHEMA.ROCKSDB_GLOBAL_INFO;
2. +--------------+--------------+-----------------------------------------+
3. | TYPE | NAME | VALUE |
4. +--------------+--------------+-----------------------------------------+
5. | BINLOG | FILE | mysql-bin.000003 |
6. | BINLOG | POS | 18957545 |
7. | BINLOG | GTID | b89fb268-0b22-11e7-a0ce-2c44fd7a5210:27 |
8. | MAX_INDEX_ID | MAX_INDEX_ID | 264 |
9. | CF_FLAGS | 0 | default [0] |
10. | CF_FLAGS | 1 | __system__ [0] |
11. | CF_FLAGS | 2 | cf_1 [0] |
12. | CF_FLAGS | 3 | cf_2 [0] |
13. +--------------+--------------+-----------------------------------------+
15. select * from INFORMATION_SCHEMA.ROCKSDB_DBSTATS;
16. +-------------------------+-------+
17. | STAT_TYPE | VALUE |
18. +-------------------------+-------+
19. | DB_BACKGROUND_ERRORS | 0 |
20. | DB_NUM_SNAPSHOTS | 0 |
21. | DB_OLDEST_SNAPSHOT_TIME | 0 |
22. | DB_BLOCK_CACHE_USAGE | 1119 |
23. +-------------------------+-------+
25. select * from INFORMATION_SCHEMA.ROCKSDB_CFSTATS where CF_NAME='cf_1';
26. +---------+-------------------------------+----------+
27. | CF_NAME | STAT_TYPE | VALUE |
28. +---------+-------------------------------+----------+
29. | cf_1 | NUM_IMMUTABLE_MEM_TABLE | 0 |
30. | cf_1 | MEM_TABLE_FLUSH_PENDING | 0 |
31. | cf_1 | COMPACTION_PENDING | 0 |
32. | cf_1 | CUR_SIZE_ACTIVE_MEM_TABLE | 44739520 |
33. | cf_1 | CUR_SIZE_ALL_MEM_TABLES | 44739520 |
34. | cf_1 | NUM_ENTRIES_ACTIVE_MEM_TABLE | 1048574 |
35. | cf_1 | NUM_ENTRIES_IMM_MEM_TABLES | 0 |
36. | cf_1 | NON_BLOCK_CACHE_SST_MEM_USAGE | 0 |
37. | cf_1 | NUM_LIVE_VERSIONS | 1 |
38. +---------+-------------------------------+----------+
- 性能相关
INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT_GLOBAL 是全局的性能信息, 而INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT是以表为单位的性能信息。
性能统计由参数rocksdb_perf_context_level控制,取值范围如下
1. enum PerfLevel : unsigned char {
2. kUninitialized = 0, // unknown setting
3. kDisable = 1, // disable perf stats
4. kEnableCount = 2, // enable only count stats
5. kEnableTimeExceptForMutex = 3, // Other than count stats, also enable time
6. // stats except for mutexes
7. kEnableTime = 4, // enable count and time stats
8. kOutOfBounds = 5 // N.B. Must always be the last value!
9. };
示例如下:
2. select * from INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT_GLOBAL;
3. +---------------------------------+-------------+
4. | STAT_TYPE | VALUE |
5. +---------------------------------+-------------+
6. | USER_KEY_COMPARISON_COUNT | 565061179 |
7. | BLOCK_CACHE_HIT_COUNT | 26 |
8. | BLOCK_READ_COUNT | 2 |
9. | BLOCK_READ_BYTE | 145 |
10. | BLOCK_READ_TIME | 684522 |
11. | BLOCK_CHECKSUM_TIME | 8380 |
12. | BLOCK_DECOMPRESS_TIME | 10825 |
13. | INTERNAL_KEY_SKIPPED_COUNT | 3371079 |
14. | INTERNAL_DELETE_SKIPPED_COUNT | 0 |
15. | GET_SNAPSHOT_TIME | 2409821566 |
16. | GET_FROM_MEMTABLE_TIME | 68354733245 |
17. | GET_FROM_MEMTABLE_COUNT | 4194309 |
18. | GET_POST_PROCESS_TIME | 3421224444 |
19. | GET_FROM_OUTPUT_FILES_TIME | 8016972510 |
20. | SEEK_ON_MEMTABLE_TIME | 277621 |
21. | SEEK_ON_MEMTABLE_COUNT | 33 |
22. | SEEK_CHILD_SEEK_TIME | 1700582 |
23. | SEEK_CHILD_SEEK_COUNT | 54 |
24. | SEEK_IN_HEAP_TIME | 101201 |
25. | SEEK_INTERNAL_SEEK_TIME | 2019275 |
26. | FIND_NEXT_USER_ENTRY_TIME | 3997301676 |
27. | WRITE_WAL_TIME | 410899041 |
28. | WRITE_MEMTABLE_TIME | 23580852751 |
29. | WRITE_DELAY_TIME | 0 |
30. | WRITE_PRE_AND_POST_PROCESS_TIME | 1117611 |
31. | DB_MUTEX_LOCK_NANOS | 237804 |
32. | DB_CONDITION_WAIT_NANOS | 0 |
33. | MERGE_OPERATOR_TIME_NANOS | 0 |
34. | READ_INDEX_BLOCK_NANOS | 0 |
35. | READ_FILTER_BLOCK_NANOS | 0 |
36. | NEW_TABLE_BLOCK_ITER_NANOS | 1109437 |
37. | NEW_TABLE_ITERATOR_NANOS | 308214 |
38. | BLOCK_SEEK_NANOS | 1290004508 |
39. | FIND_TABLE_NANOS | 0 |
40. | IO_THREAD_POOL_ID | 102 |
41. | IO_BYTES_WRITTEN | 54016973 |
42. | IO_BYTES_READ | 145 |
43. | IO_OPEN_NANOS | 0 |
44. | IO_ALLOCATE_NANOS | 0 |
45. | IO_WRITE_NANOS | 116163102 |
46. | IO_READ_NANOS | 664547 |
47. | IO_RANGE_SYNC_NANOS | 0 |
48. | IO_LOGGER_NANOS | 0 |
49. +---------------------------------+-------------+
51. select * from INFORMATION_SCHEMA.ROCKSDB_PERF_CONTEXT where table_name='t1' limit 1;
52. +--------------+------------+----------------+---------------------------+-----------+
53. | TABLE_SCHEMA | TABLE_NAME | PARTITION_NAME | STAT_TYPE | VALUE |
54. +--------------+------------+----------------+---------------------------+-----------+
55. | test | t1 | NULL | USER_KEY_COMPARISON_COUNT | 565060904 |
56. +--------------+------------+----------------+---------------------------+-----------+
- COMPACTION相关
1. select * from INFORMATION_SCHEMA.ROCKSDB_COMPACTION_STATS where CF_NAME='cf_1' limit 3;
2. +---------+-------+-----------+-------+
3. | CF_NAME | LEVEL | TYPE | VALUE |
4. +---------+-------+-----------+-------+
5. | cf_1 | L0 | AvgSec | 0 |
6. | cf_1 | L0 | CompCount | 2 |
7. | cf_1 | L0 | CompSec | 0 |
8. +---------+-------+-----------+-------+
具体可以参考下节SHOW ENGINE ROCKSDB STATUS。
- 参数配置
每个column family 都是独立的配置信息
1. select * from INFORMATION_SCHEMA.ROCKSDB_CF_OPTIONS where CF_NAME='cf_1' limit 3;
2. +---------+-------------------+------------------+
3. | CF_NAME | OPTION_TYPE | VALUE |
4. +---------+-------------------+------------------+
5. | cf_1 | COMPARATOR | RocksDB_SE_v3.10 |
6. | cf_1 | MERGE_OPERATOR | NULL |
7. | cf_1 | COMPACTION_FILTER | NULL |
8. +---------+-------------------+------------------+
SHOW ENGINE ROCKSDB STATUS
show engine rocksdb status.结果主要分为三部分
1)DB Stats
2)Compaction Stats
3)Memory_Stats
show engine rocksdb status\G结果节选
1. show engine rocksdb status\G
2. *************************** 1. row ***************************
3. Type: DBSTATS
4. Name: rocksdb
5. Status:
6. ** DB Stats **
7. Uptime(secs): 211548.0 total, 8140.1 interval
8. Cumulative writes: 58 writes, 2097K keys, 58 commit groups, 1.0 writes per commit group, ingest: 0.10 GB, 0.00 MB/s
9. Cumulative WAL: 58 writes, 13 syncs, 4.14 writes per sync, written: 0.05 GB, 0.00 MB/s
10. Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent
11. Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s
12. Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s
13. Interval stall: 00:00:0.000 H:M:S, 0.0 percent
15. ......(省略)
17. *************************** 3. row ***************************
18. Type: CF_COMPACTION
19. Name: cf_1
20. Status:
21. ** Compaction Stats [cf_1] **
22. Level Files Size(MB} Score Read(GB} Rn(GB} Rnp1(GB} Write(GB} Wnew(GB} Moved(GB} W-Amp Rd(MB/s} Wr(MB/s} Comp(sec} Comp(cnt} Avg(sec} KeyIn KeyDrop
23. ----------------------------------------------------------------------------------------------------------------------------------------------------------
24. L0 1/0 0.00 0.2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.3 0 2 0.004 0 0
25. Sum 1/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.1 0.2 0 3 0.004 2 2
26. Int 0/0 0.00 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0 0 0.000 0 0
27. Uptime(secs): 210665.0 total, 210665.0 interval
28. Flush(GB): cumulative 0.000, interval 0.000
29. AddFile(GB): cumulative 0.000, interval 0.000
30. AddFile(Total Files): cumulative 0, interval 0
31. AddFile(L0 Files): cumulative 0, interval 0
32. AddFile(Keys): cumulative 0, interval 0
33. Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
34. Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds
35. Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count
37. ......(省略)
39. *************************** 6. row ***************************
40. Type: Memory_Stats
41. Name: rocksdb
42. Status:
43. MemTable Total: 93675232
44. MemTable Unflushed: 93673184
45. Table Readers Total: 0
46. Cache Total: 1119
47. Default Cache Capacity: 0
48. 6 rows in set (0.00 sec)
- DB Stats
其中: Interval stall: 此值受max_write_buffer_number,level0_slowdown_writes_trigger、soft_pending_compaction_bytes_limit等参数的影响, 具体参考(SetupDelay)
- Compaction Stats
其中 Rn(GB} = bytes_read_non_output_levels / kGB Rnp1(GB} = bytes_read_output_level / kGB W-Amp = bytes_written/bytes_read_non_output_levels
此部分内容与 INFORMATION_SCHEMA.ROCKSDB_COMPACTION_STATS有部分重合。
- Memory_Stats
MemTable Total: 对应DB::Properties::kSizeAllMemTables MemTable Unflushed:对应DB::Properties::kCurSizeAllMemTables Table Readers Total: 对应DB::Properties::kEstimateTableReadersMem Cache Total: 表示已使用的内存 Default Cache Capacity: 使用默认blockcache的总量(basetable没有指定blockcache时使用默认的8M的blockcache)
SHOW ENGING ROKSDB TRANCTION STATUS
显示当前正在运行的事务语句
1. show engine rocksdb transaction status\G
2. *************************** 1. row ***************************
3. Type: SNAPSHOTS
4. Name: rocksdb
5. Status:
6. ============================================================
7. 2017-03-20 07:49:22 ROCKSDB TRANSACTION MONITOR OUTPUT
8. ============================================================
9. ---------
10. SNAPSHOTS
11. ---------
12. LIST OF SNAPSHOTS FOR EACH SESSION:
13. ---SNAPSHOT, ACTIVE 5 sec
14. MySQL thread id 12, OS thread handle 0x7fd23a1d0700, query id 187 127.0.0.1 root Searching rows for update
15. update t1 set b='cc' where a=2
16. lock count 72822, write count 2
17. -----------------------------------------
18. END OF ROCKSDB TRANSACTION MONITOR OUTPUT
19. =========================================
总结
以上粗略介绍了myrocks的监控信息,具体还需要在实践中灵活运用。myrocks的监控信息也在不断完善中。
