南大通用GBase 8a分布策略distribution的不同pattern样例

本文介绍GBase 8a的分布策略distribution的不同方案pattern的结果样例,方便用户创建高可用程度更高的集群。所有的策略,都是为了在节点故障时,更合理有效的利用副本。

环境

我们准备了9个IP,分成3组(rack),代表3个机柜,看每种策略的主备策略的结果。

gcChangeInfo.xml

<?xml version="1.0" encoding="utf-8"?>
<servers>
    <rack>
        <node ip="10.0.2.103"/>
        <node ip="10.0.2.104"/>
        <node ip="10.0.2.105"/>
    </rack>
    <rack>
        <node ip="10.0.2.106"/>
        <node ip="10.0.2.107"/>
        <node ip="10.0.2.108"/>
    </rack>
    <rack>
        <node ip="10.0.2.109"/>
        <node ip="10.0.2.110"/>
        <node ip="10.0.2.111"/>
    </rack>
</servers>

parttern 1负载均衡默认

目标:

  • 任意一台服务器离线
  • 任意一个rack整体离线
  • 任意横向同层(同上下位置)的机器离线

其负载要均衡在不同的机器上;

第一个分片的副本1:放下一个rack的一个位置,副本2,则放在上一个rack的一个位置。

第二个分片的副本1:放下一个rack的下下一个位置,副本2,则放在上一个rack的上上一个位置。

第三个分片的副本1:放下一个rack的下下下一个位置,副本2,则放在上一个rack的上上上一个位置。

以此类推

在考虑副本前提下,一个主分片的副本,必须落在第2台机器上,第二个主分片,必须放在第三台机器上。所以主分片数量必须小于rack内的节点数量。

同时建议不同rack内机器数量相同,否则都是10个节点的rack,有一个rack是3个节点。当10节点rack故障时,3节点的rack负载必然比别的rack的高。如果不同rack无法整齐,不如用pattern 2或者自定义。

注意:parrten 1时,建议不要使用单rack,无意义。

1个副本的1+1模式

副本位置是下一个rack的下一个位置,比如

  • rack1的节点1,副本1放到rack2的节点2
  • rack1的节点2,副本1放到rack2的节点3
  • 同理
  • rack2的节点3,副本1放到rack3的节点4

如果节点超过了末尾,则转到开头,比如rack之后3个节点,那么

  • rack1的节点3,副本1放到rack2的节点1(回到开头)

实际样例

GBase 8a分布策略distribution的不同pattern样例 pattern 1 ,1副本
GBase 8a分布策略distribution的不同pattern样例 pattern 1 ,1副本
[gbase@gbase_rh7_003 gcinstall]$ gcadmin distribution gcChangeInfo.xml p 1 d 1 pattern 1
gcadmin generate distribution ...

NOTE: node [10.0.2.103] is coordinator node, it shall be data node too
gcadmin generate distribution successful

[gbase@gbase_rh7_003 gcinstall]$ gcadmin showdistribution

                                 Distribution ID: 1 | State: new | Total segment num: 9

             Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                   10.0.2.103                   |         1          |                   10.0.2.107                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.104                   |         2          |                   10.0.2.108                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.105                   |         3          |                   10.0.2.106                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.106                   |         4          |                   10.0.2.110                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.107                   |         5          |                   10.0.2.111                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.108                   |         6          |                   10.0.2.109                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.109                   |         7          |                   10.0.2.104                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.110                   |         8          |                   10.0.2.105                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.111                   |         9          |                   10.0.2.103                   |
========================================================================================================================

2个副本的1+2模式

副本2,则放在上一个rack的上一个位置,如果小于第一个,则返回末尾一个。

GBase 8a分布策略distribution的不同pattern样例 pattern 1 ,2副本
GBase 8a分布策略distribution的不同pattern样例 pattern 1 ,2副本
[gbase@gbase_rh7_003 gcinstall]$ gcadmin distribution gcChangeInfo.xml p 1 d 2 pattern 1
gcadmin generate distribution ...

gcadmin generate distribution successful

[gbase@gbase_rh7_003 gcinstall]$ gcadmin showdistribution

                                 Distribution ID: 2 | State: new | Total segment num: 9

             Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                   10.0.2.103                   |         1          |                   10.0.2.107                   |
|                                                |                    |                   10.0.2.111                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.104                   |         2          |                   10.0.2.108                   |
|                                                |                    |                   10.0.2.109                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.105                   |         3          |                   10.0.2.106                   |
|                                                |                    |                   10.0.2.110                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.106                   |         4          |                   10.0.2.110                   |
|                                                |                    |                   10.0.2.105                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.107                   |         5          |                   10.0.2.111                   |
|                                                |                    |                   10.0.2.103                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.108                   |         6          |                   10.0.2.109                   |
|                                                |                    |                   10.0.2.104                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.109                   |         7          |                   10.0.2.104                   |
|                                                |                    |                   10.0.2.108                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.110                   |         8          |                   10.0.2.105                   |
|                                                |                    |                   10.0.2.106                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.111                   |         9          |                   10.0.2.103                   |
|                                                |                    |                   10.0.2.107                   |
========================================================================================================================

如果只有2个rack,采用了2副本

副本1不变,下一个rack的下一个位置,副本2则变成了本rack的下下一个位置,也就是位置+2。如下是2个rack的样例。

gcChangeInfo_rack2.xml

[gbase@gbase_rh7_003 gcinstall]$ cat gcChangeInfo_rack2.xml
<?xml version="1.0" encoding="utf-8"?>
<servers>
    <rack>
        <node ip="10.0.2.103"/>
        <node ip="10.0.2.104"/>
        <node ip="10.0.2.105"/>
        <node ip="10.0.2.106"/>
    </rack>
    <rack>
        <node ip="10.0.2.107"/>
        <node ip="10.0.2.108"/>
        <node ip="10.0.2.109"/>
        <node ip="10.0.2.110"/>
    </rack>

结果样例

[gbase@gbase_rh7_003 gcinstall]$ gcadmin distribution gcChangeInfo_rack2.xml p 1 d 2 pattern 1
gcadmin generate distribution ...

gcadmin generate distribution successful

[gbase@gbase_rh7_003 gcinstall]$ gcadmin showdistribution

                                 Distribution ID: 4 | State: new | Total segment num: 8

             Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                   10.0.2.103                   |         1          |                   10.0.2.108                   |
|                                                |                    |                   10.0.2.105                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.104                   |         2          |                   10.0.2.109                   |
|                                                |                    |                   10.0.2.106                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.105                   |         3          |                   10.0.2.110                   |
|                                                |                    |                   10.0.2.103                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.106                   |         4          |                   10.0.2.107                   |
|                                                |                    |                   10.0.2.104                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.107                   |         5          |                   10.0.2.104                   |
|                                                |                    |                   10.0.2.109                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.108                   |         6          |                   10.0.2.105                   |
|                                                |                    |                   10.0.2.110                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.109                   |         7          |                   10.0.2.106                   |
|                                                |                    |                   10.0.2.107                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.110                   |         8          |                   10.0.2.103                   |
|                                                |                    |                   10.0.2.108                   |

pattern 2 高可用模式

该模式忽视rack,按照出现先后顺序,副本1在下一个节点,副本2在上一个节点。如果超过末尾,则返回第一个,如果低于第一个,则返回末尾。

无论多少个主分片,副本1都在下一个节点,副本2都在上一个节点。

如果有多个分片,副本策略同第一个相同,也就是无论多少个分片,其第一个副本都在下一个节点,第二个副本在下下一个。

1主分片+2副本输出样例

[gbase@gbase_rh7_003 gcinstall]$ gcadmin distribution gcChangeInfo.xml p 1 d 2 pattern 2
gcadmin generate distribution ...

NOTE: node [10.0.2.103] is coordinator node, it shall be data node too
gcadmin generate distribution successful

[gbase@gbase_rh7_003 gcinstall]$ gcadmin showdistribution

                                 Distribution ID: 3 | State: new | Total segment num: 9

             Primary Segment Node IP                   Segment ID                 Duplicate Segment node IP
========================================================================================================================
|                   10.0.2.103                   |         1          |                   10.0.2.104                   |
|                                                |                    |                   10.0.2.111                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.104                   |         2          |                   10.0.2.105                   |
|                                                |                    |                   10.0.2.103                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.105                   |         3          |                   10.0.2.106                   |
|                                                |                    |                   10.0.2.104                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.106                   |         4          |                   10.0.2.107                   |
|                                                |                    |                   10.0.2.105                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.107                   |         5          |                   10.0.2.108                   |
|                                                |                    |                   10.0.2.106                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.108                   |         6          |                   10.0.2.109                   |
|                                                |                    |                   10.0.2.107                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.109                   |         7          |                   10.0.2.110                   |
|                                                |                    |                   10.0.2.108                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.110                   |         8          |                   10.0.2.111                   |
|                                                |                    |                   10.0.2.109                   |
------------------------------------------------------------------------------------------------------------------------
|                   10.0.2.111                   |         9          |                   10.0.2.103                   |
|                                                |                    |                   10.0.2.110                   |
========================================================================================================================

多主分片 2分片+2副本

[gbase@localhost gcinstall_9524410p5]$ cat gcChangeInfo.xml 
<?xml version="1.0" encoding="utf-8"?>
<servers>
 <rack>
  <node ip="10.0.2.102"/>
  <node ip="10.0.2.202"/>
  <node ip="10.0.2.203"/>
 </rack>
</servers>
[gbase@localhost gcinstall_9524410p5]$ gcadmin showdistribution

              Distribution ID: 2 | State: new | Total segment num: 6

    Primary Segment Node IP         Segment ID         Duplicate Segment node IP
====================================================================================
|       10.0.2.102             |       1          |        10.0.2.202              |
|                              |                  |        10.0.2.203              |
------------------------------------------------------------------------------------
|       10.0.2.202             |       2          |        10.0.2.203              |
|                              |                  |        10.0.2.102              |
------------------------------------------------------------------------------------
|       10.0.2.203             |       3          |        10.0.2.102              |
|                              |                  |        10.0.2.202              |
------------------------------------------------------------------------------------
|       10.0.2.102             |       4          |        10.0.2.202              |
|                              |                  |        10.0.2.203              |
------------------------------------------------------------------------------------
|       10.0.2.202             |       5          |        10.0.2.203              |
|                              |                  |        10.0.2.102              |
------------------------------------------------------------------------------------
|       10.0.2.203             |       6          |        10.0.2.102              |
|                              |                  |        10.0.2.202              |
====================================================================================

总结

为了最大限度的高可用,建议不小于3个机柜,避免某个机柜整体出问题,导致集群整体不可用的情况出现。