[MongoDB#2] How To Create a Sharded Cluster in MongoDB Using an Ubuntu 14.04 LTS

How To Create a Sharded Cluster in MongoDB Using an Ubuntu 14.04 LTS

예전 로컬 가상환경에서 구성했던 경험을 바탕으로 개발머신 5대에 Replication

및 Sharding 환경을 구성해 보았다.

기존 자료 참조 : http://rocksea.tistory.com/13

System Layout.

[그림 1] MongoDB Process Layout

환경 구성에 앞서 각각의 프로세스에 대한 설명을 짚고넘어갈 필요가 있어 정리해봤다.

Mongos : 분산처리를 위한 라우팅을 담당하는 프로세스.

Mongod : MongoDB 시스템의 기본 데몬프로세스로 DB 정보 저장 및 요청에 대한 핸들링, 데이터 access등의 관리등을 담당한다.

Arbiter : Primary 장애 시 Secondary를 Primary로 승격시키기 위한 선출만을 담당한다. 

Config : Config 데몬의 경우 1개 또는 세개가 아니면 정상동작 하지 않기 때문에 갯수를 꼭 1개 또는 3개로 맞춰야 한다.

step 1. 공개키 서버 설정

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 7F0CEB10

step 2. MongoDB Repository 설정

$ echo 'deb http://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen' | sudo tee /etc/apt/sources.list.d/mongodb.list

$ sudo apt-get update

step 3. mongo db 설치

$ sudo apt-get install mongodb-org

$ sudo apt-get install mongodb-org=2.6.6 mongodb-org-server=2.6.6 mongodb-org-shell=2.6.6 mongodb-org-mongos=2.6.6 mongodb-org-tools=2.6.6

step 4. Replica Set 설정.

저장 디렉토리 생성.

$ sudo mkdir -p /data/mongodb

$ sudo mkdir -p /data/mongodb/first

$ sudo mkdir -p /data/mongodb/second

$ sudo mkdir -p /data/mongodb/third

$ sudo mkdir -p /data/mongodb/config

$ sudo mkdir -p /data/mongodb/arbiter

$ sudo mkdir -p /data/mongodb/logs

$ sudo chown cloud -R /data/mongodb

step 5. Mongo 데몬 프로세스 실행

cloud 0

$ sudo mongod --dbpath /data/mongodb/first --port 10001 --replSet first --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/first.log

$ sudo mongod --dbpath /data/mongodb/second --port 10002 --replSet second --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/second.log

$ sudo mongod --configsvr --dbpath /data/mongodb/config --port 10000  --fork --quiet --logpath /data/mongodb/logs/config.log 

$ sudo mongod --port 11000 --dbpath /data/mongodb/arbiter --replSet first --fork --quiet --logpath /data/mongodb/logs/arbiter.log

cloud1 

$ sudo mongod --dbpath /data/mongodb/third --port 10003 --replSet third --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/third.log

$ sudo mongod --port 11000 --dbpath /data/mongodb/arbiter --replSet second --fork --quiet --logpath /data/mongodb/logs/arbiter.log

$ sudo mongod --configsvr --dbpath /data/mongodb/config --port 10000  --fork --quiet --logpath /data/mongodb/logs/config.log 

cloud2

$ sudo mongod --dbpath /data/mongodb/first --port 10001 --replSet first --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/first.log

$ sudo mongod --configsvr --dbpath /data/mongodb/config --port 10000  --fork --quiet --logpath /data/mongodb/logs/config.log 

$ sudo mongod --port 11000 --dbpath /data/mongodb/arbiter --replSet third --fork --quiet --logpath /data/mongodb/logs/arbiter.log

cloud3

$ sudo mongod --dbpath /data/mongodb/first --port 10001 --replSet first --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/first.log

$ sudo mongod --dbpath /data/mongodb/second --port 10002 --replSet second --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/second.log

$ sudo mongod --dbpath /data/mongodb/third --port 10003 --replSet third --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/third.log

cloud4

$ sudo mongod --dbpath /data/mongodb/second --port 10002 --replSet second --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/second.log

$ sudo mongod --dbpath /data/mongodb/third --port 10003 --replSet third --fork --quiet --oplogSize 1024 --logpath /data/mongodb/logs/third.log

공통

모든 config server 띄운 후 mongos 수행

$ sudo mongos --configdb cloud0:10000,cloud1:10000,cloud2:10000 -chunkSize 64 --port 12000 --fork --quiet --logpath /data/mongodb/logs/mongos.log  

Mongos 오류

2015-01-14T16:27:38.454+0900 [mongosMain] waited 442s for distributed lock configUpgrade for upgrading config database to new format v5

이런 오류가 발생하는 경우가 있다. 원인은 시간동기화가 안되었을 경우 발생하니 필히 

서버 시간 동기화 설정을 해두도록 하자.

참고 : http://blog.pincoin.co.kr/2012/12/24/ntp-%EC%8B%9C%EA%B0%81-%EB%8F%99%EA%B8%B0%ED%99%94/

step 6. Replica set 설정

replication을 설정하기 위해 Arbiter설정과 각 Replication의 Primary, Second에 대한

Priority 값을 설정한다. ( Priority 값이 제일 높은값이 Primary가 되며 기본값은 1이다. )

1. first replica set 설정

$ mongo cloud0:10001/admin

> db.runCommand({"replSetInitiate" : 

{"_id" : "first", 

 "members" : [

               {"_id" : 1, "host" : "cloud0:11000",  "arbiterOnly":true}, 

               {"_id" : 2, "host" : "cloud0:10001", "priority":2}, 

               {"_id" : 3, "host" : "cloud2:10001", "priority":1},

               {"_id" : 4, "host" : "cloud3:10001", "priority":1}

             ]

}});

{

"info" : "Config now saved locally. Should come online in about a minute.",

"ok" : 1

}

2. second replica set 설정

$ mongo cloud3:10002/admin

> db.runCommand({"replSetInitiate" : 

{"_id" : "second", 

 "members" : [

               {"_id" : 1, "host" : "cloud1:11000",  "arbiterOnly":true}, 

               {"_id" : 2, "host" : "cloud0:10002", "priority":1}, 

               {"_id" : 3, "host" : "cloud3:10002", "priority":2},

               {"_id" : 4, "host" : "cloud4:10002", "priority":1}

             ]

}});

3. third replica set 설정

$ mongo cloud4:10003/admin

> db.runCommand({"replSetInitiate" : 

{"_id" : "third", 

 "members" : [

               {"_id" : 1, "host" : "cloud2:11000",  "arbiterOnly":true}, 

               {"_id" : 2, "host" : "cloud1:10003", "priority":1}, 

               {"_id" : 3, "host" : "cloud3:10003", "priority":1},

               {"_id" : 4, "host" : "cloud4:10003", "priority":2}

             ]

}});

step 7. Shard 설정

$ mongo cloud0:12000/admin

mongos> db.runCommand({addshard:"first/cloud0:10001"})

mongos> db.runCommand({addshard:"second/cloud3:10002"})

mongos> db.runCommand({addshard:"third/cloud4:10003"})

db.runCommand({listshards:1})

{

        "shards" : [

                {

                        "_id" : "first",

                        "host" : "first/cloud0:10001,cloud2:10001,cloud3:10001"

                },

                {

                        "_id" : "second",

                        "host" : "second/cloud0:10002,cloud3:10002,cloud4:10002"

                },

                {

                        "_id" : "third",

                        "host" : "third/cloud1:10003,cloud3:10003,cloud4:10003"

                }

        ],

        "ok" : 1

}

테스트용 collection을 생성한다.

mongod> use test

mongod> people = ["Marc", "Bill", "George", "Eliot", "Matt", "Trey", "Tracy", "Greg", "Steve", "Kristina", "Katie", "Jeff"];

mongod> for(var i=0; i<100; i++){

name = people[Math.floor(Math.random()*people.length)];

user_id = i;

boolean = [true, false][Math.floor(Math.random()*2)];

added_at = new Date();

number = Math.floor(Math.random()*10001);

db.test_collection.save({"name":name, "user_id":user_id, "boolean": boolean, "added_at":added_at, "number":number });

}

index 설정 및 sharding을 위한 기준이 되는 shard key를 설정한다.

mongos> use test

mongos> db.test_collection.ensureIndex({number:1})

mongos> use admin

mongos> db.runCommand( { enablesharding : "test" } )

mongos> db.runCommand( { shardcollection : "test.test_collection", key : {"number":1} })

chunk 들이 분할되어가는 모습을 확인한다.

mongos> db.printShardingStatus();

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" : ObjectId("54b72888d660b109dcd9e30f")

}

  shards:

        {  "_id" : "first",  "host" : "first/cloud0:10001,cloud2:10001,cloud3:10001" }

        {  "_id" : "second",  "host" : "second/cloud0:10002,cloud3:10002,cloud4:10002" }

        {  "_id" : "third",  "host" : "third/cloud1:10003,cloud3:10003,cloud4:10003" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "test",  "partitioned" : true,  "primary" : "first" }

                test.test_collection

                        shard key: { "number" : 1 }

                        chunks:

                                second  1

                                first   5

                        { "number" : { "$minKey" : 1 } } -->> { "number" : 0 } on : second Timestamp(2, 0)

                        { "number" : 0 } -->> { "number" : 1494 } on : first Timestamp(2, 2)

                        { "number" : 1494 } -->> { "number" : 2992 } on : first Timestamp(2, 3)

                        { "number" : 2992 } -->> { "number" : 5985 } on : first Timestamp(1, 1)

                        { "number" : 5985 } -->> { "number" : 8988 } on : first Timestamp(1, 2)

                        { "number" : 8988 } -->> { "number" : { "$maxKey" : 1 } } on : first Timestamp(1, 3)

mongos> db.printShardingStatus();

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" : ObjectId("54b72888d660b109dcd9e30f")

}

  shards:

        {  "_id" : "first",  "host" : "first/cloud0:10001,cloud2:10001,cloud3:10001" }

        {  "_id" : "second",  "host" : "second/cloud0:10002,cloud3:10002,cloud4:10002" }

        {  "_id" : "third",  "host" : "third/cloud1:10003,cloud3:10003,cloud4:10003" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "test",  "partitioned" : true,  "primary" : "first" }

                test.test_collection

                        shard key: { "number" : 1 }

                        chunks:

                                second  1

                                third   1

                                first   4

                        { "number" : { "$minKey" : 1 } } -->> { "number" : 0 } on : second Timestamp(2, 0)

                        { "number" : 0 } -->> { "number" : 1494 } on : third Timestamp(3, 0)

                        { "number" : 1494 } -->> { "number" : 2992 } on : first Timestamp(3, 1)

                        { "number" : 2992 } -->> { "number" : 5985 } on : first Timestamp(1, 1)

                        { "number" : 5985 } -->> { "number" : 8988 } on : first Timestamp(1, 2)

                        { "number" : 8988 } -->> { "number" : { "$maxKey" : 1 } } on : first Timestamp(1, 3)

mongos> db.printShardingStatus();

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" : ObjectId("54b72888d660b109dcd9e30f")

}

  shards:

        {  "_id" : "first",  "host" : "first/cloud0:10001,cloud2:10001,cloud3:10001" }

        {  "_id" : "second",  "host" : "second/cloud0:10002,cloud3:10002,cloud4:10002" }

        {  "_id" : "third",  "host" : "third/cloud1:10003,cloud3:10003,cloud4:10003" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "test",  "partitioned" : true,  "primary" : "first" }

                test.test_collection

                        shard key: { "number" : 1 }

                        chunks:

                                second  2

                                third   1

                                first   4

                        { "number" : { "$minKey" : 1 } } -->> { "number" : 0 } on : second Timestamp(2, 0)

                        { "number" : 0 } -->> { "number" : 1494 } on : third Timestamp(3, 0)

                        { "number" : 1494 } -->> { "number" : 2992 } on : second Timestamp(4, 0)

                        { "number" : 2992 } -->> { "number" : 4487 } on : first Timestamp(4, 2)

                        { "number" : 4487 } -->> { "number" : 5985 } on : first Timestamp(4, 3)

                        { "number" : 5985 } -->> { "number" : 8988 } on : first Timestamp(1, 2)

                        { "number" : 8988 } -->> { "number" : { "$maxKey" : 1 } } on : first Timestamp(1, 3)

mongos> db.printShardingStatus();

--- Sharding Status ---

  sharding version: {

        "_id" : 1,

        "version" : 4,

        "minCompatibleVersion" : 4,

        "currentVersion" : 5,

        "clusterId" : ObjectId("54b72888d660b109dcd9e30f")

}

  shards:

        {  "_id" : "first",  "host" : "first/cloud0:10001,cloud2:10001,cloud3:10001" }

        {  "_id" : "second",  "host" : "second/cloud0:10002,cloud3:10002,cloud4:10002" }

        {  "_id" : "third",  "host" : "third/cloud1:10003,cloud3:10003,cloud4:10003" }

  databases:

        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

        {  "_id" : "test",  "partitioned" : true,  "primary" : "first" }

                test.test_collection

                        shard key: { "number" : 1 }

                        chunks:

                                second  2

                                third   2

                                first   3

1억건을 Insert한 뒤 chunk가  아래와 같이 균등히 분산되어 있다는 것을 확인 할 수 있었다.

second  89

third   86

first   86