GlusterFS Installation

GlusterFS 는 petabytes급 (실제로는 72 brontobytes) 으로 스케일링이 가능한 오픈소스 분산파일 시스템입니다.

HDFS와는 다르게 SPOF 및 performance bottleneck 의 원인이 될 수 있는 Master node(namenode)가 없다는것이 특징입니다.

cassandra 의 consistent hashing algorithm 에 따른 분산처리처럼
GlusterFS 는 elastic hashing algorithm 에 따라 분산 저장 및 재균형(rebalance)이 이루어지기 때문입니다.


GlusterFS 의 장단점은 아래와 같습니다.


Disadvantages

• If you lose a single server, you lose access to all the files that are hosted on that server. This is why distribute is typically graphed to the replicate translator.
• If your file is larger than the subvolume, writing your file will fail.
• If the file is not where the hash code calculates to, an extra lookup operation must be performed, adding slightly to latency.

Advantages

• The more servers you add, the better this scales in terms of random file access. As long as clients aren't all retrieving the same file, their access should be spread pretty evenly across all the servers.
• Increasing volume can be done by adding a new server. Adding servers can be done on-the-fly (Since 3.1.0).

GlusterFS 에서 알아야 할 용어는 아래와 같습니다.

• brick - The brick is the storage filesystem that has been assigned to a volume.
• client - The machine which mounts the volume (this may also be a server).
• server - The machine (virtual or bare metal) which hosts the actual filesystem in which data will be stored.
• subvolume - a brick after being processed by at least one translator.
• volume - The final share after it passes through all the translators.

이하의 설치 과정은 CentOS 6.3 을 기준으로 진행합니다.
gluster01(192.168.10.211), gluster02(192.168.10.212), gluster03(192.168.10.213), gluster(192.168.10.214)  총 네개의 노드에 Replicated + Distributed Volume ( replica 2 transport)으로 구성하는 예입니다.

1. 사전 준비 사항
1) 64비트 OS
2) 네트워크 가능 환경
3) 최소 두개의 Storage Device ( OS 설치용과, GlusterFS운영용으로 분리하여 실제 라이브 환경과 동일하도록 가상디스크 구성 )

2. bricks 의 마운트와 포멧

[root@gluster01 dev]# mkfs.xfs -i size=512 /dev/sdb
meta-data=/dev/sdb               isize=512    agcount=4, agsize=65536 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=262144, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
[root@gluster01 dev]# 

3. etc/fstab 에 아래와 같이 /dev/sdb /export/brick1 xfs defaults 1 2 추가

[root@gluster01 export]# mkdir -p /export/brick1
[root@gluster01 export]# vi /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Oct 12 17:08:48 2012
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/VolGroup-lv_root /                       ext4    defaults        1 1
UUID=92897b22-fed7-4a4a-bfed-500f128ba963 /boot                   ext4    defaults        1 2
/dev/mapper/VolGroup-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/dev/sdb                /export/brick1          xfs     defaults        1 2

4. 마운트

[root@gluster01 export]# mount -a && mount

#source download 
[root@gluster01 src]# wget http://download.gluster.org/pub/gluster/glusterfs/3.3/3.3.1/glusterfs-3.3.1.tar.gz
[root@gluster01 src]# tar -xvf glusterfs-3.3.1.tar.gz  

[root@gluster01 glusterfs-3.3.1]# yum install flex # 필요시 의존 라이브러리
[root@gluster01 glusterfs-3.3.1]# yum install bison-devel # 필요시 의존 라이브러리
[root@gluster01 glusterfs-3.3.1]# yum install openssl-devel #필요시 의존 라이브러리
[root@gluster01 glusterfs-3.3.1]# ./configure --prefix=/srv/glusterfs
[root@gluster01 glusterfs-3.3.1]# make
[root@gluster01 glusterfs-3.3.1]# make install

# service 시작
[root@gluster01 glusterfs]# service glusterd start

5. Trusted Pool 설정

# 더 많은 정보는 gluster peer help 
[root@gluster01 /]# /srv/glusterfs/sbin/gluster peer probe 다른 노드의 아이피 혹은 호스트명을 입력

주) Probe returned with unknown errno 107 오류시 아래와 같이 방화벽 포트 추가 (gluster log 확인)

[root@gluster01 /]# iptables -I INPUT -p tcp --dport 24007 -j ACCEPT
[root@gluster01 /]# service iptables save
[root@gluster01 /]# service iptables restart

아래와 같이 각 노드에서 상태 확인

[root@gluster02 /]# /srv/glusterfs/sbin/gluster peer status
Number of Peers: 3

Hostname: 192.168.10.211
Uuid: 47005db6-f72d-492d-8bbd-5b759b89a51a
State: Peer in Cluster (Connected)

Hostname: gluster03
Uuid: 26c9eea8-0296-41f5-a1ad-81f81e1739e2
State: Peer in Cluster (Connected)

Hostname: gluster04
Uuid: e41d7b67-73a4-4a07-8372-2654ace82da8
State: Peer in Cluster (Connected)

[root@gluster02 /]# 

6. Gluster Volume 설정

[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume info #볼륨 정보 확인
No volumes present
[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume create help #볼륨 생성 도움말
# 아래 볼륨 생성 : Replicated + Distributed Volume ( replica 2 transport)
[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume create uvu_glusterfs_volume replica 2 transport gluster01:/export/brick1 gluster02:/export/brick1 gluster03:/export/brick1 gluster04:/export/brick1
Creation of volume myvolume0 has been successful. Please start the volume to access data.
[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume info #볼륨 정보 확인
 
Volume Name: luvu_glusterfs_volume
Type: Distributed-Replicate
Volume ID: 27e8b2eb-1fb0-4e79-af74-ba20cab74854
Status: Created
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gluster01:/export/brick1
Brick2: gluster02:/export/brick1
Brick3: gluster03:/export/brick1
Brick4: gluster04:/export/brick1

[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume start luvu_glusterfs_volume #볼륨 시작
Starting volume luvu_glusterfs_volume has been successful

[root@gluster01 /]# /srv/glusterfs/sbin/gluster volume info #볼륨 정보 확인
 
Volume Name: luvu_glusterfs_volume
Type: Distributed-Replicate
Volume ID: 27e8b2eb-1fb0-4e79-af74-ba20cab74854
Status: Started
Number of Bricks: 2 x 2 = 4
Transport-type: tcp
Bricks:
Brick1: gluster01:/export/brick1
Brick2: gluster02:/export/brick1
Brick3: gluster03:/export/brick1
Brick4: gluster04:/export/brick1

[root@vm00 /]# 

7. 클라이언트에서 테스트
(참고 : http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Performing_a_Source_Installation)

#주 마운트가 안될 경우 gluster 서버 방화벽 포트 확인 및 추가 ( 24009 )
[root@vm luvu]# mount -t glusterfs gluster01:/luvu_glusterfs_volume /mnt
[root@vm /]# for i in `seq -w 1 100`; do cp -rp /var/log/messages* /mnt/copy-test-$i; done
# 마운트 포인트 확인
[root@vm02 /]# ls -lA /mnt | wc -l
101
[root@vm02 /]# 

각 gluster 서버에서 다음과 같이 확인하면 데이터가 복제된것을 확인할 수 있다.

ls -lA /export/brick1

주) replica 값이 2 이므로 각 서버마다 동일한 파일이 복제된것을 확인할 수 있다. 만약 replication 을 사용하지 않을 경우 각 서버마다 약 50개씩의 파일로 나뉘어 저장되는것을 확인할 수 있다.


  참고
- http://www.gluster.org
- http://blog.syszone.co.kr/3038