在Ubuntu系统上部署分布式系统Ceph(ubuntu xenial)

推荐整理分享在Ubuntu系统上部署分布式系统Ceph(ubuntu xenial)，希望有所帮助，仅作参考，欢迎阅读内容。

文章相关热门搜索词:ubuntu xvfb,ubuntu sharing,ubuntu x,ubuntu xenial,ubuntu zed,ubuntu xvfb,ubuntu sharing,ubuntu x,内容如对您有帮助，希望把文章链接给更多的朋友！

Ceph是统一存储系统，支持三种接口。Object：有原生的API，而且也兼容Swift和S3的APIBlock：支持精简配置、快照、克隆File：Posix接口，支持快照Ceph也是分布式存储系统，它的特点是：高扩展性：使用普通x服务器，支持~台服务器，支持TB到PB级的扩展。高可靠性：没有单点故障，多数据副本，自动管理，自动修复。高性能：数据分布均衡，并行化度高。对于objects storage和block storage,不需要元数据服务器。

架构

Ceph的底层是RADOS，它的意思是“A reliable, autonomous, distributed object storage”。 RADOS由两个组件组成：OSD： Object Storage Device，提供存储资源。Monitor：维护整个Ceph集群的全局状态。RADOS具有很强的扩展性和可编程性，Ceph基于RADOS开发了Object Storage、Block Storage、FileSystem。Ceph另外两个组件是：MDS：用于保存CephFS的元数据。RADOS Gateway：对外提供REST接口，兼容S3和Swift的API。

映射

Ceph的命名空间是 (Pool, Object)，每个Object都会映射到一组OSD中(由这组OSD保存这个Object)：(Pool, Object) → (Pool, PG) → OSD set → DiskCeph中Pools的属性有：Object的副本数Placement Groups的数量所使用的CRUSH Ruleset在Ceph中，Object先映射到PG(Placement Group)，再由PG映射到OSD set。每个Pool有多个PG，每个Object通过计算hash值并取模得到它所对应的PG。PG再映射到一组OSD（OSD的个数由Pool 的副本数决定），第一个OSD是Primary，剩下的都是Replicas。数据映射(Data Placement)的方式决定了存储系统的性能和扩展性。(Pool, PG) → OSD set 的映射由四个因素决定：CRUSH算法：一种伪随机算法。OSD MAP：包含当前所有Pool的状态和所有OSD的状态。CRUSH MAP：包含当前磁盘、服务器、机架的层级结构。CRUSH Rules：数据映射的策略。这些策略可以灵活的设置object存放的区域。比如可以指定 pool1中所有objecst放置在机架1上，所有objects的第1个副本放置在机架1上的服务器A上，第2个副本分布在机架1上的服务器B上。 pool2中所有的object分布在机架2、3、4上，所有Object的第1个副本分布在机架2的服务器上，第2个副本分布在机架3的服 器上，第3个副本分布在机架4的服务器上。

Client从Monitors中得到CRUSH MAP、OSD MAP、CRUSH Ruleset，然后使用CRUSH算法计算出Object所在的OSD set。所以Ceph不需要Name服务器，Client直接和OSD进行通信。伪代码如下所示：

复制代码代码如下: locator = object_name obj_hash = hash(locator) pg = obj_hash % num_pg osds_for_pg = crush(pg) # returns a list of osds primary = osds_for_pg[0] replicas = osds_for_pg[1:]

这种数据映射的优点是：把Object分成组，这降低了需要追踪和处理metadata的数量(在全局的层面上，我们不需要追踪和处理每个object的metadata和placement，只需要管理PG的metadata就可以了。PG的数量级远远低于object的数量级)。增加PG的数量可以均衡每个OSD的负载，提高并行度。分隔故障域，提高数据的可靠性。

强一致性

Ceph的读写操作采用Primary-Replica模型，Client只向Object所对应OSD set的Primary发起读写请求，这保证了数据的强一致性。由于每个Object都只有一个Primary OSD，因此对Object的更新都是顺序的，不存在同步问题。当Primary收到Object的写请求时，它负责把数据发送给其他Replicas，只要这个数据被保存在所有的OSD上时，Primary才应答Object的写请求，这保证了副本的一致性。

容错性

在分布式系统中，常见的故障有网络中断、掉电、服务器宕机、硬盘故障等，Ceph能够容忍这些故障，并进行自动修复，保证数据的可靠性和系统可用性。Monitors是Ceph管家，维护着Ceph的全局状态。Monitors的功能和zookeeper类似，它们使用Quorum和Paxos算法去建立全局状态的共识。OSDs可以进行自动修复，而且是并行修复。故障检测：OSD之间有心跳检测，当OSD A检测到OSD B没有回应时，会报告给Monitors说OSD B无法连接，则Monitors给OSD B标记为down状态，并更新OSD Map。当过了M秒之后还是无法连接到OSD B，则Monitors给OSD B标记为out状态(表明OSD B不能工作），并更新OSD Map。备注：可以在Ceph中配置M的值。故障恢复：当某个PG对应的OSD set中有一个OSD被标记为down时(假如是Primary被标记为down，则某个Replica会成为新的Primary，并处理所有读写 object请求)，则该PG处于active+degraded状态，也就是当前PG有效的副本数是N-1。过了M秒之后，假如还是无法连接该OSD，则它被标记为out，Ceph会重新计算PG到OSD set的映射(当有新的OSD加入到集群时，也会重新计算所有PG到OSD set的映射)，以此保证PG的有效副本数是N。新OSD set的Primary先从旧的OSD set中收集PG log，得到一份Authoritative History(完整的、全序的操作序列)，并让其他Replicas同意这份Authoritative History(也就是其他Replicas对PG的所有objects的状态达成一致)，这个过程叫做Peering。当Peering过程完成之后，PG进 入active+recoverying状态，Primary会迁移和同步那些降级的objects到所有的replicas上，保证这些objects 的副本数为N。

下面来看一下部署与配置

系统环境：Ubuntu ..2复制代码代码如下:hostname:s1 osd.0/mon.a/mds.a ip:...hostname:s2 osd.1/mon.b/mds.b ip:...hostname:s3 osd.2/mon.c/mds.c ip:...hostname:s4 client ip:...免密钥：s1/s2/s3 启用root，相互之间配置免密钥。复制代码代码如下:cat id_rsa.pub_s* >> authorized_keys安装：复制代码代码如下:apt-get install ceph ceph-common ceph-fs-common （ceph-mds）更新到新版本：复制代码代码如下:wget -q -O- ‘ sudo apt-key add -echo deb $(lsb_release -sc) main | tee /etc/apt/sources.list.d/ceph.listapt-get updateapt-get install ceph分区及挂载(使用btrfs)：复制代码代码如下:root@s1:/data/osd.0# df -h|grep osd/dev/sdb1 G M G 1% /data/osd.0root@s2:/data/osd.1# df -h|grep osd/dev/sdb1 G M G 1% /data/osd.1root@s3:/data/osd.2# df -h|grep osd/dev/sdb1 G M G 1% /data/osd.2root@s1:~/.ssh# mkdir -p /tmp/ceph/（每个server上执行）配置：

复制代码代码如下:root@s1:/data/osd.0# vim /etc/ceph/ceph.conf[global]auth cluster required = noneauth service required = noneauth client required = none[osd]osd data = /data/$name[mon]mon data = /data/$name[mon.a]host = s1mon addr = ...:[mon.b]host = s2mon addr = ...:[mon.c]host = s3mon addr = ...:[osd.0]host = s1brtfs devs = /dev/sdb1[osd.1]host = s2brtfs devs = /dev/sdb1[osd.2]host = s3brtfs devs = /dev/sdb1[mds.a]host = s1[mds.b]host = s2[mds.c]host = s3

同步配置：复制代码代码如下:root@s1:~/.ssh# scp /etc/ceph/ceph.conf s2:/etc/ceph/ceph.conf % 0.5KB/s :root@s1:~/.ssh# scp /etc/ceph/ceph.conf s3:/etc/ceph/ceph.conf % 0.5KB/s :所有server上执行：复制代码代码如下:rm -rf /data/$name/* /data/mon/*（初始化前保持没有任何数据）root@s1:~/.ssh# mkcephfs -a -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.keyringtemp dir is /tmp/mkcephfs.qLmwP4Nd0Gpreparing monmap in /tmp/mkcephfs.qLmwP4Nd0G/monmap/usr/bin/monmaptool –create –clobber –add a ...: –add b ...: –add c ...: –print /tmp/mkcephfs.qLmwP4Nd0G/monmap/usr/bin/monmaptool: monmap file /tmp/mkcephfs.qLmwP4Nd0G/monmap/usr/bin/monmaptool: generated fsid cfac--f-a6ac-3dcdfepoch 0fsid cfac--f-a6ac-3dcdflast_changed -- ::.created -- ::.: ...:/0 mon.a1: ...:/0 mon.b2: ...:/0 mon.c/usr/bin/monmaptool: writing epoch 0 to /tmp/mkcephfs.qLmwP4Nd0G/monmap (3 monitors)=== osd.0 ===** WARNING: No osd journal is configured: write latency may be high.If you will not be using an osd journal, write latency may berelatively high. It can be reduced somewhat by loweringfilestore_max_sync_interval, but lower values mean lower writethroughput, especially with spinning disks.-- ::. bcc created object store /data/osd.0 for osd.0 fsid cfac--f-a6ac-3dcdfcreating private key for osd.0 keyring /tmp/mkcephfs.qLmwP4Nd0G/keyring.osd.0creating /tmp/mkcephfs.qLmwP4Nd0G/keyring.osd.0=== osd.1 ===pushing conf and monmap to s2:/tmp/mkfs.ceph.** WARNING: No osd journal is configured: write latency may be high.If you will not be using an osd journal, write latency may berelatively high. It can be reduced somewhat by loweringfilestore_max_sync_interval, but lower values mean lower writethroughput, especially with spinning disks.-- ::. b created object store /data/osd.1 for osd.1 fsid cfac--f-a6ac-3dcdfcreating private key for osd.1 keyring /tmp/mkfs.ceph./keyring.osd.1creating /tmp/mkfs.ceph./keyring.osd.1collecting osd.1 key=== osd.2 ===pushing conf and monmap to s3:/tmp/mkfs.ceph.** WARNING: No osd journal is configured: write latency may be high.If you will not be using an osd journal, write latency may berelatively high. It can be reduced somewhat by loweringfilestore_max_sync_interval, but lower values mean lower writethroughput, especially with spinning disks.-- ::. bb created object store /data/osd.2 for osd.2 fsid cfac--f-a6ac-3dcdfcreating private key for osd.2 keyring /tmp/mkfs.ceph./keyring.osd.2creating /tmp/mkfs.ceph./keyring.osd.2collecting osd.2 key=== mds.a ===creating private key for mds.a keyring /tmp/mkcephfs.qLmwP4Nd0G/keyring.mds.acreating /tmp/mkcephfs.qLmwP4Nd0G/keyring.mds.a=== mds.b ===pushing conf and monmap to s2:/tmp/mkfs.ceph.creating private key for mds.b keyring /tmp/mkfs.ceph./keyring.mds.bcreating /tmp/mkfs.ceph./keyring.mds.bcollecting mds.b key=== mds.c ===pushing conf and monmap to s3:/tmp/mkfs.ceph.creating private key for mds.c keyring /tmp/mkfs.ceph./keyring.mds.ccreating /tmp/mkfs.ceph./keyring.mds.ccollecting mds.c keyBuilding generic osdmap from /tmp/mkcephfs.qLmwP4Nd0G/conf/usr/bin/osdmaptool: osdmap file ‘/tmp/mkcephfs.qLmwP4Nd0G/osdmap’-- ::. be adding osd.0 at {host=s1,pool=default,rack=unknownrack}-- ::. be adding osd.1 at {host=s2,pool=default,rack=unknownrack}-- ::. be adding osd.2 at {host=s3,pool=default,rack=unknownrack}/usr/bin/osdmaptool: writing epoch 1 to /tmp/mkcephfs.qLmwP4Nd0G/osdmapGenerating admin key at /tmp/mkcephfs.qLmwP4Nd0G/keyring.admincreating /tmp/mkcephfs.qLmwP4Nd0G/keyring.adminBuilding initial monitor keyringadded entity mds.a auth auth(auid = key=AQB3O2tTwDNwLRAAofpkrOMqtHCPTFXEKAMA== with 0 caps)added entity mds.b auth auth(auid = key=AQB8O2tT8H8nIhAAq1O2lh4IV/cQFUUTOUug== with 0 caps)added entity mds.c auth auth(auid = key=AQB9O2tTWIfsKRAAVYeueMToCtRSvlslV/jQ== with 0 caps)added entity osd.0 auth auth(auid = key=AQBrO2tTOLQpEhAA4MSCnJRYAkoxrFSvC3aQ== with 0 caps)added entity osd.1 auth auth(auid = key=AQB1O2tTME0eChAA7U4xSrv7MJUZ8vxcEkILbw== with 0 caps)added entity osd.2 auth auth(auid = key=AQB7O2tT0FUKERAAQ/EJT5TclI2XSCLAWAZZOw== with 0 caps)=== mon.a ===/usr/bin/ceph-mon: created monfs at /data/mon for mon.a=== mon.b ===pushing everything to s2/usr/bin/ceph-mon: created monfs at /data/mon for mon.b=== mon.c ===pushing everything to s3/usr/bin/ceph-mon: created monfs at /data/mon for mon.cplacing client.admin keyring in /etc/ceph/ceph.keyring上面提示了没有配置journal。复制代码代码如下:root@s1:~# /etc/init.d/ceph -a start=== mon.a ===Starting Ceph mon.a on s1…already running=== mds.a ===Starting Ceph mds.a on s1…already running=== osd.0 ===Starting Ceph osd.0 on s1…** WARNING: Ceph is still under development. Any feedback can be directed **** at ceph-devel@vger.kernel.org or **starting osd.0 at 0.0.0.0:/ osd_data /data/osd.0 (no journal)查看状态：复制代码代码如下:root@s1:~# ceph -s-- ::. pg v: pgs: active+clean; bytes data, MB used, MB / MB avail-- ::. mds e: 1/1/1 up {0=a=up:active}, 2 up:standby-- ::. osd e: 3 osds: 3 up, 3 in-- ::. log -- ::. mds.0 ...:/ 1 : [INF] closing stale session client. ...:0/ after .-- ::. mon e1: 3 mons at {a=...:/0,b=...:/0,c=...:/0}</p><p>root@s1:~# for i in 1 2 3 ;do ceph health;done-- ::. mon <- [health]-- ::. mon.1 -> ‘HEALTH_OK’ (0)-- ::. mon <- [health]-- ::. mon.2 -> ‘HEALTH_OK’ (0)-- ::. mon <- [health]-- ::. mon.0 -> ‘HEALTH_OK’ (0)并同时查看 s1、s2、s3 log可以看到，证明3个节点都正常：复制代码代码如下:-- ::. b4bfeb mon.a@0(leader) e1 handle_command mon_command(health v 0) v-- ::. b4bfeb mon.a@0(leader).osd e e: 3 osds: 3 up, 3 in-- ::. b4bfeb mon.a@0(leader).osd e e: 3 osds: 3 up, 3 in-- ::. b4bfeb mon.a@0(leader) e1 handle_command mon_command(health v 0) v-- ::. b4bfeb mon.a@0(leader) e1 handle_command mon_command(health v 0) v1log 里面会看到如下时间未同步信息：复制代码代码如下:-- ::. bfcb log [WRN] : message from mon.0 was stamped 6.s in the future, clocks not synchronized-- ::. bfcb log [WRN] : message from mon.0 was stamped 6.s in the future, clocks not synchronized-- ::. bfcb log [WRN] : message from mon.0 was stamped 6.s in the future, clocks not synchronized-- ::. bfcb log [WRN] : message from mon.0 was stamped 6.s in the future, clocks not synchronized所以我们在做集群之前最好能在集群内部做好ntp服务器，确保各节点之前时间一致。

3. 接下来在客户机s4上进行验证操作：复制代码代码如下:root@s4:/mnt# mount -t ceph s1::/ /mnt/s1fs/root@s4:/mnt# mount -t ceph s2::/ /mnt/s2fs/root@s4:/mnt# mount -t ceph s3::/ /mnt/s3fs/root@s4:~# df -hFilesystem Size Used Avail Use% Mounted on/dev/sda1 G 1.3G G 2% /udev M 4.0K M 1% /devtmpfs M K M 1% /runnone 5.0M 0 5.0M 0% /run/locknone M 0 M 0% /run/shm...::/ G 3.6G G 6% /mnt/s3fs...::/ G 3.6G G 6% /mnt/s2fs...::/ G 3.6G G 6% /mnt/s1fs</p><p>root@s4:/mnt/s2fs# touch aaroot@s4:/mnt/s2fs# ls -al /mnt/s1fstotal 4drwxr-xr-x 1 root root 0 May 8 : ./drwxr-xr-x 7 root root May 8 : ../-rw-r–r– 1 root root 0 May 8 : aaroot@s4:/mnt/s2fs# ls -al /mnt/s3fstotal 4drwxr-xr-x 1 root root 0 May 8 : ./drwxr-xr-x 7 root root May 8 : ../-rw-r–r– 1 root root 0 May 8 : aa</p><p>root@s4:/mnt/s2fs# rm -f aaroot@s4:/mnt/s2fs# ls -al /mnt/s1fs/total 4drwxr-xr-x 1 root root 0 May 8 ./drwxr-xr-x 7 root root May 8 : ../root@s4:/mnt/s2fs# ls -al /mnt/s3fs/total 4drwxr-xr-x 1 root root 0 May 8 : ./drwxr-xr-x 7 root root May 8 : ../接下来我们验证单点故障：将s1服务停掉，复制代码代码如下:root@s1:~# /etc/init.d/ceph stop=== mon.a ===Stopping Ceph mon.a on s1…kill …done=== mds.a ===Stopping Ceph mds.a on s1…kill …done=== osd.0 ===Stopping Ceph osd.0 on s1…kill …dones2上log 立马显示：省掉了很多，基本的意思就是mon监控中心发现，剔除故障节点，进行自动切换，集群恢复。复制代码代码如下:-- ::. a5af0b — ...:/ >> ...:/ pipe(0xb1e1b1a8 sd= pgs=3 cs=3 l=0).fault with nothing to send, going to standby-- ::. afeb — ...:/ >> ...:0/ pipe(0xa6ed sd= pgs=1 cs=1 l=0).fault with nothing to send, going to standby-- ::. af9b — ...:0/ >> ...:/ pipe(0xb1ed sd= pgs=1 cs=1 l=0).fault with nothing to send, going to standby-- ::. bb mds.0.6 handle_mds_map i am now mds.0.-- ::. bb mds.0.6 handle_mds_map state change up:reconnect –> up:rejoin-- ::. bb mds.0.6 rejoin_joint_start-- ::. bb mds.0.6 rejoin_done==> /var/log/ceph/mon.b.log <==-- ::. bb log [INF] : mds.0 ...:/ up:active==> /var/log/ceph/mds.b.log <==-- ::. bb mds.0.6 handle_mds_map i am now mds.0.-- ::. bb mds.0.6 handle_mds_map state change up:rejoin --> up:active-- ::. bb mds.0.6 recovery_done — successful recovery!-- ::. bb mds.0.6 active_start-- ::. bb mds.0.6 cluster recovered.==> /var/log/ceph/mds.b.log <==-- ::. bb mds.0.6 handle_mds_map i am now mds.0.-- ::. bb mds.0.6 handle_mds_map state change up:rejoin --> up:active-- ::. bb mds.0.6 recovery_done — successful recovery!-- ::. bb mds.0.6 active_start-- ::. bb mds.0.6 cluster recovered.==> /var/log/ceph/mon.b.log <==-- ::. bb mon.b@1(leader) e1 handle_command mon_command(health v 0) v-- ::. bb mon.b@1(leader) e1 handle_command mon_command(health v 0) v-- ::. bb mon.b@1(leader) e1 handle_command mon_command(health v 0) v-- ::. bb log [INF] : osd.0 out (down for .)-- ::. bb mon.b@1(leader).osd e e: 3 osds: 2 up, 2 in-- ::. bb mon.b@1(leader).osd e e: 3 osds: 2 up, 2 inroot@s2:~# ceph -s-- ::. pg v: pgs: active+clean; bytes data, MB used, MB / MB avail-- ::. mds e: 1/1/1 up {0=b=up:active}, 1 up:standby-- ::. osd e: 3 osds: 2 up, 2 in-- ::. log -- ::. mon.1 ...:/0 : [INF] osd.0 out (down for .)-- ::. mon e1: 3 mons at {a=...:/0,b=...:/0,c=...:/0}root@s1:~# ceph health-- ::. mon <- [health]-- ::. mon.2 -> ‘HEALTH_WARN 1/3 in osds are down; 1 mons down, quorum 1,2′ (0)root@s2:~# ceph health-- ::. mon <- [health]-- ::. mon.2 -> ‘HEALTH_WARN 1 mons down, quorum 1,2′ (0)root@s3:~# ceph health-- ::. mon <- [health]-- ::. mon.1 -> ‘HEALTH_WARN 1 mons down, quorum 1,2′ (0)再接下来，关闭s2，只开启s3：s3上log显示大量复制代码代码如下:==> /var/log/ceph/mds.c.log <==-- ::. bb mds.-1.0 ms_handle_connect on ...:/0</p><p>==> /var/log/ceph/osd.2.log <==-- ::. bb osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. a7be9b osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. a7be9b osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. bb osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. a7be9b osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. bb osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)-- ::. a7be9b osd.2 heartbeat_check: no heartbeat from osd.1 since -- ::. (cutoff -- ::.)健康检测不能从s2上的osd.1 获取no heartbeat 。s1、s2、s3上都有mon、mds、osd。但是总个集群中只有一个节点，所以不能提供服务。

CentOS 7.0怎么配置mail定时发送svn日志邮件? 公司搭建了svn服务器，现在的需求是需要每天定时给上司发送svn日志邮件，所以就一步一步来配置。第一步:配置mail发送邮件1、查看mail是否安装2、配置

为Ubuntu系统装上Mac OS的主题 一.先上两张图看看二.具体安装方法1.设置软件源，这些源在国外，速度有点慢复制代码代码如下:sudoadd-apt-repositoryppa:noobslab/themessudoadd-apt-repositoryppa:noobs

Debian系统的VPS上iptables的配置分享 看了下secure日志和access的日志，一大半都是暴力破解和扫描，虽密码极其复杂，不过总被这么消耗服务器资源也不是事，索性还是把ssh端口和ftp改了然