happyfish100 / fastcfs

A high performance distributed file system which can be used as the back-end storage of databases, K8s and VM etc.

FastCFS -- a high performance general distributed file system for databases, K8s and KVM etc.

English | Chinese

1. About

FastCFS is a general distributed file system with strong consistency, high performance, high availability and supporting 10 billion massive files. FastCFS can be used as the back-end storage of databases (MySQL, PostgreSQL, Oracle etc.), K8s, KVM, FTP, SMB and NFS.

Main Features

• High performance on the premise of strong data consistency: performance is more better than ceph
• Fully compatible with POSIX, supporting file lock and 10 billion massive files
• High availability: no single point & automatic failover
• Easy to use:
  • simple and efficient architecture and native implementation
  • independent of third-party components
  • built-in operation and maintenance tools
• Strong random write performance: FCFS allocates space based on the trunk file, converts random writes from the client to sequential writes

Classical Application Scene

• Database: supports two storage methods (conventional exclusive data and high-level shared data) and database cloudification
• File Storage: documents, pictures, videos, etc., FastCFS is easier to integrate with general software than the object storage
• Unified Storage: database and file storage share a storage cluster, which significantly improves the utilization of storage resources
• High Performance Computing: FastCFS with high reliability and high performance is naturally suitable for the HPC scene
• Video monitoring: smooth writing for multi-channel videos with HDD such as SATA because FastCFS uses sequential writing approach

2. Current Version: V5.3

3. Supported Platforms

• Linux: Kernel version >= 3.10 (Full support, >= 4.18 is recommended)
• MacOS or FreeBSD (Only server side)

4. Dependencies

• libfuse (version 3.9.4 or newer, 3.10.5 is recommended)
  • Python (version 3.5 or newer)
  • Ninja (version 1.7 or newer)
  • gcc (version 4.7.0 or newer)
• libfastcommon (tag: V1.0.73)
• libserverframe (tag: V1.2.3)
• libdiskallocator (tag: V1.1.7)
• fastDIR (tag: V5.3.0)
• faststore (tag: V5.3.0)
• FastCFS (tag: V5.3.0)

5. Installation

5.1 DIY installation

you can use Cluster Operation Tool to deploy FastCFS

step by step please see INSTALL

recommend to execute libfuse_setup.sh for compiling and installing libfuse

5.2 easy installation

libfastcommon, libserverframe, fastDIR, faststore and FastCFS can be compiled, installed and auto configurated by fastcfs.sh

fastcfs.sh can automatically pull or update above six projects codes from GitHub, compile and install according to dependency orders, and automatically generate cluster related configuration files according to the config templates.

git clone https://github.com/happyfish100/FastCFS.git; cd FastCFS/

fastcfs.sh usage:

* install: pull/update codes from gitee, then make and install
* config: copy config files and configure them with local ip
* start | stop | restart: for service processes control

one click to build (deploy and run) single node demo environment (MUST run by root):

./helloWorld.sh

or execute following commands (MUST run by root):

./fastcfs.sh install
./fastcfs.sh config --force
./fastcfs.sh restart

now you can see the mounted path of FastCFS by the command:

df -h /opt/fastcfs/fuse | grep fuse

6. Benchmark

FastCFS has huge better performance than Ceph: the IOPS ratio of sequential write is 6.x, sequential read is 2.x, random write is about 2.0.

• FastCFS vs. Ceph benchmark

7. K8s CSI Driver

fastcfs-csi

8. Chinese Relative articles

CSDN blog

9. TODO List

• [fstore] data recovery after single disk fault (done)
• [fstore] after the machine recovery, the data masters should be rebalanced (done)
• [fauth & fdir & fstore] leader election uses more than half principle to prevent brain-split (done)
• [fauth & fdir & fstore] import public vote node under 2 copies scene to prevent brain-split (done)
• [fdir & fstore] binlog deduplication and historical data deletion (done)
• [fdir & fstore] data submission by majority confirmation to ensure data consistency (done)
• [fdir & fstore] for two replicas, the data submission majority mechanism is intelligent (done)
• [fstore] file read & write performance optimization (done)
• [fdir & api] POSIX compatibility test and improvement (done)
• [fstore] slice storage engine plugin to support massive data with limited memory (done)
• [all] adapt to RDMA network to break through network bottlenecks (done)
• [api] provide Java native API (doing)
• [all] cluster online expansion

10. Business Support

We provide technical support service and customized development. Welcome to use WeChat or email for discuss.

11. Contact us

email: 384681(at)qq(dot)com

WeChat subscription: search "fastdfs" for the related articles (Chinese Only)

fastcfs's Issues

客户端配置需要包含元数据和存储节点信息吗，对于扩容和缩容的灵活性有影响吗？

先赞一下大神的项目。关于客户端配置文件中，看配置例子中要说要包含fastdir和faststore的服务器信息，这个在线上做扩容，缩容，存储节点维护(慢盘等)是都需要修改客户端的配置吗？这个不能改成自动方式吗？客户端的配置更新会中断当前已经挂载的文件访问吗？谢谢

集群部署工具不如用python写

现在集群部署工具的shell脚本限制在centos,还要ssh免密访问，不如用python写。
在主机(虽然fastcfs没有主从一说)用Python脚本起一个http服务，在从机上用python脚本发个加入集群的请求，主机把请求的从机加入集群配置，并返回全部集群内服务器ip，从机同步更新本地配置。当有第二台、第三台从机加入，前面加入的从机再请求一次获取最新集群即可。

关于ls和ll会中断写入的问题

三副本，同时向一个节点或不同节点上传文件，在上传的过程中，在任意节点上执行ls或ll命令都会中断在该节点的写入，导致文件大小就停留在执行ll看到的那个大小，造成这个现象的原因是？
这个是在上传过程中执行了ll后的文件大小

这个是不执行ll，等文件都上传完后再执行ll的文件大小

FastCFS支持sparse file 和reflink 吗？

请问FastCFS支持sparse file 和reflink 吗？

db/../server_types.h:34:10: fatal error: diskallocator/binlog/common/binlog_types.h: No such file or directory

执行
git clone https://gitee.com/fastdfs100/FastCFS.git;
cd FastCFS/
./helloWorld.sh

报如下错误(Rocky Linux 8.5)：

In file included from db/dentry_lru.h:20,
from server_func.c:33:
db/../server_types.h:34:10: fatal error: diskallocator/binlog/common/binlog_types.h: No such file or directory
#include "diskallocator/binlog/common/binlog_types.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:36: server_func.o] Error 1

In file included from storage/slice_op.c:26:
storage/../dio/trunk_read_thread.h:22:10: fatal error: libaio.h: No such file or directory
#include <libaio.h>
^~~~~~~~~~
compilation terminated.
make: *** [Makefile:56: storage/slice_op.o] Error 1
/usr/bin/ld: cannot find -laio

INFO: Execute command (./make.sh --exclude=auth_client) in path (/u01/FastCFS)
In file included from async_reporter.h:21,
from fcfs_api.c:34:
fcfs_api_types.h:25:10: fatal error: fastsore/api/fs_api.h: No such file or directory
#include "fastsore/api/fs_api.h"
^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:42: fcfs_api.o] Error 1
In file included from ../../include/fastcfs/api/fcfs_api.h:21,
from test_file_op.c:25:
../../include/fastcfs/api/fcfs_api_types.h:25:10: fatal error: fastsore/api/fs_api.h: No such file or directory
#include "fastsore/api/fs_api.h"
^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:17: test_file_op] Error 1
In file included from ../include/fastcfs/api/std/../fcfs_api.h:21,
from ../include/fastcfs/api/std/api_types.h:22,
from ../include/fastcfs/api/std/posix_api.h:23,
from types.h:19,
from global.h:20,
from global.c:17:
../include/fastcfs/api/std/../fcfs_api_types.h:25:10: fatal error: fastsore/api/fs_api.h: No such file or directory
#include "fastsore/api/fs_api.h"
^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:22: global.lo] Error 1
In file included from ../include/fastcfs/api/fcfs_api.h:21,
from global.h:20,
from fuse_wrapper.c:24:
../include/fastcfs/api/fcfs_api_types.h:25:10: fatal error: fastsore/api/fs_api.h: No such file or directory
#include "fastsore/api/fs_api.h"
^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:20: fuse_wrapper.o] Error 1

waiting services ready ...
some mistake happen!
tail -n 20 /opt/fastcfs/fcfs/logs/fcfs_fused.log
tail: cannot open '/opt/fastcfs/fcfs/logs/fcfs_fused.log' for reading: No such file or directory

可否关闭compress_binlog？

我是用的版本是v6.07, 目前我用的配置中：compress_binlog=true。
最近发现机器在compress_binlog的时候固定会单核CPU跑高，并且无法继续服务如图：

时间点是0300到0400，看监控：


关于binlog的配置如下：

想知道这个配置是否可以关掉，或者有什么方法可以缓解/改善/解决这种问题？

非常期待作者大佬可以回复这个问题！感恩！

能不能用于生产环境

用于k8s, vitess, mysql

在3副本的情况下，我将其中一个节点关闭模拟节点故障，集群无法正常运行

如题，三副本都通过EXSI部署在同一台服务器里，在关闭节点之前集群里的auth和vote是正常运行的，一台机器上同时部署所有服务。我把其中一个节点关掉之后，服务好像就崩溃了，无法读写，没有选出新的master节点。

能不能给出性能数据: fastCFS, ceph, local-storage

要是能给出性能比较的数据，就很容易了解

完全新手，不知道我测试的有问题么？用 dd 测试吞吐量只有 37MB/s ， nfs 还有 85MB/s

执行命令：dd if=/dev/zero of=test.file1 bs=1G count=2

fastcfs 5.2:
2 个 fstore
2 个 fdir
2 个 auth
3 个 vote

nfs 单机暴露

配置都是 4核8G, 机械硬盘 原磁盘直接运行 dd 吞吐量在 500MB/s
fastcfs 开启了共享数据

Fastcfs和fastdft有什么区别？

Fastcfs和fastdft有什么区别？我看这两个的介绍都是分布式文件系统，这两个有什么区别吗？应用场景有什么不同吗？
如果我想了解这两个的架构，可以在哪里找到资料呢？

咨询一个问题， fastCFS 是否适用于 循环存储？

有了解的吗，介绍一下，是否适合循环存储？ 比如存储监控视频资料。
我看到了二进制日志，我尝试存储内容，然后删除，可能因为测试的内容较少，并没有看到回收机制，只看到了存储的文件在逐渐变大。
所以，想咨询一下上述问题，是否适合循环存储？

How to control replication on server?

English:
Good day.
There are a couple of questions.

1. Could you please explain how replications on servers can be controlled? For example, there are 5 storage servers and I need to always have 2 replications on these servers, no more. What is the best way to achieve this? Can I manually specify replication servers? For example, one contains a replica in one data center and the second one is directed to another.
2. Is there any utility/interface or API where I can view the storage state? For example, to find out if one of the disks is dropped or not in replication state?
   中文：
   下午好。
   有几个问题。
3. 请解释一下如何控制服务器上的复制？例如，有 5 台存储服务器，我需要在这些服务器上始终有 2 个复制，不能再多了。实现这一目标的最佳方法是什么？能否手动指定复制服务器？例如，在一个数据中心包含一个复制，然后将第二个复制指向另一个数据中心。
4. 是否有任何实用程序/接口或应用程序接口可以查看存储的状态？例如，了解其中一个磁盘是否掉线或不在复制状态？

/opt/fastcfs/fstore/下的serverd.pid会丢失

如题，三副本，突然01和02的faststore下线，查询日志发现是因为找不到/opt/fastcfs/fstore/serverd.pid这个文件，如果手动创建的话，再重启还是会失败，并且文件会被删除，请问这个应该如何解决？

使用fuse_splice_write fuse_splice_read优化读写性能，避免内核额外的data copy

fuse在内核逻辑下的通信特别慢，特别是在大带宽的模型下，推荐用fuse_splice_write fuse_splice_read避免额外的数据复制

使用集群工具部署的文档还能再优化优化不

使用集群工具部署的文档还能再优化优化不，有点儿看的不是太明白。。。

能不能给个例子

能不能给一个最少并行节点数量的实例，以及前端数据作为并行文件系统的文件系统的客户端配置。比如要作为ORACLE、SYSBASE、MYSQL、GP数据库的配置过程。我觉得这样可以更好的普及FastCFS。希望FastDFS和FastCFS能够早日成为各类Gnu/Linux发行版的默认支持组件功能！！！

fastcfs扩容

对容量扩容支持怎么样呢?比如3个store机器(3 copies),再加一台机器.

使用systemctl 启动fastcfs 异常退出

systemctl restart fastcfs 正常运行，但程序启动后，命令行不会退出。
df -h可以看到挂载的文件系统。大约30秒后，程序异常退出，报错如下：
[2021-01-04 15:34:44] INFO - file: sf_nio.c, line: 205, connect to server :11012 successfully
[2021-01-04 15:35:43] CRIT - file: sf_service.c, line: 523, catch signal 15, program exiting...
[2021-01-04 15:35:44] INFO - file: combine_handler.c, line: 132, combine_handler_terminate, running: 4, waiting_slice_count: 0
[2021-01-04 15:35:44] INFO - file: async_reporter.c, line: 292, async_reporter_terminate, remain count: 0
[2021-01-04 15:35:44] INFO - file: fcfs_fused.c, line: 152, program exit normally.

采用fcfs_fused /etc/fastcfs/fcfs/fuse.conf restart 可以正常启动。

另外，安装fused时，/etc/fastcfs/fcfs/下只有一个配置文件，启动会报错，需要增加cluster.conf 和servers.conf两个配置文件。
请大佬帮忙解惑，感谢！

agpl的开源协议，可以免费商用吗

Java API 什么时候会出啊

参阅 apt 安装文档安装配置完成后fastcfs服务无法正常启动,fastcfs.service: Can't open PID file /opt/fastcfs/fcfs/fused.pid (yet?) after start: Operation not permitted

可能是虚拟机下没有 加载fuse的原因
查看 /opt/fastcfs/fcfs/logs/fcfs_fused.log 日志,如下
[2022-05-06 06:42:37] INFO - base_path=/opt/fastcfs/fcfs, max_connections=256, connect_timeout=2, network_timeout=10, thread_stack_size=512 KB, max_pkg_size=256 KB, min_buff_size=256 KB, max_buff_size=256 KB, tcp_quick_ack=1, log_level=INFO, run_by_group=, run_by_user=, error-log: {sync_log_buff_interval=1, rotate_everyday=0, rotate_time=00:00, rotate_on_size=0, compress_old=0, compress_days_before=1, keep_days=0, delete_old_time=01:30}, port=0, bind_addr=, accept_threads=1, work_threads=1
[2022-05-06 06:42:37] INFO - FastDIR v3.3.0, connect_timeout=2, network_timeout=10, read_rule: master only, retry_interval_mode=multiple, retry_max_interval_ms=3000 ms, connect_retry_times=200, connect_retry_interval_ms=100 ms, network_retry_times=200, network_retry_interval_ms=100 ms, dir_server_count=1, use_sys_lock_for_append: 0, async_report { enabled: 1, async_report_interval_ms: 100, shared_allocator_count: 11, hashtable_sharding_count: 17, hashtable_total_capacity: 1403641 } , auth {enabled: 0}
[2022-05-06 06:42:37] INFO - FastStore v3.3.0, connect_timeout: 2, network_timeout: 10, read_rule: master only, retry_interval_mode=multiple, retry_max_interval_ms=3000 ms, connect_retry_times=200, connect_retry_interval_ms=100 ms, network_retry_times=200, network_retry_interval_ms=100 ms, server group count: 1, data group count: 64, write_combine { enabled: 1, buffer_size: 256 KB, min_wait_time_ms: 20 ms, max_wait_time_ms: 1000 ms, skip_combine_on_slice_size: 256 KB, skip_combine_on_last_merged_slices: 1, timer_shared_lock_count: 163, shared_allocator_count: 17, hashtable_sharding_count: 163, hashtable_total_capacity: 1403641, thread_pool_max_threads: 16, thread_pool_min_idle_count: 2, thread_pool_max_idle_time: 300 s }, read_ahead { enabled: 1, cache_ttl_ms: 1000 ms, min_buffer_size: 4 KB, max_buffer_size: 128 KB, skip_preread_on_slice_size: 64 KB, shared_lock_count: 1361 }, auth {enabled: 0}
[2022-05-06 06:42:37] INFO - FastCFS V3.3.0, FUSE library version 3.10.4, FastDIR namespace: fs, FastDIR idempotency_enabled=1, FastStore idempotency_enabled=1, channel_htable_capacity=1361, channel_heartbeat_interval=3s, channel_max_idle_time=300s, FUSE mountpoint: /opt/fastcfs/fuse, owner_type: caller, singlethread: 0, clone_fd: 1, max_idle_threads: 10, allow_others: all, auto_unmount: 0, attribute_timeout: 5.0s, entry_timeout: 5.0s, xattr_enabled: 0
[2022-05-06 06:42:38] INFO - file: sf_nio.c, line: 221, connect to server 192.168.110.15:11012 successfully
fuse: device not found, try 'modprobe fuse' first
[2022-05-06 06:42:38] CRIT - file: sf_service.c, line: 562, catch signal 2, program exiting...
[2022-05-06 06:42:38] CRIT - file: fcfs_fused.c, line: 256, program exit abnormally with errno: -1!

FastCFS fuse 用于 kvm 的块存储性能怎么样

我维护了一套虚拟化系统. ceph 作为后端的块存储系统. 很长时间以来一直苦于IO性能的问题. 如果 FastCFS 可以通过 Fuse 挂载到本地. 那么我用这个挂载点存放块文件. 性能会如何? 数据强一致性可以保证吗. 如果FastCFS 可以作为数据库的存储来用. 那么也许作为kvm的后端性能上也是足够的. 好像 qemu-kvm + GlusterFS 也是这么玩的.