happyfish100 / fastcfs Goto Github PK
View Code? Open in Web Editor NEWA high performance distributed file system which can be used as the back-end storage of databases, K8s and VM etc.
License: GNU Affero General Public License v3.0
A high performance distributed file system which can be used as the back-end storage of databases, K8s and VM etc.
License: GNU Affero General Public License v3.0
执行
git clone https://gitee.com/fastdfs100/FastCFS.git;
cd FastCFS/
./helloWorld.sh
waiting services ready ...
some mistake happen!
tail -n 20 /opt/fastcfs/fcfs/logs/fcfs_fused.log
tail: cannot open '/opt/fastcfs/fcfs/logs/fcfs_fused.log' for reading: No such file or directory
执行命令:dd if=/dev/zero of=test.file1 bs=1G count=2
fastcfs 5.2:
2 个 fstore
2 个 fdir
2 个 auth
3 个 vote
nfs 单机暴露
配置都是 4核8G, 机械硬盘 原磁盘直接运行 dd 吞吐量在 500MB/s
fastcfs 开启了共享数据
您好,我第一次接触分布式文件系统。
我在计划一个数据条目管理项目,其中每个条目的数据结构灵活性很大,使我考虑使用文件系统存放json或xml之类的代替数据库。
如果我像数据库那样频繁的对FastCFS挂载点,进行 列表遍历 文件读取 文件写入,那所涉及的文件会被缓存在内存中吗?
以及文件是否可以携带元数据,已通过元数据快速搜索文件列表,就像数据库。
Fastcfs和fastdft有什么区别?我看这两个的介绍都是分布式文件系统,这两个有什么区别吗?应用场景有什么不同吗?
如果我想了解这两个的架构,可以在哪里找到资料呢?
使用集群工具部署的文档还能再优化优化不,有点儿看的不是太明白。。。
我维护了一套虚拟化系统. ceph 作为后端的块存储系统. 很长时间以来一直苦于IO性能的问题. 如果 FastCFS 可以通过 Fuse 挂载到本地. 那么我用这个挂载点存放块文件. 性能会如何? 数据强一致性可以保证吗. 如果FastCFS 可以作为数据库的存储来用. 那么也许作为kvm的后端性能上也是足够的. 好像 qemu-kvm + GlusterFS 也是这么玩的.
English:
Good day.
There are a couple of questions.
请问FastCFS支持sparse file 和reflink 吗?
如题,三副本都通过EXSI部署在同一台服务器里,在关闭节点之前集群里的auth和vote是正常运行的,一台机器上同时部署所有服务。我把其中一个节点关掉之后,服务好像就崩溃了,无法读写,没有选出新的master节点。
要是能给出性能比较的数据,就很容易了解
您好,我在测试FastCFS的时候遇到一个问题,麻烦您帮忙看看。集群包含两台Rockey Linux服务器。之前启动过一次,后来修改了配置,启动fstore的时候报错。使用的是工具fcfs.sh start fstore。已经按照提示在 /etc/fastcfs/fstore/cluster.conf配置文件中新增了use_hash_func=true。但是仍然报这个问题。
CRIT - fs_serverd.c:438 program exit abnormally with errno: 22
[2024-07-17 17:45:34] WARNING - file: ../common/fs_cluster_cfg.c, line: 306, config file: /etc/fastcfs/fstore/cluster.conf, 1 server ids: 2 NOT used (NOT contain any data group)!
[2024-07-17 17:45:34] ERROR - file: server_func.c, line: 374, use_hash_func in cluster.conf changed, old: true, new: false, you must restore use_hash_func to true
用于k8s, vitess, mysql
可能是虚拟机下没有 加载fuse的原因
查看 /opt/fastcfs/fcfs/logs/fcfs_fused.log 日志,如下
[2022-05-06 06:42:37] INFO - base_path=/opt/fastcfs/fcfs, max_connections=256, connect_timeout=2, network_timeout=10, thread_stack_size=512 KB, max_pkg_size=256 KB, min_buff_size=256 KB, max_buff_size=256 KB, tcp_quick_ack=1, log_level=INFO, run_by_group=, run_by_user=, error-log: {sync_log_buff_interval=1, rotate_everyday=0, rotate_time=00:00, rotate_on_size=0, compress_old=0, compress_days_before=1, keep_days=0, delete_old_time=01:30}, port=0, bind_addr=, accept_threads=1, work_threads=1
[2022-05-06 06:42:37] INFO - FastDIR v3.3.0, connect_timeout=2, network_timeout=10, read_rule: master only, retry_interval_mode=multiple, retry_max_interval_ms=3000 ms, connect_retry_times=200, connect_retry_interval_ms=100 ms, network_retry_times=200, network_retry_interval_ms=100 ms, dir_server_count=1, use_sys_lock_for_append: 0, async_report { enabled: 1, async_report_interval_ms: 100, shared_allocator_count: 11, hashtable_sharding_count: 17, hashtable_total_capacity: 1403641 } , auth {enabled: 0}
[2022-05-06 06:42:37] INFO - FastStore v3.3.0, connect_timeout: 2, network_timeout: 10, read_rule: master only, retry_interval_mode=multiple, retry_max_interval_ms=3000 ms, connect_retry_times=200, connect_retry_interval_ms=100 ms, network_retry_times=200, network_retry_interval_ms=100 ms, server group count: 1, data group count: 64, write_combine { enabled: 1, buffer_size: 256 KB, min_wait_time_ms: 20 ms, max_wait_time_ms: 1000 ms, skip_combine_on_slice_size: 256 KB, skip_combine_on_last_merged_slices: 1, timer_shared_lock_count: 163, shared_allocator_count: 17, hashtable_sharding_count: 163, hashtable_total_capacity: 1403641, thread_pool_max_threads: 16, thread_pool_min_idle_count: 2, thread_pool_max_idle_time: 300 s }, read_ahead { enabled: 1, cache_ttl_ms: 1000 ms, min_buffer_size: 4 KB, max_buffer_size: 128 KB, skip_preread_on_slice_size: 64 KB, shared_lock_count: 1361 }, auth {enabled: 0}
[2022-05-06 06:42:37] INFO - FastCFS V3.3.0, FUSE library version 3.10.4, FastDIR namespace: fs, FastDIR idempotency_enabled=1, FastStore idempotency_enabled=1, channel_htable_capacity=1361, channel_heartbeat_interval=3s, channel_max_idle_time=300s, FUSE mountpoint: /opt/fastcfs/fuse, owner_type: caller, singlethread: 0, clone_fd: 1, max_idle_threads: 10, allow_others: all, auto_unmount: 0, attribute_timeout: 5.0s, entry_timeout: 5.0s, xattr_enabled: 0
[2022-05-06 06:42:38] INFO - file: sf_nio.c, line: 221, connect to server 192.168.110.15:11012 successfully
fuse: device not found, try 'modprobe fuse' first
[2022-05-06 06:42:38] CRIT - file: sf_service.c, line: 562, catch signal 2, program exiting...
[2022-05-06 06:42:38] CRIT - file: fcfs_fused.c, line: 256, program exit abnormally with errno: -1!
能不能给一个最少并行节点数量的实例,以及前端数据作为并行文件系统的文件系统的客户端配置。比如要作为ORACLE、SYSBASE、MYSQL、GP数据库的配置过程。我觉得这样可以更好的普及FastCFS。希望FastDFS和FastCFS能够早日成为各类Gnu/Linux发行版的默认支持组件功能!!!
有了解的吗,介绍一下,是否适合循环存储? 比如存储监控视频资料。
我看到了二进制日志,我尝试存储内容,然后删除,可能因为测试的内容较少,并没有看到回收机制,只看到了存储的文件在逐渐变大。
所以,想咨询一下上述问题,是否适合循环存储?
先赞一下大神的项目。关于客户端配置文件中,看配置例子中要说要包含fastdir和faststore的服务器信息,这个在线上做扩容,缩容,存储节点维护(慢盘等)是都需要修改客户端的配置吗?这个不能改成自动方式吗?客户端的配置更新会中断当前已经挂载的文件访问吗?谢谢
systemctl restart fastcfs 正常运行,但程序启动后,命令行不会退出。
df -h可以看到挂载的文件系统。大约30秒后,程序异常退出,报错如下:
[2021-01-04 15:34:44] INFO - file: sf_nio.c, line: 205, connect to server :11012 successfully
[2021-01-04 15:35:43] CRIT - file: sf_service.c, line: 523, catch signal 15, program exiting...
[2021-01-04 15:35:44] INFO - file: combine_handler.c, line: 132, combine_handler_terminate, running: 4, waiting_slice_count: 0
[2021-01-04 15:35:44] INFO - file: async_reporter.c, line: 292, async_reporter_terminate, remain count: 0
[2021-01-04 15:35:44] INFO - file: fcfs_fused.c, line: 152, program exit normally.
采用fcfs_fused /etc/fastcfs/fcfs/fuse.conf restart 可以正常启动。
另外,安装fused时,/etc/fastcfs/fcfs/下只有一个配置文件,启动会报错,需要增加cluster.conf 和servers.conf两个配置文件。
请大佬帮忙解惑,感谢!
现在集群部署工具的shell脚本限制在centos,还要ssh免密访问,不如用python写。
在主机(虽然fastcfs没有主从一说)用Python脚本起一个http服务,在从机上用python脚本发个加入集群的请求,主机把请求的从机加入集群配置,并返回全部集群内服务器ip,从机同步更新本地配置。当有第二台、第三台从机加入,前面加入的从机再请求一次获取最新集群即可。
fuse在内核逻辑下的通信特别慢,特别是在大带宽的模型下,推荐用fuse_splice_write fuse_splice_read避免额外的数据复制
对容量扩容支持怎么样呢?比如3个store机器(3 copies),再加一台机器.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.