helyim / file-system-paper

Must-read Papers for File System (FS)

File System

• 1.Local File System

  • 1.1 Kernel File System

  • 1.2 User-Space File System

  • 1.3 Crash Consistency

    • 1.3.1 File System Check

  • 1.4 Fragmentation

  • 1.5 Multicore/Manycore Scalability

• 2.Distributed File System

  • 2.1 General Purpose File System

  • 2.2 Big Data

  • 2.3 High Performance Computing

    • 2.3.1 Burst Buffer File System

  • 2.4 Cloud Computing

  • 2.5 Data Distribution

  • 2.6 Metadata Management

  • 2.7 Erasure Coding

• 3.Surveys

• 4.Analysis

• 5.Object Storage

• 6.AI + Storage

Local File System

Kernel File System

Linux File System

[1984 TOCS] FFS: A Fast File System for UNIX. [PDF]

[1986 USENIX Summer] Vnodes: An Architecture for Multiple File System Types in Sun UNIX. [PDF]

[1991 USENIX Winter] Extent−like Performance from a UNIX File System. [PDF]

[1991 SOSP] LFS: The Design and Implementation of a Log-Structured File System. [PDF]

[1995] Design and Implementation of the Second Extended Filesystem. [PDF]

[1996 ATC] Scalability in the XFS File System. [PDF]

[1998 LinuxExpo] ext3: Journaling the Linux ext2fs Filesystem. [PDF]

[2005 ATC] Analysis and Evolution of Journaling File Systems. [PDF]

[2006 SIGOPS Operating Systems Review] NILFS: The Linux implementation of a log-structured file system. [PDF]

[2007] The new ext4 filesystem: Current status and future plans. [PDF]

[2013 FAST] A Study of Linux File System Evolution. [PDF] [Slides]

[2013 TOS] BTRFS: The Linux B-Tree Filesystem. [PDF]

[2015 FAST] F2FS: A New File System for Flash Storage. [PDF] [Slides]

User-Space File System

FUSE: Filesystem in USErspace, https://github.com/libfuse/libfuse

[2004 ATC] Wayback: A User-level Versioning File System for Linux. [PDF]

[2010 SAC] Performance and Extension of User Space File Systems. [PDF]

[2011 EuroSys] Refuse to crash with Re-FUSE. [PDF]

[2015 HotStorage] Terra Incognita: On the Practicality of User-Space File Systems. [PDF] [Slides]

[2017 FAST] To FUSE or Not to FUSE: Performance of User-Space File Systems. [PDF] [Slides]

[2019 TOS] Performance and Resource Utilization of FUSE User-Space File Systems. [PDF]

[2019 ATC] ExtFUSE: Extension Framework for File Systems in User space. [PDF] [Slides]

[2021 ATC] XFUSE: An Infrastructure for Running Filesystem Services in User Space. [PDF] [Slides]

Crash Consistency

[2012 FAST] Consistency Without Ordering. [PDF] [Slides] [Codes]

[2013 SOSP] Optimistic Crash Consistency. [PDF] [Slides] [Codes]

[2017 TOS] Application Crash Consistency and Performance with CCFS. [PDF]

File System Check

[1983] Fsck − The UNIX File System Check Program. [PDF]

[2008 OSDI] SQCK: A Declarative File System Checker. [PDF]

[2013 FAST] ffsck: The Fast File System Checker. [PDF]

[2018 TOS] Towards Robust File System Checkers. [PDF]

[2021 FAST] pFSCK: Accelerating File System Checking and Repair for Modern Storage. [PDF]

Fragmentation

[2016 HotStorage] An Empirical Study of File-System Fragmentation in Mobile Storage Systems. [PDF] [Slides]

[2017 FAST] File Systems Fated for Senescence? Nonsense, Says Science! [PDF] [Slides]

[2017 ATC] Improving File System Performance of Mobile Storage Systems Using a Decoupled Defragmenter. [PDF] [Slides]

Multicore/Manycore Scalability

[2016 ATC] Understanding Manycore Scalability of File Systems. [PDF] [Slides]

[2017 SOSP] ScaleFS: Scaling a File System to Many Cores Using an Operation Log. [PDF]

[2022 FAST] ScaleXFS: Getting scalability of XFS back on the ring. [PDF]

Distributed File System

General Purpose File System

[1985] Design and Implementation of the Sun Network Filesystem. [PDF]

[1987 SOSP & 1988 TOCS] AFS: Scale and performance in a distributed file system. [PDF]

[1993 SOSP & 1995 TOCS] The Zebra Striped Network File System. [PDF] [Ph.D. Thesis]

[2003 MSST] zFS - A Scalable Distributed File System Using Object Disks. [PDF] [Slides]

[2006 OSDI] Ceph: A Scalable, High-Performance Distributed File System. [PDF]

[2007 SC] RADOS: A Scalable, Reliable Storage Service for Petabyte-scale Storage Clusters. [PDF]

[2007 Ph.D. Thesis@UCSC] Ceph: Reliable, Scalable, and High-Performance Distributed Storage. [PDF]

[2011 ATC] TidyFS: A Simple and Small Distributed File System. [PDF] [Slides]

[2019 SOSP] File systems unfit as distributed storage backends: lessons from 10 years of Ceph evolution. [PDF] [Slides]

Big Data

[2003 SOSP] GFS: The Google file system. [PDF]

[2010 MSST] The Hadoop Distributed File System. [PDF]

[2021 FAST] Facebook’s Tectonic Filesystem: Efficiency from Exascale. [PDF]

High Performance Computing (HPC)

[2000] PVFS: A Parallel File System for Linux Clusters. [PDF]

[2002 FAST] GPFS: A Shared-Disk File System for Large Computing Clusters. [PDF]

[2003] Lustre: Building a File System for 1,000-node Clusters. [PDF]

[2008 FAST] Scalable Performance of the Panasas Parallel File System. [PDF]

Burst Buffer File System

[2016 SC] BurstFS: An Ephemeral Burst-Buffer File System for Scientific Applications. [PDF]

[2018 CLUSTER & 2020 JCST] GekkoFS – A temporary distributed file system for HPC applications. [CLUSTER PDF] [JCST PDF] [Codes]

Cloud Computing

[2018 VLDB] PolarFS: An Ultra-low Latency and Failure Resilient Distributed File System for Shared Storage Cloud Database. [PDF]

[2019 SIGMOD] CFS: A Distributed File System for Large Scale Container Platforms. [PDF]

Data Distribution

[2006 SC] CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data. [PDF]

[2020 FAST] MAPX: Controlled Data Migration in the Expansion of Decentralized Object-Based Storage Systems. [PDF]

Metadata Management

[2003 MSST] Efficient Metadata Management in Large Distributed Storage Systems. [PDF] [Slides]

[2004 SC] Dynamic Metadata Management for Petabyte-scale File Systems. [PDF]

• Key-Value Store based

[2013 ATC] TABLEFS: Enhancing Metadata Efficiency in the Local File System. [PDF]

[2014 SC] IndexFS: Scaling File System Metadata Performance with Stateless Caching and Bulk Insertion. [PDF] [Slides]

[2015 SoCC] ShardFS vs. IndexFS: Replication vs. Caching Strategies for Distributed Metadata Management in Cloud Storage Systems. [PDF]

[2015 PDSW] DeltaFS: Exascale File Systems Scale Better Without Dedicated Servers. [PDF]

[2017 SC] LocoFS: A Loosely-Coupled Metadata Service for Distributed File Systems. [PDF]

[2018 TPDS] A Flattened Metadata Service for Distributed File Systems. [PDF]

[2022 FAST] InfiniFS: An Efficient Metadata Service for Large-Scale Distributed Filesystems. [PDF] [Slides]

• Load Balance

[2015 SC] Mantle: a programmable metadata load balancer for the ceph file system. [PDF]

[2021 SC] Lunule: An Agile and Judicious Metadata Load Balancer for CephFS. [PDF]

Erasure Coding

[2015 FAST] A Tale of Two Erasure Codes in HDFS. [PDF] [Slides]

Surveys

[1989] A Survey of Distributed File Systems. [PDF]

[1990] Distributed File Systems: Concepts and Examples. [PDF]

[2008 NCM] A Taxonomy and Survey on Distributed File Systems. [PDF]

[2015] Analysis of Six Distributed File Systems. [PDF]

[2016 ICCCA] Evolution and Analysis of Distributed File Systems in Cloud Storage: Analytical Survey. [PDF]

[2018 CSUR] Scalable Metadata Management Techniques for Ultra-Large Distributed Storage Systems – A Systematic Review. [PDF]

[2022 TOS] Survey of Distributed File System Design Choices. [PDF]

Analysis

[2000 ATC] A Comparison of File System Workloads. [PDF]

Object Storage

[2003 MSST] Towards an Object Store. [PDF] [Slides]

[2004 MSST] OBFS: A File System for Object-based Storage Devices. [PDF] [Slides]

[2010 OSDI] Finding a needle in Haystack: Facebook’s photo storage. [PDF] [Slides]

[2020 SCFA] DAOS: A Scale-Out High Performance Storage Stack for Storage Class Memory. [PDF]

AI + Storage

[2022 CCF Transactions on High Performance Computing] A survey on AI for storage. [PDF]