file_re is a Python library written in Rust aimed at providing robust and efficient regular expression operations on large files, including compressed files such as .gz and .xz. The goal of this library is to handle huge files in the order of gigabytes (GB) seamlessly.
- Fast and efficient: Utilizes Rust for performance improvements.
- Supports Large Files: Capable of parsing files in gigabytes.
- Compressed Files: Supports reading and searching within
.gzand.xzcompressed files. - Flexible: Similar interface to Python's built-in
remodule.
from file_re import file_re
from pathlib import Path
# Define the path to the file
file_path = Path('path/to/your/big_file.txt')
# Search for a specific pattern
match = file_re.search(r"(\d{3})-(\d{3})-(\d{4})", file_path)
# Mimic the behavior of Python's re.search
print("Full match:", match.group(0))
print("Group 1:", match.group(1))
print("Group 2:", match.group(2))
print("Group 3:", match.group(3))
match = file_re.search(r"(?P<username>[\w\.-]+)@(?P<domain>[\w]+)\.\w+", file_path)
# Mimic the behavior of Python's re.search with named groups
print("Full match:", match.group(0))
print("Username:", match.group("username"))
print("Domain:", match.group("domain"))
# Find all matches
matches = file_re.findall(r"(\d{3})-(\d{3})-(\d{4})", file_path)
print(matches)
# You can read direclty from compressed files
file_path = Path('path/to/your/big_file.txt.gz')
matches = file_re.findall(r"(\d{3})-(\d{3})-(\d{4})", file_path)
# For regex that requires multiple lines you have to enable the multiline mode
matches = file_re.search(r"<body>[\s\S]+</body>", file_path, multiline=True)
print(matches.group(0))-
Default Line-by-Line Processing:
- Memory Efficiency: By default,
file_rereads files line by line and applies the regular expression to each line individually. This approach is memory efficient as it avoids loading the entire file into RAM. - Pattern Constraints: This mode may not work effectively for regex patterns that span across multiple lines.
- Memory Efficiency: By default,
-
Multiline Mode:
- Full File Loading: When the multiline mode is enabled, the entire file is loaded into RAM to perform the regex operation. This is necessary for regex patterns that require matching across multiple lines.
- Increased RAM Usage: Loading large files (in gigabytes) into RAM can lead to significant memory consumption. This may not be suitable for systems with limited memory.
- Performance Trade-offs: While enabling multiline mode can result in faster
findalloperations for certain patterns, it comes at the cost of higher memory usage.
-
Limited Flag Support:
- Flag Limitations: Currently, flags such as
re.IGNORECASEorre.MULTILINEare not supported. - Future Enhancements: Support for these flags is planned for future releases, which will enhance the flexibility and usability of the library.
- Flag Limitations: Currently, flags such as
Users are encouraged to assess their specific needs and system capabilities when using file_re, especially when working with extremely large files or complex multiline regex patterns.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent