remikalbe / cunw

A CLI tool that generates a structured Markdown representation of a codebase, making it easy to provide context to a large language model (LLM).

cunw

cunw (codebase unwrap) is a command-line interface (CLI) tool that generates a structured representation of a codebase, making it easy to provide context to a large language model (LLM). It recursively traverses a directory, collects file content, and generates a single file that represents the structure and content of the codebase.

🌟 Features

• Recursively traverses a directory and collects file content
• Generates a file representing the codebase structure and content
• Supports excluding files based on glob patterns
• Respects .gitignore files by default (can be disabled)
• Allows specifying the maximum depth of directory traversal
• Supports following symbolic links (disabled by default)

📦 Installation

Precompiled Binaries

You can easily install cunw through cargo:

cargo install cunw

Or download the precompiled binaries from the releases page.

From Source

To install cunw, ensure you have Rust and Cargo installed on your system. Then, clone the repository and build the project:

git clone https://github.com/RemiKalbe/cunw.git
cd cunw
cargo build --release

The compiled binary will be available at target/release/cunw.

🚀 Usage

cunw [OPTIONS]

Options

• -p, --path <PATH>: The path to the directory containing the codebase.
• -o, --output <FILE>: The path of the output file. Default: output.txt
• -e, --exclude <PATTERN>: Exclude files or directories matching the specified glob pattern.
• --do-not-consider-ignore-files: Do not consider .gitignore files when filtering. Default: false
• --dangerously-allow-dot-git-traversal: Include .git directory in the search. Default: false
• -d, --max-depth <DEPTH>: Maximum depth to walk into the directory tree.
• -f, --follow-symbolic-links: Follow symbolic links. Default: false
• -v, --verbose: Set the verbosity level. Can be used multiple times to increase verbosity.

Example

To generate a file representation of a codebase located at path/to/codebase, excluding files matching *.txt and save the output to codebase.md:

cunw path/to/codebase -o codebase.md -e "*.txt"

📝 Output Format

The generated file will have the following structure:

<directory_structure>
.
└─ .
├─ ./src
│ ├─ main.rs
│ └─ lib.rs
├─ .gitignore
├─ Cargo.lock
└─ Cargo.toml
</directory_structure>

<file path="Cargo.toml">
[package]
name = "cunw"
version = "0.1.0"
edition = "2021"

[dependencies]

<!-- ... -->
</file>

<file path="src/main.rs">
fn main() {
    println!("Hello, world!");
}
</file>

<!-- ... -->

The <directory_structure> section represents the directory tree of the codebase, and each <file> section contains the content of a specific file.

🤝 Contributing

Contributions are welcome! If you find any issues or have suggestions for improvements, please open an issue or submit a pull request on the GitHub repository.

📄 License

This project is licensed under the MIT License.

cunw's Issues

Fix unintuitive glob issue with `.`

If we run the CLI like so cunw ., then the glob patterns are expected to start with ./, so for example if we want to ignore all node_module, we have to write ./**/node_module. Which I don't find really intuitive.

Automatically ignore non `utf-8` files

Right now the cli will fail as soon as it tries to open a non utf-8 files which means we have to each time add a file or specific file type to the ignore arg, which become annoying really quickly, instead we should just warn about files that couldn't be processed but still continue with the other ones.

• Should we just ignore and continue on error, or first inspect the file with the file-format crate?

Not respecting nested `.gitignore`

It looks like only the .gitignore in the same dir as where the command was run is respected.

Switch from `nuclei`

It seems like nuclei is no longer maintained, which is problematic as the build fails on windows.