mmatongo / chew

Chew is a Go library for processing various content types into markdown/plaintext.

A Go library for processing various content types into markdown/plaintext..

About

Chew is a Go library that processes various content types into markdown or plaintext. It supports multiple content types, including HTML, PDF, CSV, JSON, YAML, DOCX, PPTX, Markdown, Plaintext, MP3, FLAC, and WAVE.

Installation

go get github.com/mmatongo/chew

Usage

Here's a basic example of how to use Chew:

package main

import (
    "context"
    "fmt"
    "log"
	"time"

    "github.com/mmatongo/chew"
)

func main() {
    urls := []string{
        "https://example.com",
    }

	// The context is optional
	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

    chunks, err := chew.Process(urls, ctx)
    if err != nil {
		if err == context.DeadlineExceeded {
			log.Println("Operation timed out")
		} else {
			log.Printf("Error processing URLs: %v", err)
		}
		return
    }

    for _, chunk := range chunks {
        fmt.Printf("Source: %s\nContent: %s\n\n", chunk.Source, chunk.Content)
    }
}

Output

Source: https://example.com
Content: Example Domain

Source: https://example.com
Content: This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

Source: https://example.com
Content: More information...

You can find more examples in the examples directory as well as instructions on how to use Chew with Ruby and Python.

Contributing

Contributions are welcome! Feel free to open an issue or submit a pull request if you have any suggestions or improvements.

License

This project is licensed under the MIT License - see the LICENSE file for details.

Logo

The logo was made by the amazing MariaLetta.

Similar Projects

docconv

Roadmap

The roadmap for this project is available here. It's meant more as a guide than a strict plan because I only work on this project in my free time.

chew's Issues

Reorganise internal

• Refactor and reorganise the code under internal
  Currently the folder is a bit of a mess (subjective?) and so this is mostly housekeeping to reorganise and restructure it

Write testcases for chew.go

Add an open source transcription option

Right now the only options are Google and OpenAI (under development), this doesn't leave much room for people that would rather not use these.

Task

• Implement a generic interface to allow chew to work with other open source alternatives for transcription

Blocker

• I need to first implement the Whisper intergration so that I can have an idea of how I can work around this.

Notes

• Whisper is open source (research)

For context this is my first public project with golang and considering that I am still learning most of my process involves a lot of iterative refinement

Refactor code under internal/speech

• Google is badly written and that makes it hard to test because of how it explicitly needs certain things so mocking presents itself as a challenge.

Enhance selection for HTML and EPUB

Currently selectors are hardcoded making it tricky for anyone who wants to target specific selectors.

	doc.Find("p, h1, h2, h3, h4, h5, h6, li").Each(func(_ int, s *goquery.Selection) {
		text := strings.TrimSpace(s.Text())
		if text != "" {
			chunks = append(chunks, common.Chunk{Content: text, Source: url})
		}
	})

This is far from ideal and doesn't allow for much flexibility so users should be able to specify selectors

Implement transcribing using the whisper api

I tested out whisper and wow, it's much faster and much more accurate the google's cloud speech api. Implementing this will require a rework and reorganisation of the speech module so as to avoid spaghettification.

Write testcases for everything under internal

Fix flakey process test

{
	name: "multiple URLs",
	args: args{
		urls: []string{server.URL + "/text", server.URL + "/html"},
	},
	want: []common.Chunk{
		{Content: "A plain text file.", Source: server.URL + "/text"},
		{Content: "An HTML file.", Source: server.URL + "/html"},
	},
	wantErr: false,
},

test fails on the returned order
it can be text/html or html/text