muesli / reflow

A collection of (ANSI-sequence aware) text reflow operations & algorithms

reflow

A collection of ANSI-aware methods and io.Writers helping you to transform blocks of text. This means you can still style your terminal output with ANSI escape sequences without them affecting the reflow operations & algorithms.

Word-Wrapping

The wordwrap package lets you word-wrap strings or entire blocks of text.

import "github.com/muesli/reflow/wordwrap"

s := wordwrap.String("Hello World!", 5)
fmt.Println(s)

Result:

Hello
World!

The word-wrapping Writer is compatible with the io.Writer / io.WriteCloser interfaces:

f := wordwrap.NewWriter(limit)
f.Write(b)
f.Close()

fmt.Println(f.String())

Customize word-wrapping behavior:

f := wordwrap.NewWriter(limit)
f.Breakpoints = []rune{':', ','}
f.Newline = []rune{'\r'}

Unconditional Wrapping

The wrap package lets you unconditionally wrap strings or entire blocks of text.

import "github.com/muesli/reflow/wrap"

s := wrap.String("Hello World!", 7)
fmt.Println(s)

Result:

Hello W
orld!

The unconditional wrapping Writer is compatible with the io.Writer interfaces:

f := wrap.NewWriter(limit)
f.Write(b)

fmt.Println(f.String())

Customize word-wrapping behavior:

f := wrap.NewWriter(limit)
f.Newline = []rune{'\r'}
f.KeepNewlines = false
f.PreserveSpace = true
f.TabWidth = 2

Tip: This wrapping method can be used in conjunction with word-wrapping when word-wrapping is preferred but a line limit has to be enforced:

wrapped := wrap.String(wordwrap.String("Just an example", 5), 5)
fmt.Println(wrapped)

Result:

Just
an
examp
le

ANSI Example

s := wordwrap.String("I really \x1B[38;2;249;38;114mlove\x1B[0m Go!", 8)
fmt.Println(s)

Result:

Indentation

The indent package lets you indent strings or entire blocks of text.

import "github.com/muesli/reflow/indent"

s := indent.String("Hello World!", 4)
fmt.Println(s)

Result:

    Hello World!

There is also an indenting Writer, which is compatible with the io.Writer interface:

// indent uses spaces per default:
f := indent.NewWriter(width, nil)

// but you can also use a custom indentation function:
f = indent.NewWriter(width, func(w io.Writer) {
    w.Write([]byte("."))
})

f.Write(b)
f.Close()

fmt.Println(f.String())

Dedentation

The dedent package lets you dedent strings or entire blocks of text.

import "github.com/muesli/reflow/dedent"

input := `    Hello World!
  Hello World!
`

s := dedent.String(input)
fmt.Println(s)

Result:

  Hello World!
Hello World!

Padding

The padding package lets you pad strings or entire blocks of text.

import "github.com/muesli/reflow/padding"

s := padding.String("Hello", 8)
fmt.Println(s)

Result: Hello___ (the underlined portion represents 3 spaces)

There is also a padding Writer, which is compatible with the io.WriteCloser interface:

// padding uses spaces per default:
f := padding.NewWriter(width, nil)

// but you can also use a custom padding function:
f = padding.NewWriter(width, func(w io.Writer) {
    w.Write([]byte("."))
})

f.Write(b)
f.Close()

fmt.Println(f.String())

reflow's Issues

Working with tabs?

Hello and thank you for making this!
I was just wondering if you have a solution for tabs, which reflow seems to count as one column, but their width is really dependent on the terminal and their position with the tabstop.

For now I've got my pager program replacing tabs with eight spaces but it's not ideal for the navigation and copy pasting.
I'm not sure how programs like Vim (or the libraries) cope with tabs and truncating etc.

Thanks again,
John

Add unconditional (non-word) wrapping

I think general non-word wrapping would be a great addition to this library. This algorithm should just break as soon as the limit is reached. This would also be a great addition to wordwrap, as wordwrap does currently not enforce the given limit if the length of single words already exceeds the limit.

A typical usecase would be to stay confined to the terminal width: In this case word-wrapping is nice but the limit has to be strictly enforced so word-wrapping could be applied first and then the general wrapping.

package main

import (
	"fmt"

	"github.com/muesli/reflow/wordwrap"
	"github.com/muesli/reflow/wrap"

)

func main() {
	test := "AAAAA BBBBBBBBBBB"

	fmt.Println(wordwrap.String(test, 8))
	// output:
	// AAAAA
	// BBBBBBBBBBB

	fmt.Println(wrap.String(wordwrap.String(test, 8), 8))
	// output:
	// AAAAA
	// BBBBBBBB
	// BBB
}

Can't use padding Writer instance twice

I don't know if it is a bug or an intended/designed choose.

package main

import (
	"fmt"
	"io"

	"github.com/muesli/reflow/padding"
)

func main() {
	width := uint(6)
	b := []byte("foo")

	// padding uses spaces per default:
	f := padding.NewWriter(width, nil)

	// but you can also use a custom padding function:
	f = padding.NewWriter(width, func(w io.Writer) {
		w.Write([]byte("."))
	})

	f.Write(b)
	f.Close()
	
	// foo...
	fmt.Println(f.String())

	f.Write(b)
	f.Close()

	// foo...foo
	fmt.Println(f.String())
}

Don't wrap at non breaking space.

unicode.IsSpace also returns true for a non breaking space (0xA0), which isn't supposed to be treated as a space in this context.

reflow/wordwrap/wordwrap.go

Line 127 in 00a9f5c

} else if unicode.IsSpace(c) {

Truncate support

Maybe add support for truncating strings as in https://github.com/cli/cli/blob/trunk/pkg/text/truncate.go.

Provide option to specify a different indent level for first line

I would be great to have the option to specify a different indent level for the first line.

That would help fix charmbracelet/glamour#167. The idea would be to use a block element for list items with an indent of 0 for the first line, to account for the list item prefix.

What do you think?

Question: should wordwrap support a "hard wrap if needed" option?

I'm learning bubbletea and lipgloss, and reading a lot of the code because I'm not finding any descriptions about the "box model" (so to speak) that explains how width, padding, border, and margins interact. In doing so, I ran across this code in lipgloss/style.go:

str = wordwrap.String(str, wrapAt)
str = wrap.String(str, wrapAt) // force-wrap long strings

If you have a series of words just longer than the limit, you will get weird, orphaned pieces. For example, wrapping 123456789 123 123456789 123 123456789 123 at length 8 using the above code, the initial wordbreak will leave the length-9 words intact, but add newlines after them, causing the short words to always be on their own lines:

wordwrap    ==> wrap
--------|       --------|   
12345678|9      12345678|  (hard break)  
123     |       9       |
12345678|9      123     |   
123     |       12345678|  (hard break)
12345678|9      9       |
                123     |   
                12345678|  (hard break)
                9       |

If wordbreak had a "hard break if needed", the calling code would just call "wordwrap with hard-breaks", and get:

wordwrap+hard
--------|
12345678|  (hard break)
9 123   |  ("123" fits, soft-wrap afterwards)
12345678|  (hard break)
9 123   |  ("123" fits, soft-wrap afterwards)
12345678|  (hard break)
9

Would that be useful?

Extra newlines when combining unconditional and standard word wrapping

When combining word-wrapping with unconditional wrapping as described in the README, extra linebreaks can sometimes be found in the output.

For example:

const str = "the quick brown foxxxxxxxxxxxxxxxx jumped over the lazy dog."
const limit = 16
wrapped := wrap.String(wordwrap.String(str, limit), limit)
fmt.Println(wrapped)

Outputs:

the quick brown
foxxxxxxxxxxxxxx
xx
jumped over the
lazy dog.

However, I'd expect it to be:

the quick brown
foxxxxxxxxxxxxxx
xx jumped over 
the lazy dog.

Playground Example

Add dedent package

Awesome work with this package, really what I was looking for. I could clean up a lot of duplicated helper functions by replacing it with this library.

I have a feature request: Dedentation. It is quite helpful. I currently use a helper function and spotted some code in https://github.com/cli/cli doing the same. It is essentially:

func dedent(s string) string {
	lines := strings.Split(s, "\n")
	minIndent := -1

	for _, l := range lines {
		if len(l) == 0 {
			continue
		}

		indent := len(l) - len(strings.TrimLeft(l, " "))
		if minIndent == -1 || indent < minIndent {
			minIndent = indent
		}
	}

	if minIndent <= 0 {
		return s
	}

	var buf bytes.Buffer
	for _, l := range lines {
		fmt.Fprintln(&buf, strings.TrimPrefix(l, strings.Repeat(" ", minIndent)))
	}
	return strings.TrimSuffix(buf.String(), "\n")
}

The algorithm behind that is simple: Find the longest indentation possible for all lines and dedent each line by that.

If you like the idea, I can make a PR and add some tests.

Issue with wrapping CP437 ANSI file

I'm attempting to wrap a CP437 ANSI file, with a wrap after column 80. However, something is causing to to not wrap correctly.

I'm decoding before using reflow, and then re-encoding CP437 afterward, and printing...

when using reflow:

the actual art file for reference:

Not sure if there's something obvious I'm missing...? Happy to take to SO if this is not the forum for this type of question.

Thanks for a great library!

(code for reference)

wordwrap: Support for URL encoding of the space character (%20)?

Any chance you could consider support URL encoding of the space characters (%20) for word wrapping?

This would make long URLs such as those from Jira wrap nicely rather than the possibility of wrapping or truncating at say ending % on one line and then starting 20... on the other line.

Use case is with the https://github.com/42wim/matterircd project.

New optimized write performance seems to break glamor

The recent change (in ansi.Writer) to iterating over bytes instead of runes seems to break glamor's UTF-8 processing here: https://github.com/charmbracelet/glamour/blob/master/ansi/margin.go

The first indent pipe (using ansi.Writer) breaks up UTF-8 byte sequences into single byte writes, which are misinterpreted when ingested into the second padding pipe, as they go through a byte->string->[]byte conversion.

Here's an illustration of the effect:
https://play.golang.org/p/N-caKzuAwdQ

Originally posted by @zavislak in #10 (comment)

wordwrap.String - like API which returns []string?

I'm writing a chat client and I'd like to iterate over the lines I wrap. I can strings.Split(..., "\n" over what I return but it seems like such a common things to need to do...

Hyphens can cause wordwrap to exceed the limit

When wordwrap.Write() encounters a hyphen, it immediately adds the existing space, word, and hyphen, without first checking to see if this would exceed the limit. This can cause the hyphen to extend beyond the limit.

Example: wordwrap.String("foo foo-foobar", 7) should return

1234567|  <-- limit
foo    |
foo-   |
foobar |

but actually returns:

1234567|  <-- limit
foo foo|- <-- hyphen beyond limit
foobar |

New release that contains truncate functionality

Would you please tag a new release.

It would help packaging your library for e.g. guix

v0.3.0 broke dedent package

Assume the following (in real life dynamically created) string:

Your bank accounts:

  DKB
    ✓ Logged in as jamesbond
  ING
    ✓ Logged in as jamesbond

The indentation is there for a reason. At one stage a special character is inserted to denote the selected bank account (if an account is actually selected):

Your bank accounts:

➜ DKB
    ✓ Logged in as jamesbond
  ING
    ✓ Logged in as jamesbond

Regardless if an account is selected or not, this is run through dedent.String() to get rid of the indentation of the first example above.

With v0.2.0 and v0.3.0, this works fine:

Your bank accounts:

DKB
  ✓ Logged in as jamesbond
ING
  ✓ Logged in as jamesbond

However, when the arrow is present as in the second example above, dedentation gets mangled with v0.3.0:

Your bank accounts:

➜DKB
  ✓ Logged in as jamesbond
ING
  ✓ Logged in as jamesbond

Not even the whitespace between the char and the next string is preserved.

To sum this up and give a working example, try the following code with v0.2.0 and v0.3.0:

const s = `Your bank accounts:

➜ DKB
    ✓ Logged in as jamesbond
  ING
    ✓ Logged in as jamesbond
`

fmt.Print(dedent.String(s))

Replace go-runewidth with uniseg

There are a number of issues with the go-runewidth package that makes it problematic when used with emojis. Working with runes is a flawed approach as explained by the author of uniseg:

That's because a single code point (rune) is not always a complete character. That's the whole basis of the uniseg package. The README explains this in detail. One example given there is the country flag emoji. These emojis must always consist of two runes. What's the width of only one of these runes then? It's basically undefined because it's an incomplete grapheme cluster. The mattn/go-runewidth package gets this fundamentally wrong and I have tried for a long time to help them do it right but there was never much interest in following up on it.
rivo/uniseg#48 (comment)

All references to RuneWidth and StringWidth need to be replaced to improve the width calculations when dealing with emojis.

It is trivial to replace StringWidth from go-runewidth to uniseg as they provide the same input and output. There is also a performance increase of 4x migrating to using the uniseg version so there are some very clearly benefits here.

The complexity lies with what to do about the RuneWidth function. The right choice will be stop thinking in runes and switch over to grapheme clusters. This will require logic changes to any function using runes.

From an acceptance criteria perspective, the goals should be:

1. Remove go-runewidth as a dependency.
2. No regressions in performance.

We can expect different results for some of the calculations but these should be considered improvements and bug fixes. It is likely this will cause breaking changes to applications that depend on this package. This may warrant a major version bump to reflect that. We should also make it clear in the documentation that the calculations involving some emojis will have changed.

Coloring issue when used in association with lipgloss.JoinHorizontal

When the wrapping is done on the right of the terminal, the start of the line is highlighted (aapt in the screenshot).

Here is the method setting the preview

func (c *List) RefreshPreview() {
	previewWidth := c.viewport.Width - 2 // padding
	previewContent := wrap.String(wordwrap.String(c.previewContent, previewWidth), previewWidth)

	c.viewport.SetContent(previewContent)
}

and here is the view method:

func (c List) View() string {
	if c.actions.Focused() {
		return c.actions.View()
	}

	if c.ShowPreview {
		var separatorChars = make([]string, c.viewport.Height)
		for i := 0; i < c.viewport.Height; i++ {
			separatorChars[i] = "│"
		}
		separator := strings.Join(separatorChars, "\n")
		view := lipgloss.JoinHorizontal(lipgloss.Top, c.filter.View(), separator, c.viewport.View())

		return lipgloss.JoinVertical(lipgloss.Top, c.header.View(), view, c.footer.View())
	}

	return lipgloss.JoinVertical(lipgloss.Left, c.header.View(), c.filter.View(), c.footer.View())
}

Consider adding extra repository write users

This package is a core component of the Bubble Tea ecosystem as it is heavily used by Lip Gloss and Bubbles.

There has been no new release since May 2021.

There are also a number of open pull requests that would provide many benefits that have not been reviewed or merged.

While I appreciate the work @muesli has done maintaining this package, I am wondering if it may help to have some more active developers with write access. There is a lot of small changes that could be done that would have benefits for many Bubble Tea components.

I'd also add that it would be very useful to automatically add reviewers to newly created pull requests. Currently, it is hard to know who has write access to request them to review a change.

Hyperlinks consume space though the link isn't printed

I was modifying glamour to output links without the anchor text, but this uses up "space" in wordwrap.go even though it ideally would know that the anchor text does not get displayed by the terminal.

Here's a printf example that you can use to see what the link output looks like:

$ printf '\e]8;;file:./SECURITY.md\e\\This is a link\e]8;;\e\\\n'

So the bytes from file:./SECURITY.md should not be counted towards the block's wordwrapping. Here's an example of how it fails to look correct:

$ ./glow ~/projects/kitty/CONTRIBUTING.md 

  ### Reporting bugs                                                                                
                                                                                                    
  Please first search existing bug reports (especially closed ones) for a report that matches your  
  issue.                                                                                            
                                                                                                  
  When reporting a bug, provide full details of your environment, that means, at a minimum, kitty   
  version, OS and OS version, kitty config (ideally a minimal config to reproduce the issue with).  
                                                                                                    
  ### Contributing code                                                                             
                                                                                                    
  Install the dependencies using your favorite  
  package manager. Build and run kitty from source.                                                                                
                                                                                                  
  Make a fork, submit your Pull Request. If it's a large/controversial change, open an issue        
  beforehand to discuss it, so that you don't waste your time making a pull request that gets       
  rejected.                                                                                         
                                                                                                  
  If the code you are submitting is reasonably easily testable, please contribute tests as well (see
  the  kitty_tests/  sub-directory for existing tests, which can be run with  ./test.py ).          
                                                                                                  
  That's it.                                                                                        

Note how the middle paragraph that contains a link for "the dependencies" and a link for "from source" cause the paragraph to be badly wrapped, the same as if the URL was inline:

$ glow ~/projects/kitty/CONTRIBUTING.md 

  ### Reporting bugs                                                                                
                                                                                                    
  Please first search existing bug reports (especially closed ones) for a report that matches your  
  issue.                                                                                            
                                                                                                  
  When reporting a bug, provide full details of your environment, that means, at a minimum, kitty   
  version, OS and OS version, kitty config (ideally a minimal config to reproduce the issue with).  
                                                                                                    
  ### Contributing code                                                                             
                                                                                                    
  Install the dependencies https://sw.kovidgoyal.net/kitty/build/#dependencies using your favorite  
  package manager. Build and run kitty from source https://sw.kovidgoyal.net/kitty/build/#install-and-
  run-from-source.                                                                                  
                                                                                                  
  Make a fork, submit your Pull Request. If it's a large/controversial change, open an issue        
  beforehand to discuss it, so that you don't waste your time making a pull request that gets       
  rejected.                                                                                         
                                                                                                  
  If the code you are submitting is reasonably easily testable, please contribute tests as well (see
  the  kitty_tests/  sub-directory for existing tests, which can be run with  ./test.py ).          
                                                                                                  
  That's it.                                                                                        

Middle string truncation

Hey, I'm looking for something that can help truncate strings and add an ellipsis, similar to how macOS's Finder will truncate filenames in the middle of the string.

Would something that either achieves this or helps with it be considered within scope?

(Happy to try and write up a PR if so.)

For the former (achieving this), I'd imagine adding a new package that does this middle truncation, and similar to truncate would take a parameter similar to tail. (Not sure what a good name for the param would be off the top of my head, though.)

For the latter (helping with this), I have in mind a version of truncate that truncates counting from the right. (I guess it could almost be thought of like an RTL truncate?) An end-user would then end up using both truncate-from-left and truncate-from-right.

FYI: similar project

Just a FYI to maybe have some cross pollination, I wrote a similar utility for the need of the git-bug project: https://github.com/MichaelMure/go-term-text

It doesn't support the io.Writer interface though, that's a cool idea.

Is a command-line interface planned?

OSC sequences are mis-counted in `PrintableRuneWidth`

OSC sequences are sequences beginning with OSC (ESC ]) and ending in ST (ESC ], or BEL which is 0x07). Note that that's a right bracket, not a left one like CSI sequences.

Those sequences should be ignored, but currently they entirely misinterpreted.

Although PrintableRuneWidth ignores characters after ESC (0x1B), isTerminator does not correctly recognize either variant of ST: (ESC \\, or BEL).

In the former, it will mis-count the slash after the ESC; and the latter fails because BEL is not matched as a sequence terminator.

Specifically, this breaks text using OSC8 for describing URL links.
OSC8 is a mechanism for describing URI/URL links in terminals, described very digestibly here:
https://gist.github.com/egmontkob/eb114294efbcd5adb1944c9f3cb5feda#the-escape-sequence

It's widely supported by many modern terminals (https://github.com/Alhadis/OSC8-Adoption/), and those that don't support it fall back to omitting the URL part. (Same text, just not clickable).

On a whim, I wanted to make the GitHub cli (https://github.com/cli/cli) support those links, but hit a stumbling block: its tableprinter measures column text width ultimately through muesli/reflow.

It could be fixed in their intermediate go-gh package, but it seems like this is the "right" place to fix it, or at least to have the issue logged.

Dedent incorrectly removes spaces that aren't at the start of a line

When a string has lines with no indentation and lines with indentations, the dedented string will remove the first spaces within the line instead of just at the start.

package main

import (
	"fmt"

	"github.com/muesli/reflow/dedent"
)

func main() {
	input := `This is an example
	
  with no shared indent.`
	fmt.Println(dedent.String(input))
}

The output will be:

Thisisan example

with no shared indent.

https://go.dev/play/p/Q13CeRD8Tb2

Truncate does not work for blocks of text

All the functions referenced in the README.md are advertised to work on blocks of text, but truncate does not:

package main

import (
	"fmt"

	"github.com/muesli/reflow/truncate"
)

func main() {
	test := "123456789\nhidden"
	fmt.Println(truncate.String(test, 5)) // output: 12345
}

BytesWithTail and StringsWithTail should put tail between escape sequences

When a trail is added to stylized (e.g. color) text, it would be great if the tail (e.g. ellipsis) was included in the sequence.

For example, consider the string: \x1b[0;31mexample\x1b[0m. If part of the string is replaced, it's the string within the vterm sequence that is replaced; thus, the ellipsis should be before the vterm sequence as well: \x1b[0;31mex...\x1b[0m

Consistency with widths: int vs uint

I've noticed, while using various parts Reflow together, that most packages int to describe widths, while the newer truncate package uses uint. For example:

// In the ansi sub-package
func PrintableRuneWidth(s string) int

// In the truncate sub-package
func String(s string, width uint) string

Why not standardize on or the other? If not breaking the API is a concern, then it would be safer to stay with int as the truncate package is still unreleased.