buckket / twtxt

Decentralised, minimalist microblogging service for hackers.

twtxt

twtxt is a decentralised, minimalist microblogging service for hackers.

So you want to get some thoughts out on the internet in a convenient and slick way while also following the gibberish of others? Instead of signing up at a closed and/or regulated microblogging platform, getting your status updates out with twtxt is as easy as putting them in a publicly accessible text file. The URL pointing to this file is your identity, your account. twtxt then tracks these text files, like a feedreader, and builds your unique timeline out of them, depending on which files you track. The format is simple, human readable, and integrates well with UNIX command line utilities.

tl;dr: twtxt is a CLI tool, as well as a format specification for self-hosted flat file based microblogging.

Features

• A beautiful command-line interface thanks to click.
• Asynchronous HTTP requests thanks to asyncio/aiohttp and Python 3.
• Integrates well with existing tools (scp, cut, echo, date, etc.) and your shell.
• Don’t like the official client? Tweet using echo -e "`date +%FT%T%:z`\tHello world!" >> twtxt.txt!

Documentation

Check out the full documentation at: http://twtxt.readthedocs.org/en/latest/

Community

• twtxt IRC channel: #twtxt on irc.libera.chat

Contributions

• A curated list of active twtxt users by yarn.social: https://git.mills.io/yarnsocial/we-are-twtxt
• A web-based directory of twtxt users by reednj: http://twtxt.reednj.com/
• A web-based twtxt feed hoster for the masses by plomlompom: https://github.com/plomlompom/htwtxt
• A twtxt-to-atom converter in sh by erlehmann: http://news.dieweltistgarnichtso.net/bin/twtxt2atom
• A twitter-to-twtxt converter in node.js by DracoBlue: https://gist.github.com/DracoBlue/488466eaabbb674c636f
• A port to node.js / npm by Melvin Carvalho: https://github.com/webize/twtxt
• A patched version of TweetNest, which serves TweetNest archives in twtxt format, by texttheater: https://github.com/texttheater/tweetnest/tree/feat/twtxt
• A twtxt registry api by DracoBlue: https://registry.twtxt.org
• A twtxt client written in perl by mdom: https://github.com/mdom/txtnix
• A twtxt client with minimal dependencies by mdom: https://github.com/mdom/txtnish
• A twtxt client written in C by dertuxmalwieder: https://hub.darcs.net/dertuxmalwieder/twtxtc
• A read-only timeline of the last 3000 tweets via gopher by trqx: gopher://shroom.party/1/twtxt
• A bot for using twtxt over xmpp by mdosch: https://salsa.debian.org/mdosch-guest/goxtxt
• twtxt registry server written in Go by gbmor: https://github.com/gbmor/getwtxt-ng
• A twtxt parsing library written in Rust by gbmor: https://github.com/rustwtxt/rustwtxt
• A twtxt WordPress plugin, that provides the blog-posts as twtxt file, written by pfefferle: https://github.com/pfefferle/wordpress-twtxt
• A twtxt client for Emacs by deadblackclover: https://github.com/deadblackclover/twtxt-el
• An php interface for publishing to your selfhosted twtxt.txt by sorenpeter: https://github.com/sorenpeter/phpub2twtxt/
• A graphical twtxt client written in Tcl/Tk, RSS-to-twtxt converter, and mentions extractor by dbohdan: https://gitlab.com/dbohdan/twtxt.tcl
• twtwt: a really fast UNIX only twtxt client written in C by win0err: https://github.com/win0err/twtwt

License

twtxt is released under the MIT License. See the bundled LICENSE file for details.

twtxt's Issues

How to pronounce 'twtxt'?

Suggestions anyone? 😬

Option for timeline command to style output in twtxt format

Aggregate content but keep it in twtxt format.
Should allow other programs to easily work with that data.

twtxt has very low color contrast in terminal emulator

See picture (using dev version with Atom/RSS support, following a podcast feed):

Terminal emulator is ROXTerm. Any idea what could cause this?

Config update removes comments

After eg. the follow command, the config gets rewritten and all comments are lost. In my case it was a commented out value of a variable when I did a quick-test of different value.
Would be good to preserve the comments or document that they could get removed.

Pre/post hooks should check return value

post_tweet_hook = /bin/false followed by 'twxtxt tweet' succeeds, no messages. While the current semantics don't specify the behaviour, I'd expect that there should be some way how to get the feedback from the hooks.
Possible semantics:

• if post/pre _tweet_hook returns with nonzero code, print the stdout/stderr
• keep current as-is, add new hooks that will be strict about the hook result

Include user’s twtxt nick and URL in their user-agent string

Like proposed here: #3 (comment)
This allows searching the webserver’s access log for one’s followers.

Format: twtxt/1.2.3 (+https://example.com/twtxt.txt; @somebody)

Any other suggestions for the format we should use?

add_local_tweet is not atomic

https://github.com/buckket/twtxt/blob/master/twtxt/file.py#L29-L36

This function is not atomic. Should writing to the file fail for any reason, the file may be corrupted and require manual cleanup.

It would be better to copy the existing file to a temporary file, write to the temporary file, and rename the temp file to the existing file. rename(2) is an atomic operation, so at worst, if writing the temporary file fails, you can easily abort, and the existing file is left unscathed.

Follower url works in browser, does not in twtxt

HTTP:

twtxt -v follow mdom http://www.domgoergen.com/twtxt.txt
...
twtxt.twhttp DEBUG    400, message='deflate'

HTTPS

...
twtxt.twhttp DEBUG    hostname 'www.domgoergen.com' doesn't match either of '*.kasserver.com', 'kasserver.com'

works in the browser.

How to build from source?

The project looks very interesting but I couldn't find instructions to build from source.

More place holders for use in post/pre_tweet_hook

Currently we support "{twtfile}" which expands to the location of the local twtxt file.

While playing around with https://github.com/plomlompom/htwtxt I thought about also making the tweet itself (in it’s raw format) available, so that we could post our new tweet to a remote server via a HTTP POST request, say by using curl.

Example:
curl --data "username=foo&password=bar&tweet=FOOBAR" https://example.com/append

Plugin system

There were some Issues and PRs for features which I think should be implement as Plugins/Extensions to twtxt rather then implementing directly in the main code base.

I'd like to collect ideas and proposals about possible interfaces here and discuss about the implementation of some prototypes as a first proof of concept.

Should ‘\S+’ be ‘\w+’?

In https://github.com/buckket/twtxt/blob/master/twtxt/mentions.py#L16 we have

short_mention_re = re.compile(r'@(?P<name>\S+)')

Shouldn't the ‘\S+’ be a ‘\w+’? It seems that if people mention several users separated by a comma, the ‘\S+’ form will swallow the comma:

import re

test = '@alpha, @beta, . . . @omega'

rx_s = re.compile(r'@(\S+)')
print(rx_s.sub('CENSORED', test))

rx_w = re.compile(r'@(\w+)')
print(rx_w.sub('CENSORED', test))

When run, the above code will show

CENSORED CENSORED . . . CENSORED
CENSORED, CENSORED, . . . CENSORED

Or will ‘\w+’ just cause another problem?

Increase default message size

140 characters is an archaic limitation that twitter only used so that single tweets could fit in an SMS. it's equivalent to not allowing lines of source code over 80 characters because then they won't fit on a punch card. Even doubling it, or 256 characters, would be an awesome improvement.

Add command to view feed of single source

Display absolute time instead of relative time

The relative time format displayed by twtxt (“2 hours ago”) makes it harder to comprehend the timeline.

First, it is relative not to the current time (when the user is reading the text), but to the time when the command was invoked. If one invokes the command and reads the output later, the information is no longer accurate. This also means one cannot use the twtxt client to generate a web page with a timeline unless it is regenerated very frequently.

Second, it has a very low resolution – I now have three entries in my timeline that have a time of “9 days ago”. I have no way to find out if that was 9 days ago in the morning or 9 days ago in the evening with the current display.

I therefore propose displaying the time always in an absolute form, taking the user's timezone into account. This could also eliminate the dependency on the “humanize“ library, which twtxt uses to turn the timestamps into the fuzzy human readable times shown right now.

failing to write to the twtxt file doesn't abort execution

https://github.com/buckket/twtxt/blob/master/twtxt/cli.py#L72-L73

If that fails, a nice error message is reported and execution continues as usual, including executing the post_tweet_hook. I feel like this is incorrect behavior, and that it should abort immediately if writing the file fails.

Use RFC3339 for timestamps

Would it be possible to change the specification of a message to include a rfc3339 timestamp? ISO 8601 is a lot harder to parse and allows constructs like 2016-W15-03 that are not parsed correctly by the current parser. I checked 55 twtxt files and everybody is using rfc3339, so it would be easy to make the change now.

Revamp Config object

Currently we’re almost always parsing the config file again when accessing a property. This should be done only once, and then saving the resulting object for later use.

Implement caching to avoid downloading whole files over and over again

Idea from detaro on Hacker News:

Make the ETAG the size, send a range request for only the bytes you haven't seen already?

Sounds cool, but If someone changes or deletes older status updates this might cause problems.

Simpler: If-Modified-Since would help to avoid downloading the file if nothing changed, but then again if something changed we would have to grab the whole file.

Evil escape sequences

Just a warning for those of you who are writing twtxt terminal clients: Please visit https://mosh.mit.edu/ and search for “Careful terminal emulation”. While we still ought to allow unicode, we should probably think about sanitizing each tweet before displaying it.

~@kas

Config file sanity checking, better error reporting

I did a stupid typo Fasle instead of False in the config. This is reported as raw python exception. It would be good to print at least the line, or eg. the expected value type.
The overall config sanity could look for uknown key names.

can't (figure out how to) run coverage

I'm hoping someone here can point me in the right direction- I can bump up the unit test coverage if I can get pycov working.

$ py.test --cov-config .coveragerc --cov=twtxt 
========================================== test session starts ==========================================
platform darwin -- Python 3.4.3, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /Users/ted/git/twtxt, inifile: 
plugins: cov-2.2.1
collected 6 items 

tests/test_types.py ......Coverage.py warning: No data was collected.

$ py.test --cov-config .coveragerc --cov=twtxt tests/
========================================== test session starts ==========================================
platform darwin -- Python 3.4.3, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /Users/ted/git/twtxt, inifile: 
plugins: cov-2.2.1
collected 6 items 

tests/test_types.py ......Coverage.py warning: No data was collected.

This one gives approximately the same coverage percent (13%) as the repo shows, but is obviously incorrect:

$ py.test --cov-config .coveragerc --cov=. tests/
========================================== test session starts ==========================================
platform darwin -- Python 3.4.3, pytest-2.8.7, py-1.4.31, pluggy-0.3.1
rootdir: /Users/ted/git/twtxt, inifile: 
plugins: cov-2.2.1
collected 6 items 

tests/test_types.py ......
---------------------------- coverage: platform darwin, python 3.4.3-final-0 ----------------------------
Name               Stmts   Miss  Cover
--------------------------------------
setup                  5      5     0%
tests/test_types      63      0   100%
twtxt/__init__         0      0   100%
twtxt/cli            114    114     0%
twtxt/config          87     87     0%
twtxt/file            20     20     0%
twtxt/helper          47     47     0%
twtxt/http            49     49     0%
twtxt/log             10     10     0%
twtxt/parser          23     23     0%
twtxt/types           51     51     0%
--------------------------------------
TOTAL                469    406    13%

======================================= 6 passed in 0.09 seconds ========================================

Please make mentions case-insensistive

Here's an excerpt from a recent twtxt conversation with @buckket:

2016-02-13T10:07:37+0000    Are #twtxt @usernames case-sensitive?
2016-02-13T11:52:04+01:00   @<kas https://enotty.dk/twtxt.txt> As far as mentions go: yes.
2016-02-13T11:13:48+0000    @buckket [Yes, I was referring to mentions.] Then my next question is: does case-sensitivity make any sense in that context? #twtxt
2016-02-13T13:15:22+01:00   @<kas https://enotty.dk/twtxt.txt> None, really. We could make it case-insensitive.

I suggest we make mention matching case-insensitive. The rationale is that it places the burden of formatting and recognizing mentions/usernames on the software, and makes it easier for end users. We should never place that burden on end users.

Add config option for 'minimum interval between checks'

During experimenting, the 'timeline' option can be run quite often though there are no changes. The cache works, but still has to ping the remote servers for no reason.

I'd like to see a config option that would let the 'timeline' command use just the cache and skip connecting to the remotes if this happens within the configured period. For parity, there would need to be a commandline option to override the behaviour.
The default could be a few seconds or a minute, whatever.

Create a Code of Conduct

A code of conduct is a common practice to foster a kind, inclusive, cooperative, and harassment-free community. Adopting a code of conduct would demonstrate that new contributors will be welcomed and respected.

Add support for microformats and/or web mentions

I think twtxt would fit well into the IndieWebCamp ecosystem, particularly with support for webmentions and microformats

Pluggable config files/Separate following list

It would be a very nice addition to add an include directive to the config file, or define an optional following file.

I realized this when I tried to use twtxt with my home computer (I started it in my work). I got my public file but there is no way to "sync" my following list. You might be thinking I should also sync my config file. But my config is osx/linux especific and my home computer is a windows one (:P).

It will be even more convenient to add "includes" so one can have platform specific settings and plug in a following section in a more 'generic' sense.

Create a package for Debian

to make it non-Python users easier to install twtxt.

Quickstart: Dialog to set the configfile filepath

SO ITS NOT OVERWRITING MY SHIT

Timeline sorting

At the moment I'm not sure if I'm happy with the default sorting order (descending) of the tweets in the timeline...

If I don't use a pager I usually want the tweets in an ascending order - the newest tweets in the bottom because these are those I care most about and I do not want to scroll to the beginning. This is different if I use a pager because these start at the top ...

What are your thoughts?
What do you think about changing the default sorting order dynamically depending on the given pager option?

sync following list

This is something I'd like to see but don't want to implement right this moment (I should be getting ready for the superbowl party I'm going to, not writing python, dammit!), but I thought I'd share to see if others had ideas for how this might work.

I use 2 computers pretty regularly, so being able to synchronize the list of folks I follow is going to be a critical feature for adoption for me :)

I had something in mind similar to pre_tweet_hook from #32 to have it pull that down when you look at timeline or follow someone, and a post hook to publish it. Currently the list of who I follow is not public, and I'm not sure if that should be (I personally don't care, but others might), or I would say just add a url to the config where my following list is published. Implementing it as a pair of hooks makes it so it can be as public or private as I want it to be.

Thoughts?

Installing not as root

Tip: Instead of installing the package globally (as root), you may want to install this package locally by passing --user to pip, but you then have to make sure that you have included ~/.local/bin/ in your $PATH. Using pyvenv and running twtxt from within a virtualenv is of course also an option!

Could you elaborate on this one for python noobs?

When I try pip3 install twtxt --user=Adam I get "--user option does not take a value"

Create a Formula for Homebrew

Don’t own a Mac any more, so I can’t test or write it myself, but it would be nice to have.

Ask to turn on disclose_identity when running quickstart

How to store metadata about a feed

A number of different issues and ideas have made clear the need for a place to specify metadata about a twtxt.txt feed. For instance, essentially every idea for notifications so far needs to know where the notifications should go (technical details vary based on the proposal). The question then is how to store metadata.

Discussion in #22 has suggested a general comment character, thus allowing clients to handle individually how the metadata would be stored. I suggest building on this, allowing for general comments, but make the following format specifically for metadata:

# this is a regular comment

# the next line is a metadata entry
# nick = benaiah

This echoes the .ini format of the twtxt config file, which I think gives it a nice consistency.

The other main suggestion for metadata is to have another file. I dislike this approach because it complicates the protocol, significantly increases how much twtxt has to hit the network, and requires either a second URL for each person (for the metadata file), switching twtxt.txt to hold metadata and having another file hold the feed, or putting a metadata entry in twtxt.txt that points to the metadata file.

Improve testing

• Increase unit test coverage
• Use tox to test against multiple python versions locally
• Implement integration and functional tests

IPFS support

IPFS seems like a neat way to share twtxt feeds. Currently it’s a bit complicated to distribute mutable content to the network, but it’s possible via ipfs publish. [1]

This should also be implemented as an optional, but official, plugin.

Regarding the implementation we could make use of IPFS’s HTTP/JSON API.

Add # for comments in Feed

Hello,

it would be nice, if the client ignores lines starting with #. This way the feed format would be extensible with e.g. the date of creation or other meta fields (e.g. # nickname dracoblue is a hint for clients, who get it).

Configurable time-out for HTTP requests

Move documentation away from README.rst

Instead use Sphinx to document twtxt.

Allow to limit timeline by timestamp

I'd like to do eg. "twtxt timeline --since yesterday", or what's new since last time I check (provided that I save the timestamp of previous check).

Decentral Solution for @mentions or search

Since everyone has his/her own twtxt.txt, we cannot support @mention like on twitter locally. Except if we follow those people already.

If we say that we have registries - custom hosted aggregators like http://twtxt.reednj.com/ - they could provide an api to get all mentions for a twtxt uri (e.g. http://example.org/twtxt.txt like suggested in #6, if a twtxt tweet looks like this: TIMESTAMP @<http://example.org/twtxt.txt> how are you

Api Response could look like this:

TIMESTAMP https://dracoblue.net/twtxt @<http://example.org/twtxt.txt> how are you

if I messaged example.org with a mention.

The advantage of this approach is, that every client can decide at which registry he is registered and which he wants to query for possible mentions. Like a federated web search ;).

UPDATE 2

An api-doc with swagger UI can be found here https://registry.twtxt.org/swagger-ui/ and a PR is made here #74

UPDATE

Implemented a server for this at https://github.com/DracoBlue/twtxt-registry - will work on a PR now.

A demo runs on https://registry.twtxt.org/api/plain/mentions?url=https://buckket.org/twtxt.txt and gives:

@<dracoblue https://dracoblue.net/twtxt.txt>    2016-02-09T12:57:59.000Z    @<buckket https://buckket.org/twtxt.txt> something like https://gitter.im/ or a freenode channel?
@<dracoblue https://dracoblue.net/twtxt.txt>    2016-02-08T22:51:47.000Z    @<buckket https://buckket.org/twtxt.txt> looks nice ;)

Discovery

A way to discover people you might want to follow would be cool.

Maybe by exposing a list of followers-of-followers?

Sort timeline in ascending or descending order

Implement a config option to switch between an ascending or descending timeline.

Resolve Nicknames, which are Domains

I host a twtxt file at https://dracoblue.net/twtxt.txt

If I reply to somebody like @buckket it's not clea if other people saved him as @buckket, too. Or a different person.

Would it be possible to have something like @example.org resolves preferably to the person who hosts https://example.org/twtxt.txt?

Hide posts from the future

At the moment I see this in my timeline:

➤ erlehmann (82 years from now):
I am also from the far future, but with another timestamp.

➤ erlehmann (82 years from now):
I am from the far future, bow before me!

since http://daten.dieweltistgarnichtso.net/tmp/docs/twtxt posted:

2099-01-01T00:00:00Z    I am from the far future, bow before me!
2099-01-01T00:00:01+00:00   I am also from the far future, but with another timestamp.

to be on top of every timeline ;).

Maybe an option to hide posts from the future would be useful?

Change config URL of source when receiving HTTP 301 status code

Low priority. And only ask when calling twtxt followings.

Tweet Spell check

I'm not particularly good at spelling, and rely heavily on spellcheck to get things right.
It should be quite simple to pipe a tweet into aspell before actually sending it, notifying the user if there are any words spelt incorrectly.

I should be able to do this myself, Just thought I'd post an issue to get suggestions before attempting it. (read - Give me a few days, I'd like to do it myself, don't do it for me!.)

Support for RSS and Atom feeds

A lot of websites already provide feeds in RSS or Atom format. Popular browsers, like Mozilla Firefox, Microsoft Internet Explorer or Safari include features to view such feeds and subscribe to them. Using the Python feedparser module twtxt could be used to subscribe to such a feed without lots of additional code.

This would also sidestep any debate regarding embedded media, metadata etc. as those who want metadata could always just provide Atom feeds (RSS is less capable). I have written a script to convert twtxt files to Atom feeds which could be patched for more metadata if so desired: http://news.dieweltistgarnichtso.net/bin/twtxt2atom

Improve documentation

I just submited a PR #53 to establish a sphinx documentation.
It would be nice to extend this documentation to cover the major features and implementation details of twtxt.