mrusme / superhighway84

USENET-inspired, uncensorable, decentralized internet discussion system running on IPFS & OrbitDB

Superhighway84

===============================================================================
                        INTERACTIVE ASYNC / FULL DUPLEX
===============================================================================

                              Dial Up To 19.2 Kbps
                                  
                                      with

    _  _ _ __ ____                  __   _      __                   ___  ____
   /  / / // / __/_ _____  ___ ____/ /  (_)__ _/ / _    _____ ___ __( _ )/ / /
  _\ _\_\_\\_\ \/ // / _ \/ -_) __/ _ \/ / _ \/ _ \ |/|/ / _ \/ // / _  /_  _/
 /  / / // /___/\_,_/ .__/\__/_/ /_//_/_/\_, /_//_/__,__/\_,_/\_, /\___/ /_/
                   /_/                  /___/                /___/

 ::: UNCENSORABLE USENET-INSPIRED DECENTRALIZED INTERNET DISCUSSION SYSTEM :::


   The V.H.S. (Very High Speed) Superhighway84 platform is more than just the
  fastest decentralized, uncensorable, USENET-inspired communications platform 
         available. It is also the first one to be based on the latest 
                        IPFS technology available today!

    Superhighway84 offers the most spectacular features under the Spectrum.
                                       
                             100% Error Protection
                         Data and Character Compression
                         Alternate Bell Compatible Mode
                         Long Haul Satellite Operation
                              Network Diagnostics
                                 Fallback Mode
                                   And More!


                    The Superhighway84 modern, uncensorable, 
                   decentralized internet discussion system.
                       It should cost a lot more than $0.

Superhighway84 is an open source, terminal-based, IPFS-powered, USENET-inspired, uncensorable, decentralized peer-to-peer internet discussion system with retro aesthetics.

More info here.

Installation

Prerequisites

Download the kubo 0.16 release and unpack it:

$ tar -xzf ./kubo_*.tar.gz

If you haven't used IPFS so far, initialize the IPFS repository using the following command:

$ ./kubo/ipfs init

If you had used IPFS an already have an IPFS repository in place, either (re)move it from ~/.ipfs or make sure to export IPFS_PATH before running the ipfs init command, e.g.:

$ export IPFS_PATH=~/.ipfs-sh84
$ ./go-ipfs/ipfs init

From Release

Download the latest release and unpack it:

$ tar -xzf ./superhighway84_*.tar.gz
$ ./superhighway84

If you initialized the IPFS repo under in a custom location, you need to prefix IPFS_PATH:

$ IPFS_PATH=~/.ipfs-sh84 ./superhighway84

The binary superhighway84 can be moved wherever you please.

From Source

Clone this repository

• from GitHub

  $ git clone [email protected]:mrusme/superhighway84.git

• from Radicle

  $ rad clone rad:z4JkpNjyUemfCeU85iqssNEukoBn1

Then cd into the cloned directory and run:

$ go build .

The binary will be available at ./superhighway84 and can be moved wherever you please.

Running

First, check ulimit -n and verify that it's at a reasonable amount. IPFS requires it to be large enough (>= 2048) in order to work properly over time.

Second, if your hardware shouldn't be a beefy computer but instead one of those flimsy MacBooks, older hardware, a Raspberry or a low-memory VPS it is advisable to set the previously created IPFS repository to the lowpower profile.

$ ipfs config profile apply lowpower

This should help with CPU usage, file descriptors and the amount of network connections. While during the startup period you might still see peers peaking between 1k and 3k, connections should ultimately settle somewhere between 100 and 300 peers.

Afterwards you can simply launch the binary:

$ superhighway84

A setup wizard will help you with initial configuration. Please make sure to have at least HOME and EDITOR exported in your environment.

In case you're intending to run the official IPFS daemon and Superhighway84 in parallel, be sure to adjust the ports in their respective IPFS repos (e.g. ~/.ipfs and ~/.ipfs-sh84) so that they won't utilize the same port numbers. The ports 4001, 5001 and 8080 are relevant and should be adjusted to something other for every new repo/IPFS node that will run in parallel, e.g.:

  "Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4002",
      "/ip6/::/tcp/4002",
      "/ip4/0.0.0.0/udp/4002/quic",
      "/ip6/::/udp/4002/quic"
    ],
    "Announce": [],
    "NoAnnounce": [],
    "API": "/ip4/127.0.0.1/tcp/5002",
    "Gateway": "/ip4/127.0.0.1/tcp/8081"
  },

NOTE: When running Superhighway84 for the first time it might seem like it's "hanging" at the command prompt. Usually it isn't hanging but rather searching for peer it can connect to in order to synchronize the database. Depending on how many people are online, this process might take some time, please be patient.

Connectivity

If you're having trouble connecting to the IPFS network that might be due to your network setup. Please try the IPFS AutoRelay feature in such a case:

$ ipfs config --json Swarm.RelayClient.Enabled true

More information on this can be found here: https://github.com/ipfs/kubo/blob/master/docs/experimental-features.md#autorelay

Configuration

Superhighway84 will guide you through the basic configuration on its first run. The configuration is stored at the path that you specified in the setup wizard. After it was successfully created, it can be adjusted manually and will take effect on the next launch of Superhighway84.

Configuration options that might be of interest:

ArticlesListView =
  The view to be used for the articles lit. Possible values:
  0 - threaded view, latest thread at the top
  1 - list view, latest article at the top

[Profile]
  From =
    The identifier that is being shown when posting an article, e.g. your name,
    username or email that you'd like to display

  Organization =
    An optional organization that you'd like to display affiliation with

[Shortcuts]
  The shortcuts for navigating Superhighway84, can be reset to its defaults by
  simply removing the whole [Shortcuts] block and launching Superhighway84

  The structure is as following:

  `<key code> = "event"`

  The key codes can be looked up under the following link:

  https://pkg.go.dev/github.com/gdamore/tcell/v2#Key

  For simple ASCII characters use their ASCII code, e.g. `114` for the character 
  `r`.

Usage

The default keyboard shortcuts are:

     C-r: Refresh
     C-h: Focus groups list
C-l, C-k: Focus articles list
     C-j: Focus preview pane
     C-q: Quit
       k: Move up in list
       j: Move down in list
       h: Move left in list
       l: Move right in list
       g: Move to the beginning of list/text
       G: Move to the end of list/text
      CR: Select item in list
       n: Publish new article
       r: Reply to selected article

However, you are free to customize these within your configuration file, under the section Shortcuts.

Submit Article

When submitting a new article or a reply to an article, the $EDITOR is launched in which a document with a specific structure will be visible. This structure consists of the HEADER, a SEPARATOR and the BODY and looks like this:

Subject: This is the subject of the article
Newsgroup: test.sandbox
= = = = = =
This is the multiline
body of the article

The HEADER contains all headers that are required for an article to be submitted. These are:

• Subject:
  The subject of the article that will be shown in the articles list. The subject must only contain of printable ASCII characters.

• Newsgroup:
  The newsgroup under which the article will be submitted, this can either be an existing group or a new group. Please try to follow the convention when creating new groups. The newsgroup must only contain of printable ASCII characters.

The SEPARATOR contains of 6 equal signs and 5 spaces, alternating each other, followed by a new line.

The BODY can contain of multiline text.

Known Limitations

• The OrbitDB that Superhighway84 uses is a public database, meaning everyone can alter its data. Since its using a standard docstore, PUT and DELETE events can alter existing data. This issue will be solved in the future by customizing the store to ignore these types of events.

• Superhighway84 is bound to the version of IPFS that Berty decides to support for go-orbit-db. go-orbit-db updates, on the other hand, seem to introduce breaking changes from time to time, which are hard to debug as someone without in-depth knowledge nor documentation. Since Superhighway84 is pretty much a one-man-show it would be quite challenging to fork go-orbit-db in order to keep it up to date with IPFS and make its interface more stable. Unfortunately there doesn't seem to be an alternative to Berty's go-orbit-db as of right now, so Superhighway84 is basically stuck with it. If you happen to know your way around IPFS and maybe even go-orbit-db, and would like to support this project, please get in touch!

• If you have a newer IPFS version installed than the one used by Superhighway84, please make sure to not upgrade the IPFS_REPO that Superhighway84 is using. Otherwise you will get an error when starting Superhighway84 that will tell you that there is an IPFS repository mismatch:

  > panic: Your programs version (11) is lower than your repos (12).
  

  If this should be the case, please follow the instructions provided here:

  #42 (comment)

• If you encounter the following issue your IPFS repo version might be older than what Superhighway84 is using:

  > panic: ipfs repo needs migration
  

  In this case you might want to follow the IPFS migration guide here:

  https://github.com/ipfs/fs-repo-migrations/blob/master/run.md

  Alternatively use the same IPFS version as used by Superhighway84 to initialize a dedicated Superhighway84 repository. Please refer to the INSTALLATION part for how to do so.

Credits

• Superhighway84 name, code and graphics by mrusme
• Logo backdrop by Swift

superhighway84's Issues

Confirmation dialog

When I finish my post, I get a confirmation dialog, but the only thing visible is 'yes' or 'no'. Maybe this should display the post text?

Since I can't see what I wrote any more, I basically always hit 'yes'.

Message read/unread status

As mentioned here it would be great if there was a way to see which messages were read already.

The easiest solution would be to simply store that information somewhere on the local disk.

Search

As the content grows, the ability to search becomes more important. Need to figure out a way to quickly fuzzy-search through all articles and display those that match the criteria.

Conversation threading

As more and more people start to use this platform (hehehe :-) 👍🏼 ) conversation threading needs to be implemented in order to enhance the reading experience.

On the back-end, this was already done, with articles having the InReplyToID field. What's left to be done is the front-end code in tui/mainscreen.go, specifically the Refresh function that renders the list.

Many posts cannot be viewed

Love this project! About a half hour after installing and running it, I still can't view a lot of the posts - specifically the older ones that from their subject lines appear to be reposts of news articles (e.g. Anti-mimetic tactics for living a counter-cultural life, Gene Therapy for a Failing Heart).

The body of these posts just shows up as "NA."

I am using the linux binary release on Ubuntu 21.04 with ipfs installed from the snap repository.

When I start the client I get the warning failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details. Maybe these are longer posts and this is somehow related?

Allow for initialization of repository on startup

As pointed out in #42, if the local ipfs version is newer than the node used by Superhighway84, then people are unable to initialize a repo for Superhighway84.

It would be helpful if there was a way to make the IPFS node used within Superhighway84 initialize its own repository in case there are problem with the exiting one. This way we wouldn't be depending on the external ipfs binary anymore and users could simply run Superhighway84 without caring too much for IPFS internals.

Save IPFS_PATH and EDITOR in config

Was wondering if it would be a simple change to add the IPFS_PATH and EDITOR variables to the config.
Saves having to export them in every new shell.

Show Poster and Subject in editor view.

Often I lose track. The article browser shows this info but if an article does not fit in the preview window I open it such that it opens in my editor. It would be nice to populate the top of the article with the poster and subject as I sometimes lose my place.

Content moderation

As this seems to be a topic many people are interested in, here's a rough idea of how this could be implemented in future. I've detailed this on Superhighway84 already and just copypasta it here for the sake of documentation.

BBS were centralized, USENET as well (in that sense that there were moderators
who could decide what goes and what not), so it was basically just a thing of
having someone around to do the job. However, even these platforms struggled
eventually, e.g.

https://www.techdirt.com/articles/20200918/15082545338/content-moderation-case-study-usenet-has-to-figure-out-how-to-deal-with-spam-april-1994.shtml

I believe a viable solution for Superhighway84 would be client-side filtering,
maybe with the help of distributed lists that people could subscribe to if they
felt like. Think of it as how e.g. uBlock Origin works in your browser.

That way users could also create their own lists and publish them for others to
subscribe to. OrbitDB allows for entities to be restricted to writing from
certain peers, meaning that these lists could be restricted to just a handful of
people that maintain them. But each user would basically have the possibility to
start their own list if they'd be dissatisfied with the filtering they get from
others.

Just an idea. Basically like reddit but you can decide on your own who you
want your platform moderators to be.

Feel free to discuss/suggest other approaches and ideas.

Database not syncing anymore since IPFS 12.2 upgrade

Not sure yet what's going on, also reported to go-orbit-db.

Feedback on navigation

I received excellent feedback on the navigation through replies on Superhighway84 (thanks for that!) and understood that these would be some great improvements (to be extended):

• Use Tab for switching between lists
• Use Ctrl-Left/-Right for switching between lists
• Use single keys over Ctrl+Key combinations for easier navigation
• Add ? key shortcut to display navigation help

Feel free to add to this list. I will start implementing a few things soon.

Mix of tabs and spaces

This codebase is a mixture of spaces and tabs.

Would you mind doing a pass and switching everything to one or the other?

(@mrusme I'm surprised you haven't noticed, I'm guessing you don't have invisibles turned on in your editor, and probably use different editors?)

I'd send a PR myself, but I don't want to 'steal' all of these LOCs from you! 🙂

$ grep -e  '^\t' *.go */*.go | wc -l
     245
$ grep -e  '^  ' *.go */*.go | wc -l
    1257

Preview pane: Color for quoted text, detect position of reply

It would probably increase readability if quoted text in the preview pane was a different color.

Also it was suggested to maybe try and detect where the actual reply is and automatically scroll to it in the preview. Not quite sure if that really helps with readability or confuses more than it should though.

Scorefiles

I got an interesting suggestion on reddit:

Biggest immediate question: Can it interoperate with Gnus and scorefiles?

(The things I most miss from Usenet were literally the tools that let you make it reflect your interests and needs and not what someone else thought you ought to prefer.)

I haven't used the mentioned method so far, but maybe it makes sense to incorporate it as part of #19 and the overall subscribing/unsubscribing features that are still due to be implemented.

superhighway84 hangs on first launch / can't seem to retrieve default database

I tried with the binary version and one built from git.

my system is amd64, ipfs version 0.10.0, go version go1.17.5 linux/amd64

Seems to be locked up on Windows? Never fetches groups

Have installed Kubo 0.16 as only system install of IPFS, and SH84 v0.2.4, and trying to use it with Powershell 7 x64 7.3.4.

Sadly it just sits and looks like this, for hours:

I left it overnight and it's still stuck. Ctrl+r makes the little thing at the bottom left green for a second, and then back to grey. Pressing r without ctrl produces the below output:

panic: runtime error: index out of range [0] with length 0 [recovered]
        panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/rivo/tview.(*Application).Run.func1()
        /home/runner/go/pkg/mod/github.com/rivo/[email protected]/application.go:243 +0x4d
panic({0x212d1a0, 0xc0084266f0})
        /opt/hostedtoolcache/go/1.18.9/x64/src/runtime/panic.go:838 +0x207
github.com/mrusme/superhighway84/tui.(*Mainscreen).HandleInput(0xc00007b180, 0x285beb8?)
        /home/runner/work/superhighway84/superhighway84/tui/mainscreen.go:355 +0x4e5
github.com/mrusme/superhighway84/tui.(*TUI).initInput.func1(0xc006812b40)
        /home/runner/work/superhighway84/superhighway84/tui/tui.go:143 +0x13e
github.com/rivo/tview.(*Application).Run(0xc0003b8000)
        /home/runner/go/pkg/mod/github.com/rivo/[email protected]/application.go:318 +0x782
github.com/mrusme/superhighway84/tui.(*TUI).Launch(0xc00007af00)
        /home/runner/work/superhighway84/superhighway84/tui/tui.go:165 +0xa5
main.main()
        /home/runner/work/superhighway84/superhighway84/superhighway84.go:144 +0x95b

I love the idea of the project and I'm happy to do some work. How can I start to debug it?

Limit number of connections

It has been reported that the number of connections IPFS creates makes it tricky to run Superhighway84 on especially older hardware.

It was mentioned that apparently this is a known issue in IPFS:

The libp2p team is currently refactoring the "dialer" system in a way that'll make it easy for us to configure a maximum number of outbound connections. Unfortunately, there's really nothing we can do about inbound connections except kill them as soon as we can. On the other hand, having too many connections usually comes from dialing.

I couldn't find out what the status of the libp2p refactoring is though. However, people are mentioning that disabling QUIC in the IPFS repository has helped a bit with the issue, e.g.:

... ipfs init --profile server, and (3) I removed the lines of quic under Addresses.Swarm in ~/.ipfs/config.

and

I fust followed the advice here of disabling QUIC
ipfs config --json Swarm.Transports.Network.QUIC false

Another idea could be to try and press Superhighway84 into a set of iptables rules that limit its available bandwidth and the amount of connections artificially. I didn't investigate whether that really works, but apparently other users have done that for IPFS.

Feature Request: Show group name when browsing `*`

When browsing *, the threads should indicate which group they belong to. That way you know where to go to see more like that.

Unable to open database

Hi,

I cannot get superhighway84 to go thru the first run. This is what gets output:

2022/05/30 19:42:53 initializing logger ...
2022/05/30 19:42:53 initializing cache ...
2022/05/30 19:42:53 initializing TUI and loading database, please wait ...
2022/05/30 19:42:53 initializing database ...
2022/05/30 19:42:54 connecting database ...
2022/05/30 19:52:54 unable to open database: unable to fetch database manifest: failed to get block for bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4: context deadline exceeded
panic: unable to open database: unable to fetch database manifest: failed to get block for bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4: context deadline exceeded

	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1292fe7]

goroutine 1 [running]:
github.com/mrusme/superhighway84/database.(*Database).Disconnect(0xc00218d2c0)
	/home/runner/work/superhighway84/superhighway84/database/database.go:224 +0x27
panic({0x13e68e0, 0xc004195c40})
	/opt/hostedtoolcache/go/1.18.1/x64/src/runtime/panic.go:838 +0x207
log.Panicln({0xc0055ebf48?, 0xc002030e70?, 0x1832397?})
	/opt/hostedtoolcache/go/1.18.1/x64/src/log/log.go:399 +0x65
main.main()
	/home/runner/work/superhighway84/superhighway84/superhighway84.go:124 +0x8dd

The log file says:

2022-05-30T20:32:19.632+0200	DEBUG	database/database.go:148	getting config root path ...
2022-05-30T20:32:19.634+0200	DEBUG	database/database.go:154	setting up plugins ...
2022-05-30T20:32:19.634+0200	DEBUG	database/database.go:159	creating IPFS node ...
2022-05-30T20:32:20.287+0200	INFO	database/database.go:171	connecting to peers ...
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:124	connected!	{"peerID": "QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"}
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:124	connected!	{"peerID": "QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"}
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:124	connected!	{"peerID": "QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"}
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:124	connected!	{"peerID": "QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"}
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:124	connected!	{"peerID": "QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ"}
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:177	connected to peer!
2022-05-30T20:32:20.635+0200	INFO	database/database.go:182	initializing database connection ...
2022-05-30T20:32:20.635+0200	DEBUG	database/database.go:51	initializing NewOrbitDB ...
2022-05-30T20:32:20.706+0200	DEBUG	database/database.go:79	initializing OrbitDB.Docs ...
2022-05-30T20:32:20.706+0200	DEBUG	baseorbitdb/orbitdb.go:531	open orbitdb store	{"address": "/orbitdb/bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4/superhighway84"}
2022-05-30T20:32:20.707+0200	DEBUG	baseorbitdb/orbitdb.go:553	Open database 	{"dbAddress": "/orbitdb/bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4/superhighway84"}
2022-05-30T20:32:20.707+0200	DEBUG	baseorbitdb/orbitdb.go:560	Look from 	{"directory": "/home/milan/.cache/superhighway84/database"}
2022-05-30T20:32:20.707+0200	DEBUG	baseorbitdb/orbitdb.go:575	address '/orbitdb/bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4/superhighway84' is valid
2022-05-30T20:32:20.707+0200	DEBUG	cacheleveldown/leveldown.go:88	opening cache db	{"path": "/home/milan/.cache/superhighway84/database/bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4/superhighway84"}
2022-05-30T20:52:20.636+0200	ERROR	database/database.go:186	%s	{"error": "unable to open database: unable to fetch database manifest: failed to get block for bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4: context deadline exceeded", "errorVerbose": "failed to get block for bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4: context deadline exceeded\nunable to fetch database manifest\nberty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Open\n\t/home/milan/go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:597\nberty.tech/go-orbit-db.(*orbitDB).Docs\n\t/home/milan/go/pkg/mod/berty.tech/[email protected]/orbitdb.go:143\ngithub.com/mrusme/superhighway84/database.(*Database).init\n\t/home/milan/work/superhighway84-0.2.0/database/database.go:80\ngithub.com/mrusme/superhighway84/database.(*Database).Connect\n\t/home/milan/work/superhighway84-0.2.0/database/database.go:184\nmain.main\n\t/home/milan/work/superhighway84-0.2.0/superhighway84.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581\nunable to open database\nberty.tech/go-orbit-db.(*orbitDB).Docs\n\t/home/milan/go/pkg/mod/berty.tech/[email protected]/orbitdb.go:145\ngithub.com/mrusme/superhighway84/database.(*Database).init\n\t/home/milan/work/superhighway84-0.2.0/database/database.go:80\ngithub.com/mrusme/superhighway84/database.(*Database).Connect\n\t/home/milan/work/superhighway84-0.2.0/database/database.go:184\nmain.main\n\t/home/milan/work/superhighway84-0.2.0/superhighway84.go:108\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:255\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1581"}
github.com/mrusme/superhighway84/database.(*Database).Connect
	/home/milan/work/superhighway84-0.2.0/database/database.go:186
main.main
	/home/milan/work/superhighway84-0.2.0/superhighway84.go:108
runtime.main
	/usr/local/go/src/runtime/proc.go:255

The database LOG file in .cache says:

=============== May 30, 2022 (CEST) ===============
19:42:54.143405 log@legend F·NumFile S·FileSize N·Entry C·BadEntry B·BadBlock Ke·KeyError D·DroppedEntry L·Level Q·Se
qNum T·TimeElapsed
19:42:54.161259 db@open opening
19:42:54.161488 version@stat F·[] S·0B[] Sc·[]
19:42:54.168994 db@janitor F·2 G·0
19:42:54.169012 db@open done T·7.728795ms

This is version 0.2.0 and IPFS is 0.12.2 running on Ubuntu 20.04.4.
I ran ipfs init and accepted all the default values offered by superhighway84.

I've tried everything I could find without effect:

1. Setting ulimit to 2048
2. Setting net.core.rmem_max to 2500000
3. Setting ipfs lowpower
4. Also tried it on macOS
5. Tried the previous version with older IPFS, also unsuccessful
6. Tried compiled and prebuilt
7. Tried increasing the timeouts in database.go from 600 to 1200

What else can I try? I would appreciate some hints.

Thnx.

unable to close emitter {"error": "closed an emitter more than once"}

2022-05-29T01:00:24.304-0500	WARN	replicator/replicator.go:120	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/replicator.(*replicator).Stop
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/replicator/replicator.go:120
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:309
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	replicator/replicator.go:120	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/replicator.(*replicator).Stop
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/replicator/replicator.go:120
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:309
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	replicator/replicator.go:120	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/replicator.(*replicator).Stop
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/replicator/replicator.go:120
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:309
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250
2022-05-29T01:00:24.305-0500	WARN	basestore/base_store.go:319	unable to close emitter	{"error": "closed an emitter more than once"}
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:319
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
	/home/mrus/.go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
	/home/mrus/projects/@mrusme/superhighway84/database/database.go:226
main.main
	/home/mrus/projects/@mrusme/superhighway84/superhighway84.go:144
runtime.main
	/usr/lib/go/src/runtime/proc.go:250

../../go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/load_unix.go:8:2: package plugin is not in GOROOT (/usr/src/plugin)

╰─ go build .              
../../go/pkg/mod/github.com/ipfs/[email protected]/plugin/loader/load_unix.go:8:2: package plugin is not in GOROOT (/usr/src/plugin)

Getting the following error when building.

Error reporting update available

stuck on "connecting database..."

So after a long process of attempting to get this software to run, raising my send buffer to 2500000 and dealing with IPFS versions and what not, upon launch of the superhighway84 binary it gets stuck on "connecting database..." I have no idea what to make of this, i was able to sort out all the other issues with my system and IPFS and their dependancy shenanigans but I have no information to work with beyond what the superhighway84 prompt is giving me upon initialization.

Thanks.

Implement "factory reset"

With some users unable to sync the default database instantly ( #1 ) and hence likely to test and maybe break things, a factory reset would make sense.

The factory reset should offer to delete the OrbitDB cache, as well as the IPFS repo.

Reformat line length

Not sure how others feel about this, but maybe Superhighway84 should enforce a line length (e.g. 72 or 80) by simply reformatting the text upon submission.

Message view pane

It was pointed out that an integrated message view pane would increase comfort, which I agree with. For the first shot $EDITOR was sufficient, but there needs to be a message view integrated into Superhighway84.

failed to enable TCP keepalives

This appeared after I did a successful? post.

2021-12-27T22:57:55.466-0500u knERROR	tcp-tpt	[email protected]/tcp.go:52	failed to enable TCP keepalivemes.mor{"error": "set tcp4 192.168.1.196:4001->135.125.8.109:57940: setsockopt: invalid argument"}

Running MacOS on MacBook Air, Apple Silicon.

Unable to use absolute path to log file on Windows

Summary

Setting an absolute path to the log file (the default) prevents the application from launching on Windows.

Steps to Reproduce

Environment: superhighway84 v0.0.4 on Windows build 22000.348

1. Execute superhighway84.exe
2. Accept the defaults for the Initial Setup prompts
3. Specify a From name and accept other profile defaults

Expected Result

The application launches.

Actual Result

The application crashes with the following message:

SUPERHIGHWAY84

Initial Setup
-------------

Database connection string [/orbitdb/bafyreifdpagppa7ve45odxuvudz5snbzcybwyfer777huckl4li4zbc5k4/superhighway84]:
Database cache path [C:\Users\username/.cache/superhighway84]:
Logfile path [C:\Users\username/.cache/superhighway84.log]:

Profile information
-------------------

From [@localhost]: username123
Organization []:
2021/12/28 00:45:28 couldn't open sink "C:\\Users\\username/.cache/superhighway84.log": no sink found for scheme "c"
panic: couldn't open sink "C:\\Users\\username/.cache/superhighway84.log": no sink found for scheme "c"


goroutine 1 [running]:
log.Panicln({0xc00171ff50, 0x28, 0x29c6847bb11})
        /opt/hostedtoolcache/go/1.17.5/x64/src/log/log.go:368 +0x65
main.main()
        /home/runner/work/superhighway84/superhighway84/superhighway84.go:42 +0x1a5

Workaround

Set a relative path for the log file.

Unfortunately, this is not a permanent fix as the log file will be recreated several times based on the working directory from which the application was launched.

Notes

This seems related to parsing Windows paths in the logging module.

failed to enable TCP keepalivemes

This appeared after I did a successful? post.

2021-12-27T22:57:55.466-0500u knERROR	tcp-tpt	[email protected]/tcp.go:52	failed to enable TCP keepalivemes.mor{"error": "set tcp4 192.168.1.196:4001->135.125.8.109:57940: setsockopt: invalid argument"}

Running MacOS on MacBook Air, Apple Silicon.

"too many open files"

I see one post to alt.social, but soon afterwards the system crashes with

ERROR┐╔═blockstore══════[email protected]/blockstore.go:218═════blockstore.AllKeysChan got err: walk failed: open /Users/emv/.ipfs-sh84/blocks: too many open files

Error on Blank Post

Minor issue, when no post is made, ie. make new post and don't save on exit, it goes to an error.
It should simply go back to the tui view.

Run `gofmt` so your PR's don't have formatting issues

Most people use plugins for their text editors that run gofmt automatically (for example this is default behavior of vim-go as well as vscode's go plugin), which means that if I submit a PR, it is going to be 90% formatting changes.

I would consider running gofmt yourself on all your go files so that people don't have to struggle with this.

Redraw TUI shortcut

Occasionally errors will post log messages to the terminal. This writes over the top of the TUI.

Sometimes moving between threads cases a redraw of that area but not always.

It would be handy if refresh (or some other shortcut) allowed a total TUI redraw to clean up after this happens.

I don't have a clearer screenshot as an example but the one attached shows some date info after the download total due to an error message that was printed.

.

Add Dockerfile

This project should contain a Dockerfile that can be run by anyone who's not keen running it natively. Since it's a TUI, it should be possible to run it attached.

Running it in a container also has the benefit that it allows people to restrict IPFS' resource use even further and more strict.

Unresponsive after return from editor on Windows

Summary

The application is unresponsive after returning from the editor.

Steps to Reproduce

Application v0.0.4
Windows build 22000.348

1. Launch the application
2. Open a post
3. Close the editor application
4. Navigate to the next post

Expected Result

Navigation to the next post is possible.

Actual Result

No keyboard commands have any effect.

Notes

I've tried setting several different editors without any change in behavior. No keyboard commands seem to work, including the quit command. Force-quitting and re-launching the application restores functionality until a post is opened.

CPU Usage

On my mac superhighway84 slams all the cores on startup for a few minutes, then ramps down to just using one entire core. This seems ...suboptimal.

If there's anything I can do to help debug let me know.

Can people get messages on v0.2?

I wasn't getting any messages with the new update.
I checked my logs and saw this occasionally,

2022-05-08T16:16:14.384-0700 WARN replicator/replicator.go:120 unable to close emitter {"error": "closed an emitter more than
once"}
berty.tech/go-orbit-db/stores/replicator.(*replicator).Stop
/home/anon/go/pkg/mod/berty.tech/[email protected]/stores/replicator/replicator.go:120
berty.tech/go-orbit-db/stores/basestore.(*BaseStore).Close
/home/anon/go/pkg/mod/berty.tech/[email protected]/stores/basestore/base_store.go:309
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).closeAllStores
/home/anon/go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:208
berty.tech/go-orbit-db/baseorbitdb.(*orbitDB).Close
/home/anon/go/pkg/mod/berty.tech/[email protected]/baseorbitdb/orbitdb.go:478
github.com/mrusme/superhighway84/database.(*Database).Disconnect
/zmedia/scratch/software/github/superhighway84/database/database.go:226
main.main
/zmedia/scratch/software/github/superhighway84/superhighway84.go:144
runtime.main
/usr/lib/go-1.17/src/runtime/proc.go:255

Although, I'm not seeing it on other machines.

Trustless gateways

I was reading your recent blog post and saw you mentioned that open files were causing memory issues, just wanted to open this issue about considering using Trustless IPFS gateways for fetching content as a way to reduce memory consumption from clients.

By using cid.ipfs.gateway.tld/?format=car, you can fetch a CAR file that allows "light IPFS clients" to fetch content without needing to worry about gateways mistakenly or maliciously giving the wrong content (since the cid won't match the root CID in the CAR file if it's been modified).

New IPFS Repo Version

go-ipfs v0.12 was released. This introduced repo version 12. I upgraded IPFS, upgraded my repo, but now superhighway84 doesn't like it anymore, which is understandable:

panic: Your programs version (11) is lower than your repos (12).
Please update ipfs to a version that supports the existing repo, or run
a migration in reverse.

See https://github.com/ipfs/fs-repo-migrations/blob/master/run.md for details.


goroutine 1 [running]:
log.Panicln({0xc001d59f48, 0xc0000ec400, 0xc000181500})
        /opt/hostedtoolcache/go/1.17.6/x64/src/log/log.go:368 +0x65
main.main()
        /home/runner/work/superhighway84/superhighway84/superhighway84.go:87 +0x5e6

What's the plan? I'm using a separate IPFS_PATH for superhighway84 with repo version 11 now, which works, but I guess we'll have to upgrade at some point...

Themes / color adjustments

Image for reference

Upgrade to quic-go 0.28 to fix building with Go 1.19

It currently can't build if you're on Go 1.19 -- here's the issue in question quic-go/quic-go#3491

Integrate with IPFS file management to share large content

is there a solution to "Key: 'Article.Body' Error:Field validation for 'Body' failed on the 'max' tag"?

thank you

[Bug] "initializing TUI and loading database, please wait ..." wait forever

$ ipfs version
ipfs version 0.12.2

$ git clone --depth 1 https://github.com/mrusme/superhighway84
$ cd superhighway84
$ nano go.mod        #modify ipfs version from "github.com/ipfs/go-ipfs v0.11.0" to "github.com/ipfs/go-ipfs v0.12.2"
$ go mod tidy
$ go build .
$ killall ipfs
$ ./superhighway84
2022/04/16 00:55:38 loading configuration ...
2022/04/16 00:55:38 initializing logger ...
2022/04/16 00:55:38 initializing cache ...
2022/04/16 00:55:38 initializing TUI and loading database, please wait ...

# It seems it will wait forever

Reply notifications

It would be great to have some notification list that lists e.g. replies to own articles or replies to articles one was participating in.

Beginner app development

Hey community, i hope everybody is doing fine, i am looking some assistance in creating the first application through github using codes for my newly launched beauty business. This app would be blogging, articles as well as the products all in one place.

Looking forward to the responses.

Thanks.

Validate reply body to contain an actual reply rather than just quoted text

Superhighway84 needs to validate replies in order to validate they not only contain lines starting with >, as it seems that sometimes people submit replies without an actual reply and the validations in place won't hinder them to do so.