nwtgck / piping-server

Infinitely transfer between every device over pure HTTP with pipes or browsers

Piping Server

Infinitely transfer between every device over HTTP/HTTPS

Transfer

Piping Server is simple. You can transfer as follows.

# Send
echo 'hello, world' | curl -T - https://ppng.io/hello

# Get
curl https://ppng.io/hello > hello.txt

Piping Server transfers data to POST /hello or PUT /hello into GET /hello. The path /hello can be anything such as /mypath or /mypath/123/. A sender and receivers who specify the same path can transfer. Both the sender and the recipient can start the transfer first. The first one waits for the other.

You can also use Web UI like https://ppng.io on your browser. A more modern UI is found in https://piping-ui.org, which supports E2E encryption.

Stream

The most important thing is that the data are streamed. This means that you can transfer any data infinitely. The demo below transfers an infinite text stream with seq inf.

Ideas

Piping Server is designed based on the ideas as follows.

• Infinite transfer: You can transfer any kind of data infinitely on a stream. Streams are very efficient in terms of both time and space.
• Zero installation: All you need is to have either a Web browser or curl, which are widely pre-installed. You do not need to install any extra software.
• Simpleness: Making simple makes it more secure.
• Storageless: The server makes transfer more secure since the server never stores your data.
• Purity: The server streams over pure HTTP, which makes integration easier with other softwares.
• Engineer friendly: Also designed for Unix/Linux users, who use pipes, not only for Web browser users.

Applications

Any data such as text streams, video streams and protocols can be streamed over Piping Server. Here are applications that fully use the power of pure HTTP.
See: "The Power of Pure HTTP – screen share, real-time messaging, SSH and VNC"

Text stream chat	Screen share	Drawing share

E2E encryption file transfer	SSH on Web browser	VNC on Web browser

The most important thing is that Piping Server stays simple. The applications use Piping Server as a core of data communication. It transfers data to POST /thepath into GET /thepath streamingly. The stream makes real-time communications over every device possible.

See "Ecosystem around Piping Server · nwtgck/piping-server Wiki" to find more about softwares using Piping Server.

Power of HTTP

In my experiment, Piping Server transferred 1,110TB (≈ 1PB) in a single HTTP request for 64 days and 2 hours at least. This means that it can transfer huge data and keep a request for about 2 months.

Engineer-friendly help

Get help and version only with curl.

curl https://ppng.io/help

curl https://ppng.io/version

Transfer to multiple receivers

You can transfer to multiple receivers. In the demo below, query parameter ?n=3 is specified to allow three receivers.

Public servers

Here are public Piping Servers.

• https://ppng.io
• https://piping.nwtgck.repl.co
• https://piping.glitch.me
• https://piping-47q675ro2guv.runkit.sh

Self-host on Docker

Run a Piping Server on http://localhost:8080 as follows.

docker run -p 8080:8080 nwtgck/piping-server

Run a server in background and it automatically always restarts.

docker run -p 8080:8080 -d --restart=always nwtgck/piping-server

Self-host using portable binaries

Run a Piping Server on http://localhost:8080 as follows on Linux.

curl -L https://github.com/nwtgck/piping-server-pkg/releases/download/v1.12.0/piping-server-pkg-linuxstatic-x64.tar.gz | tar xzvf -
./piping-server-pkg-linuxstatic-x64/piping-server --http-port=8080

The binaries are for multi-platform including Linux on x64, Linux on ARM64, Linux on ARMv7, Intel macOS, Apple Silicon macOS and Windows on x64. See https://github.com/nwtgck/piping-server-pkg to run on the other platform.

Self-host on free services

Here are easier ways to own your Piping Server.

• Remix on Glitch https://glitch.com/~piping
• Clone on RunKit: https://runkit.com/nwtgck/piping
• Fork on Repl.it: https://repl.it/@nwtgck/piping

See "Ecosystem around Piping Server · nwtgck/piping-server Wiki" to find more about self-hosting.

Headers passed from sender to receivers

Here are headers passed to receivers.

• Content-Type
• Content-Length
• Content-Disposition
• X-Piping: custom header whose value is freely specified by the sender

Server options

Here is available CLI options by piping-server --help.

Options:
  --help          Show help                                            [boolean]
  --version       Show version number                                  [boolean]
  --host          Bind address (e.g. 127.0.0.1, ::1)                    [string]
  --http-port     Port of HTTP server                            [default: 8080]
  --enable-https  Enable HTTPS                        [boolean] [default: false]
  --https-port    Port of HTTPS server                                  [number]
  --key-path      Private key path                                      [string]
  --crt-path      Certification path                                    [string]

Piping Server written in Rust

Piping Server is also developed in Rust.
https://github.com/nwtgck/piping-server-rust

piping-server's Issues

Request: option to use subdomains

Hi, I'd like to use subdomains like this: some_name.ppng.io
I guess this can be done by a simple proxy, but maybe there is general interest to have this functionality built into piping-server.

Unable to play video streamed using piping server on safari.

[INFO] Waiting for 1 receiver(s)...
[INFO] A receiver was connected.
[INFO] Start sending to 1 receiver(s)!
[INFO] All receiver(s) was/were closed halfway.

Streaming images

Thank you for the very interesting project!

Can the Piping Server run on iOS Device?

Can the Piping Server run on iOS Device?

Feature request: write url to clipboard after click "Send"

After click the button Send, write the URL https://ppng.io/test to OS clipboard.

Need full nginx or caddy reverse proxy example conf with ssl termination for piping-server

Need full nginx or caddy reverse proxy example conf with ssl termination for piping-server.

I can't find any official documentation for this. Please help. Thanks.

Action required: Greenkeeper could not be activated 🚨

🚨 You need to enable Continuous Integration on Greenkeeper branches of this repository. 🚨

To enable Greenkeeper, you need to make sure that a commit status is reported on all branches. This is required by Greenkeeper because it uses your CI build statuses to figure out when to notify you about breaking changes.

Since we didn’t receive a CI status on the greenkeeper/initial branch, it’s possible that you don’t have CI set up yet. We recommend using Travis CI, but Greenkeeper will work with every other CI service as well.

If you have already set up a CI for this repository, you might need to check how it’s configured. Make sure it is set to run on all new branches. If you don’t want it to run on absolutely every branch, you can whitelist branches starting with greenkeeper/.

Once you have installed and configured CI on this repository correctly, you’ll need to re-trigger Greenkeeper’s initial pull request. To do this, please click the 'fix repo' button on account.greenkeeper.io.

How do I cancel a request?

Can I use piping server to stream image

I have a local rtsp connection. I want to publish rtsp to the internet, It is always in listening mode, wait for other receiver connecting to. Can this be done in piping server?

nginx. stream cuts off after a certain point

i am using a portable bin and set up nginx as a proxy

when trying to read an infinite stream from the pipe server it cuts off at a certain point
seq inf only return up to about 393936. sends for less then a second

receiver error curl: (18) transfer closed with outstanding read data remaining
sender error curl: (55) OpenSSL SSL_write: Broken pipe, errno 32

2022/02/25 23:16:12 [error] 433215#433215: *288 upstream timed out (110: Connection timed out) while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "PUT /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"

2022/02/25 23:16:12 [error] 433215#433215: *290 upstream prematurely closed connection while reading upstream, client: 72.178.17.200, server: x.grimmygrom.com, request: "GET /testing HTTP/1.1", upstream: "http://127.0.0.1:8080/testing", host: "x.grimmygrom.com"

my nginx config is like this

server {

        server_name x.grimmygrom.com;
        client_max_body_size 0;
        proxy_http_version      1.1; # default: 1.0, Streaming (chunked transfer encoding) requires at least 1.1.
        proxy_buffering         off;
        proxy_request_buffering off;

        location / {
                proxy_pass http://127.0.0.1:8080/;
        }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/x.grimmygrom.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/x.grimmygrom.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {

    if ($host = x.grimmygrom.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        server_name x.grimmygrom.com;
        listen 80;
    return 404; # managed by Certbot
    
}

video of me showing the problem
https://youtu.be/WyqEju096so

thank you

[ERROR] The number of receivers has reached limits

An error occurred in my long-running script, what could be the cause?

Script look like this:

# ...

while true; do
    # ...
    curl --silent --max-time 3600 "${piping_server}/${piping_path}"
    # ...
    sleep 1
    # ...
done

# ...

Error when trying to pipe large binary file

Hi there,

I'm getting an error when trying to pipe a binary file (a 3D model in .glb). Here is the server error:

[ERROR] default - on uncaughtException Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at new NodeError (node:internal/errors:405:5)
at ServerResponse.end (node:_http_outgoing:1017:15)
at PassThrough.<anonymous> (/app/dist/src/piping.js:315:40)
at PassThrough.emit (node:events:514:28)
at errorEventQueue (/app/node_modules/multiparty/index.js:645:18)
at handleError (/app/node_modules/multiparty/index.js:211:9)
at IncomingMessage.onReqAborted (/app/node_modules/multiparty/index.js:197:5)
at IncomingMessage.emit (node:events:514:28)
at IncomingMessage._destroy (node:_http_incoming:224:10)
at _destroy (node:internal/streams/destroy:109:10) {
code: 'ERR_STREAM_WRITE_AFTER_END'
}

The file is 47 MB.

What I'm doing to get there:

• Reading a file from disk
• Creating a blob from this file
• Putting the blob in a form data
• Sending the form data with a POST request

Let me know if you need more details.

Environment:
Deployed from the Dockerfile on Railway.app

Screensharing firefox “text/plain” is not supported.

At first - nice application!

I try to screen share like the article below explained, but i get an error message in Firefox.

https://dev.to/nwtgck/the-power-of-pure-http-screen-share-real-time-messaging-ssh-and-vnc-5ghc

HTTP “Content-Type” of “text/plain” is not supported. Load of media resource http://****:8181/meintest2 failed.

Using this HTML code:
<!-- viewer --> <meta charset="UTF-8"> <video src="http://*********:8181/meintest2" autoplay muted></video>
Have someone an idea?

[Suggestion/Idea] piping-serial-web

hi/hello is possible to make (after chat, phone, etc) for serial port?

same method for draw chat etc but with a serial port, the chorme browser is supported

the steps is for two computer with serial port ::::::

1. plug serial port
2. use this webpage
3. if complet, click on connect and choose port.

this useful to connect remote serial, so far!

Newa Notre Maitre.

Add 'engine' field in package.json

The CI files and others seem to follow the LTS, but it was unclear.

https://docs.npmjs.com/cli/v6/configuring-npm/package-json#engines

Too early to destroy?

piping-server/src/piping.ts

Line 472 in 0095a03

receiver.res.connection.destroy();

At this point, I think some readers may still be reading.
Client often fails with error like:
"transfer closed with outstanding read data remaining."

Problem with video winking when working with docker

When raising the server through the docker, I find a problem. Constant" flashing " of the video.
At the same time, if you use the servers from your example, then there is no such problem.

Only 20MB of RAM is used (16 gigabytes of gigabytes are available) and 0.1% of the CPU capacity.

Can you tell me how I can solve the problem and what capacities are on your servers?

Link screen - https://i.ibb.co/MNLCqCq/19-07-2021-23-30-45.gif

Pipi

command line "http basic auth" support

Please add http basic auth to piping-server command line.

The lastest version of piping-server must use with nginx or caddy to enable http basic auth.

Thanks.

Listen uncaughtException

code: https://github.com/nwtgck/http-knocking/blob/84845b5954c71fecb81ce05d8138aa15f545aea5/src/main.ts#L246

Can "piping server" be used as a websocket alternative?

Let me start by saying I really like your project and the approach.

I was wondering if this idea could be used as a websocket alternative and how would one do that?

Request: support defining the options in a .env file

Hey there!

First of all, thank you for this great project. I want to deploy my own instance on Railway.app (similar to Heroku). I choose the option to deploy from a GitHub repo directly. It apparently uses the Dockerfile in the project and deploys the thing. It all starts to fall apart when Railway tries to launch the server: the port should not be defined manually but instead Railway should inject it when starting the server.

Right now I see no ability to get this port variable outside of the args from the start command. Is there any chance that it exists and I didn't notice?

Self hosted piping with ui

I am very interested by your project. Especially thinking about using it to transfer files between my local network devices.
Is there a docker or something for the server + piping ui ?
Thanks

Hosting piping-server on Dokku

I'm trying to setup piping-server in Dokku (Digital Ocean) for use it with piping-vnc-web

After deploy, the piping-server is working:

• I can successfully browse http://my_dokku_ip

• I can successfully exchange ASCII text with:

shell1:     curl -sSN http://my_dokku_ip/test
shell2:    echo "this is a test" | curl -sSNT - http://my_dokku_ip/test

But piping-vnc-web is not working.
The log I get in Dokku is:

2022/05/17 08:51:19 [error] 351780#351780: *37 upstream timed out (110: Connection timed out) while reading response header from upstream, client: client_ip, server: , request: "GET /s HTTP/1.1", upstream: "http://docker_internal_ip:5000/s", host: "my_dokku_ip", referrer: "http://localhost/"

If I try the same local machine setup and use the official server ppng.io, everything work well.
So there must be something wrong with my Dokku setup.

My local machine setup is as follow

• piping-vnc-web is served on my development machine on IIS 10
• vnc.html page is open in Chrome
• on a shell I have the following cmd:
  curl -sSN http://my_dokku_ip/c | ncat 192.168.0.90 5900 | curl -sSNT - http://my_dokku_ip/s

My Dokku nginx config is as follow:

server {
listen [::]:80;
listen 80;

access_log /var/log/nginx/a-access.log;
error_log /var/log/nginx/a-error.log;

location / {

gzip on;
gzip_min_length  1100;
gzip_buffers  4 32k;
gzip_types    text/css text/javascript text/xml text/plain text/x-component application/javascript application/x-javascript application/json application/xml  application/rss+xml font/truetype application/x-font-ttf font/opentype application/vnd.ms-fontobject image/svg+xml;
gzip_vary on;
gzip_comp_level  6;

proxy_pass  http://a-5000;
proxy_http_version 1.1;
proxy_read_timeout 60s;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $http_connection;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Request-Start $msec;

}
include /home/dokku/a/nginx.conf.d/*.conf;

error_page 400 401 402 403 405 406 407 408 409 410 411 412 413 414 415 416 417 418 420 422 423 424 426 428 429 431 444 449 450 451 /400-error.html;
location /400-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}

error_page 404 /404-error.html;
location /404-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}

error_page 500 501 502 503 504 505 506 507 508 509 510 511 /500-error.html;
location /500-error.html {
root /var/lib/dokku/data/nginx-vhosts/dokku-errors;
internal;
}

}

upstream a-5000 {

server 172.17.0.2:5000;
}

Any help would be appreciated.
Thank You

how it works

How are files temporarily stored during the file transfer period ?

1. Is there a expiry time. If Yes, what is the expiry time for those files. Can we change that time?

2. And is there any limit we can set on cache files storage?

How to run piping-server with http disabled and https only?

How to run piping-server with http disabled and https only? Thanks.

How to use piping-server to foward TCP port with socat/netcat

I know that piping-server can for foward VNC stream with special NoVNC. But how to do with common TCP stream ? Thanks

Variable receivers count

Hi there!

Thank you for this nice tool!
I see how it can be handy for connecting decoupled or closed ecosystems.

While I was imagining this tool used to push some webhooks info to CD pipelines I wondered how easy it would be to have a variable number of receivers.
I'm not talking about allowing new receivers once the sender started to send but only allowing new receivers while the sender didn't connect it and then making it send right away in that particular case.
I tried to hack it and it seems this is not quite complicated. would you be open to having a PR to support this?

Add /version and /help to README.md

Proposal

/version routing and /help routing should be added to README.md

Test for multipart

#6 doesn't care about test of multipart.
Working test was done by human. This should be automated.

Handling delay when Video Streaming (live) from readableStream with piping server for screen sharing

Hi nwtgck,

I've come across your project which was really intriguing to handle live streaming an app window over HTTP(s?) to another browser. But in my own experience with my own code I'm experiencing an issue where the video on the receiving end is X seconds behind the video on the sender side. For a live stream I want the latest available frame to be seen. I thought it was a buffering/backpressure issue at first with my own code.

But even when using piping-server and the "friend-password (proof of concept for screen sharing with readable stream" repository which also builds something similar the delay is there. Do you have any insight as to how we can improve the delay/configuration when handling live streams, for regular video streaming this wouldn't be an issue, but for a screen sharing application this is quite problematic.

https://github.com/ci7lus/friend-password

Migrate TSLint to ESLint

References

• palantir/tslint#4534

.github/workflows/ci.yml: 'version' has been deprecated

Replace `url.resolve` and `url.parse` which were deprecated since v11.0.0 with the WHATWG URL API

piping-server/src/piping.ts

Lines 105 to 107 in 6b58205

	url.resolve(
	"/",
	optMap(url.parse, req.url).pathname?.

piping-server/src/piping.ts

Line 83 in 6b58205

(optMap(url.parse, reqUrl, true) as OptionalProperty<url.UrlWithParsedQuery>)

url.resolve and url.parse were deprecated since v11.0.0. Instead of above apis, the WHATWG URL API should be used. This API was introduced since v7, therefore it can be introduced.

At the end of April 2021, Node.js 10 will be end-of-life.
You should probably migrate to the new API before you discontinue support for Node 10.