HLS Proxy : HTTP Live Streaming Proxy
- proxy .m3u8 files, and the video segments (.ts files) they internally reference
- to all proxied files:
- add permissive CORS response headers
- to .m3u8:
- modify contents such that URLs in the playlist will also pass through the proxy
- inject custom HTTP headers in all outbound proxied requests
- prefetch video segments (.ts files)
- use a hook function to conditionally decide which video segments to prefetch
- use a hook function to conditionally redirect URLs in the playlist (before and/or after they're modified to pass through the proxy)
- use a hook function to conditionally rewrite URLs after they're received by the proxy
- any video player (on the LAN) can access the proxied video stream
- including Chromecast
- prefetch and caching of video segments ahead-of-time makes playback of the video stream very stable
- solves buffering problems
- the proxy can easily be configured to bypass many of the security measures used by video servers to restrict access:
- CORS response headers (to XHR requests)
- used by web browsers to enforce a security policy that limits which website(s) may access the content
- HTTP request headers
OriginandRefererare often inspected by the server- when these headers don't match the site hosting the content, a
403 Forbiddenresponse is returned (in lieu of the requested data)
- when these headers don't match the site hosting the content, a
- restricted access to encryption keys
- often times the encrypted video segments (.ts files) are readily available, but the encryption keys are well protected
- if the keys can be obtained from another source, then a hook function can be used to redirect only those URL requests
- often times the encrypted video segments (.ts files) are readily available, but the encryption keys are well protected
- CORS response headers (to XHR requests)
- [example Javascript]: construction of URL to HLS Proxy for video stream
{ const proxy_url = 'http://127.0.0.1:8080' const video_url = 'https://example.com/video/master.m3u8' const file_extension = '.m3u8' const hls_proxy_url = `${proxy_url}/${ btoa(video_url) }${file_extension}` }
- [example Javascript]: construction of URL to HLS Proxy for video stream w/ "Referer" request header
{ const proxy_url = 'http://127.0.0.1:8080' const video_url = 'https://example.com/video/master.m3u8' const referer_url = 'https://example.com/videos.html' const file_extension = '.m3u8' const hls_proxy_url = `${proxy_url}/${ btoa(`${video_url}|${referer_url}`) }${file_extension}` }
- [example Bash]: construction of URL to HLS Proxy for video stream
proxy_url='http://127.0.0.1:8080' video_url='https://example.com/video/master.m3u8' file_extension='.m3u8' hls_proxy_url="${proxy_url}/"$(echo -n "$video_url" | base64 --wrap=0)"$file_extension"
- [example Bash]: construction of URL to HLS Proxy for video stream w/ "Referer" request header
proxy_url='http://127.0.0.1:8080' video_url='https://example.com/video/master.m3u8' referer_url='https://example.com/videos.html' file_extension='.m3u8' hls_proxy_url="${proxy_url}/"$(echo -n "${video_url}|${referer_url}" | base64 --wrap=0)"$file_extension"
- adding a file extension to the base64 encoded video URL is highly recommended
- the following file extension values have important significance to indicate the type of file being requested:
.m3u8
HLS manifest.ts
media segment.key
encryption key.json
JSON data
- though currently,
.m3u8
is the only file extension that receives special treatment- all other file types (including those without any file extension) are piped directly to the HTTP response
- the following file extension values have important significance to indicate the type of file being requested:
- refer to the section: Other Projects
npm install --global "@warren-bank/hls-proxy"hlsd <options>
options:
========
--help
--version
--tls
--host <host>
--port <number>
--copy-req-headers
--req-headers <filepath>
--origin <header>
--referer <header>
--useragent <header>
--header <name=value>
--req-options <filepath>
--req-insecure
--req-secure-honor-server-cipher-order
--req-secure-ciphers <string>
--req-secure-protocol <string>
--req-secure-curve <string>
--hooks <filepath>
--prefetch
--max-segments <number>
--cache-timeout <number>
--cache-key <number>
--cache-storage <adapter>
--cache-storage-fs-dirpath <dirpath>
-v <number>
--acl-ip <ip_address_list>
--acl-pass <password_list>
--http-proxy <http[s]://[user:pass@]hostname:port>
--tls-cert <filepath>
--tls-key <filepath>
--tls-pass <filepath>
--manifest-extension <ext>
--segment-extension <ext>- --tls is a flag to start HTTPS proxy, rather than HTTP
- used as shorthand to automatically configure the following options:
- --tls-cert
- --tls-key
- --tls-pass
- the values assigned to these options enable the use of a self-signed security certificate that is included in both the git repo and npm package, within the directory:
- when all of these option are properly specified:
- the
https:protocol is used by all URLs in modified HLS manifests
- the
- used as shorthand to automatically configure the following options:
- --host is an IP or hostname with an optional port number that can be resolved and is reachable by clients
- ex:
192.168.0.100:8080 - used to modify URLs in .m3u8 files
- when this option is specified without a port number:
- the value of the --port option is appended
- when this option is specified and the port number is
443:- the
https:protocol is used by all URLs in modified HLS manifests
- the
- when this option is not specified:
- the value of the "Host" HTTP request header is used
- ex:
- --port is the port number that the server listens on
- ex:
8080 - when this option is not specified:
- HTTP proxy binds to:
80 - HTTPS proxy binds to:
443
- HTTP proxy binds to:
- ex:
- --copy-req-headers is a flag to enable the duplication of all HTTP request headers sent to the proxy → to the request made by the proxy to the video server
- --req-headers is the filepath to a JSON data Object containing key:value pairs
- each key is the name of an HTTP header to send in every outbound request
- --origin is the value of the corresponding HTTP request header
- --referer is the value of the corresponding HTTP request header
- --useragent is the value of the corresponding HTTP request header
- --header is a single name:value pair
- this option can be used multiple times to include several HTTP request headers
- the pair can be written:
- "name: value"
- "name=value"
- "name = value"
- --req-options is the filepath to a JSON data Object
- exposes the options Object passed to low-level network request APIs:
- advanced https request options:
- context of the secure request is obtained by passing the request options Object to:
tls.createSecureContext(options) - configuration for the context of the secure request can be merged with the request options Object
- configuration keys of particular interest:
honorCipherOrder- default value:
false
- default value:
cipherssecureProtocol- default value:
"TLS_method"
- default value:
ecdhCurve- default value:
tls.DEFAULT_ECDH_CURVE- the exact value depends on the version of node
- most commonly:
- older versions of node:
"prime256v1" - newer versions of node:
"auto"
- older versions of node:
- default value:
- context of the secure request is obtained by passing the request options Object to:
- --req-insecure is a flag to override the following environment variable to disable certificate validation for secure https requests:
NODE_TLS_REJECT_UNAUTHORIZED= 0- equivalent to:
curl --insecurewget --no-check-certificate
- equivalent to:
- --req-secure-honor-server-cipher-order is a flag to set the following key in the request options Object to configure the context for secure https requests:
{honorCipherOrder: true}
- --req-secure-ciphers is the value to assign to the following key in the request options Object to configure the context for secure https requests:
{ciphers: value}
- --req-secure-protocol is the value to assign to the following key in the request options Object to configure the context for secure https requests:
{secureProtocol: value}
- --req-secure-curve is the value to assign to the following key in the request options Object to configure the context for secure https requests:
{ecdhCurve: value}
- --hooks is the filepath to a CommonJS module that exports a single JSON Object
- each key is the name of a hook function
- each value is the implementation of the corresponding Function
- hook function signatures:
"add_request_options": (url, is_m3u8) => request_options- conditionally add HTTP request options
- inputs:
url- string URL
is_m3u8- boolean that indicates whether
urlwill request an HLS manifest
- boolean that indicates whether
- return value:
- Object having attributes that are combined with --req-options and used to send the outbound request to
url
- Object having attributes that are combined with --req-options and used to send the outbound request to
"add_request_headers": (url, is_m3u8) => request_headers- conditionally add HTTP request headers
- return value:
- Object containing key:value pairs that are combined with --req-headers
- each key is the name of an HTTP header to send in the outbound request to
url
- each key is the name of an HTTP header to send in the outbound request to
- Object containing key:value pairs that are combined with --req-headers
"modify_m3u8_content": (m3u8_content, m3u8_url) => new_m3u8_content- conditionally modify the content of .m3u8 files before they are parsed to extract URLs
"redirect": (url) => new_url- conditionally redirect the URLs encountered in .m3u8 files before they are modified to pass through the proxy
"redirect_final": (url) => new_url- conditionally redirect the URLs encountered in .m3u8 files after they are modified to pass through the proxy
"rewrite": (url) => new_url- conditionally rewrite the URLs requested by clients before they are proxied
"prefetch": (url) => boolean- conditionally decide whether to prefetch video segments on a per-URL basis
- return value must be a strict boolean type (ie:
trueorfalse) - otherwise, the default behavior supersedes
- to only prefetch .ts files
"prefetch_segments": (prefetch_urls, max_segments, is_vod, seg_duration_ms, perform_prefetch) => new_prefetch_urls- conditionally filter the list of video segment URLs that are pending prefetch, when more than --max-segments are contained in an HLS manifest
- inputs:
prefetch_urls- array of string video segment URLs
max_segments- integer that denotes the max length of the return value
is_vod- boolean that indicates whether the HLS manifest is for video-on-demand
- if true:
- the video is not a live stream
- the HLS manifest is complete and contains URLs for every video segment that would be needed to play the entire stream from start to finish
- if true:
- boolean that indicates whether the HLS manifest is for video-on-demand
seg_duration_ms- integer that represents the duration (ms) of each video segment in the HLS manifest
perform_prefetch- function that accepts an array of string video segment URLs, and immediately begins to prefetch all corresponding segments
- return value:
- array of string video segment URLs that is a subset of
prefetch_urls- can be emtpy (ex: when using
perform_prefetch)
- can be emtpy (ex: when using
- array of string video segment URLs that is a subset of
- pre-conditions:
- the length of
prefetch_urlsis >max_segments
- the length of
- post-conditions:
- the length of the return value array is <=
max_segments
- the length of the return value array is <=
"request_intervals": (add_request_interval) => {}- enables the use of a cookie jar for all outbound HTTP requests
- adds any number of timers that each execute at individually specified intervals
- when each timer executes, it is passed an HTTP request client that is preconfigured to:
- include the request headers that are specified by other relevant options
- utilize the same cookie jar as all other outbound HTTP requests
- this allows the implementation of custom logic that may be required by one or more video hosts to periodically refresh or update session cookies
- an example would better illustrate usage:
module.exports = { request_intervals: (add_request_interval) => { add_request_interval( 0, // run timer once at startup to initialize cookies (request) => { request('https://example.com/login', {username: 'me', password: '12345'}) } ) add_request_interval( (1000 * 60 * 5), // run timer at 5 minute interval to refresh cookies (request) => { request('https://example.com/heart-beat') } ) } }
- more advanced configuration of the call to the HTTP request client is possible
- the 1st parameter is required, and must be a URL string
- the 2nd parameter is optional, and can contain POST data
- the 3rd parameter is optional, and can be used for more advanced configuration options
- usage of this HTTP request client is documented here
- specifically, pay careful attention to the signatures for:
- the latter two input parameters
- the attributes of the Object that is resolved by the Promise in the return value (if the content of the response is needed)
- specifically, pay careful attention to the signatures for:
- --prefetch is a flag to enable the prefetch and caching of video segments
- when .m3u8 files are downloaded and modified inflight, all of the URLs in the playlist are known
- at this time, it is possible to prefetch the video segments (.ts files)
- when the video segments (.ts files) are requested at a later time, the data is already cached (in memory) and can be returned immediately
- --max-segments is the maximum number of video segments (.ts files) to hold in each cache
- this option is only meaningful when --prefetch is enabled
- a cache is created for each unique HLS manifest URL
- all of the video segments (.ts files) associated with each distinct video stream are stored in isolation
- when any cache grows larger than this size, the oldest data is removed to make room to store new data
- when this option is not specified:
- default value:
20
- default value:
- --cache-timeout is the maximum number of seconds that any segment cache can remain unused before its contents are cleared (to reduce wasted space in memory)
- this option is only meaningful when --prefetch is enabled
- when this option is not specified:
- default value:
60
- default value:
- --cache-key sets the type of string used to represent keys in the cache hashtable when logged
- this option is only meaningful when --prefetch is enabled
- scope:
- v0.16.0 and earlier
- keys in the cache hashtable used this string representation
- v0.16.1 and later
- keys in the cache hashtable are full URLs
- the data structure to cache video segments (.ts files) was updated
- each unique HLS manifest is associated with a distinct FIFO list that holds --max-segments
- when a video segment is requested
- the proxy needs to search every FIFO list for a match
- when keys in the cache hashtable lose fidelity, collisions can occur and the wrong video segment can be returned
- full URLs are unique and guarantee correct behavior
- keys in the cache hashtable are full URLs
- v0.16.0 and earlier
0(default):- sequence number of .ts file w/ .ts file extension (ex: "123.ts")
- pros:
- shortest type of string
- makes the log output easiest to read
- cons:
- in the wild, I've encountered video servers that assign each .ts file a unique filename that always terminate with the same static sequence number
- this is a strange edge case, but this option provides an easy workaround
- in the wild, I've encountered video servers that assign each .ts file a unique filename that always terminate with the same static sequence number
- pros:
- sequence number of .ts file w/ .ts file extension (ex: "123.ts")
1:- full filename of .ts file
2:- full URL of .ts file
- --cache-storage selects a storage adapter that is used to hold the cache of prefetched video segments
- this option is only meaningful when --prefetch is enabled
memory(default):- uses RAM
filesystem:- each video segment is written to a new file within a specified directory
- filenames are random and unique
- --cache-storage-fs-dirpath specifies the directory in which to save video segments when using a filesystem-based cache storage adapter
- this option is only meaningful when --prefetch is enabled and --cache-storage is
filesystem
- this option is only meaningful when --prefetch is enabled and --cache-storage is
- -v sets logging verbosity level:
-1:- silent
0(default):- show errors only
1:- show an informative amount of information
2:- show technical details
3:- show an enhanced technical trace (useful while debugging unexpected behavior)
4:- show the content of .m3u8 files (both before and after URLs are modified)
- --acl-ip restricts proxy server access to clients at IP addresses in whitelist
- ex:
"192.168.1.100,192.168.1.101,192.168.1.102"
- ex:
- --acl-pass restricts proxy server access to requests that include a
passwordquerystring parameter having a value in whitelist- ex:
"1111,2222,3333,4444,5555"
- ex:
- --http-proxy enables all outbound HTTP and HTTPS requests from HLS-Proxy to be tunnelled through an additional external web proxy server
- SOCKS proxies are not supported
- ex:
http://myusername:[email protected]:1234
- --tls-cert is the filepath to a security certificate to use for HTTPS
- --tls-key is the filepath to the private key for the --tls-cert security certificate
- --tls-pass is the filepath to a text file containing the security passphrase for the --tls-key private key
- optional, not required when the --tls-key private key was created without a security passphrase
- --manifest-extension is the file extension associated with HLS manifests
- default value:
m3u8
- default value:
- --segment-extension is the file extension associated with media segments
- default value:
ts
- default value:
-
print help
hlsd --help -
print version
hlsd --version -
start HTTP proxy at default host:port
hlsd -
start HTTP proxy at default host and specific port
hlsd --port "8080" -
start HTTP proxy at specific host:port
hlsd --host "192.168.0.100" --port "8080" -
start HTTPS proxy at default host:port
hlsd --tls -
start HTTPS proxy at specific host:port
hlsd --tls --host "192.168.0.100" --port "8081" -
start HTTPS proxy at default host:port and send specific HTTP headers
hlsd --tls --req-headers "/path/to/request/headers.json" -
start HTTPS proxy at default host:port and enable prefetch of 10 video segments
hlsd --tls --prefetch --max-segments 10 -
start HTTPS proxy using a non-generic security certificate
hlsd --tls-cert "/path/to/cert.pem" --tls-key "/path/to/key.pem" --tls-pass "/path/to/pass.phrase"
git clone "https://github.com/warren-bank/HLS-Proxy.git"
cd "HLS-Proxy"
npm install# ----------------------------------------------------------------------
# If using a port number >= 1024 on Linux, or
# If using Windows:
# ----------------------------------------------------------------------
npm start [-- <options>]
# ----------------------------------------------------------------------
# https://www.w3.org/Daemon/User/Installation/PrivilegedPorts.html
#
# Linux considers port numbers < 1024 to be privileged.
# Use "sudo":
# ----------------------------------------------------------------------
npm run sudo [-- <options>]- identical to the command-line binary
-
print help
npm start -- --help -
start HTTP proxy at specific host:port
npm start -- --host "192.168.0.100" --port "8080" -
start HTTPS proxy at specific host:port
npm start -- --host "192.168.0.100" --port "8081" --tls -
start HTTP proxy at default host:port with escalated privilege
npm run sudo -- --port "80" -
start HTTPS proxy at default host:port with escalated privilege
npm run sudo -- --port "443" --tls -
start HTTP proxy at specific port and send custom "Referer" request header for specific video stream
npm start -- --port "8080"
h_referer='http://XXX:80/page.html'
URL='https://httpbin.org/headers'
URL="${URL}|${h_referer}"
URL=$(echo -n "$URL" | base64 --wrap=0)
URL="http://127.0.0.1:8080/${URL}.json"
# URL='http://127.0.0.1:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJzfGh0dHA6Ly9YWFg6ODAvcGFnZS5odG1s.json'
curl --silent "$URL"- start HTTP proxy at specific port and send custom request headers
headers_file="${TMPDIR}/headers.json"
echo '{"Origin" : "http://XXX:80", "Referer": "http://XXX:80/page.html"}' > "$headers_file"
npm start -- --port "8080" --req-headers "$headers_file"
URL='https://httpbin.org/headers'
URL=$(echo -n "$URL" | base64 --wrap=0)
URL="http://127.0.0.1:8080/${URL}.json"
# URL='http://127.0.0.1:8080/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent "$URL"- start HTTPS proxy at specific port and send custom request headers
headers_file="${TMPDIR}/headers.json"
echo '{"Origin" : "http://XXX:80", "Referer": "http://XXX:80/page.html"}' > "$headers_file"
npm start -- --port "8081" --req-headers "$headers_file" --tls -v 1
URL='https://127.0.0.1:8081/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent --insecure "$URL"- start HTTPS proxy at specific port and send custom request headers
h_origin='http://XXX:80'
h_referer='http://XXX:80/page.html'
h_useragent='Chromium'
h_custom_1='X-Foo: 123'
h_custom_2='X-Bar: baz'
npm start -- --port "8081" --origin "$h_origin" --referer "$h_referer" --useragent "$h_useragent" --header "$h_custom_1" --header "$h_custom_2" --tls -v 1
URL='https://127.0.0.1:8081/aHR0cHM6Ly9odHRwYmluLm9yZy9oZWFkZXJz.json'
curl --silent --insecure "$URL"- when playing the proxied HLS video stream in an HTML5 player in a Chromium web browser (ex: THEOplayer)
- if the page hosting the HTML5 video player is served from HTTPS:
- when running only the HTTP proxy server:
- the XHR requests from the player to the HTTP proxy server raise a security warning (insecure content)
- the XHR requests get elevated to HTTPS, which are unanswered (since the HTTPS proxy server isn't running)
- when running only the HTTPS proxy server:
- the XHR requests from the player to the HTTPS proxy server will silently fail
- this is because the HTTPS proxy server is using a self-signed security certificate
- this certificate needs to be (temporarily) allowed
- once it is, the video stream works perfectly
- to allow the certificate:
- browse to a URL hosted by the proxy server ( example )
- you should see the warning:
NET::ERR_CERT_AUTHORITY_INVALIDYour connection is not private - click:
Advanced - click:
Proceed to 127.0.0.1 (unsafe) - done
- to allow the certificate:
- when running only the HTTP proxy server:
- if the page hosting the HTML5 video player is served from HTTPS:
- when playing the proxied HLS video stream on a Chromecast
- the HTTP proxy server works perfectly
- the HTTPS proxy server doesn't begin playback
- not sure why..
- probably has something to do with the Chromecast's browser security policies
- a more respectable security certificate (ie: more expensive) would probably fix it
-
error:
ssl3_check_cert_and_algorithm:dh key too small- attempted fix:
--req-secure-ciphers "AES128-SHA"
- attempted fix:
-
error:
SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure- attempted fix:
--req-secure-protocol "SSLv3_method"- result:
Error: SSLv3 methods disabled - issue:
- result:
- attempted fix:
--req-secure-curve "auto"
- attempted fix:
- Webcast-Reloaded
- consists of 2 parts:
- a Chromium web browser extension (.crx)
- on each browser tab, it's silently watching the URL of all outbound requests
- every requested URL matching a regex pattern that identifies it to be a video file is displayed in the modal window that toggles open when the extension's icon is clicked
- links in this modal window open to URLs of component #2
- a static website
- a Chromium web browser extension (.crx)
- consists of 2 parts:
- Streamlink
- notes:
- this project has way more features, and is way more polished
- though its main purpose is to transcode online video with ffmpeg and pipe the output into another program, it can be configured to not load a video player and instead start a web server
- it can strongly support individual websites through single-purpose plugins
- it can also support streams via direct URLs
- using URLs from the wild will have mixed results, since cookies and headers and authentication aren't being managed by any plugin
- docs:
- binaries:
- Windows portable
- minimum system requirements:
- Windows 7 SP1
- .NET Framework 4.5
- minimum system requirements:
- Windows portable
- usage test:
streamlink --player-external-http --player-external-http-port 8080 --default-stream best --http-ignore-env --http-no-ssl-verify --url "https://XXX/video.m3u8"
- usage test result:
- doesn't appear to work with HTML5 video players or Chromecast
- the server starts and works as it was intended, but something about the format of the data it "streams" is incompatible
- VLC can play the video stream from the server, and be used to render the video on Chromecast
- notes:
v1.x- commit history is in branch:
v01 - summary:
- m3u8 manifest parser uses regex patterns to identify all URL patterns without any special knowledge of the m3u8 manifest specification
- internal
proxymodule exports a function that accepts an instance ofhttp.Serverand adds event listeners to process requests
- system requirements:
- Node.js version: v8.6.0 (and higher)
- transitive dependency requirements:
- v08.06.00+:
@warren-bank/node-process-argv - v08.06.00+:
@warren-bank/node-request
- v08.06.00+:
- transitive dependency requirements:
- Node.js version: v8.6.0 (and higher)
- commit history is in branch:
v2.x- commit history is in branch:
v02 - summary:
- m3u8 manifest parser uses regex patterns to identify all URL patterns without any special knowledge of the m3u8 manifest specification
- internal
proxymodule exports an Object containing event listeners to process requests that can be either:- added to an instance of
http.Server - added to an
Express.jsapplication as middleware to handle a custom route- important limitation: since
/is a valid character in a base64 encoded URL, the path for a custom route needs to end with a character that is not allowed in base64 encoding (ex:'/proxy_/*')
- important limitation: since
- added to an instance of
- system requirements:
- Node.js version: v8.6.0 (and higher)
- transitive dependency requirements:
- v08.06.00+:
@warren-bank/node-process-argv - v08.06.00+:
@warren-bank/node-request
- v08.06.00+:
- transitive dependency requirements:
- Node.js version: v8.6.0 (and higher)
- commit history is in branch:
v3.x- commit history is in branch:
v03 - summary:
- m3u8 manifest parser uses special knowledge of the m3u8 manifest specification to contextually identify URLs
- internal
proxymodule exports an Object containing event listeners to process requests that can be either:- added to an instance of
http.Server - added to an
Express.jsapplication as middleware to handle a custom route- important requirement: the path for a custom route needs to include exactly one unnamed parameter that matches the base64 encoded URL and (optionally) a file extension (ex:
'/proxy/*') - the use of nested routers is supported
- important requirement: the path for a custom route needs to include exactly one unnamed parameter that matches the base64 encoded URL and (optionally) a file extension (ex:
- added to an instance of
- system requirements:
- Node.js version: v8.6.0 (and higher)
- transitive dependency requirements:
- v08.06.00+:
@warren-bank/node-process-argv - v08.06.00+:
@warren-bank/node-request
- v08.06.00+:
- transitive dependency requirements:
- Node.js version: v8.6.0 (and higher)
- commit history is in branch:
- copyright: Warren Bank
- license: GPL-2.0
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent
