jtmoon79 / goto_http_redirect_server Goto Github PK
View Code? Open in Web Editor NEWThe "Go To" HTTP Redirect Server for sharing dynamic shortcut URLs on your network.
License: MIT License
The "Go To" HTTP Redirect Server for sharing dynamic shortcut URLs on your network.
License: MIT License
POST
or other verbs. Just modify the currently supported GET
(and possible HEAD
#10 ) verbs.README.md
and --help
to read RFC 2616 HTTP 1.1. compliant!Currently, the Azure Pipeline does everything interesting within one task
. This means nearly everything that occurs does so in one visual block, specifically the task
named task: <Bash@3> Bash .azure-pipelines/build-install-run.sh
The Azure Pipeline should have more distinct parts that use more jobs
, job
, steps
and task
.
There should be distinct parts for
It would be easy to add task
for each, however, I'm not yet sure if that is the optimal level of Pipeline object for each part listed. More research required.
Currently, successful redirects are tracked by tuple of (requested path, returned to path).
Given a redirects entry /a https://server.corp.com/search/id=${query}
an attacker could iterate requests goto/a?1
, goto/a?2
, ... and overwhelm memory.
Only count successful redirects via requested path. This would have to handle how #5 works when that is implemented.
Allow counters to be turn-on options. A command-line parameter like --stats-count-redirects
would suffice.
Currently the status page dumps these counters. The status page would disclude such information if it is not present.
This approach also allows for future additions to statistics, e.g. --stats-count-IP-addresses
or --stats-count-browsers
.
This approach might warrant it's own Issue.
A user requested
I like the fact that goto/reload simply ignores entries that it finds issues with but it would be even better if it told me what it found that it didn't understand. 9 times out of 10 it is because I forgot to put a tab somewhere which raises the second request - can you please switch to JSON or something like that for the data file format CSV is just too hard to get right every time.
Either that or just get rid of the need to enter a name and a date/time on every entry. They serve no purpose AFAICT
This is a user-requested feature.
Given a path like https://service.corp.com/a/1234
I would like a shortcut path http://goto/a/1234
to redirect to https://service.corp.com/a/1234
.
Currently, it cannot.
The user requested path /a/1234
does not match the Administrator-set redirect entry /a
.
Allowing the user to declare redirects where the from path could be a bit smarter would be ideal.
Allow strings-style Template Syntax in the from redirect path. Matching the request path would vary from current strictly string compare of requested from_path == redirect from_path
to something smarter.
For example, redirects file entry like:
/a${path} https://service.corp.com/a${path} user
Request http://goto/a/1234
would redirect to https://service.corp.com/a/1234
.
Request http://goto/a
would redirect to https://service.corp.com/a
.
In effect, all requested paths /a*
would match.
Something difficult about this is allowing multiple similar redirect paths. For example, imagine redirects entrys
/a${path} https://service.corp.com/a${path} user
/alpha https://build-server.corp.com/builds/release-alpha/ user
All requests to http://goto/alpha
would match to entry /a${path}
(user would end up at unwanted location https://service.corp.com/alpha
). The matching algorithm should allow matching these as the Administrator would expect:
/alpha
-> https://build-server.corp.com/builds/release-alpha/
a/123
-> https://service.corp.com/a/123
Otherwise, a redirect entry like /a${path}
"smothers" any further redirect entries that start with substring /a
.
Currently, the --status-path
output shows the path of --reload-path
by way of showing the command line invoked. Reload path should not be shown on the status page, it is presumed that the reload path should be hidden.
Remove the date requirement when adding a redirect entry. It's tedious.
This change would be affected by #19 .
This is a user-requested change.
The status page orders the list of redirects by ord
of the string characters.
Instead, list redirects in the order they were processed. Typically, a redirects.csv
will have ad-hoc ordering significant to the user. Follow that ordering.
It would be nice if user's could view redirects by categories.
Categories would be administrator-defined. Would be ideal if users could add their own categories.
Examples of categories:
Teams:
or
Expertise Areas:
When entering a new redirect, lack of category would be allowed.
This is a user-requested feature.
Holding the TCP connection open causes all requests to goto_http_redirect_server to stall until the TCP connection is closed. No TCP response is sent to any proceeding HTTP requests while the TCP port remains open. User-agents also stall waiting for an HTTP response from the goto_http_redirect_server.
Web browsers commonly leave open TCP connections so it is easy to accidentally cause this to occur.
start the goto_http_redirect_server.py
assuming the server is listening on localhost:80
.
in a shell window, make repeated HTTP requests
Note that curl
closes the TCP connection after the HTTP request returns.
while sleep 0.01; do (set -x; curl -vv "http://localhost:80/${RANDOM}";); done
in a different shell window, open a TCP connection and leave it open
netcat localhost 80
BUG: a curl
request will hang after the HTTP request is sent. The server has stalled because netcat
has an open TCP connection.
EXPECT: no server stalls.
kill the netcat
process.
The curl
process receives a response and the loop of curl
requests resume.
socketserver.TCPServer
has an option to do this?I wrote a hack shell script to inotify when a redirect file changed (see tools/goto-http-redirect-server-inotify.sh
). Then it would signal goto_http_redirect_server
process instance.
Would be useful to put this behavior into goto_http_redirect_server.py
. For the hat trick, do this without referencing a 3rd party module.
The current HTTP redirect response HTTP 308 Permanent Redirect causes web browsers to cache the response.
If a user updates a redirect target path then web browsers that have already used the redirect will not see the update until that web browser expires the related cache.
The default HTTP redirect response should be HTTP 307 Temporary Redirect, which web browsers do not cache.
Since 32353d5 , pressing ctrl+C
at the console does not cause an exit of the script.
Instead, pressing ctrl+C
results in
2019-12-09 20:55:11 goto_http_redirect_server INFO: Calling shutdown on Redirect_Server <__main__.RedirectServer object at 0x00000214FF683348> (0x214ff683348)
Exception ignored in: <module 'threading' from 'C:\\Python\\python.org.3.7.4\\lib\\threading.py'>
Traceback (most recent call last):
File "C:\Python\python.org.3.7.4\lib\threading.py", line 1308, in _shutdown
lock.acquire()
KeyboardInterrupt
Tested in Powershell on Windows 10. Haven't checked this behavior on Linux but it's probably the same.
python 3.5 pytest running in Azure fails.
From here
+ python -m pip install --user -e '.[development-pytest]'
Obtaining file:///home/vsts/work/1/s
ERROR: Command errored out with exit status 1:
command: /opt/hostedtoolcache/Python/3.5.9/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/vsts/work/1/s/setup.py'"'"'; __file__='"'"'/home/vsts/work/1/s/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
cwd: /home/vsts/work/1/s/
Complete output (22 lines):
warning: no files found matching '*.dynlib' under directory 'tests'
no previously-included directories found matching 'tests/testdata/*/build'
no previously-included directories found matching 'tests/testdata/*/dist'
no previously-included directories found matching 'tests/testdata/*/*.egg-info'
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
zip_safe flag not set; analyzing archive contents...
Installed /home/vsts/work/1/s/.eggs/UNKNOWN-0.0.0-py3.5.egg
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/vsts/work/1/s/setup.py", line 267, in <module>
include_package_data=True,
File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/distutils/core.py", line 108, in setup
_setup_distribution = dist = klass(attrs)
File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/setuptools/dist.py", line 315, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
replace_conflicting=True,
File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/pkg_resources/__init__.py", line 853, in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'wheel' distribution was not found and is required by the application
----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
##[error]Bash exited with code '1'.
Finishing: task: «Bash@3» Python 3.5
Notable error pkg_resources.DistributionNotFound: The 'wheel' distribution was not found and is required by the application
.
The .azure-pipelines/azure-pipelines.yml
has effective code
- job: job_pytest
displayName: 'job: «pytest»'
strategy:
matrix:
Python35:
python.release: '3.5'
pool:
vmImage: 'ubuntu-18.04'
steps:
- task: Bash@3
displayName: 'task: «Bash@3» Python $(python.release)'
inputs:
targetType: inline
script: |
set -eux
python --version
python -m pip install --user -e '.[development-pytest]'
The '.[development-pytest]'
is from setup.py
entry:
'development-pytest': [
'pytest',
'pytest-cov',
'pytest-timeout',
],
Pytest using Python 3.7 or Python 3.6 succeeds.
Currently, a not found redirect path returns no HTTP body. The user just sees a blank page. It appears the redirect server is failing to respond.
Return HTML content in the HTTP body informing the user the request was not found.
Be more lenient about datetimes in redirect entries.
Often, they are not very important so humans are not very meticulous about them. Make an attempt to parse a variety of datetimes including formats like
YYYY-MM-DDTHH:MM
YYYY-MM-DD HH:MM
YYYY/MM/DD HH:MM
YYYY-MM-DD
YYYY/MM/DD
As an Administrator, I'd like to customize the message on the status page with more information. e.g.
Hello, this HTTP redirect server is run by the IT-Southeast Team.
See the project page at https://docs.mega-corp.com/d/it-southeast/projects/redirect-server
email Alice Bobby at [email protected] for further questions.
This would be a command-line parameter that takes a file path, --status-page-custom-note note.html
. It can presume the note is already in HTML form that can be directly input in the HTML that makes up the status page.
Part of new+improved list of redirects would be:
Relates to #20
This is a user-requested feature. The original request was "allow filtering by user".
At least one of the CIs should have a code coverage report created.
The README.md shoud have a badge link to that report.
The status page view of redirect entries is very diagnostic. Provide something more pleasant to view.
Either, a new page path (e.g. --redirect-list-path
) or just re-arrange the current status page.
This is a user-requested enhancement.
Add HTTP HEAD
request responses.
CircleCI fails to run job that uses docker image python:3.7-windowsservercore
.
The console output reads:
Build-agent version 1.0.15410-75b89bda (2019-09-13T15:43:07+0000)
Docker Engine Version: 18.09.6
Kernel Version: Linux 0e78b5f7cd6b 4.15.0-1043-aws #45-Ubuntu SMP Mon Jun 24 14:07:03 UTC 2019 x86_64 Linux
Starting container python:3.7-windowsservercore
image cache not found on this host, downloading python:3.7-windowsservercore
3.7-windowsservercore: Pulling from library/python
no matching manifest for linux/amd64 in the manifest list entries
CircleCI job code:
build_install_run__python_3_7_windowsservercore:
docker:
- image: python:3.7-windowsservercore
steps:
- checkout
- run:
name: build, install, and run in Python 3.7 on Windows Server Core
command: |
set -ex
chmod +x ./.circleci/build-install-run.sh # force +x
./.circleci/build-install-run.sh
Allow passing a custom .CSS file to be used for the status page custom style sheet code. Currently, CSS is hardcoded.
This is a user requested enhancement.
During CircleCI job build_install_run__python_3_6_alpine
, script .circleci/build-install-run.sh
installs twine
.
Installing twine
will install cffi
which builds OpenSSL cryptography stuff.
From .circleci/build-install-run.sh
script output
...
+ python -m pip install --quiet --user twine mypy
ERROR: Command errored out with exit status 1:
command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpv70un2v9
cwd: /tmp/pip-install-xgrf9zc3/cryptography_fdd804f73ec544009d8b07de0ca674e6
...
copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
creating build/temp.linux-x86_64-3.9
generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
running build_rust
=============================DEBUG ASSISTANCE=============================
If you are seeing a compilation error please try the following steps to
successfully install cryptography:
1) Upgrade to the latest pip and try again. This will fix errors for most
users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
2) Read https://cryptography.io/en/latest/installation.html for specific
instructions for your platform.
3) Check our frequently asked questions for more information:
https://cryptography.io/en/latest/faq.html
4) Ensure you have a recent Rust toolchain installed:
https://cryptography.io/en/latest/installation.html#rust
5) If you are experiencing issues with Rust for *this release only* you may
set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
=============================DEBUG ASSISTANCE=============================
error: Can not find Rust compiler
----------------------------------------
ERROR: Failed building wheel for cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly
Each run of Azure Pipeline repeats this warning several times:
##[warning]Please use Download Build Artifact task for downloading Build Artifact type artifact.
https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/utility/download-build-artifacts?view=azure-devops
Use Download Build Artifact task
within ./.azure-pipelines/azure-pipelines.yml
.
There are still some spots of code that have old Python 3.5 idioms.
goto_http_redirect_server/goto_http_redirect_server/goto_http_redirect_server.py
Lines 1484 to 1488 in c8f9c27
goto_http_redirect_server/goto_http_redirect_server/goto_http_redirect_server.py
Lines 1491 to 1495 in c8f9c27
Update the code for Python 3.6
For the security sensitive, logging the full request may be a problem.
Given a request like http://goto/other-place?password=1234
, if the Administator has turned on logging (--log file
), then /other-place?password=1234
will be written to file
.
Would be cool if user's could add redirect entries via a web interface.
This is a user-requested feature.
Currently, a change in the passed --status-note-file FILE.HTML
requires restarting the process.
Also, attempt to reload the FILE.HTML
contents upon receipt of a reload signal or request.
This would make changes to the FILE.HTML
easier to update.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.