Coder Social home page Coder Social logo

jtmoon79 / goto_http_redirect_server Goto Github PK

View Code? Open in Web Editor NEW
3.0 2.0 0.0 401 KB

The "Go To" HTTP Redirect Server for sharing dynamic shortcut URLs on your network.

License: MIT License

PowerShell 4.45% Shell 16.96% Python 78.52% HTML 0.07%
http-server redirect-urls redirector shortcuts shorturl shorturls shorturl-services python

goto_http_redirect_server's Introduction

Hello

I am James Thomas Moon. 👋 I am a software engineer. 🤓 Most of my experience is within testing and tools. 👷

My Open-Source Work Samples

The following lists are pubicly available examples of my work:

github Projects


Contribution Stats


Continuous Integration and Code Coverage

I've used several Continuous Integration (CI) services for the sake of learning about them. Here are example runs of each.

The archived links are provided because most CI Service providers expire detailed records.

Azure Pipelines

CircleCI

codecov.io

Github Actions

Travis CI

StackExchange Questions and Answers

Some of my favorite StackExchange posts:

Bug Reports and Feature Requests

Some public bug reports and feature requests I have made:






Pull Requests

My other github commits:

github Forum Posts

My favorite github forum posts:

Other Links

Contributing to Open Source

Among many other software projects and organizations that I have voluntarily donated!

Recommended podcasts

Software-oriented podcasts that I listen to irregularly.

profile for @JamesThomasMoon on Stack Exchange, a network of free, community-driven Q&A sites

goto_http_redirect_server's People

Contributors

jaraco avatar jtmoon79 avatar

Stargazers

avatar avatar avatar

Watchers

avatar avatar

goto_http_redirect_server's Issues

CircleCI job fails to using image "python:3.7-windowsservercore"

CircleCI fails to run job that uses docker image python:3.7-windowsservercore.

The console output reads:

Build-agent version 1.0.15410-75b89bda (2019-09-13T15:43:07+0000)
Docker Engine Version: 18.09.6
Kernel Version: Linux 0e78b5f7cd6b 4.15.0-1043-aws #45-Ubuntu SMP Mon Jun 24 14:07:03 UTC 2019 x86_64 Linux
Starting container python:3.7-windowsservercore
  image cache not found on this host, downloading python:3.7-windowsservercore
3.7-windowsservercore: Pulling from library/python
no matching manifest for linux/amd64 in the manifest list entries

CircleCI job code:

  build_install_run__python_3_7_windowsservercore:
    docker:
      - image: python:3.7-windowsservercore
    steps:
      - checkout
      - run:
          name: build, install, and run in Python 3.7 on Windows Server Core
          command: |
            set -ex
            chmod +x ./.circleci/build-install-run.sh  # force +x
            ./.circleci/build-install-run.sh

provide cleaner view of redirects

The status page view of redirect entries is very diagnostic. Provide something more pleasant to view.
Either, a new page path (e.g. --redirect-list-path) or just re-arrange the current status page.

This is a user-requested enhancement.

Azure pipelines should have distinct parts

Currently, the Azure Pipeline does everything interesting within one task. This means nearly everything that occurs does so in one visual block, specifically the task named task: <Bash@3> Bash .azure-pipelines/build-install-run.sh

The Azure Pipeline should have more distinct parts that use more jobs, job, steps and task.

First draft

There should be distinct parts for

  1. building + installing
  2. mypy
  3. pytest
  4. server self-test

It would be easy to add task for each, however, I'm not yet sure if that is the optimal level of Pipeline object for each part listed. More research required.

lenient datetimes

Be more lenient about datetimes in redirect entries.

Often, they are not very important so humans are not very meticulous about them. Make an attempt to parse a variety of datetimes including formats like

  • YYYY-MM-DDTHH:MM
  • YYYY-MM-DD HH:MM
  • YYYY/MM/DD HH:MM
  • YYYY-MM-DD
  • YYYY/MM/DD

redirect from path should allow smarter replacements

problem

Given a path like https://service.corp.com/a/1234 I would like a shortcut path http://goto/a/1234 to redirect to https://service.corp.com/a/1234.
Currently, it cannot.

The user requested path /a/1234 does not match the Administrator-set redirect entry /a.

Allowing the user to declare redirects where the from path could be a bit smarter would be ideal.

idea 1

Allow strings-style Template Syntax in the from redirect path. Matching the request path would vary from current strictly string compare of requested from_path == redirect from_path to something smarter.

For example, redirects file entry like:

/a${path} https://service.corp.com/a${path} user

Request http://goto/a/1234 would redirect to https://service.corp.com/a/1234.
Request http://goto/a would redirect to https://service.corp.com/a.
In effect, all requested paths /a* would match.

Something difficult about this is allowing multiple similar redirect paths. For example, imagine redirects entrys

/a${path} https://service.corp.com/a${path} user
/alpha https://build-server.corp.com/builds/release-alpha/ user

All requests to http://goto/alpha would match to entry /a${path} (user would end up at unwanted location https://service.corp.com/alpha). The matching algorithm should allow matching these as the Administrator would expect:
/alpha -> https://build-server.corp.com/builds/release-alpha/
a/123 -> https://service.corp.com/a/123
Otherwise, a redirect entry like /a${path} "smothers" any further redirect entries that start with substring /a.

failed redirects should return HTTP content

Problem

Currently, a not found redirect path returns no HTTP body. The user just sees a blank page. It appears the redirect server is failing to respond.

Solution

Return HTML content in the HTTP body informing the user the request was not found.

status should not reveal reload path

Currently, the --status-path output shows the path of --reload-path by way of showing the command line invoked. Reload path should not be shown on the status page, it is presumed that the reload path should be hidden.

Azure pytest on python 3.5 fails

python 3.5 pytest running in Azure fails.

From here

+ python -m pip install --user -e '.[development-pytest]'
Obtaining file:///home/vsts/work/1/s
    ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.5.9/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/home/vsts/work/1/s/setup.py'"'"'; __file__='"'"'/home/vsts/work/1/s/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info
         cwd: /home/vsts/work/1/s/
    Complete output (22 lines):
    warning: no files found matching '*.dynlib' under directory 'tests'
    no previously-included directories found matching 'tests/testdata/*/build'
    no previously-included directories found matching 'tests/testdata/*/dist'
    no previously-included directories found matching 'tests/testdata/*/*.egg-info'
    warning: install_lib: 'build/lib' does not exist -- no Python modules to install
    
    zip_safe flag not set; analyzing archive contents...
    
    Installed /home/vsts/work/1/s/.eggs/UNKNOWN-0.0.0-py3.5.egg
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/home/vsts/work/1/s/setup.py", line 267, in <module>
        include_package_data=True,
      File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/distutils/core.py", line 108, in setup
        _setup_distribution = dist = klass(attrs)
      File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/setuptools/dist.py", line 315, in __init__
        self.fetch_build_eggs(attrs['setup_requires'])
      File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/setuptools/dist.py", line 361, in fetch_build_eggs
        replace_conflicting=True,
      File "/opt/hostedtoolcache/Python/3.5.9/x64/lib/python3.5/site-packages/pkg_resources/__init__.py", line 853, in resolve
        raise DistributionNotFound(req, requirers)
    pkg_resources.DistributionNotFound: The 'wheel' distribution was not found and is required by the application
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

##[error]Bash exited with code '1'.
Finishing: task: «Bash@3» Python 3.5

Notable error pkg_resources.DistributionNotFound: The 'wheel' distribution was not found and is required by the application.

The .azure-pipelines/azure-pipelines.yml has effective code

      - job: job_pytest
        displayName: 'job: «pytest»'
        strategy:
          matrix:
            Python35:
              python.release: '3.5'
        pool:
          vmImage: 'ubuntu-18.04'
        steps:
          - task: Bash@3
            displayName: 'task: «Bash@3» Python $(python.release)'
            inputs:
              targetType: inline
              script: |
                set -eux
                python --version
                python -m pip install --user -e '.[development-pytest]'

The '.[development-pytest]' is from setup.py entry:

        'development-pytest': [
            'pytest',
            'pytest-cov',
            'pytest-timeout',
        ],

Pytest using Python 3.7 or Python 3.6 succeeds.

add categories to redirects

It would be nice if user's could view redirects by categories.
Categories would be administrator-defined. Would be ideal if users could add their own categories.

Examples of categories:

Teams:

  • HR Team
  • Marketing Team
  • Engineering Team
  • IT Team

or

Expertise Areas:

  • Release
  • Test
  • Fuzzy Widget
  • Project Alpha

When entering a new redirect, lack of category would be allowed.

This is a user-requested feature.

default HTTP response should be 307 Temporary Redirect

The current HTTP redirect response HTTP 308 Permanent Redirect causes web browsers to cache the response.
If a user updates a redirect target path then web browsers that have already used the redirect will not see the update until that web browser expires the related cache.

The default HTTP redirect response should be HTTP 307 Temporary Redirect, which web browsers do not cache.

leaving TCP connection open causes goto_http_redirect_server to stall

Problem

Holding the TCP connection open causes all requests to goto_http_redirect_server to stall until the TCP connection is closed. No TCP response is sent to any proceeding HTTP requests while the TCP port remains open. User-agents also stall waiting for an HTTP response from the goto_http_redirect_server.

Web browsers commonly leave open TCP connections so it is easy to accidentally cause this to occur.

Reproduction Steps

  1. start the goto_http_redirect_server.py
    assuming the server is listening on localhost:80.

  2. in a shell window, make repeated HTTP requests
    Note that curl closes the TCP connection after the HTTP request returns.
    while sleep 0.01; do (set -x; curl -vv "http://localhost:80/${RANDOM}";); done

  3. in a different shell window, open a TCP connection and leave it open
    netcat localhost 80
    BUG: a curl request will hang after the HTTP request is sent. The server has stalled because netcat has an open TCP connection.
    EXPECT: no server stalls.

  4. kill the netcat process.
    The curl process receives a response and the loop of curl requests resume.

Solution Ideas

  • force all TCP connections closed (not sure how to do this)
    • perhaps base class socketserver.TCPServer has an option to do this?
  • use a different underlying TCP Server class

CircleCI job fails "build_install_run__python_3_6_alpine", no rust compiler

During CircleCI job build_install_run__python_3_6_alpine, script .circleci/build-install-run.sh installs twine.
Installing twine will install cffi which builds OpenSSL cryptography stuff.

From .circleci/build-install-run.sh script output

...
+ python -m pip install --quiet --user twine mypy
  ERROR: Command errored out with exit status 1:
   command: /usr/local/bin/python /usr/local/lib/python3.9/site-packages/pip/_vendor/pep517/_in_process.py build_wheel /tmp/tmpv70un2v9
       cwd: /tmp/pip-install-xgrf9zc3/cryptography_fdd804f73ec544009d8b07de0ca674e6
...
copying src/cryptography/py.typed -> build/lib.linux-x86_64-3.9/cryptography
  running build_ext
  generating cffi module 'build/temp.linux-x86_64-3.9/_padding.c'
  creating build/temp.linux-x86_64-3.9
  generating cffi module 'build/temp.linux-x86_64-3.9/_openssl.c'
  running build_rust
  
      =============================DEBUG ASSISTANCE=============================
      If you are seeing a compilation error please try the following steps to
      successfully install cryptography:
      1) Upgrade to the latest pip and try again. This will fix errors for most
         users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
      2) Read https://cryptography.io/en/latest/installation.html for specific
         instructions for your platform.
      3) Check our frequently asked questions for more information:
         https://cryptography.io/en/latest/faq.html
      4) Ensure you have a recent Rust toolchain installed:
         https://cryptography.io/en/latest/installation.html#rust
      5) If you are experiencing issues with Rust for *this release only* you may
         set the environment variable `CRYPTOGRAPHY_DONT_BUILD_RUST=1`.
      =============================DEBUG ASSISTANCE=============================
  
  error: Can not find Rust compiler
  ----------------------------------------
  ERROR: Failed building wheel for cryptography
ERROR: Could not build wheels for cryptography which use PEP 517 and cannot be installed directly

reload should also reload HTML note file

Currently, a change in the passed --status-note-file FILE.HTML requires restarting the process.
Also, attempt to reload the FILE.HTML contents upon receipt of a reload signal or request.
This would make changes to the FILE.HTML easier to update.

ctrl+C does not exit script

Since 32353d5 , pressing ctrl+C at the console does not cause an exit of the script.
Instead, pressing ctrl+C results in

2019-12-09 20:55:11 goto_http_redirect_server INFO: Calling shutdown on Redirect_Server <__main__.RedirectServer object at 0x00000214FF683348> (0x214ff683348)
Exception ignored in: <module 'threading' from 'C:\\Python\\python.org.3.7.4\\lib\\threading.py'>
Traceback (most recent call last):
  File "C:\Python\python.org.3.7.4\lib\threading.py", line 1308, in _shutdown
    lock.acquire()
KeyboardInterrupt

Tested in Powershell on Windows 10. Haven't checked this behavior on Linux but it's probably the same.

status page should not re-order redirects

The status page orders the list of redirects by ord of the string characters.

Instead, list redirects in the order they were processed. Typically, a redirects.csv will have ad-hoc ordering significant to the user. Follow that ordering.

do not log entire user request, may contain sensitive data

For the security sensitive, logging the full request may be a problem.

Given a request like http://goto/other-place?password=1234, if the Administator has turned on logging (--log file), then /other-place?password=1234 will be written to file.

status page should allow custom additions

Feature Request

As an Administrator, I'd like to customize the message on the status page with more information. e.g.

Hello, this HTTP redirect server is run by the IT-Southeast Team.
See the project page at https://docs.mega-corp.com/d/it-southeast/projects/redirect-server
email Alice Bobby at [email protected] for further questions.

First draft solution

This would be a command-line parameter that takes a file path, --status-page-custom-note note.html. It can presume the note is already in HTML form that can be directly input in the HTML that makes up the status page.

allow sorting and filtering listed redirects

Part of new+improved list of redirects would be:

  • allow sorting redirects by any field
  • allow filtering redirects by user

Relates to #20

This is a user-requested feature. The original request was "allow filtering by user".

counter statistics should be reconsidered

Problem

Currently, successful redirects are tracked by tuple of (requested path, returned to path).
Given a redirects entry /a https://server.corp.com/search/id=${query} an attacker could iterate requests goto/a?1, goto/a?2, ... and overwhelm memory.

Solution 1

Only count successful redirects via requested path. This would have to handle how #5 works when that is implemented.

Solution 2

Allow counters to be turn-on options. A command-line parameter like --stats-count-redirects would suffice.
Currently the status page dumps these counters. The status page would disclude such information if it is not present.

This approach also allows for future additions to statistics, e.g. --stats-count-IP-addresses or --stats-count-browsers.

This approach might warrant it's own Issue.

allow custom .CSS

Allow passing a custom .CSS file to be used for the status page custom style sheet code. Currently, CSS is hardcoded.

This is a user requested enhancement.

give better feedback about bad entries or consider a different entry format

A user requested

I like the fact that goto/reload simply ignores entries that it finds issues with but it would be even better if it told me what it found that it didn't understand. 9 times out of 10 it is because I forgot to put a tab somewhere which raises the second request - can you please switch to JSON or something like that for the data file format CSV is just too hard to get right every time.
Either that or just get rid of the need to enter a name and a date/time on every entry. They serve no purpose AFAICT

This is a user-requested feature.

Remove Python 3.5 support

There are still some spots of code that have old Python 3.5 idioms.

Update the code for Python 3.6

monitor redirects files for changes

I wrote a hack shell script to inotify when a redirect file changed (see tools/goto-http-redirect-server-inotify.sh). Then it would signal goto_http_redirect_server process instance.
Would be useful to put this behavior into goto_http_redirect_server.py. For the hat trick, do this without referencing a 3rd party module.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.