jkehler / unshortenit Goto Github PK

Hi

It looks like adfly has just changed their algorithm. Here's some code from an adfly link:

<div style="display: none;" id="llysmm" data-ysmm="Z4TcR0oYdZHJRFwRcWzRozvZLW2VR1ydazXVZ1lULGmNdXvebn2FdVsVZwSp5kjaba2p0nvQbw30BDlZbpj9"></div>
<script type="text/javascript">
var $llysmm = $('#llysmm');

At least the value has changed.

See:

$ mkvirtualenv scratch
New python executable in scratch/bin/python
Installing setuptools, pip...done.
Downloading/unpacking ipdb
  Downloading ipdb-0.9.0.tar.gz
  Running setup.py (path:/home/rduran/.virtualenvs/scratch/build/ipdb/setup.py) egg_info for package ipdb

Downloading/unpacking ipython>=0.10 (from ipdb)
  Downloading ipython-4.1.2-py2-none-any.whl (736kB): 736kB downloaded
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/rduran/.virtualenvs/scratch/lib/python2.7/site-packages (from ipdb)
Downloading/unpacking pickleshare (from ipython>=0.10->ipdb)
  Downloading pickleshare-0.6-py2.py3-none-any.whl
Downloading/unpacking pexpect (from ipython>=0.10->ipdb)
  Downloading pexpect-4.0.1.tar.gz (143kB): 143kB downloaded
  Running setup.py (path:/home/rduran/.virtualenvs/scratch/build/pexpect/setup.py) egg_info for package pexpect

Downloading/unpacking simplegeneric>0.8 (from ipython>=0.10->ipdb)
  Downloading simplegeneric-0.8.1.zip
  Running setup.py (path:/home/rduran/.virtualenvs/scratch/build/simplegeneric/setup.py) egg_info for package simplegeneric

Downloading/unpacking traitlets (from ipython>=0.10->ipdb)
  Downloading traitlets-4.1.0-py2.py3-none-any.whl (65kB): 65kB downloaded
Downloading/unpacking decorator (from ipython>=0.10->ipdb)
  Downloading decorator-4.0.9-py2.py3-none-any.whl
Downloading/unpacking path.py>=6.2 (from pickleshare->ipython>=0.10->ipdb)
  Downloading path.py-8.1.2-py2.py3-none-any.whl
Downloading/unpacking ptyprocess>=0.5 (from pexpect->ipython>=0.10->ipdb)
  Downloading ptyprocess-0.5.1-py2.py3-none-any.whl
Downloading/unpacking ipython-genutils (from traitlets->ipython>=0.10->ipdb)
  Downloading ipython_genutils-0.1.0-py2.py3-none-any.whl
Installing collected packages: ipdb, ipython, pickleshare, pexpect, simplegeneric, traitlets, decorator, path.py, ptyprocess, ipython-genutils
  Running setup.py install for ipdb

    Installing ipdb script to /home/rduran/.virtualenvs/scratch/bin
  Running setup.py install for pexpect
      File "/home/rduran/.virtualenvs/scratch/lib/python2.7/site-packages/pexpect/async.py", line 16
        transport, pw = yield from asyncio.get_event_loop()\
                                 ^
    SyntaxError: invalid syntax


  Running setup.py install for simplegeneric

Successfully installed ipdb ipython pickleshare pexpect simplegeneric traitlets decorator path.py ptyprocess ipython-genutils
Cleaning up...
(scratch)rduran@rdc-H81M-S2H:~/workspace/unshortenit$ python setup.py test
Traceback (most recent call last):
  File "setup.py", line 6, in <module>
    import unshortenit
  File "/home/rduran/workspace/unshortenit/unshortenit/__init__.py", line 8, in <module>
    from .base import unshorten
  File "/home/rduran/workspace/unshortenit/unshortenit/base.py", line 16, in <module>
    import requests
ImportError: No module named requests

Linkbucks not working :(

I have adapted unshortenit to be compatible with python 2.7 in a separate branch at my fork.
I can create a PR if the original developer @jkehler wants.

I believe this could easily be fixed by simply adding http to the arg if it's missing, or some other change to the code, however I do not know python well nor do I know this project well.

My version from pip freeze is:

unshortenit==0.4.0

And it's throwing this error:

Traceback (most recent call last):
File "./checkurl", line 3, in
from unshortenit import UnshortenIt
File "/usr/local/lib/python2.7/dist-packages/unshortenit/init.py", line 12, in
from .unshortenit import UnshortenIt # noqa
File "/usr/local/lib/python2.7/dist-packages/unshortenit/unshortenit.py", line 24
def init(self, default_timeout: int = 30, default_headers: dict = None):
^
SyntaxError: invalid syntax

Is this because I still have Python 2.7??

hello,
unshorten it won't look for Location header in links that match adfly regex, even if i explicitly define type to something other than adf.ly links it just looks for the js variable in destination page, but returns the unmodified link after failing ...

example link: http://adf.ly/1ZpGSo
(it has a 302 redirect to google.com)

The documentation currently states:

By default the library will not make any HTTP request if the url provided does not match any of the modules url patterns. To override this you may pass force=True to the unshorten method.

However, looking at https://github.com/jkehler/unshortenit/blob/master/unshortenit/unshortenit.py#L70, it appears every URI passed to the library will results in a HTTP request, presumably to resolve 30x redirects.

Additionally, the force argument seems to be ignored.

I assume this was for the ability to make 30x resolution optional (which I PR-ed a while ago)?

It seems like the example in the docs is probably for an older version or maybe the service was removed from this library.

It results with a service unknown error.

ConnectionError: HTTPSConnectionPool(host='href.li', port=443): Max retries exceeded with url: /?https://example.com (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7f37fc102128>: 
Failed to establish a new connection: [Errno -2] Name or service not known',))

The link "http://fb.me/36vAPSlVl" causes unshortenit to churn CPU cycles and get nowhere. Stalls indefinitely, even with timeout flagged. Haven't debugged in-depth. Tested only on Ubuntu 16.04 so far.

Hello! I stumped upon your little project here, and I've noticed that you require a full Javascript engine to defeat adfly! A JS engine isn't required at all, adfly simply uses a jumbled up version of Base64. I may come back and write the algorithm in code, but for now here's the algorithm in english:

1. Grab the ysmm variable with awesome regular expressions like you do.
2. Create two temp strings: "left" and "right"
3. For each group of two characters in the ysmm from the left side:
   1. left = left + first character
   2. right = second character + right
4. Concatenate left + right, then strip the first two characters.
5. base64 decode the result

Sometimes this results in an adfly redirect. You can safely follow this link with the same HTTP setup as you did above, but this url is just base64 this time so if you'd rather save yet another HTTP request:

1. If the url doesn't match ^http(s|)://ADFLY_REGEX/go.php?u=[-A-Za-z0-9+/=]|=[^=]|={3,}$ (my regex skills are... bleh, but it works for me. Use at your own risk!), then just return
2. Otherwise, base64 decode everything after the u=

Briefly (ignore the platform and repo differences. The issue is also present in cpython, and my repo matches yours at the point I encountered the issue):

durr@rwpscrape /m/S/S/ReadableWebProxy> pypy3 -m pip install git+https://github.com/fake-name/unshortenit.git
Collecting git+https://github.com/fake-name/unshortenit.git
  Cloning https://github.com/fake-name/unshortenit.git to /tmp/pip-dcg6fu3e-build
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-dcg6fu3e-build/setup.py", line 6, in <module>
        import unshortenit
      File "/tmp/pip-dcg6fu3e-build/unshortenit/__init__.py", line 8, in <module>
        from .base import unwrap_30x_only
      File "/tmp/pip-dcg6fu3e-build/unshortenit/base.py", line 10, in <module>
        import requests
    ImportError: No module named 'requests'

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-dcg6fu3e-build/

Basically, setup.py imports unshortenit/__init__.py, which imports unshortenit/base.py, which imports requests. This is done while parsing the setup.py file, before the install_requires directives are parsed. This means it tries to import requests before it's actually processed the dependencies (and installed them).

This can be worked around by manually installing requests beforehand, but basically the way the version is currently read is broken.

I think this is what #10 was trying to get at, but was not clear with.

for this link
http://adf.ly/1ihaN4 that it is vidto url