com-puter-tips / links-extractor

Extract all internal and external links from a URL in Python.

Links-Extractor

Extract all internal and external links from a URL in Python.

• To run type:

% python3 extractor.py [http://url1] [https://url2] and so on..

• To save the output to a file you can do:

% python3 extractor.py https://com.puter.tips > out.txt

Find how the script works and the complete tutorial here -- http://com.puter.tips/2016/12/extract-all-internal-and-external-links.html

For more information/bug report you can contact me on devharsh [at] live [dot] in or com [at] puter [dot] tips.

You may also find this repository useful for SEO -- https://github.com/com-puter-tips/SEO-Analysis

Cite

Either:

  @online{DevharshLinks,
    author = {Devharsh Trivedi},
    title = {Links-Extractor Source Repository},
    howpublished = {GitHub},
    year = {2016},
    url = {https://github.com/com-puter-tips/Links-Extractor},
  }

Or if you do not have @online support:

  @misc{DevharshLinks,
    author = {Devharsh Trivedi},
    title = {Links-Extractor Source Repository},
    howpublished = {GitHub},
    year = {2016},
    note = {\url{https://github.com/com-puter-tips/Links-Extractor}},
  }

links-extractor's Issues

BUG found: unicode issue.

Can you add a feature to output to a file instead of directly in the console window? I do not want to end up as it creates new lines, the oldest lines get erased due to the buffer limits. I did try using >>Outputfile.txt, but that resulted the file containing this:

https://www.uchinokomato.me/chara/show/154058
---------------------


18 internal links found:


'charmap' codec can't encode characters in position 14-23: character maps to <undefined>