rpvelloso / webmining Goto Github PK

create dll for python import

I had some problems getting the code to run:

• I had to use pyhton-3.8 (because of my linux distro)
• some python libs needed older versions (mlxtend, skicit-learn), and fix the classifier.py code
• in util.cpp an include was missing: #include
  The readme should be updated, as there is no webmining.sh script, and the webmining-binary is generated in folder above the scripts.
  I finally succeeded.

Some more ideas:

• you rely on the dom, which may be insufficient, if pages are generated manually. I stumbled upon this thesis:
  "A Pure Visual Approach for Automatically Extracting and Aligning Structured Web Data" by Fadwa M A Estuka
  https://era.library.ualberta.ca/items/87f248e2-828c-4a88-b326-764d37027c5a
  Maybe optionally use an immediate dom representation, which completely relies on visual features could remove some problems with non-db-sourced records (but would require rendering)
  In this work maybe some other ideas, which could help to combine visual features and your approach.