vuphuctho / kaggle-dato Goto Github PK

1. Extract data from raw input (raw_html files)

• Separate contents from HTML tags. Keep relationship btw pairs of contents and tags for later use
• Minor: extract words/ phrases from URL (if possible) (advertisement might not be text-based)

2. Pre-process data

• Stemming and lemmalization
• Map HTML tags to their containing contents.

3. Extract/select features for classification

• Optional: find the advertisement before applying any text mining techniques.
• TF-IDF: can use Lucene or or Stanford NLP to calculate
• Calculate relevance between:
  • Option 1: main content and advertisement
  • Option 2: different parts of the html (eg: header and body, footer and body) if we cannot identify the advertisement.
• Important: find other features for more variables for classification