jenalgit / pylsa Goto Github PK
View Code? Open in Web Editor NEWThis project forked from kernelmachine/pylsa
Latent Semantic Analysis in Python
This project forked from kernelmachine/pylsa
Latent Semantic Analysis in Python
Here's a Latent Semantic Analysis project. This code implements SVD (Singular Value Decomposition) to determine the similarity between words. To understand SVD, check out: http://en.wikipedia.org/wiki/Singular_value_decomposition lsa.py uses TF-IDF scores and Wikipedia articles as the main tools for decomposition. Resulting vector comparisons are done with a cosine angle similarity measure. Wikipedia seems to be a useful tool in SVD-LSA because the words examined are already contained within a context, keeping important patterns from being left out from dimension reduction in the concept space. Currently the code returns a clear distinction between obviously similar (i.e. George Bush and Dick Cheney) and obviously different (i.e. Genghis Khan and Nickelodeon) words. However, the degree of detected similarity between obviously similar words seem to vary from comparison to comparison. For example, George Bush and Dick Cheney are detected as only 40 percent similar, while Barack and Michelle Obama has a similarity at about 65 percent. More tweaking has to be done. Text summarization was executed using this SVD-LSA method as well, with summary evaluation done through cosine angle similarity measure. Essentially, the better the summary, the smaller the angle between the query's vector and the summary's vector in the concept space. Summary seems to work pretty well, as seen below. In addition, as expected, summary's evaluation score (out of 100) increased dramatically from a one to two sentence summary. Check this out for more information: http://textmining.zcu.cz/publications/isim.pdf Type $ python lsa.py to see examples of the code in action. (Or just look below). Currently lsa.summarize and lsa.summarize_evaluation support both string input and url input. The url should just have the text you want to simplify in it. The less extraneous text in the webpage, the better. It is best if you input the direct link to a news article you want to summarize, for example. To use: >> import lsa >> lsa.summarize(query="my string") >> lsa.summarize_evaluation(query="my string", summary="my summary") >> lsa.compare(query1, query2) >> lsa.summarize(url="http://myurl.com/") Dependencies: Numpy, Scipy, Matplotlib (for graphing). All-in-one python scientific computing package: https://github.com/fonnesbeck/ScipySuperpack -- TESTS OF SIMILARITY -- Similarity between Facebook and Mark Zuckerberg: 0.458080054863 Similarity between Dick Cheney and George Bush: 0.37107075803 Similarity between Nickelodeon and Genghis Khan: 0.0364954778505 Similarity between Grapes and Cars: 0.0600327943326 Similarity between Brad Pitt and Angelina Jolie: 0.357673579464 Similarity between Barack Obama and Michelle Obama: 0.654795576964 ---- TESTS OF SUMMARIZATION ---- TEXT TO SUMMARIZE - PRINCIPAL COMPONENT ANALYSIS APPLIED TO FACE RECOGNITION: 'Principal component analysis (PCA), based on information theory concepts, seeks a computational model that best describes a face by extracting the most relevant information contained in that face. The Eigenfaces approach is a PCA method, in which a small set of characteristic pictures are used to describe the variation between face images. The goal is to find the eigenvectors (eigenfaces) of the covariance matrix of the distribution, spanned by training a set of face images. Later, every face image is represented by a linear combination of these eigenvectors. Recognition is performed by projecting a new image onto the subspace spanned by the eigenfaces and then classifying the face by comparing its position in the face space with the positions of known individuals.' 1 SENTENCE SUMMARY: The goal is to find the eigenvectors (eigenfaces) of the covariance matrix of the distribution, spanned by training a set of face images' TEXT TO SUMMARIZE - I HAVE A DREAM SPEECH (http://www.huffingtonpost.com/2011/01/17/i-have-a-dream-speech-text_n_809993.html) 4 SENTENCE SUMMARY: 'There will be neither rest nor tranquility in America until the Negro is granted his citizenship rights. I have a dream that one day every valley shall be every hill and mountain shall be made the rough places will be made and the crooked places will be made and the glory of the Lord shall be and all flesh shall see it together. One hundred years the Negro is still languishing in the corners of American society and finds himself an exile in his own land. We cannot be satisfied as long as a Negro in Mississippi cannot vote and a Negro in New York believes he has nothing for which to vote. But one hundred years the Negro still is not free'
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.