magsilva / dtm

This implements topics that change over time (Dynamic Topic Models) and a model of how individual documents predict that change.

Dynamic Topic Models and the Document Influence Model

This implements topics that change over time (Dynamic Topic Models) and a model of how individual documents predict that change.

This code is the result of work by David M. Blei and Sean M. Gerrish.

(C) Copyright 2006, David M. Blei

(C) Copyright 2011, Sean M. Gerrish

It includes software corresponding to models described in the following papers:

[1] D. Blei and J. Lafferty. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, 2006.

[2] S. Gerrish and D. Blei. A Language-based Approach to Measuring Scholarly Impact. In Proceedings of the 27th International Conference on Machine Learning, 2010.

These files are part of DIM.

DIM is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

DIM is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

---

A. COMPILING

You will need to have several libraries installed to compile this package: gsl gflags

Depending on your package manager, you may be able to install these with one of the following commands:

sudo apt-get install libgsl0-dev libgflags-dev # Ubuntu, Debian sudo zypper install gflags-devel gsl-devel # OpenSUSE sudo dnf install gflags-devel gsl-devel # Fedora, CentOS

You can make the main program by changing your working directory to dtm/ and typing:

make

This software has been compiled on Ubuntu 10.04, OpenSUSE 11.2, CentOS 5.5, and Fedora 27. Depending on your environment, you may need to install additional libraries.

B. RUNNING

Once everything is compiled, you can run this software by typing the command "./main ", where flags is a list of command-line options. An example command and a description of the input and output files is given in dtm/sample.sh. You can see all command-line options by typing

./main --help

(although we suggest you start out with the example in dtm/sample.sh).

You should also replace 'main' by the appropriate executable (depending on your computer architecture and operating system). We currently provide binaries for Linux (dtm-linux32 and dtm-linux64), MacOS (dtm-darwin64) and Windows (dtm-win32.exe and dtm-win64.exe).

C. SUPPORT and QUESTIONS

This software is provided as-is, without any warranty or support, WHATSOEVER. If you have any questions about running this software, you can post your question to the topic-models mailing list at [email protected]. You are welcome to submit modifications or bug-fixes of this software to the authors, although not all submissions may be posted.

D. USAGE

This progam takes as input a collection of text documents and creates as output a list of topics over time, a description of each document as a mixture of these topics, and (possibly) a measure of how "influential" each document is, based on its language.

We have provided an example dataset, instructions for formatting input data and processing output files, and example command lines for running this software in the file dtm/sample.sh.

E. CHANGES

Changes in this version include:

• Change the default top_obs_var flag to 0.5 (from -1.0)
• Change to use more iterations and a tighter convergence criterion in each doc's E-step.
• Change to initialize random topics to be a bit more "flat".

dtm's Issues

questions about document influence model

Dear Jeffrey,

Hi. My name is Sang Won Han, and I am writing to you to ask you some questions regarding your Python module, "dtmpy." I would like to use document influence model, and there are some descriptions that dim is possible, but when I executed the dtmpy, it doesn't seem that dim is possible. Is there any way that I can use document influence model with Python? I tried to compile C-based dtm/dim files posted on Blei's github, but it produces many errors so I have no idea how to deal with them. Please let me know. Thanks!

Best,

Sang Won

Invalid free error when running

Hi,

I was able to compile using make. However when I try to run using the instructions from https://markroxor.github.io/gensim/static/notebooks/dtm_example.html

I get the following error:
*** glibc detected *** /n/home09/hxue/dtm/bin/dtm-linux64: free(): invalid pointer: 0x000000000061cc18 *** ======= Backtrace: ========= /lib64/libc.so.6(+0x75e66)[0x2b396cbfbe66] /n/sw/fasrcsw/apps/Core/gcc/7.1.0-fasrc01/lib64/libstdc++.so.6(_ZNSs6assignERKSs+0x92)[0x2b396c40eca2] /n/sw/fasrcsw/apps/Core/gflags/2.1.2-fasrc01/lib/libgflags.so.2(+0x6dbb)[0x2b396c124dbb] /n/sw/fasrcsw/apps/Core/gflags/2.1.2-fasrc01/lib/libgflags.so.2(+0x8caf)[0x2b396c126caf] /n/sw/fasrcsw/apps/Core/gflags/2.1.2-fasrc01/lib/libgflags.so.2(+0x9ce7)[0x2b396c127ce7] /n/home09/hxue/dtm/bin/dtm-linux64[0x4162fc] /lib64/libc.so.6(__libc_start_main+0xfd)[0x2b396cba4d5d] /n/home09/hxue/dtm/bin/dtm-linux64[0x402ea9]
Alternatively, I also tried directly running the precompiled binary and ran into the following error:
./dtm-linux64: error while loading shared libraries: libgsl.so.23: cannot open shared object file: No such file or directory
Any idea what might be going wrong here?

Thanks!

Executable file is not working on Windows 10

Is there a release for Windows 10?

The dtm-win64.exe file is not compatible !

Compiling for the new mac architectures

Hi magsilva.

Just as a reference for future users (and, quite likely, myself), let me leave a record of how to compile this project for M1 and M2 mac users.

I installed gsl and gflags using brew.

brew install gsl
brew install gflags

brew places the headers and the libraries in

/opt/homebrew/Cellar/gsl/2.7.1
/opt/homebrew/Cellar/gflags/2.2.2

This means that one needs to change the Makefile's CXX flags as follows:

# Line 8 in the Makefile
CXXFLAGS += -I /opt/homebrew/Cellar/gsl/2.7.1/include -L /opt/homebrew/Cellar/gsl/2.7.1/lib -I /opt/homebrew/Cellar/gflags/2.2.2/include -L /opt/homebrew/Cellar/gflags/2.2.2/lib -I ${LIB} -I ${LIB}/math -I ${GSLWRAP_LIB}/include -I ${GSLWRAP_LIB}/include/gslwrap 

This change allows you to compile in the new architectures.

Calculate perplexity

I'm trying to calculate the perplexity of a model based on the lhoods.dat file but I can find how to do it. What would be the correct way to calculate it? Thanks.

Executable file not working

When I tried to run the dtm-win64.exe file, its opening and momentarily getting closed. Please provide me with a solution for this.