Parallel Implementation of Deepwalk about deepwalk

[GTmac]

When you set the number of workers to 5, I assume that DeepWalk will run on a single instance with 5 threads. Thus, you need to write some additional code to generate walks on each instance and then merge the walk files together.

Perhaps the reason it got killed is that your machine run out of memory. Can you provide more detailed information on that?

[rkchunduri]

I am running it on AWS ec2 at this moment,ec2 has 8V cpus and I am configuring number of workers to 8.It is running fine at this moment but the process is still slow.Can you please help me to write the additional code to generate walks on each instance and then merging the walk files. I am also thinking to make this code to run using apache spark .Can you please share your thoughts and help me in this

[phanein]

If you generate the random walks across different machines, be sure to use a different seed so they actually generate different walks. You may want to check out the Deeplearning4j project, which has (I think) a distributed implementation. (Note that I haven't personally used it) https://deeplearning4j.org/doc/org/deeplearning4j/graph/ models/deepwalk/DeepWalk.html

…

On Apr 4, 2017 11:57 PM, "Haochen Chen" ***@***.***> wrote: Closed #33 <#33>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEo0dgVhKXBqcEi5YgSllITFb9XRiztks5rsmhOgaJpZM4MwZRB> .

[rkchunduri]

Thanks Phanein. The DL4j has some limitations .Can you please give an outline to implement distributed machines for your python code. I executed your code on 16v cpus machine and it really worked well.I want to implement it using spark. Thanks On Tue, Apr 4, 2017 at 8:40 PM Bryan Perozzi <[email protected]> wrote:

If you generate the random walks across different machines, be sure to use a different seed so they actually generate different walks. You may want to check out the Deeplearning4j project, which has (I think) a distributed implementation. (Note that I haven't personally used it) https://deeplearning4j.org/doc/org/deeplearning4j/graph/ models/deepwalk/DeepWalk.html On Apr 4, 2017 11:57 PM, "Haochen Chen" ***@***.***> wrote: > Closed #33 <#33>. > > — > You are receiving this because you are subscribed to this thread. > Reply to this email directly, view it on GitHub > <#33 (comment)>, or mute > the thread > < https://github.com/notifications/unsubscribe-auth/ABEo0dgVhKXBqcEi5YgSllITFb9XRiztks5rsmhOgaJpZM4MwZRB > > . > — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFG9EJFvaPFD4gwEJyz9ejfXgolXUOJ6ks5rsvD0gaJpZM4MwZRB> .

-- Sent from my iPhone

[phanein]

I think it's straight forward - you can generate the random walks in parallel (as the multi-threaded implementation does now). Then just use Spark's word2vec implementation (it seems like MLLib has one: https://spark.apache.org/docs/latest/mllib-feature-extraction.html#word2vec )

…

On Tue, Apr 4, 2017 at 9:43 PM, rkchunduri ***@***.***> wrote: Thanks Phanein. The DL4j has some limitations .Can you please give an outline to implement distributed machines for your python code. I executed your code on 16v cpus machine and it really worked well.I want to implement it using spark. Thanks On Tue, Apr 4, 2017 at 8:40 PM Bryan Perozzi ***@***.***> wrote: > If you generate the random walks across different machines, be sure to use > a different seed so they actually generate different walks. > > You may want to check out the Deeplearning4j project, which has (I think) a > distributed implementation. (Note that I haven't personally used it) > > https://deeplearning4j.org/doc/org/deeplearning4j/graph/ > models/deepwalk/DeepWalk.html > > On Apr 4, 2017 11:57 PM, "Haochen Chen" ***@***.***> wrote: > > > Closed #33 <#33>. > > > > — > > You are receiving this because you are subscribed to this thread. > > Reply to this email directly, view it on GitHub > > <#33 (comment)>, or > mute > > the thread > > < > https://github.com/notifications/unsubscribe-auth/ ABEo0dgVhKXBqcEi5YgSllITFb9XRiztks5rsmhOgaJpZM4MwZRB > > > > . > > > > — > You are receiving this because you authored the thread. > Reply to this email directly, view it on GitHub > <#33 (comment)>, > or mute the thread > <https://github.com/notifications/unsubscribe-auth/ AFG9EJFvaPFD4gwEJyz9ejfXgolXUOJ6ks5rsvD0gaJpZM4MwZRB> > . > -- Sent from my iPhone — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#33 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEo0Z2AmGuvWsX9R5xMB-xArG4F7I8uks5rsvG1gaJpZM4MwZRB> .

[thomasniebler]

How about using Spark all along? We did that for our RandomWalks implementation, but trained the word2vec models locally, since the spark word2vec implementation is nowhere as good as the C variant from Mikolov himself, e.g. the maximum vocabulary size is heavily restricted.

[rkchunduri]

HI @thomasniebler Can you please share your spark code for random walk generation,so that I will integrate my word2vec code with that and see how it is working.