dmarcous / spark-betweenness Goto Github PK

View Code? Open in Web Editor NEW

54.0 8.0 17.0 699 KB

k Betweenness Centrality algorithm for Spark using GraphX

License: Apache License 2.0

Scala 48.44% Python 2.92% CSS 8.38% XSLT 40.26%

betweenness-centrality spark graph-algorithms social-network-analysis centrality big-data

spark-betweenness's People

Contributors

Stargazers

Watchers

Forkers

codeaudit giserh intellifora salgorithom haanns kunmei zbn123 agalazis caiwenderek dujiaxin jodk drownfish19 embedxj shiyufeng03 avyhlj fatimaezzahrabadadoui wwwqqqlll

spark-betweenness's Issues

Task not serializable

First, thanks for a great example! I really appreciate it, and it does exactly what I've been looking for.

Second, I copied/pasted your code into a larger project I'm working on, so this isn't directly an issue with what you did, but more of a question that you might be able to answer based on your experiences. When I run the (copy/pasted/modified) code, when it gets into the Pregel I get an error:

Exception in thread "main" org.apache.spark.SparkException: Task not serializable
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:304)
    at org.apache.spark.util.ClosureCleaner$.org$apache$spark$util$ClosureCleaner$$clean(ClosureCleaner.scala:294)
    at org.apache.spark.util.ClosureCleaner$.clean(ClosureCleaner.scala:122)
    at org.apache.spark.SparkContext.clean(SparkContext.scala:2055)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:707)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1.apply(RDD.scala:706)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
    at org.apache.spark.rdd.RDD.mapPartitions(RDD.scala:706)
    at org.apache.spark.graphx.impl.VertexRDDImpl.mapVertexPartitions(VertexRDDImpl.scala:96)
    at org.apache.spark.graphx.impl.GraphImpl.mapVertices(GraphImpl.scala:132)
    at org.apache.spark.graphx.Pregel$.apply(Pregel.scala:122)
    at Main$KBetweenness$2$.createKGraphlets(Main.scala:672)
    at Main$KBetweenness$2$.run(Main.scala:636)
    at Main$.do_spark_3$1(Main.scala:783)
    at Main$.main(Main.scala:801)
    at Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)
Caused by: java.io.NotSerializableException: Main$KBetweenness$2$
Serialization stack:
    - object not serializable (class: Main$KBetweenness$2$, value: Main$KBetweenness$2$@4fc142ec)
    - field (class: Main$KBetweenness$2$$anonfun$createKGraphlets$1, name: $outer, type: class Main$KBetweenness$2$)
    - object (class Main$KBetweenness$2$$anonfun$createKGraphlets$1, <function3>)
    - field (class: org.apache.spark.graphx.Pregel$$anonfun$1, name: vprog$1, type: interface scala.Function3)
    - object (class org.apache.spark.graphx.Pregel$$anonfun$1, <function2>)
    - field (class: org.apache.spark.graphx.impl.GraphImpl$$anonfun$5, name: f$1, type: interface scala.Function2)
    - object (class org.apache.spark.graphx.impl.GraphImpl$$anonfun$5, <function1>)
    - field (class: org.apache.spark.graphx.impl.VertexRDDImpl$$anonfun$1, name: f$1, type: interface scala.Function1)
    - object (class org.apache.spark.graphx.impl.VertexRDDImpl$$anonfun$1, <function1>)
    at org.apache.spark.serializer.SerializationDebugger$.improveException(SerializationDebugger.scala:40)
    at org.apache.spark.serializer.JavaSerializationStream.writeObject(JavaSerializer.scala:47)
    at org.apache.spark.serializer.JavaSerializerInstance.serialize(JavaSerializer.scala:101)
    at org.apache.spark.util.ClosureCleaner$.ensureSerializable(ClosureCleaner.scala:301)
    ... 22 more

I'm feeding it a graph from GraphGenerators.logNormalGraph if that makes any difference. Any thoughts off the top of your head?

Thanks!

Exception ClassCastException

Hi
I am trying to execute one of your test examples ( with KBetweenness.run() ) and i get this exception.

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD.

During compile time i don't get any errors. I use Spark 2.2. and Scala 2.11. My maven dependencies are

<dependency>
        <groupId>dmarcous</groupId>
        <artifactId>spark-betweenness</artifactId>
        <version>1.0-s_2.10</version>
 </dependency>
<repository>
      <id>spark-repo</id>
      <name>Spark Packages Repository</name>
      <url>https://dl.bintray.com/spark-packages/maven/</url>
 </repository>

Any ideas?
Thanks in advance

Cache recommandation for performance improvement

vertexKBcgraph in aggregateGraphletsBetweennessScores() can be cached, for it is used by multi actions.

How to print sortedVBC?

Hi, I am trying to print the sortedVBC. I do not get errors by building the .jar file, but anything is shown when I run it. Please, this is my source code to do so:

val graph = Graph(vertices, edges, defaultVertex)

val k = 3

val kBCGraph = KBetweenness.run(graph, k)
   
val verticesBetweenness = kBCGraph.vertices.collect()
  
val sortedVBC = verticesBetweenness.sortWith((x,y) => x._1 < y._1)

println(s"${sortedVBC(0)._1} should equal (1L)")

Best and thanks in advance

Running KBetweenness with k = 1 returns zero for all nodes.

I just want to calculate the betweenness centrality for the nodes of a graph but i noticed that for different k's it returns different results , i assume that what i want is for k equal to 1 , i try to run the algorithm with k=1 but it returns zero for all nodes , i assumed that this was because the graph is directed so i added more edges , for example if there is an edge x -> y i added also the edge y -> x. I run the algorithm again and the result is still 0 for all nodes , the api is straightforward but i can not find the solution to my problem. I would appreciate it very much if you could help me.

Thanks.

small diameter

could you clarify what

The method above ilustrated that spark-betweenness works best for graphs with a small diameter. We actually hold all k-graphlets in memory for Brandes calculation as they are the core of parallelizing this algorithm. Therefore, we manage to compute kBC on millions of nodes and vertices with large diameter graphs (such as road networks), but fail miserabely to do so on small diameter graphs (such as social networks).

means?
First you say a smal diameter is good, then the opposite.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.