[Feature] Support master output

Apache HugeGraph-Computer

The hugegraph-computer is a distributed graph processing system for hugegraph. It is an implementation of Pregel. It runs on Kubernetes or YARN framework.

Features

Support distributed MPP graph computing, and integrates with HugeGraph as graph input/output storage.
Based on BSP(Bulk Synchronous Parallel) model, an algorithm performs computing through multiple parallel iterations, every iteration is a superstep.
Auto memory management. The framework will never be OOM(Out of Memory) since it will split some data to disk if it doesn't have enough memory to hold all the data.
The part of edges or the messages of super node can be in memory, so you will never lose it.
You can load the data from HDFS or HugeGraph, output the results to HDFS or HugeGraph, or adapt any other systems manually as needed.
Easy to develop a new algorithm. You just need to focus on a vertex only processing just like as in a single server, without worrying about message transfer and memory/storage management.

Learn More

The project homepage contains more information about hugegraph-computer.

And here are links of other repositories:

hugegraph-server (graph's core component - OLTP server)
hugegraph-toolchain (include loader/dashboard/tool/client)
hugegraph-commons (include common & rpc module)
hugegraph-website (include doc & website code)

Note

If some classes under computer-k8s cannot be found, you need to execute mvn clean install in advance to generate corresponding classes.

Contributing

Welcome to contribute to HugeGraph, please see How to Contribute for more information.
Note: It's recommended to use GitHub Desktop to greatly simplify the PR and commit process.
Thank you to all the people who already contributed to HugeGraph!

License

hugegraph-computer is licensed under Apache 2.0 License.

Contact Us

GitHub Issues: Feedback on usage issues and functional requirements (quick response)
Feedback Email: [email protected] (subscriber only)
WeChat public account: Apache HugeGraph, welcome to scan this QR code to follow us.

	@Override
	public void exceptionCaught(TransportException cause,
	ConnectionId connectionId) {
	// TODO: implement failover
	LOG.error("Channel for connectionId {} occurred exception",
	connectionId, cause);
	DataClientManager.this.connManager.closeClient(connectionId);
	}
	}

	@Override
	public void exceptionCaught(TransportException cause,
	ConnectionId connectionId) {
	// TODO: implement failover
	LOG.warn("Exception caught for connection:{}, root cause:",
	connectionId, cause);
	}

	public WorkerStat input() {
	WorkerStat workerStat = new WorkerStat();
	this.recvManager.waitReceivedAllMessages();

	Map<Integer, PeekableIterator<KvEntry>> vertices =
	this.recvManager.vertexPartitions();
	Map<Integer, PeekableIterator<KvEntry>> edges =
	this.recvManager.edgePartitions();

	// TODO: parallel input process
	for (Map.Entry<Integer, PeekableIterator<KvEntry>> entry :
	vertices.entrySet()) {
	int partition = entry.getKey();

apache / incubator-hugegraph-computer Goto Github PK

incubator-hugegraph-computer's Introduction

Apache HugeGraph-Computer

Features

Learn More

Note

Contributing

License

Contact Us

incubator-hugegraph-computer's People

Contributors

Stargazers

Watchers

Forkers

incubator-hugegraph-computer's Issues

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)

High Priority (高优先级)

Medium Priority (中优先级)

Low Priority (低优先级)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Feature Description (功能描述)

Feature Description (功能描述)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)

Bug Type (问题类型)

Before submit

Environment (环境信息)

Expected & Actual behavior (期望与实际表现)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Feature Description (功能描述)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Feature Description (功能描述)

Problem Type (问题类型)

Before submit

Environment (环境信息)

Your Question (问题描述)

Vertex/Edge example (问题点 / 边数据举例)

Schema [VertexLabel, EdgeLabel, IndexLabel] (元数据结构)

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)

Bug Type (问题类型)

Before submit

Environment (环境信息)

Expected & Actual behavior (期望与实际表现)

Feature Description (功能描述)

Feature Description (功能描述)

Feature Description (功能描述)