Really cool visualization, amazing how it resembles a neural network.
chrisweekly•26m ago
> "I can compute PageRank on a directed graph with one billion edges (graph500-26 from the Graphalytics dataset) using 5 GB of memory. Alternatively, I can identify all the weakly connected components in a graph with two billion edges (twitter_mpi from the same dataset collection) using 10 GB of memory. Neither NetworkX nor Igraph can do this; most existing graph algorithms require the graph to fit into memory. Previously, I thought you needed Apache Spark and GraphFrames for billion-scale graph analytics. Now, however, I think all you need is a laptop. I have completely changed my old opinion about using Apache DataFusion for graph analytics."
Impressive!
anon7725•4m ago
> most existing graph algorithms require the graph to fit into memory.
You can get pretty far with sparse graphs, which are just arrays, in combination with memory mapping.
esafak•21m ago
Does it support out-of-core or multi-processor processing?
theLiminator•15m ago
DataFusion is really cool, it's kind of like the LLVM of the OLAP world.
slopblast•1h ago