Getting started with XGI

XGI is a Python library to make working with and analyzing complex systems with higher-order interactions easy. We have collected a lot of useful functions, algorithms, and tools for working with hypergraphs and simplicial complexes to make life easier.

We will

1. Create and load hypergraphs

2. Visualize hypergraphs and simplicial complexes

3. Show how to use the stats interface

4. Give an example of how to compare an empirical dataset to a null model.

We start off by loading the XGI library.

import matplotlib.pyplot as plt
import networkx as nx
import numpy as np
import pandas as pd
from IPython.display import display
from matplotlib.pyplot import cm

import xgi

You can easily check the version of XGI that you have installed by using the __version__ property:

xgi.__version__

'0.10.1'

Creating a hypergraph

We want to start with a hypergraph, and this can be done in several ways:

1. Build a hypergraph node-by-node and edge-by-edge (less common, but can be helpful in writing your own generative models)

2. Load an existing dataset

3. Sample from a random generative model

Let’s start with the first method.

Building a hypergraph

H_build = xgi.Hypergraph()
H_build.add_edge([1, 2], idx="a")
H_build.add_node(0)
H_build.add_edges_from([[3, 4], [0, 2, 3]])
H_build.add_nodes_from([9, 10])

XGI automatically assigns unique edge IDs (if a user doesn’t specify the ID)

H_build.edges

EdgeView(('a', 0, 1))

Why NodeViews and EdgeViews? These allow users to access many different properties and data structures from nodes and edges in a much simpler way. We will cover this more in depth later. For now, we can get the edges of which each node is a part and the nodes in each edge as follows:

H_build.edges

EdgeView(('a', 0, 1))

print(H_build.nodes.memberships())
print(H_build.edges.members())
print(H_build.nodes.memberships(2))
print(H_build.edges.members("a"))

{1: {'a'}, 2: {'a', 1}, 0: {1}, 3: {0, 1}, 4: {0}, 9: set(), 10: set()}
[{1, 2}, {3, 4}, {0, 2, 3}]
{'a', 1}
{1, 2}

Loading datasets

Moving on to method 2, one can load datasets in several different ways. First, we provide a companion data repository, xgi-data, where users can easily load several datasets in standard format:

H_enron = xgi.load_xgi_data("email-enron")

This dataset, for example, has a corresponding datasheet explaining its characteristics. The nodes (individuals) in this dataset contain associated email addresses and the edges (emails) contain associated timestamps. These attributes can be accessed by simply typing H.nodes[id] or H.edges[id] respectively.

print(f"The hypergraph has {H_enron.num_nodes} nodes and {H_enron.num_edges} edges")

# We can also print a summary of the hypergraph
print(H_enron)

The hypergraph has 148 nodes and 10885 edges
Hypergraph named email-Enron with 148 nodes and 10885 hyperedges

print("The first 10 node IDs are:")
print(list(H_enron.nodes)[:10])
print("\nThe first 10 edge IDs are:")
print(list(H_enron.edges)[:10])
print("\nThe attributes of node '4' are")
print(H_enron.nodes["4"])
print("\nThe attributes of edge '6' are")
print(H_enron.edges["6"])

The first 10 node IDs are:
['4', '1', '117', '129', '51', '41', '65', '107', '122', '29']

The first 10 edge IDs are:
['0', '1', '2', '3', '4', '5', '6', '7', '8', '9']

The attributes of node '4' are
{'name': 'robert.badeer@enron.com'}

The attributes of edge '6' are
{'timestamp': '2000-02-22T08:07:00'}

We can clean up this dataset to remove isolates, multi-edges, singletons, and to replace all IDs with integer IDs.

H_enron_cleaned = H_enron.cleanup(
    multiedges=False, singletons=False, isolates=False, relabel=True, in_place=False
)

print(H_enron_cleaned)

print("The first 10 node IDs are:")
print(list(H_enron_cleaned.nodes)[:10])

Hypergraph named email-Enron with 143 nodes and 1459 hyperedges
The first 10 node IDs are:
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Random null models

Lastly, we can create synthetic hypergraphs using random generative models. For example, we can extract the degree and edge size sequence from this dataset and wire them together at random (according to the Chung-Lu model) to create a random null model:

D = H_enron_cleaned.dual()
k = H_enron_cleaned.nodes.degree.asdict()
s = H_enron_cleaned.edges.size.asdict()

H_random = xgi.chung_lu_hypergraph(k, s)

We check whether this new hypergraph is connected and if not, the sizes of the connected components:

connected = xgi.is_connected(H_random)
if not connected:
    size, num = np.unique(
        [len(cc) for cc in xgi.connected_components(H_random)], return_counts=True
    )
    print(size)
    print(num)
    print("The connected components:")
    display(pd.DataFrame([size, num], columns=["Component size", "Number"]))

Converting between different representations

Incidence matrix

Any hypergraph can be expressed as an \(N \times M\) incidence matrix, \(I\), where \(N\) is the number of nodes and \(M\) is the number of edges. Rows indicate the node ID and the columns indicate the edge ID. \(I_{i,j}=1\) if node \(i\) is a member of edge \(j\) and zero otherwise.

I = xgi.incidence_matrix(H_enron_cleaned, sparse=False)
plt.spy(I, aspect="auto")
plt.xlabel("Hyperedges", fontsize=16)
plt.ylabel("Nodes", fontsize=16)
plt.show()

Bipartite network

Any hypergraph can be mapped to an equivalent bipartite network, \(G\), where nodes become nodes of the first type and edges become nodes of the second type. A bipartite link \((i, j)\) of \(G\) indicates that node \(i\) is a member of edge \(j\).

G = xgi.to_bipartite_graph(H_enron_cleaned)
nx.draw(G, node_color="black", node_size=10, width=0.5)

Adjacency matrix

We can represent a hypergraph with an \(N\times N\) adjacency matrix, \(A\), where \(N\) is the number of nodes. The adjacency matrix is a lossy format; different hypergraphs can create the same adjacency matrix. \(A_{i,j} = 1\) if there is at least one hyperedge containing both nodes \(i\) and \(j\).

A = xgi.adjacency_matrix(H_enron_cleaned, sparse=False)
plt.spy(A)

<matplotlib.image.AxesImage at 0x7f306aae5b50>

Line graph

The \(s\)-line graph is a network where nodes are hyperedges and a link is formed between two edges if they share at least \(s\) nodes in common.

G = xgi.to_line_graph(H_enron_cleaned, s=2)
nx.draw(G, node_color="black", node_size=10, width=0.5)

The dual

The dual is a hypergraph whose vertices and edges are interchanged. This can be pictured by swapping the nodes of the first and second types in the bipartite representation.

D = H_enron_cleaned.dual()
print(D)

Hypergraph named email-Enron with 1459 nodes and 143 hyperedges

We can do much more such as computing different properties, reading/writing, converting to/from different data structures, hypergraph null models and much more!

See the Read The Docs for more information: https://xgi.readthedocs.io

Visualization

The first step for drawing a hypergraph is to choose a layout for the nodes. At the moment the available layouts are:

• random_layout: positions nodes uniformly at random in the unit square.

• pairwise_spring_layout: positions the nodes using the Fruchterman-Reingold force-directed algorithm on the projected graph.

• barycenter_spring_layout and weighted_barycenter_spring_layout: slight modification of pairwise_spring_layout

Each layout returns a dictionary that maps nodes ID into (x, y) coordinates.

seed = 0
H_viz = xgi.random_hypergraph(20, [0.08, 0.01, 0.001], seed=seed)
pos1 = xgi.random_layout(H_viz)
pos2 = xgi.pairwise_spring_layout(H_viz)
pos3 = xgi.barycenter_spring_layout(H_viz)
pos4 = xgi.weighted_barycenter_spring_layout(H_viz)

/home/lucasm/WORK/SCIENCE/xgi/xgi/generators/random.py:150: UserWarning: This method is much slower than fast_random_hypergraph
  warn("This method is much slower than fast_random_hypergraph")

xgi.draw(H_viz, pos1)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f306a43c0d0>,
  <matplotlib.collections.LineCollection at 0x7f30660ba2d0>,
  <matplotlib.collections.PatchCollection at 0x7f3066379b50>))

We can easily add labels to the nodes to more easily compare the visualization to the hypergraph:

xgi.draw(H_build, node_labels=True, hyperedge_labels=True)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f306a4f9b50>,
  <matplotlib.collections.LineCollection at 0x7f306a463bd0>,
  <matplotlib.collections.PatchCollection at 0x7f306a460f10>))

Colors of the hyperedges are designed to match the hyperedge size, but the colormap can be tweaked:

# Sequential colormap
cmap = cm.Paired

xgi.draw(H_viz, pos4, edge_fc_cmap=cmap)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f306612edd0>,
  <matplotlib.collections.LineCollection at 0x7f3066118890>,
  <matplotlib.collections.PatchCollection at 0x7f3065e88e10>))

Other parameters can be changed as well:

cmap = cm.Reds
dyad_color = "gray"
dyad_lw = 4
node_fc = "black"
node_ec = "white"
node_lw = 2
node_size = 20

xgi.draw(
    H_viz,
    pos4,
    edge_fc_cmap=cmap,
    dyad_color=dyad_color,
    dyad_lw=dyad_lw,
    node_fc=node_fc,
    node_ec=node_ec,
    node_lw=node_lw,
    node_size=node_size,
)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f3065ee0690>,
  <matplotlib.collections.LineCollection at 0x7f3065f726d0>,
  <matplotlib.collections.PatchCollection at 0x7f3065ee09d0>))

Simplicial complexes can be visualized using the same functions for node layout and drawing.

Technical note

By definition, a simplicial complex object contains all sub-simplices. This would make the visualization heavy since all sub-simplices contained in a maximal simplex would overlap. The automatic solution for this, implemented by default in all the layouts, is to convert the simplicial complex into a hypergraph composed by solely by its maximal simplices.

Visual note

To visually distinguish simplicial complexes from hypergraphs, the draw function will also show all links contained in each maximal simplices (while omitting simplices of intermediate orders).

SC = xgi.SimplicialComplex()
SC.add_simplices_from([[3, 4, 5], [3, 6], [6, 7, 8, 9], [1, 4, 10, 11, 12], [1, 4]])

pos = xgi.pairwise_spring_layout(SC)
xgi.draw(SC, pos)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f30660ba7d0>,
  <matplotlib.collections.LineCollection at 0x7f306615dd50>,
  <matplotlib.collections.PatchCollection at 0x7f30667f78d0>))

We can even color/draw the nodes and edges based on statistics!

n = 100
is_connected = False
while not is_connected:
    H = xgi.random_hypergraph(n, [0.03, 0.0002, 0.00001])
    is_connected = xgi.is_connected(H)
pos = xgi.barycenter_spring_layout(H)

/home/lucasm/WORK/SCIENCE/xgi/xgi/generators/random.py:150: UserWarning: This method is much slower than fast_random_hypergraph
  warn("This method is much slower than fast_random_hypergraph")

plt.figure(figsize=(10, 10))
ax = plt.subplot(111)
xgi.draw(
    H,
    pos,
    node_size=H.nodes.degree,
    node_lw=H.nodes.average_neighbor_degree,
    node_fc=H.nodes.degree,
    ax=ax,
)

(<Axes: >,
 (<matplotlib.collections.PathCollection at 0x7f3065feaf90>,
  <matplotlib.collections.LineCollection at 0x7f306a2703d0>,
  <matplotlib.collections.PatchCollection at 0x7f306a7f9450>))

The stats package

The stats package is one of the features that sets xgi apart from other libraries. It provides a common interface to all statistics that can be computed from a network, its nodes, or edges.

Consider the degree of the nodes of a hypergraph H. After computing the values of the degrees, one may wish to store them in a dict, a list, an array, a dataframe, etc. Through the stats package, xgi provides a simple interface that seamlessly allows for this type conversion. This is done via the NodeStat class.

import numpy as np
import pandas as pd

import xgi

H_stats = xgi.Hypergraph([[1, 2, 3], [2, 3, 4, 5], [3, 4, 5]])
H_stats.degree()

{1: 1, 2: 2, 3: 3, 4: 2, 5: 2}

H_stats.nodes.degree

NodeStat('degree')

This NodeStat object allows us to choose the datatype, specify keywords, and get basic statistics from these properties

print("As a dictionary:")
print(H_stats.nodes.degree.asdict())
print("\nAs a list:")
print(H_stats.nodes.degree.aslist())
print("\nAs a numpy array:")
print(H_stats.nodes.degree.asnumpy())
print("\nAs a pandas dataframe:")
print(H_stats.nodes.degree.aspandas())

As a dictionary:
{1: 1, 2: 2, 3: 3, 4: 2, 5: 2}

As a list:
[1, 2, 3, 2, 2]

As a numpy array:
[1 2 3 2 2]

As a pandas dataframe:
1    1
2    2
3    3
4    2
5    2
Name: degree, dtype: int64

H_stats.nodes.degree.aspandas().plot()

<Axes: >

print(H_stats.nodes.degree(order=2).asdict())

{1: 1, 2: 1, 3: 2, 4: 1, 5: 1}

st = H_stats.nodes.degree
np.round([st.max(), st.min(), st.mean(), st.median(), st.var(), st.std()], 3)

array([3.   , 1.   , 2.   , 2.   , 0.4  , 0.632])

Likewise, for edges,

st = H_stats.edges.size
st

EdgeStat('size')

np.round([st.max(), st.min(), st.mean(), st.median(), st.var(), st.std()], 3)

array([4.   , 3.   , 3.333, 3.   , 0.222, 0.471])

The interface for attributes is very similar. If we add nodal attributes, for example

H_stats.add_nodes_from(
    [
        (1, {"color": "red", "name": "horse"}),
        (2, {"color": "blue", "name": "pony"}),
        (3, {"color": "yellow", "name": "zebra"}),
        (4, {"color": "red", "name": "orangutan", "age": 20}),
        (5, {"color": "blue", "name": "fish", "age": 2}),
    ]
)
print(H_stats.nodes.attrs("color").asdict())

{1: 'red', 2: 'blue', 3: 'yellow', 4: 'red', 5: 'blue'}

print(H_stats.nodes.attrs[1])

{'color': 'red', 'name': 'horse'}

One can also filter nodes and edges by their attributes as well as any associated statistic. For example,

print(H_stats.nodes.filterby("degree", 2))
print(H_stats.nodes.filterby_attr("color", "blue"))

[2, 4, 5]
[2, 5]

H_stats.edges.filterby("size", 3).members()

[{1, 2, 3}, {3, 4, 5}]

H_stats.nodes.multi(["degree", "clustering_coefficient"]).aspandas().groupby(
    "degree"
).agg("mean")

	clustering_coefficient
degree
1	1.000000
2	0.888889
3	0.666667

(
    H_stats.nodes.multi(["degree", "clustering_coefficient", "average_neighbor_degree"])
    .aspandas()
    .groupby("degree")
    .agg("mean")
    .plot(marker="o")
)

<Axes: xlabel='degree'>

For even more functionality, see the Read The Docs: https://xgi.readthedocs.io

Let’s compare the assortativity of an empirical dataset (Let’s choose the email-enron dataset) to a random null model. We start by loading the Enron dataset and constructing a Chung-Lu hypergraph using the dataset’s degree sequence and edge size sequence:

H = H_enron
k = H.nodes.degree.asdict()
s = H.edges.size.asdict()

H_null = xgi.chung_lu_hypergraph(k, s)

Now we use the definitions of assortativity in “Configuration models of random hypergraphs” by Phil Chodrow to compare the assortativity of the random null model to the original dataset.

labels = ["uniform", "top-2", "top-bottom"]

assort_orig = []
assort_null = []
for l in labels:
    assort_orig.append(xgi.degree_assortativity(H, kind=l))
    assort_null.append(xgi.degree_assortativity(H_null, kind=l))

x = np.arange(len(labels))
width = 0.35

fig, ax = plt.subplots()
rects1 = ax.bar(x - width / 2, assort_orig, width, label="Original")
rects2 = ax.bar(x + width / 2, assort_null, width, label="Null model")
ax.set_xticks(x)
ax.set_xticklabels(labels, fontsize=16)
ax.set_ylabel("Assortativity", fontsize=16)
ax.legend()
plt.tight_layout()
plt.show()

From this plot we can see that the original dataset is more assortative than the random null model.

For more details, visit https://xgi.readthedocs.io