Add benchmark for PC and GES causal discovery algorithms on simulated Linear Gaussian data #7

Vanshitaaa20 · 2025-07-15T21:44:32Z

This PR introduces a new benchmark to evaluate the performance of PC and GES structure learning algorithms on Linear Gaussian Bayesian networks.

Vanshitaaa20 · 2025-07-16T15:13:57Z

@ankurankan can you pls review this?

ankurankan · 2025-07-25T16:50:57Z

@Vanshitaaa20 Sorry for the delay in reviewing this. I still haven't got a chance to look at it. In the meantime, I would suggest restructuring your code similar to the code that we currently have for the Conditional Independence test benchmarks. So, the script should write a CSV file with all the results, and the frontend should read in the CSV and make the plots.

@Nimish-4 @RudraCodesForU Would any of you be interested in reviewing this?

RudraCodesForU · 2025-07-26T06:36:05Z

benchmarks/causal_discovery.py

+num_trials = 10
+shd_pc_list = []
+shd_ges_list = []
+


@Vanshitaaa20 , add the algo equations like the benchmarking script in the "doc string" form

RudraCodesForU · 2025-07-26T06:37:46Z

benchmarks/causal_discovery.py

+from pgmpy.factors.continuous import LinearGaussianCPD
+from pgmpy.models import LinearGaussianBayesianNetwork as LGBN
+
+


@Vanshitaaa20 , Add def of GES and PC in doc string form in the script.

RudraCodesForU · 2025-07-26T06:39:51Z

@ankurankan, I will be reviewing it since the benchmarking and web part had been coded by me. @Vanshitaaa20 ,

@Vanshitaaa20 Sorry for the delay in reviewing this. I still haven't got a chance to look at it. In the meantime, I would suggest restructuring your code similar to the code that we currently have for the Conditional Independence test benchmarks. So, the script should write a CSV file with all the results, and the frontend should read in the CSV and make the plots.

@Nimish-4 @RudraCodesForU Would any of you be interested in reviewing this?

@ankurankan , I am reviewing this , as the benchmark and web part is assigned to me.

RudraCodesForU · 2025-07-27T07:21:53Z

benchmarks/causal_discovery.py

+
+    print(" SHD (PC):", shd_pc)
+    print(" SHD (GES):", shd_ges)
+


add a custom csv import instead of print , which gets stored in causalbench/results folder. Perform this changes first . @Vanshitaaa20

ankurankan · 2025-07-28T17:35:07Z

benchmarks/causal_discovery.py

+        dag.add_node(f"X_{i}")
+    return dag
+
+def compute_shd_direct(true_dag, learned_dag) -> int:


We have an implementation of SHD at pgmpy.metrics.SHD.

ankurankan · 2025-08-07T20:11:36Z

@Vanshitaaa20 Any updates on this?

Vanshitaaa20 · 2025-08-08T12:27:09Z

@Vanshitaaa20 Any updates on this?

I replaced the custom SHD function with pgmpy.metrics.SHD as suggested. However, I'm getting the following error during execution:

networkx.exception.NetworkXError: Node X_3 in nodelist is not in G

It seems that true_dag and learned_dag might have different sets of nodes. Should I align their node sets explicitly before computing SHD, or is there a preferred way to handle this?

ankurankan · 2025-08-08T12:30:42Z

@Vanshitaaa20 could also potentially be a bug in the SHD method. Is it possible for you to print out the models that is resulting in the error? We can then run SHD on them and debug the issue.

Vanshitaaa20 · 2025-08-08T12:51:25Z

@Vanshitaaa20 could also potentially be a bug in the SHD method. Is it possible for you to print out the models that is resulting in the error? We can then run SHD on them and debug the issue.

Sure, I’ll print the true_dag and learned_dag and share the ones causing the error so we can debug SHD on them directly. Will update soon.

ankurankan · 2025-08-08T12:54:21Z

@Vanshitaaa20 Make sure to print both the edge list and the node list, as printing edges would leave out any disconnected nodes. Also, feel free to ask any questions on discord if you get stuck anywhere.

Vanshitaaa20 · 2025-08-23T13:52:00Z

@ankurankan sorry for the delay! i updated the SHD benchmarking code to print both the edge list and node list for easier debugging and alignment verification, and also added a skip_trial mechanism so that any trial raising a ValueError is skipped gracefully without interrupting the benchmarking run.

ankurankan · 2025-08-26T13:10:34Z

benchmarks/causal_discovery.py

+    for i in range(num_nodes):
+        dag.add_node(f"X_{i}")


Why do we need to add the nodes here? Doesn't the get_random method already give the DAG on the specified number of nodes?

ankurankan · 2025-08-26T13:11:52Z

benchmarks/causal_discovery.py

+    lgbn = LGBN(true_dag.edges())
+    lgbn.add_nodes_from(true_dag.nodes())
+    for node in true_dag.nodes():
+        parents = list(lgbn.get_parents(node))
+        beta = [0.0] + list(np.random.uniform(0.5, 1.5, size=len(parents)))
+        cpd = LinearGaussianCPD(variable=node, beta=beta, std=1, evidence=parents)
+        lgbn.add_cpds(cpd)


LinearGaussianBayesianNetwork has a get_random method that should give a full randomly generated model.

ankurankan · 2025-08-26T13:12:20Z

benchmarks/causal_discovery.py

+    data = lgbn.simulate(n=1000)
+
+    # PC Estimation
+    try:


No need to do a try-except. Better to let it fail; it will help in detecting bugs.

ankurankan · 2025-08-26T13:12:27Z

benchmarks/causal_discovery.py

+        continue
+
+    # GES Estimation
+    try:


ankurankan · 2025-08-26T13:13:04Z

benchmarks/causal_discovery.py

+    # Ensure node alignment
+    all_nodes = sorted(set(true_dag.nodes()).union(
+        set(learned_dag_pc.nodes())).union(set(learned_dag_ges.nodes())))
+    true_dag.add_nodes_from(all_nodes)
+    learned_dag_pc.add_nodes_from(all_nodes)
+    learned_dag_ges.add_nodes_from(all_nodes)


Why is this required? Both the true model and the learned dag would have the same nodes/variables as in the dataset. Right?

ankurankan · 2025-08-26T13:14:02Z

benchmarks/causal_discovery.py

+
+    # Compute SHD using built-in method
+    try:
+        shd_pc = SHD(true_dag, learned_dag_pc)


Try-except should only be used when we know in what situation the code will throw an error, and that is the expected behavior. Not required here.

@Vanshitaaa20 , Like @ankurankan said, remove try-catch exceptions , let the test cases fail, so that we could identify the potential causes and make changes on it.

RudraCodesForU · 2025-09-13T04:55:04Z

benchmarks/causal_discovery.py

+
+    # Compute SHD using built-in method
+    try:
+        shd_pc = SHD(true_dag, learned_dag_pc)


@Vanshitaaa20 , Like @ankurankan said, remove try-catch exceptions , let the test cases fail, so that we could identify the potential causes and make changes on it.

Vanshitaaa20 · 2025-09-13T05:53:11Z

@ankurankan @RudraCodesForU I previously tried fixing the error by explicitly aligning nodes across the graphs, but that doesn’t seem to help, so I removed that part. I also removed the try-except block so the actual error shows up directly. Could you please help me identify the real cause of this issue?

Vanshitaaa20 · 2025-09-13T06:00:40Z

I noticed that SHD throws a ValueError: The graphs must have the same nodes during benchmarking. After adding debug prints, the issue seems to be that the learned DAGs from PC and GES sometimes exclude isolated nodes.

For example in one trial :-

Trial 5/10 INFO:pgmpy: Datatype (N=numerical, C=Categorical Unordered, O=Categorical Ordered) inferred from data: {'X_0': 'N', 'X_2': 'N', 'X_1': 'N', 'X_4': 'N', 'X_3': 'N'} Working for n conditional variables: 2: 40%|██████████████████████████████████ | 2/5 [00:00<00:00, 374.79it/s] INFO:pgmpy: Datatype (N=numerical, C=Categorical Unordered, O=Categorical Ordered) inferred from data: {'X_0': 'N', 'X_2': 'N', 'X_1': 'N', 'X_4': 'N', 'X_3': 'N'} INFO:pgmpy: Datatype (N=numerical, C=Categorical Unordered, O=Categorical Ordered) inferred from data: {'X_0': 'N', 'X_2': 'N', 'X_1': 'N', 'X_4': 'N', 'X_3': 'N'} INFO:pgmpy: Datatype (N=numerical, C=Categorical Unordered, O=Categorical Ordered) inferred from data: {'X_0': 'N', 'X_2': 'N', 'X_1': 'N', 'X_4': 'N', 'X_3': 'N'} True nodes: {'X_3', 'X_0', 'X_1', 'X_4', 'X_2'} PC nodes: {'X_3', 'X_4', 'X_0', 'X_2'} GES nodes: {'X_0', 'X_4', 'X_3', 'X_2'} Traceback (most recent call last): File "c:\causalbench\causalbench\benchmarks\causal_discovery.py", line 65, in <module> shd_pc = SHD(true_dag, learned_dag_pc) File "C:\Users\vansh\AppData\Local\Programs\Python\Python313\Lib\site-packages\pgmpy\metrics\metrics.py", line 437, in SHD raise ValueError("The graphs must have the same nodes.") ValueError: The graphs must have the same nodes

ankurankan · 2025-09-14T07:09:56Z

@Vanshitaaa20 The error is because in Line 52 and Line 57, you are constructing DiGraph objects using only the edges from the learned models. As a result, if there are any disconnected nodes in the learned model, it doesn't get added to the DiGraph object, and hence those are missing in the model as you can see in your error message. So,

What are we not directly using the learned DAG objects for SHD?
If DiGraph objects need to be created, you can add a statement like: learned_dag_pc.add_nodes_from(pc_est.nodes()) to make sure that disconnected nodes get added too.

added benchmark for pc and ges

e7be305

RudraCodesForU suggested changes Jul 26, 2025

View reviewed changes

RudraCodesForU suggested changes Jul 27, 2025

View reviewed changes

ankurankan reviewed Jul 28, 2025

View reviewed changes

ankurankan moved this to PR under progress in Mentored Projects Aug 7, 2025

ankurankan added this to Mentored Projects Aug 7, 2025

ankurankan requested review from ankurankan and RudraCodesForU August 7, 2025 20:54

ankurankan assigned Vanshitaaa20 Aug 7, 2025

Print both edge list and node list in SHD benchmarking

9632233

ankurankan requested changes Aug 26, 2025

View reviewed changes

RudraCodesForU suggested changes Sep 13, 2025

View reviewed changes

Remove redundant node alignment; keep raw DAGs for debugging

e49851a

		from pgmpy.factors.continuous import LinearGaussianCPD
		from pgmpy.models import LinearGaussianBayesianNetwork as LGBN

Add benchmark for PC and GES causal discovery algorithms on simulated Linear Gaussian data #7

Are you sure you want to change the base?

Add benchmark for PC and GES causal discovery algorithms on simulated Linear Gaussian data #7

Uh oh!

Conversation

Vanshitaaa20 commented Jul 15, 2025

Uh oh!

Vanshitaaa20 commented Jul 16, 2025

Uh oh!

ankurankan commented Jul 25, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RudraCodesForU commented Jul 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ankurankan commented Aug 7, 2025

Uh oh!

Vanshitaaa20 commented Aug 8, 2025

Uh oh!

ankurankan commented Aug 8, 2025

Uh oh!

Vanshitaaa20 commented Aug 8, 2025

Uh oh!

ankurankan commented Aug 8, 2025

Uh oh!

Vanshitaaa20 commented Aug 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vanshitaaa20 commented Sep 13, 2025

Uh oh!

Vanshitaaa20 commented Sep 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ankurankan commented Sep 14, 2025

Uh oh!

Uh oh!

Vanshitaaa20 commented Aug 23, 2025 •

edited

Loading

Vanshitaaa20 commented Sep 13, 2025 •

edited

Loading