Skip to content

Tests for causal prediction #1321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

Tests for causal prediction #1321

wants to merge 2 commits into from

Conversation

maartenvanhooftds
Copy link

@maartenvanhooftds maartenvanhooftds commented Jun 2, 2025 •
edited
Loading

Inspired by Issue 1313.

Changes:

  • Added some unit tests
  • Created a test for ERM and CACM based on this notebook, but with some changes here and there, e.g. such that we don't have to download data during tests.

First contribution here, please review critically for any mistakes or inconsistencies!

@maartenvanhooftds maartenvanhooftds force-pushed the causal-prediction-tests branch 2 times, most recently from 5df95b8 to 58a2504 Compare June 2, 2025 05:56
@maartenvanhooftds maartenvanhooftds changed the title Tests for causal prediction algorithms Tests for causal prediction Jun 2, 2025
maartenvanhooftds
maartenvanhooftds commented Jun 2, 2025
View reviewed changes
@maartenvanhooftds maartenvanhooftds marked this pull request as ready for review June 2, 2025 06:00
@maartenvanhooftds maartenvanhooftds force-pushed the causal-prediction-tests branch from 58a2504 to 424755d Compare June 2, 2025 12:43
@maartenvanhooftds
Tests for causal prediction
17a97cb 
Signed-off-by: maartenvanhooft <[email protected]>
@maartenvanhooftds maartenvanhooftds force-pushed the causal-prediction-tests branch from 424755d to 17a97cb Compare June 2, 2025 12:44
maartenvanhooftds
maartenvanhooftds commented Jun 2, 2025
View reviewed changes
amit-sharma
amit-sharma reviewed Jun 8, 2025
View reviewed changes
Copy link
Member

@amit-sharma amit-sharma left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for adding this PR, @maartenvanhooftds . The tests make sense, but I'm wondering if we can have a stronger test that compares CACM and ERM.
How about the following property: difference in accuracy between a test dataset from the same distribution as train, and the main test dataset? That would be higher for ERM and we can check as a comparison assert. Can you add this for your setup?

tests/causal_prediction/test_causal_prediction_algorithms.py Outdated
results = trainer.test(algorithm, dataloaders=loaders["test_loaders"])
assert isinstance(results, list)
assert len(results) > 0
for r in results:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does the code assume that both ERM and CACM will lead to >0.7 accuracy? What is the exclusive property that we can testing for CACM?

@maartenvanhooftds
Copy link
Author

maartenvanhooftds commented Jun 10, 2025 •
edited
Loading

Great feedback, thanks! Will implement it later this week.

Edit: Sorry, it has been taking a bit longer, just started a new job. It's still on my mind though.

@maartenvanhooftds maartenvanhooftds force-pushed the causal-prediction-tests branch from 2fec8f0 to f228a4e Compare June 22, 2025 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@amit-sharma amit-sharma Awaiting requested review from amit-sharma

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@maartenvanhooftds @amit-sharma