Skip to content

Commit 3e06602

Browse files
suryanshgargbpgcsuryanshgargbpgc
committed
solves issue projectmesa#129
1. Adds Github actions workflow similar to https://github.com/projectmesa/mesa/blob/main/.github/workflows/benchmarks.yml 2. Documentation added (benchmark.md) 3.updated mkdocs.yml to include new documentation in the navigation
1 parent a097949 commit 3e06602

File tree

3 files changed

+290
-0
lines changed

3 files changed

+290
-0
lines changed

‎.github/workflows/benchmarks.yml

+214
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
name: Benchmarks
2+
3+
on:
4+
pull_request:
5+
branches: [ main ]
6+
types: [ opened, synchronize, reopened, ready_for_review ]
7+
push:
8+
branches: [ main ]
9+
workflow_dispatch:
10+
11+
jobs:
12+
benchmark:
13+
name: Run Performance Benchmarks
14+
runs-on: ubuntu-latest
15+
if: github.event.pull_request.draft == false
16+
17+
steps:
18+
- name: Checkout Repository
19+
uses: actions/checkout@v3
20+
21+
- name: Set up Python
22+
uses: actions/setup-python@v4
23+
with:
24+
python-version: '3.10'
25+
cache: 'pip'
26+
27+
- name: Install dependencies
28+
run: |
29+
python -m pip install --upgrade pip
30+
python -m pip install -e ".[dev]"
31+
python -m pip install perfplot matplotlib seaborn
32+
33+
- name: Try to set up GPU support (optional)
34+
run: |
35+
# Try to install CUDA dependencies for GPU-accelerated benchmarks
36+
# This won't fail the workflow if it doesn't work, as most CI runners don't have GPUs
37+
python -m pip install numba cupy-cuda11x || echo "GPU support not available (expected for most CI runners)"
38+
39+
- name: Run SugarScape Benchmark (Small Dataset)
40+
run: |
41+
cd examples/sugarscape_ig
42+
python -c "
43+
import sys
44+
import time
45+
from performance_comparison import SugarScapeSetup, mesa_frames_polars_numba_parallel, mesa_implementation
46+
47+
# Run a smaller subset for CI benchmarks (faster execution)
48+
setup = SugarScapeSetup(50000)
49+
50+
print('Running mesa-frames implementation...')
51+
start_time = time.time()
52+
mf_model = mesa_frames_polars_numba_parallel(setup)
53+
mf_time = time.time() - start_time
54+
print(f'mesa-frames implementation completed in {mf_time:.2f} seconds')
55+
56+
print('Running mesa implementation...')
57+
start_time = time.time()
58+
mesa_model = mesa_implementation(setup)
59+
mesa_time = time.time() - start_time
60+
print(f'mesa implementation completed in {mesa_time:.2f} seconds')
61+
62+
print('Benchmark complete!')
63+
64+
# Save timing results for the PR comment
65+
with open('sugarscape_results.txt', 'w') as f:
66+
f.write(f'mesa-frames: {mf_time:.2f}s\\n')
67+
f.write(f'mesa: {mesa_time:.2f}s\\n')
68+
f.write(f'speedup: {mesa_time/mf_time:.2f}x\\n')
69+
"
70+
71+
- name: Run Boltzmann Wealth Benchmark (Small Dataset)
72+
run: |
73+
cd examples/boltzmann_wealth
74+
python -c "
75+
import sys
76+
import time
77+
from performance_plot import mesa_frames_polars_concise, mesa_implementation
78+
79+
# Run a smaller subset for CI benchmarks (faster execution)
80+
print('Running mesa-frames implementation...')
81+
start_time = time.time()
82+
mf_model = mesa_frames_polars_concise(10000)
83+
mf_time = time.time() - start_time
84+
print(f'mesa-frames implementation completed in {mf_time:.2f} seconds')
85+
86+
print('Running mesa implementation...')
87+
start_time = time.time()
88+
mesa_model = mesa_implementation(10000)
89+
mesa_time = time.time() - start_time
90+
print(f'mesa implementation completed in {mesa_time:.2f} seconds')
91+
92+
print('Benchmark complete!')
93+
94+
# Save timing results for the PR comment
95+
with open('boltzmann_results.txt', 'w') as f:
96+
f.write(f'mesa-frames: {mf_time:.2f}s\\n')
97+
f.write(f'mesa: {mesa_time:.2f}s\\n')
98+
f.write(f'speedup: {mesa_time/mf_time:.2f}x\\n')
99+
"
100+
101+
- name: Generate Simple Benchmark Visualizations
102+
run: |
103+
python -c "
104+
import matplotlib.pyplot as plt
105+
import numpy as np
106+
import os
107+
108+
# Function to read benchmark results
109+
def read_results(filename):
110+
results = {}
111+
with open(filename, 'r') as f:
112+
for line in f:
113+
key, value = line.strip().split(': ')
114+
results[key] = value
115+
return results
116+
117+
# Create visualization for Sugarscape benchmark
118+
sugarscape_results = read_results('examples/sugarscape_ig/sugarscape_results.txt')
119+
boltzmann_results = read_results('examples/boltzmann_wealth/boltzmann_results.txt')
120+
121+
# Create a simple bar chart comparing execution times
122+
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
123+
124+
# Sugarscape plot
125+
sugarscape_mesa_time = float(sugarscape_results['mesa'].replace('s', ''))
126+
sugarscape_mf_time = float(sugarscape_results['mesa-frames'].replace('s', ''))
127+
ax1.bar(['mesa-frames', 'mesa'], [sugarscape_mf_time, sugarscape_mesa_time])
128+
ax1.set_title('SugarScape Benchmark (50k agents)')
129+
ax1.set_ylabel('Execution time (s)')
130+
ax1.text(0, sugarscape_mf_time/2, f'{sugarscape_mf_time:.2f}s',
131+
ha='center', va='center', color='white', fontweight='bold')
132+
ax1.text(1, sugarscape_mesa_time/2, f'{sugarscape_mesa_time:.2f}s',
133+
ha='center', va='center', color='white', fontweight='bold')
134+
ax1.text(0.5, max(sugarscape_mf_time, sugarscape_mesa_time) * 0.9,
135+
f'Speedup: {sugarscape_results[\"speedup\"]}',
136+
ha='center', va='center', bbox=dict(facecolor='white', alpha=0.8))
137+
138+
# Boltzmann plot
139+
boltzmann_mesa_time = float(boltzmann_results['mesa'].replace('s', ''))
140+
boltzmann_mf_time = float(boltzmann_results['mesa-frames'].replace('s', ''))
141+
ax2.bar(['mesa-frames', 'mesa'], [boltzmann_mf_time, boltzmann_mesa_time])
142+
ax2.set_title('Boltzmann Wealth Benchmark (10k agents)')
143+
ax2.set_ylabel('Execution time (s)')
144+
ax2.text(0, boltzmann_mf_time/2, f'{boltzmann_mf_time:.2f}s',
145+
ha='center', va='center', color='white', fontweight='bold')
146+
ax2.text(1, boltzmann_mesa_time/2, f'{boltzmann_mesa_time:.2f}s',
147+
ha='center', va='center', color='white', fontweight='bold')
148+
ax2.text(0.5, max(boltzmann_mf_time, boltzmann_mesa_time) * 0.9,
149+
f'Speedup: {boltzmann_results[\"speedup\"]}',
150+
ha='center', va='center', bbox=dict(facecolor='white', alpha=0.8))
151+
152+
plt.tight_layout()
153+
plt.savefig('benchmark_results.png', dpi=150)
154+
print('Benchmark visualization saved as benchmark_results.png')
155+
"
156+
157+
- name: Save Benchmark Results
158+
if: always()
159+
uses: actions/upload-artifact@v3
160+
with:
161+
name: benchmark-results
162+
path: |
163+
examples/sugarscape_ig/*.png
164+
examples/sugarscape_ig/*.txt
165+
examples/boltzmann_wealth/*.png
166+
examples/boltzmann_wealth/*.txt
167+
benchmark_results.png
168+
retention-days: 90
169+
170+
- name: Add Benchmark Comment
171+
if: github.event_name == 'pull_request'
172+
uses: actions/github-script@v6
173+
with:
174+
github-token: ${{ secrets.GITHUB_TOKEN }}
175+
script: |
176+
const fs = require('fs');
177+
178+
// Read benchmark results
179+
let sugarscapeResults = '';
180+
let boltzmannResults = '';
181+
182+
try {
183+
sugarscapeResults = fs.readFileSync('./examples/sugarscape_ig/sugarscape_results.txt', 'utf8');
184+
boltzmannResults = fs.readFileSync('./examples/boltzmann_wealth/boltzmann_results.txt', 'utf8');
185+
} catch (err) {
186+
console.error('Error reading benchmark results:', err);
187+
}
188+
189+
// Create a comment with benchmark results
190+
const comment = `## 📊 Performance Benchmark Results
191+
192+
The benchmarks have been executed successfully.
193+
194+
### SugarScape Model (50k agents, 100 steps)
195+
\`\`\`
196+
${sugarscapeResults}
197+
\`\`\`
198+
199+
### Boltzmann Wealth Model (10k agents, 100 steps)
200+
\`\`\`
201+
${boltzmannResults}
202+
\`\`\`
203+
204+
![Benchmark Results](${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID}/artifacts/benchmark-results/benchmark_results.png)
205+
206+
[Click here to download full benchmark results](${process.env.GITHUB_SERVER_URL}/${process.env.GITHUB_REPOSITORY}/actions/runs/${process.env.GITHUB_RUN_ID})
207+
`;
208+
209+
github.rest.issues.createComment({
210+
issue_number: context.issue.number,
211+
owner: context.repo.owner,
212+
repo: context.repo.repo,
213+
body: comment
214+
});

‎docs/general/contributing/benchmarks.md

+75
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Benchmarking in mesa-frames
2+
3+
As a library focused on performance improvements, it's crucial that mesa-frames maintains its speed advantages over time. To ensure this, we've implemented an automated benchmarking system that runs on every pull request targeting the main branch.
4+
5+
## How the Benchmark Workflow Works
6+
7+
The automated benchmark workflow runs on GitHub Actions and performs the following steps:
8+
9+
1. Sets up a Python environment with all necessary dependencies
10+
2. Installs optional GPU dependencies (if available in the runner)
11+
3. Runs a small subset of our benchmark examples:
12+
- SugarScape model (with 50,000 agents)
13+
- Boltzmann Wealth model (with 10,000 agents)
14+
4. Generates timing results comparing mesa-frames to the original Mesa implementation
15+
5. Produces a visualization of the benchmark results
16+
6. Posts a comment on the PR with the benchmark results
17+
7. Uploads full benchmark artifacts for detailed inspection
18+
19+
## Interpreting Benchmark Results
20+
21+
When reviewing a PR with benchmark results, look for:
22+
23+
1. **Successful execution**: The benchmarks should complete without errors
24+
2. **Performance impact**: Check if the PR introduces any performance regressions
25+
3. **Expected changes**: If the PR is aimed at improving performance, verify that the benchmarks show the expected improvements
26+
27+
The benchmark comment will include:
28+
- Execution time for both mesa-frames and Mesa implementations
29+
- The speedup factor (how many times faster mesa-frames is compared to Mesa)
30+
- A visualization comparing the performance
31+
32+
## Running Benchmarks Locally
33+
34+
To run the same benchmarks locally and compare your changes to the current main branch:
35+
36+
```bash
37+
# Clone the repository
38+
git clone https://github.com/projectmesa/mesa-frames.git
39+
cd mesa-frames
40+
41+
# Install dependencies
42+
pip install -e ".[dev]"
43+
pip install perfplot matplotlib seaborn
44+
45+
# Run the Sugarscape benchmark
46+
cd examples/sugarscape_ig
47+
python performance_comparison.py
48+
49+
# Run the Boltzmann Wealth benchmark
50+
cd ../boltzmann_wealth
51+
python performance_plot.py
52+
```
53+
54+
The full benchmarks will take longer to run than the CI version as they test with more agents.
55+
56+
## Adding New Benchmarks
57+
58+
When adding new models or features to mesa-frames, consider adding benchmark tests to ensure their performance:
59+
60+
1. Create a benchmark script in the `examples` directory
61+
2. Implement both mesa-frames and Mesa versions of the model
62+
3. Use the `perfplot` library to measure and visualize performance
63+
4. Update the GitHub Actions workflow to include your new benchmark (with a small dataset for CI)
64+
65+
## Tips for Performance Optimization
66+
67+
When optimizing code in mesa-frames:
68+
69+
1. **Always benchmark your changes**: Don't assume changes will improve performance without measuring
70+
2. **Focus on real-world use cases**: Optimize for patterns that users are likely to encounter
71+
3. **Balance readability and performance**: Code should remain maintainable even while being optimized
72+
4. **Document performance characteristics**: Note any trade-offs or specific usage patterns that affect performance
73+
5. **Test on different hardware**: If possible, verify improvements on both CPU and GPU environments
74+
75+
Remember that consistent, predictable performance is often more valuable than squeezing out every last bit of speed at the cost of complexity or stability.

‎mkdocs.yml

+1
Original file line numberDiff line numberDiff line change
@@ -115,4 +115,5 @@ nav:
115115
- API Reference: api/index.html
116116
- Contributing:
117117
- Contribution Guide: contributing.md
118+
- Benchmarking: contributing/benchmarks.md
118119
- Roadmap: roadmap.md

0 commit comments

Comments
 (0)