Skip to content

[publish images] upload encrypted core dumps from CI #19189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 48 commits into
base: main
Choose a base branch
from
Open

Conversation

190n
Copy link
Collaborator

@190n 190n commented Apr 22, 2025

What does this PR do?

Makes runner.node.mjs upload core dumps from any crashes that occurred on Linux in testing as a Buildkite artifact. These are uploaded as a gzipped tarball, encrypted with an age public key. They can only be decrypted with the private key which our team will have (these contain env vars and therefore API keys so they need to be secret).

I've been testing this with a test that intentionally crashes.

To debug a core dump from CI:

  • Note the PID of the test that crashed, the URL to the core dumps artifact (this will be a file called bun-cores.tar.gz.age on the test-bun step), and the URL to the bun-profile.zip artifact for that platform.
  • Run bun scripts/debug-coredump.ts -p <pid> -b <zip file URL> -c <cores URL> -i <path to age identity file>
    • the identity file is needed to decrypt these and will be shared with the team

Example:

bun ./scripts/debug-coredump.ts -p 4725 -i ../cores-key.txt -b 'https://buildkite.com/organizations/bun/pipelines/bun/builds/15795/jobs/01968354-6002-4e33-be78-b2c1c2a673c3/artifacts/0196836e-81aa-47ae-af7e-0e1cf97a1d2c' -c 'https://buildkite.com/organizations/bun/pipelines/bun/builds/15795/jobs/01968354-6004-49fc-9904-d5779d4089d2/artifacts/01968375-9fe9-4ee4-ae6b-81703799d03b'

This will open in lldb by default, which is nice because you can debug cores from another platform. Or you can specify a different debugger with -g. CodeLLDB theoretically has support for core dumps by adding a new launch configuration, but it didn't work for me.

If interesting parts of the stack trace are in a dynamic library (libc, libc++, Node-API module...) you will need to have that library (the same exact version that was used by the CI runner) downloaded too to get the best debugging experience.

It will be nice to have these on macOS and Windows, too, but I'm not as familiar with core dumps on those platforms so I will do that later.

How did you verify your code works?

CI, plus some manual testing with machine.mjs before I opened this PR

@robobun
Copy link

robobun commented Apr 22, 2025

Updated 1:44 PM PT - May 29th, 2025

@190n, your commit cae54f8 has some failures in Build #17634


🧪   To try this PR locally:

bunx bun-pr 19189

That installs a local version of the PR into your bun-19189 executable, so you can run:

bun-19189 --bun

@190n 190n changed the title upload encrypted core dumps from CI [publish images] upload encrypted core dumps from CI Apr 30, 2025
@190n 190n marked this pull request as ready for review April 30, 2025 19:26
@190n 190n requested review from a team and cirospaciari and removed request for a team May 5, 2025 23:10
@190n 190n requested a review from Electroid May 13, 2025 18:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants