Update "Run code blocks" workflow to work with unreleased k6 features

[heitortsergent]

The run-code-blocks workflow helps us make sure that any code snippets are valid and work with the latest version of k6.

One of the problems we've run into is when opening PRs that are related to a feature that's still unreleased, and will only be available in the next version of k6.

We could use the skip option to skip any code blocks for unreleased features, and then remove those later after the new version of k6 is released, but I think this would likely lead to errors. It's also harder to do if it's a change that affects many files.

We should try to think of some alternatives for how to best handle this, in a way that's easy for maintainers, and allows us to continue to use the workflow.

[heitortsergent]

@federicotdn I opened an issue for this so we can chat about possible solutions here instead of Slack. 🤓

I was wondering if one possible solution would be to add a way to skip the workflow for a PR that doesn't involve adding any line of code to files? Like something in the PR description or title.

[federicotdn]

@heitortsergent if we skip code blocks via PR description, how can we ensure that the code block will be run later on? Someone would need to coincidentally modify the same file at some later point - and by that time (simultaneously) the latest version of k6 must also have whatever feature the originally-skipped code block uses. I think it would work sometimes but the overall system would be a bit brittle.

IMO it's better just to skip the code blocks when they are committed, and then create a new issue with "uncomment code blocks X Y and Z", and link it in the PR adding the code blocks (as a future task).

[federicotdn]

Other possible solutions:

1. Start doing nightly k6 builds and use those for the docs. However this requires some work from the Core team.
2. Compile k6 ourselves on each run of the "check code snippets" GH action (git clone ... --depth 1, go build, we could cache some files) and use that for the docs

[heitortsergent]

I think relying on folks opening an issue and going back to remove skip blocks doesn't feel very reliable... But it's probably the easiest option, as long as I can do it, or we make everyone on the team aware of it.

And, maybe this is a separate issue, but the other part of this is when someone has a PR that touches on a lot of files, adding/removing a bunch of skip blocks feels like a lot of work. I think this was the case for the crypto PR. But that shouldn't happen a lot, and that's where I was thinking about having a way where we can skip the workflow through a PR description, or any other way that involves less manual copy+pasting would be nice.

[federicotdn]

I think relying on folks opening an issue and going back to remove skip blocks doesn't feel very reliable... But it's probably the easiest option, as long as I can do it, or we make everyone on the team aware of it.

Ok, I took a crack at it: #1917 - there, we build k6 from master on each run of the script. It takes about 15 seconds to build the binary (many things are cached) so this was actually much easier than I thought it would be. WDYT? This would allow a k6 OSS developer to develop a feature, get their PR approved & merged, and then immediately start writing the documentation (or they could start writing the documentation earlier, but the script will not work until they merge the feature).

And, maybe this is a separate issue, but the other part of this is when someone has a PR that touches on a lot of files, adding/removing a bunch of skip blocks feels like a lot of work. I think this was the case for the #1886. But that shouldn't happen a lot, and that's where I was thinking about having a way where we can skip the workflow through a PR description, or any other way that involves less manual copy+pasting would be nice.

Yeah in those cases I think there's no other option than to read the PR content or something similar.
@heitortsergent

[heitortsergent]

@andrewslotin @oleiade this isn't urgent, but could I ask for your thoughts on this?

If we use the k6 master branch to build k6 and run the "Run code blocks" workflow, is that a reliable solution? 🤔

[andrewslotin]

@heitortsergent, the most reliable solution would be to run the code block with the version of k6 a particular doc has been written for, e.g. all snippets in docs/sources/k6/v0.57.x should work for k6 v0.57. The second-best would be to use the latest released version. I'd rather avoid using unreleased master or at least make this check non-mandatory.

[oleiade]

@heitortsergent Along the lines of what @andrewslotin just mentioned, we could pin specific k6 versions for past releases. I wouldn't be against using k6's master against the next version of the docs; but I don't know how much brittle that will turn out to be in the long run though?

[heitortsergent]

Right now we're only running the workflow for changes to the next folder... I guess running changes for each folder according to their version could be a separate improvement.

I'd like to avoid making this check more brittle if it means running it against k6 master... I think the alternative right now might be just making the team aware that if you're writing docs for a new feature, the workflow will fail and that's ok since it's not mandatory. What do you think?