+<GitHub discussion={{"id":"D_kwDOF8slf84AS7Ph","number":682,"author":{"login":"umm0n"},"title":"Issues when destroying modules via the gruntworks pipeline","body":"\r\nWe're using the Gruntworks Ref arch, and the Gruntworks pipeline (and associated scripts) for deploying our infra, with GHA.\r\n\r\nWe encounter this issue only when attempting to delete modules, and only under specific circumstances. To reproduce:\r\n\r\n1. We delete a bunch of modules, create a PR and push our changes.\r\n2. The pipeline runs a destroy plan on each deleted module.\r\n3. We merge the PR and the pipeline begins to run destroy on each deleted directory, as shown here:\r\n```\r\nThe following modules were deleted:\r\n<redacted>\r\nRunning apply on each deleted module.\r\nDestroying <redacted>\r\n```\r\n\r\n5. Another PR is merged while this pipe is running. The moment this happens, the pipe picks up on the latest commit (which no longer has any of the modules, as they were deleted) and fails.\r\n\r\nThis is the message we get:\r\n\r\n```\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Note: switching to 'HEAD^'.\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] You are in 'detached HEAD' state. You can look around, make experimental\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] changes and commit them, and you can discard any commits you make in this\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] state without impacting any branches by switching back to a branch.\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] If you want to create a new branch to retain commits you create, you may\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] do so (now or later) by using -c with the switch command. Example:\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -c <new-branch-name>\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Or undo this operation with:\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Turn off this advice by setting config variable advice.detachedHead to false\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] HEAD is now at 03729b5 delete <redacted> (#1090)\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] [INFO] [infrastructure-deploy-script] 2023-03-10 16:48:44 Successfully checked out HEAD^\r\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Error: Expected path /tmp/tmph4op5wbi/<redacted>/<redacted>/<redacted>/cdns/<redacted> to exist\r\n```\r\n\r\nTo be clear, if no PR is merged before the pipeline finishes, it runs perfectly well. We're not totally sure what's going on, but we suspect the pipe is picking the latest commit in master as a reference when starting each fargate task. When a new commit is added while the pipe runs, it picks this one up, and it doesn't have any of the modules, as they were deleted in a previous commit. \r\n\r\nHow can we fix this?\r\n\r\n[r:terraform-aws-ci](https://github.com/gruntwork-io/terraform-aws-ci)\r\n\r\n---\r\n\r\n<ins datetime=\"2023-03-14T09:59:11Z\">\r\n <p><a href=\"https://support.gruntwork.io/hc/requests/109987\">Tracked in ticket #109987</a></p>\r\n</ins>\r\n","bodyHTML":"<p dir=\"auto\">We're using the Gruntworks Ref arch, and the Gruntworks pipeline (and associated scripts) for deploying our infra, with GHA.</p>\n<p dir=\"auto\">We encounter this issue only when attempting to delete modules, and only under specific circumstances. To reproduce:</p>\n<ol dir=\"auto\">\n<li>We delete a bunch of modules, create a PR and push our changes.</li>\n<li>The pipeline runs a destroy plan on each deleted module.</li>\n<li>We merge the PR and the pipeline begins to run destroy on each deleted directory, as shown here:</li>\n</ol>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"The following modules were deleted:\n<redacted>\nRunning apply on each deleted module.\nDestroying <redacted>\"><pre class=\"notranslate\"><code class=\"notranslate\">The following modules were deleted:\n<redacted>\nRunning apply on each deleted module.\nDestroying <redacted>\n</code></pre></div>\n<ol start=\"5\" dir=\"auto\">\n<li>Another PR is merged while this pipe is running. The moment this happens, the pipe picks up on the latest commit (which no longer has any of the modules, as they were deleted) and fails.</li>\n</ol>\n<p dir=\"auto\">This is the message we get:</p>\n<div class=\"snippet-clipboard-content notranslate position-relative overflow-auto\" data-snippet-clipboard-copy-content=\"[ecs-deploy-runner][2023-03-10T16:48:44+0000] Note: switching to 'HEAD^'.\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] You are in 'detached HEAD' state. You can look around, make experimental\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] changes and commit them, and you can discard any commits you make in this\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] state without impacting any branches by switching back to a branch.\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] If you want to create a new branch to retain commits you create, you may\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] do so (now or later) by using -c with the switch command. Example:\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -c <new-branch-name>\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Or undo this operation with:\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Turn off this advice by setting config variable advice.detachedHead to false\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] HEAD is now at 03729b5 delete <redacted> (#1090)\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] [INFO] [infrastructure-deploy-script] 2023-03-10 16:48:44 Successfully checked out HEAD^\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Error: Expected path /tmp/tmph4op5wbi/<redacted>/<redacted>/<redacted>/cdns/<redacted> to exist\"><pre class=\"notranslate\"><code class=\"notranslate\">[ecs-deploy-runner][2023-03-10T16:48:44+0000] Note: switching to 'HEAD^'.\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] You are in 'detached HEAD' state. You can look around, make experimental\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] changes and commit them, and you can discard any commits you make in this\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] state without impacting any branches by switching back to a branch.\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] If you want to create a new branch to retain commits you create, you may\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] do so (now or later) by using -c with the switch command. Example:\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -c <new-branch-name>\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Or undo this operation with:\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] git switch -\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Turn off this advice by setting config variable advice.detachedHead to false\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] HEAD is now at 03729b5 delete <redacted> (#1090)\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] [INFO] [infrastructure-deploy-script] 2023-03-10 16:48:44 Successfully checked out HEAD^\n[ecs-deploy-runner][2023-03-10T16:48:44+0000] Error: Expected path /tmp/tmph4op5wbi/<redacted>/<redacted>/<redacted>/cdns/<redacted> to exist\n</code></pre></div>\n<p dir=\"auto\">To be clear, if no PR is merged before the pipeline finishes, it runs perfectly well. We're not totally sure what's going on, but we suspect the pipe is picking the latest commit in master as a reference when starting each fargate task. When a new commit is added while the pipe runs, it picks this one up, and it doesn't have any of the modules, as they were deleted in a previous commit.</p>\n<p dir=\"auto\">How can we fix this?</p>\n<p dir=\"auto\"><a href=\"https://github.com/gruntwork-io/terraform-aws-ci\">r:terraform-aws-ci</a></p>\n<hr>\n<ins datetime=\"2023-03-14T09:59:11Z\">\n <p dir=\"auto\"><a href=\"https://support.gruntwork.io/hc/requests/109987\" rel=\"nofollow\">Tracked in ticket #109987</a></p>\n</ins>","answer":{"body":"Hi @umm0n, thanks for the report!\r\n\r\nWe've done extensive testing for a fix for this. In short, we've made a change to calculate the source ref earlier in the process, to ensure that the ref used in the ecs-deploy-runner during delete operations will not change due to subsequent PR merges prior to the delete operation occuring.\r\n\r\nYou will need to make this same change in the ci configuration for your CI tool of choice (e.g., GH Actions, Gitlab) in your repo. For example, if you're using GH Actions, you would need to modify `infrastructure-live/.github/scripts/source-ref.sh`. \r\n\r\nWe are working to update our examples in the service-catalog, but for now, you can see how we accomplished this, in the following PR: https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1799/files#diff-406976bb201d23780f59dab210db1c29210c5e32fa2e5fe08472236f4e8f611b","bodyHTML":"<p dir=\"auto\">Hi <a class=\"user-mention notranslate\" data-hovercard-type=\"user\" data-hovercard-url=\"/users/umm0n/hovercard\" data-octo-click=\"hovercard-link-click\" data-octo-dimensions=\"link_type:self\" href=\"https://github.com/umm0n\">@umm0n</a>, thanks for the report!</p>\n<p dir=\"auto\">We've done extensive testing for a fix for this. In short, we've made a change to calculate the source ref earlier in the process, to ensure that the ref used in the ecs-deploy-runner during delete operations will not change due to subsequent PR merges prior to the delete operation occuring.</p>\n<p dir=\"auto\">You will need to make this same change in the ci configuration for your CI tool of choice (e.g., GH Actions, Gitlab) in your repo. For example, if you're using GH Actions, you would need to modify <code class=\"notranslate\">infrastructure-live/.github/scripts/source-ref.sh</code>.</p>\n<p dir=\"auto\">We are working to update our examples in the service-catalog, but for now, you can see how we accomplished this, in the following PR: <a href=\"https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1799/files#diff-406976bb201d23780f59dab210db1c29210c5e32fa2e5fe08472236f4e8f611b\">https://github.com/gruntwork-io/terraform-aws-service-catalog/pull/1799/files#diff-406976bb201d23780f59dab210db1c29210c5e32fa2e5fe08472236f4e8f611b</a></p>"}}} />
![docs-sourcer[bot]](https://avatars.githubusercontent.com/u/17118990?v=4&size=40){kind=avatar}
0 commit comments