[Bug]: [v4.66.1] aws_appautoscaling_target pollutes terraform plan when using default tags

[mkielar]

Terraform Core Version

1.4.2

AWS Provider Version

4.66.1

Affected Resource(s)

• aws_appautoscaling_target

Expected Behavior

After terraform apply, terraform plan should stop showing changes in tags_all attribute.

Actual Behavior

terraform plan keeps showing this change after terraform apply has already been applied on this change:

  # module.workload_baseline.aws_appautoscaling_target.msk_storage will be updated in-place
  ~ resource "aws_appautoscaling_target" "msk_storage" {
        id                 = "arn:aws:kafka:eu-west-1:[MASKED]:cluster/dev/[MASKED]"
        tags               = {}
      ~ tags_all           = {
          + "Env"       = "dev"
          + "ManagedBy" = "terraform"
          + "Service"   = "infra"
        }
        # (6 unchanged attributes hidden)
    }

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

The resource:

resource "aws_appautoscaling_policy" "msk_storage" {
  name               = "${aws_msk_cluster.this.cluster_name}-broker-scaling"
  policy_type        = "TargetTrackingScaling"
  resource_id        = aws_msk_cluster.this.arn
  scalable_dimension = aws_appautoscaling_target.msk_storage.scalable_dimension
  service_namespace  = aws_appautoscaling_target.msk_storage.service_namespace

  target_tracking_scaling_policy_configuration {
    predefined_metric_specification {
      predefined_metric_type = "KafkaBrokerStorageUtilization"
    }

    target_value = var.msk.storage_autoscaling.utilization_target
  }
}

Provider configuration

provider "aws" {
  region = var.region

  assume_role {
    role_arn = "arn:aws:iam::506501033716:role/infrastructure-deployer"
  }

  default_tags {
    tags = var.tags
  }
}

tags variable is set up as follows:

tags = {
  "Env"       = "dev",
  "Service"   = "infra",
  "ManagedBy" = "terraform",
}

Steps to Reproduce

1. Run terraform plan after upgrading from 4.65 to 4.66.1, and observe the change in tags_all
2. Run terraform apply - it seems to be updating the resource.
3. Run terraform plan again, and observe the change in tags_all pops up again.

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[mkielar]

It's weird...

• I have a terraform template with two instances of aws_appautoscaling_policy resource. In that template, one of the resources got updated and went silent, and the other keeps showing changes on tags_all.
• I also have another terraform template with 10 instances of aws_appautoscaling_policy resource. In that template, also one of the resources got updated and went silent, and remaining 9 pop up showing tags_all changes.

What's going on there?

[andgehri-te]

I ran into this today, as well.

According to the note at the top of the latest doc for aws_appautoscaling_target:

NOTE:
Scalable targets created before 2023-03-20 may not have an assigned arn. These resource cannot use tags or
participate in default_tags. To prevent terraform plan showing differences that can never be reconciled, use the 
[lifecycle.ignore_changes](https://developer.hashicorp.com/terraform/language/meta-arguments
/lifecycle#ignore_changes) meta-argument. See the example below.

Hopefully this answers your question, as it did for me😄

But of course, when I try to use the lifecycle meta-arg (like in the specified example), Terraform throws:

Error: Unsupported attribute
This object has no argument, nested block, or exported attribute named "tags_all".

Maybe I'll try using an older version of the provider.
Update: Using version 4.65.0 of the provider alleviated this issue for me, so if you don't need any specific features of 4.66+, this might be easiest workaround for now.

[mkielar]

Okay, so that's what was causing this weird behaviour. Some of the instances of aws_appautoscaling_target were old, and some new. I have verified this by running:

terraform state list | grep aws_appautoscaling_target

and then

terraform state show <address>

for all listed addresses of app_autoscaling_targets, and seeing that some of the indeed don't have ARNs, and that these are the same that keep popping up in terraform plans.

To work around this, I decided to re-create them and see what happens. So I made them all tainted with:

for addr in $(terraform state list | grep -E "aws_appautoscaling_(target|policy)"); do terraform taint "${addr}"; done

and then ran terraform apply. Note that I had to mark both target and policy resources as tainted to make terraform aware all of them need to be replaced. Not having done that, terraform would only recreate the targets and not notice that AWS implicitly deleted all the policies (they would pop up as "will be created" on next terraform apply but for the period between I'd not have a working autoscaling...).

After that terraform plans are back to normal.

The only issue with that workaround is that I now have ~40 git repos with terraform templates, times 4 environment statefiles per repository and would have to taint these resources in all of them. Oh joy...

[acdha]

I had the same problem and confirmed that a terraform taint cycle fixed it for resources which were created years ago. That process is functional but given the performance of the app.terraform.io service it takes about a minute per resource to update the state files.

[balonik]

I'd say it's caused by #31214
cc @ewbankkit

we might need some more tweaks for default_tags

[ewbankkit]

Relates #31180.

[socketbox]

Can confirm that this is still an issue on provider version 5.7.0