[Bug]: Changing security groups with aws_ec2_client_vpn_endpoint doesn't work as expected.

[liquidgecka]

Terraform Core Version

1.5.7

AWS Provider Version

5.21.0

Affected Resource(s)

• aws_ec2_client_vpn_endpoint

Expected Behavior

Adding security groups to a aws_ec2_client_vpn_endpoint resource using the security_group_ids parameter should leave the client vpn service in a state where the new security group controls access.

Actual Behavior

The security groups are updated in the AWS management console under: "Details", "Target Network Associations", and "Security Groups"; however the underlying security group that is used by the client VPN to control network access remains the group that was set prior to terraform running.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

Start with a config like the example config:

resource "aws_ec2_client_vpn_endpoint" "example" {
  description            = "terraform-clientvpn-example"
  server_certificate_arn = aws_acm_certificate.cert.arn
  client_cidr_block      = "10.0.0.0/16"

  authentication_options {
    type                       = "certificate-authentication"
    root_certificate_chain_arn = aws_acm_certificate.root_cert.arn
  }
}

Then add a security group via the security_group_ids parameter.

Steps to Reproduce

tl;dr: The way terraform changes security groups appears to leave the old security group as the active control for network connections on the modified client vpn instance without any indication that this has happened via terraform or the aws console.

Sorry for the verbosity of this report, this bug took me quite a bit of time to track down.

I am updating a client vpn service in our AWS account to move away from the default group for the vpc and instead use a dedicated, new security group. When applying the change to security_group_ids everything appears successful, however the default security group that was initially associated with the VPN connection remains the group that controls network access. For my test I have a EC2 instance in the same VPC as the client vpn endpoint, which allows all all networking (0.0.0.0/0 on all protocols). I setup the client vpn to be fully usable such that I can connect to this ec2 instance prior to running terraform. I create a new security group (sg-035b783b847b571b0) which allows egress on all protocols to 0.0.0.0/0, same as the default group.

This change is then applied:

  # module.networking.module.vpn[0].aws_ec2_client_vpn_endpoint.vpn will be updated in-place
  ~ resource "aws_ec2_client_vpn_endpoint" "vpn" {
        id                     = "cvpn-endpoint-02c340048f94f91ad"
      ~ security_group_ids     = [
          + "sg-032b73d4686bcff2b",
          - "sg-035b783b847b571b0", 
        ]
        # (13 unchanged attributes hidden) 
                                
         # (4 unchanged blocks hidden) 
     }                           

Plan: 0 to add, 1 to change, 0 to destroy. 

I then disconnect and reconnect from the VPN to ensure that everything is fully up to date. Everything still works, I can connect to the EC2 instance just fine. The Client VPN instance in the management console shows the security group to be the new group (sg-032b73d4686bcff2b) and all appears well. However if I remove the egress rule from the security group that no longer should be associated with the CVPN endpoint (sg-035b783b847b571b0) the EC2 instance becomes inaccessible. If I add it back and remove the egress rule from the new group such that the old groups allows egress, and the new group does not then the EC2 instance remains accessible even though the group that the client VPN is associated with shouldn't be allowing egress at all.

Now, the interesting thing is that I can re-associate the old security group through the management console, then associate the new group through the management console and it the security group appears to work as expected. Thus the management console does something different than terraform when changing security groups.

Checking CloudTrail shows that the working AWS Management Console uses the ApplySecurityGroupsToClientVpnTargetNetwork API call with the following request data:

    "requestParameters": {
        "ApplySecurityGroupsToClientVpnTargetNetworkRequest": {
            "ClientVpnEndpointId": "cvpn-endpoint-02c340048f94f91ad",
            "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
            "SecurityGroupId": {
                "tag": 1,
                "content": "sg-032b73d4686bcff2b"
            }
        }
    },

While terraform uses the ModifyClientVpnEndpoint API call with the following request data:

    "requestParameters": {
        "ModifyClientVpnEndpointRequest": {
            "ClientVpnEndpointId": "cvpn-endpoint-02c340048f94f91ad",
            "VpcId": "vpc-xxxxxxxxxxxxxxxxx",
            "SecurityGroupId": {
                "tag": 1,
                "content": "sg-032b73d4686bcff2b"
            }
        }
    },

This call comes from this section. From what I can tell the terraform use case appears to be correct, and the bug maybe in the AWS API directly (this has been reported to them already) however I figured I would report it here for documentation and to highlight it just in case it requires a fix in the terraform AWS provider as well..

Debug Output

No response

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[ewbankkit]

The call to ApplySecurityGroupsToClientVpnTargetNetworkRequest was not implemented in aws_ec2_client_vpn_endpoint when security_group_ids were added via #22911. Maybe it needed to be.

[liquidgecka]

AWS support got back to me and said that changing the security group the way that this code currently does can take up to four hours to take effect according to https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/cvpn-working-endpoints.html#cvpn-working-endpoint-modify

Modifications to Client VPN endpoints, including Certificate Revocation List (CRL) changes, will take effect up to 4 hours after a request is accepted by the Client VPN service.

If the AWS provider wants the change to take effect immediately it needs to use the API call that I mentioned in the initial bug report.

So that confirms that from AWS' perspective at least this would be considered a bug in the terraform aws provider and not an API bug.

[florian-besser]

I noted that changing the already established security group of aws_ec2_client_vpn_endpoint has a similar / same impact. When looking at the AWS console the Client VPN endpoint will use the new security group correctly, but Terraform cannot delete the old security group. The reason is that the old security group is still used by the network interfaces belonging to the ClientVPN Endpoint resource.

My changes: Before:

resource "aws_security_group" "vpn" {
  name   = "vpn"
  vpc_id = data.aws_vpc.vpc.id

  // Omitted ingress and egress as they are irrelevant
}
resource "aws_ec2_client_vpn_endpoint" "dev-vpn" {
  description            = "dev-vpn"
  client_cidr_block      = "10.0.252.0/22"
  vpc_id                 = data.aws_vpc.vpc.id
  security_group_ids     = [aws_security_group.vpn.id]
  transport_protocol     = "tcp"

  // Removed irrelevant args
}

After:

resource "aws_security_group" "vpn" {
  name   = "vpn-CHANGED"
  vpc_id = data.aws_vpc.vpc.id
  // Changing the name of the SG forces creation of a new SG
}
resource "aws_ec2_client_vpn_endpoint" "dev-vpn" {
  description            = "dev-vpn"
  client_cidr_block      = "10.0.252.0/22"
  vpc_id                 = data.aws_vpc.vpc.id
  // No changes made to aws_ec2_client_vpn_endpoint, but the changed aws_security_group.vpn.id forces an update
  security_group_ids     = [aws_security_group.vpn.id]
  transport_protocol     = "tcp"
}

As the update isn't applied properly some network interfaces keep using the old SG, thus it cannot be deleted.

[korncola]

Can confirm that in Jan 2024 terraform does still not delete the security group from the network interfaces used by ClientVPN if you remove a security group from ClientVPN.
It is removed from list of security groups but stays at the network interfaces, preventing deletion.

Deletion of security group manually from network interface leads to immediate success.

[nqivaricent]

This is not an issue with Terraform but an issue with AWS.

[liquidgecka]

No this is absolutely an issue with terraform. Terraform does not call the right API to ensure that the security groups get applied as expected, where the AWS console does.

[nqivaricent]

We deploy Client VPN Endpoint directly via Cloudformation template as per below. "IsolateVPN" is a boolean parameter that determines if "EcsServicesSecurityGroup" will be added. If True, it will not add it, and if False it will add it. The starting value is False so that endpoint has two SGs: EcsServicesSecurityGroup and VPNSecurityGroup. Now after changing it True, as expected it removed the EcsServicesSecurityGroup from the endpoint. However, the backend network ineterfaces (one per AZ), still has the two security groups attached.

VPCAccessVPNEndpoint:
Type: AWS::EC2::ClientVpnEndpoint
Properties:
AuthenticationOptions:
- Type: federated-authentication
FederatedAuthentication:
SAMLProviderArn: !Ref MainSAMLProvider
SelfServiceSAMLProviderArn: !Ref SelfServiceSAMLProvider
ClientCidrBlock: 10.1.0.0/22
ConnectionLogOptions:
Enabled: true
CloudwatchLogGroup: !Ref SecurityLogGroup
CloudwatchLogStream: !Ref VPNSecurityLogStream
DnsServers: ["10.0.0.2"]
SecurityGroupIds:
- !If [IsolateVPN, !Ref "AWS::NoValue", !Ref EcsServicesSecurityGroup]
- !Ref VPNSecurityGroup

[nqivaricent]

I have not tested it with CDK but I would expect the same issue cuz it will be converted to CF template