名称:terraform 描述:使用Terraform/OpenTofu创建和管理基础设施即代码,用于云资源部署、模块开发、状态管理和多环境部署。触发关键词:terraform, tf, opentofu, tofu, infrastructure, IaC, infrastructure as code, provision, cloud, aws, azure, gcp, kubernetes, k8s, module, provider, state, backend, plan, apply, workspace, resource, data source, output, variable, locals, import, taint, destroy. 允许工具:Read, Grep, Glob, Edit, Write, Bash
Terraform / OpenTofu
概述
本技能涵盖全面的Terraform和OpenTofu基础设施管理,包括:
- 模块设计和开发模式
- 状态管理和后端
- 多环境策略(工作区、tfvars、目录结构)
- 安全最佳实践(IAM、加密、密钥管理)
- 提供程序配置和版本管理
- 常见问题故障排除
- 迁移和重构模式
首选工具:OpenTofu - OpenTofu是Linux基金会维护的Terraform开源分支。可用时优先使用tofu命令而非terraform命令。语法和配置完全兼容。
主要用户:高级软件工程师 - 本技能支持高级软件工程师代理进行所有Terraform架构和实施工作(基础设施重点)。
指令
1. 规划基础设施架构
- 定义资源需求和依赖关系
- 规划网络拓扑(VPC、子网、路由)
- 识别模块边界和可重用性模式
- 考虑多区域/多可用区高可用性设计
- 规划灾难恢复和备份策略
- 记录安全要求(IAM、加密、网络隔离)
2. 编写Terraform配置
模块结构:
terraform/
├── modules/
│ ├── vpc/
│ │ ├── main.tf # 资源定义
│ │ ├── variables.tf # 输入变量
│ │ ├── outputs.tf # 输出值
│ │ └── versions.tf # 提供程序版本约束
│ ├── eks/
│ └── rds/
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── terraform.tfvars
│ │ └── backend.tf
│ ├── staging/
│ └── prod/
├── .terraform.lock.hcl # 提供程序版本锁文件
└── README.md
配置原则:
- 将代码结构化为具有清晰接口的可重用模块
- 定义所有变量并添加描述和验证规则
- 配置提供程序并设置版本约束
- 设置带锁定的远程状态后端
- 使用locals计算值并遵循DRY模式
- 记录输出并添加描述
- 在需要时添加生命周期规则(prevent_destroy, ignore_changes)
3. 模块开发模式
模块接口设计:
# modules/app-service/variables.tf
variable "name" {
description = "应用程序服务的名称"
type = string
validation {
condition = length(var.name) > 0 && length(var.name) <= 32
error_message = "名称长度必须在1到32个字符之间。"
}
}
variable "environment" {
description = "环境(dev, staging, prod)"
type = string
validation {
condition = contains(["dev", "staging", "prod"], var.environment)
error_message = "环境必须是dev、staging或prod。"
}
}
variable "vpc_id" {
description = "资源将创建的VPC ID"
type = string
}
variable "subnet_ids" {
description = "资源放置的子网ID列表"
type = list(string)
validation {
condition = length(var.subnet_ids) >= 2
error_message = "高可用性需要至少2个子网。"
}
}
variable "tags" {
description = "应用到资源的附加标签"
type = map(string)
default = {}
}
模块组合:
# modules/app-infrastructure/main.tf
module "vpc" {
source = "../vpc"
name = var.name
cidr_block = var.vpc_cidr
availability_zones = var.availability_zones
tags = local.common_tags
}
module "database" {
source = "../rds"
name = "${var.name}-db"
engine = "postgres"
engine_version = "15.4"
instance_class = var.db_instance_class
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnet_ids
security_groups = [module.vpc.database_security_group_id]
tags = local.common_tags
}
module "app" {
source = "../ecs-service"
name = var.name
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnet_ids
database_endpoint = module.database.endpoint
database_secret_arn = module.database.secret_arn
tags = local.common_tags
}
模块版本控制:
# 使用版本化模块以确保稳定性
module "vpc" {
source = "terraform-aws-modules/vpc/aws"
version = "~> 5.0" # 允许补丁更新,不允许次要更新
# ...
}
# 对于内部模块,使用git标签
module "internal" {
source = "git::https://github.com/org/terraform-modules.git//vpc?ref=v1.2.3"
# ...
}
4. 状态管理策略
远程状态后端配置:
AWS S3 + DynamoDB:
# backend.tf
terraform {
backend "s3" {
bucket = "myorg-terraform-state"
key = "prod/eks/terraform.tfstate"
region = "us-west-2"
encrypt = true
kms_key_id = "arn:aws:kms:us-west-2:123456789012:key/..."
dynamodb_table = "terraform-state-locks"
# 防止意外删除
lifecycle {
prevent_destroy = true
}
}
}
设置命令:
# 创建状态桶和锁表
aws s3api create-bucket \
--bucket myorg-terraform-state \
--region us-west-2 \
--create-bucket-configuration LocationConstraint=us-west-2
aws s3api put-bucket-versioning \
--bucket myorg-terraform-state \
--versioning-configuration Status=Enabled
aws s3api put-bucket-encryption \
--bucket myorg-terraform-state \
--server-side-encryption-configuration '{
"Rules": [{
"ApplyServerSideEncryptionByDefault": {
"SSEAlgorithm": "aws:kms",
"KMSMasterKeyID": "arn:aws:kms:..."
}
}]
}'
aws dynamodb create-table \
--table-name terraform-state-locks \
--attribute-definitions AttributeName=LockID,AttributeType=S \
--key-schema AttributeName=LockID,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
状态操作:
# 列出状态中的资源
tofu state list
# 显示特定资源
tofu state show aws_vpc.main
# 将资源移动到不同地址
tofu state mv aws_instance.old aws_instance.new
# 从状态中移除资源(不销毁)
tofu state rm aws_instance.temp
# 导入现有资源
tofu import aws_instance.example i-1234567890abcdef0
# 拉取远程状态以检查
tofu state pull > state.json
# 推送修改后的状态(危险 - 谨慎使用)
tofu state push state.json
# 替换提供程序地址(提供程序迁移后)
tofu state replace-provider registry.terraform.io/hashicorp/aws \
registry.opentofu.org/hashicorp/aws
状态锁定:
- 始终使用状态锁定以防止并发修改
- AWS使用DynamoDB,Google Cloud使用GCS,Azure使用Azure Storage
- 如果锁定卡住,在强制解锁前验证没有操作运行:
tofu force-unlock LOCK_ID
状态迁移:
# 从本地迁移到远程后端
# 1. 在backend.tf中配置后端
# 2. 初始化并迁移状态
tofu init -migrate-state
# 在后端之间迁移
# 1. 更新后端配置
# 2. 初始化并接受迁移
tofu init -migrate-state -backend-config="bucket=new-bucket"
5. 多环境管理
策略1:工作区(简单,相同后端)
# 创建和切换工作区
tofu workspace new dev
tofu workspace new staging
tofu workspace new prod
tofu workspace list
tofu workspace select prod
# 在配置中使用工作区
locals {
environment = terraform.workspace
instance_count = {
dev = 1
staging = 2
prod = 5
}
count = local.instance_count[local.environment]
}
策略2:目录结构(复杂,隔离)
terraform/
├── modules/ # 共享模块
├── environments/
│ ├── dev/
│ │ ├── main.tf
│ │ ├── backend.tf
│ │ ├── terraform.tfvars
│ │ └── .terraform.lock.hcl
│ ├── staging/
│ └── prod/
策略3:tfvars文件(灵活)
# environments/dev.tfvars
environment = "dev"
instance_type = "t3.small"
min_size = 1
max_size = 3
enable_monitoring = false
# environments/prod.tfvars
environment = "prod"
instance_type = "m5.large"
min_size = 3
max_size = 10
enable_monitoring = true
# 使用特定变量应用
tofu apply -var-file="environments/prod.tfvars"
6. 安全最佳实践
IAM和最小权限:
# 创建具有特定权限的角色
resource "aws_iam_role" "app" {
name = "${var.name}-app-role"
assume_role_policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Action = "sts:AssumeRole"
Effect = "Allow"
Principal = {
Service = "ec2.amazonaws.com"
}
}]
})
}
resource "aws_iam_role_policy" "app" {
name = "${var.name}-app-policy"
role = aws_iam_role.app.id
policy = jsonencode({
Version = "2012-10-17"
Statement = [{
Effect = "Allow"
Action = [
"s3:GetObject",
"s3:ListBucket"
]
Resource = [
aws_s3_bucket.data.arn,
"${aws_s3_bucket.data.arn}/*"
]
}]
})
}
密钥管理:
# 在AWS Secrets Manager中存储密钥
resource "aws_secretsmanager_secret" "db_password" {
name = "${var.name}-db-password"
recovery_window_in_days = 7
kms_key_id = aws_kms_key.secrets.id
}
resource "aws_secretsmanager_secret_version" "db_password" {
secret_id = aws_secretsmanager_secret.db_password.id
secret_string = jsonencode({
username = var.db_username
password = random_password.db_password.result
})
}
resource "random_password" "db_password" {
length = 32
special = true
}
# 在应用程序中引用密钥
resource "aws_ecs_task_definition" "app" {
# ...
container_definitions = jsonencode([{
name = "app"
image = var.app_image
secrets = [{
name = "DATABASE_PASSWORD"
valueFrom = aws_secretsmanager_secret.db_password.arn
}]
}])
}
# 将输出标记为敏感
output "database_password_arn" {
description = "数据库密码密钥的ARN"
value = aws_secretsmanager_secret.db_password.arn
sensitive = true
}
加密:
# 用于加密的KMS密钥
resource "aws_kms_key" "data" {
description = "数据加密的KMS密钥"
deletion_window_in_days = 30
enable_key_rotation = true
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Sid = "启用IAM用户权限"
Effect = "Allow"
Principal = {
AWS = "arn:aws:iam::${data.aws_caller_identity.current.account_id}:root"
}
Action = "kms:*"
Resource = "*"
},
{
Sid = "允许服务使用密钥"
Effect = "Allow"
Principal = {
Service = "s3.amazonaws.com"
}
Action = [
"kms:Decrypt",
"kms:GenerateDataKey"
]
Resource = "*"
}
]
})
}
# 带加密的S3桶
resource "aws_s3_bucket" "data" {
bucket = "${var.name}-data"
}
resource "aws_s3_bucket_server_side_encryption_configuration" "data" {
bucket = aws_s3_bucket.data.id
rule {
apply_server_side_encryption_by_default {
kms_master_key_id = aws_kms_key.data.arn
sse_algorithm = "aws:kms"
}
bucket_key_enabled = true
}
}
# 带加密的RDS
resource "aws_db_instance" "main" {
# ...
storage_encrypted = true
kms_key_id = aws_kms_key.data.arn
}
网络安全:
# 具有最小访问权限的安全组
resource "aws_security_group" "app" {
name = "${var.name}-app-sg"
description = "应用程序服务器的安全组"
vpc_id = var.vpc_id
# 仅出口
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}
tags = merge(var.tags, {
Name = "${var.name}-app-sg"
})
}
# 特定入口规则
resource "aws_security_group_rule" "app_from_alb" {
type = "ingress"
from_port = 8080
to_port = 8080
protocol = "tcp"
source_security_group_id = aws_security_group.alb.id
security_group_id = aws_security_group.app.id
}
# 网络ACL用于附加层
resource "aws_network_acl" "private" {
vpc_id = var.vpc_id
subnet_ids = var.private_subnet_ids
ingress {
protocol = "tcp"
rule_no = 100
action = "allow"
cidr_block = var.vpc_cidr
from_port = 443
to_port = 443
}
egress {
protocol = "tcp"
rule_no = 100
action = "allow"
cidr_block = "0.0.0.0/0"
from_port = 443
to_port = 443
}
tags = var.tags
}
7. 提供程序配置
版本约束:
# versions.tf
terraform {
required_version = ">= 1.6.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0" # 5.x.x,但不是6.0.0
}
kubernetes = {
source = "hashicorp/kubernetes"
version = "~> 2.23"
}
helm = {
source = "hashicorp/helm"
version = "~> 2.11"
}
}
}
provider "aws" {
region = var.region
# 应用到所有资源的默认标签
default_tags {
tags = {
Environment = var.environment
ManagedBy = "terraform"
Project = var.project
}
}
}
provider "kubernetes" {
host = module.eks.cluster_endpoint
cluster_ca_certificate = base64decode(module.eks.cluster_certificate_authority_data)
exec {
api_version = "client.authentication.k8s.io/v1beta1"
command = "aws"
args = ["eks", "get-token", "--cluster-name", module.eks.cluster_name]
}
}
提供程序别名(多区域/账户):
provider "aws" {
alias = "primary"
region = "us-west-2"
}
provider "aws" {
alias = "replica"
region = "us-east-1"
}
resource "aws_s3_bucket" "primary" {
provider = aws.primary
bucket = "${var.name}-primary"
}
resource "aws_s3_bucket" "replica" {
provider = aws.replica
bucket = "${var.name}-replica"
}
8. 高级模式
动态块:
resource "aws_security_group" "app" {
name = "${var.name}-sg"
vpc_id = var.vpc_id
dynamic "ingress" {
for_each = var.ingress_rules
content {
from_port = ingress.value.from_port
to_port = ingress.value.to_port
protocol = ingress.value.protocol
cidr_blocks = ingress.value.cidr_blocks
description = ingress.value.description
}
}
}
For表达式:
locals {
# 从列表创建映射
subnet_map = {
for subnet in aws_subnet.private :
subnet.availability_zone => subnet.id
}
# 转换和过滤
production_instances = {
for k, v in var.instances :
k => v if v.environment == "production"
}
# 复杂转换
instance_configs = [
for i in range(var.instance_count) : {
name = "${var.name}-${i}"
az = element(var.availability_zones, i)
tags = merge(var.tags, {
Index = i
})
}
]
}
Count vs For_Each:
# 不好 - count创建不稳定的地址
resource "aws_instance" "app" {
count = length(var.instance_names)
ami = var.ami_id
instance_type = var.instance_type
# 如果移除instance_names[1],instance_names[2]变为[1]
}
# 好 - for_each创建稳定的地址
resource "aws_instance" "app" {
for_each = toset(var.instance_names)
ami = var.ami_id
instance_type = var.instance_type
tags = {
Name = each.value
}
}
生命周期规则:
resource "aws_instance" "app" {
ami = var.ami_id
instance_type = var.instance_type
lifecycle {
# 销毁前创建替换
create_before_destroy = true
# 防止意外删除
prevent_destroy = true
# 忽略特定属性的更改
ignore_changes = [
tags,
user_data
]
}
}
Depends_on(显式依赖):
resource "aws_iam_role_policy" "app" {
name = "${var.name}-policy"
role = aws_iam_role.app.id
policy = jsonencode({...})
}
resource "aws_instance" "app" {
ami = var.ami_id
instance_type = var.instance_type
iam_instance_profile = aws_iam_instance_profile.app.name
# 确保在创建实例前附加策略
depends_on = [aws_iam_role_policy.app]
}
9. 常见问题故障排除
问题:状态锁超时
# 问题:另一个进程已锁定状态
# 解决方案1:等待其他进程完成
# 解决方案2:验证没有进程运行,然后强制解锁
tofu force-unlock <LOCK_ID>
问题:提供程序插件错误
# 问题:提供程序缓存损坏
# 解决方案:清除缓存并重新初始化
rm -rf .terraform/
rm .terraform.lock.hcl
tofu init
问题:资源已存在
# 问题:资源存在但不在状态中
# 解决方案:导入现有资源
tofu import aws_instance.example i-1234567890abcdef0
# 或:从代码中移除并在Terraform外管理
问题:循环依赖
# 问题:资源相互依赖
# 解决方案1:使用单独的apply步骤
resource "aws_security_group" "app" {
# ...
}
resource "aws_security_group_rule" "app_to_db" {
type = "egress"
source_security_group_id = aws_security_group.app.id
security_group_id = aws_security_group.db.id
}
# 解决方案2:重构依赖关系为单向
问题:状态中的敏感数据
# 问题:状态文件中可见密码/密钥
# 解决方案:始终使用带加密的远程状态
# 永远不要将状态文件提交到版本控制
# 使用AWS Secrets Manager / Vault管理密钥
问题:资源漂移
# 检测漂移
tofu plan -refresh-only
# 查看当前与期望状态
tofu show
# 刷新状态以匹配现实
tofu apply -refresh-only
# 覆盖漂移(恢复到代码定义)
tofu apply
问题:模块源更改
# 问题:模块源或版本更改
# 解决方案:重新初始化并升级
tofu init -upgrade
# 锁定提供程序版本
tofu providers lock
问题:大型计划输出
# 过滤计划输出
tofu plan | grep "will be created"
# 保存计划以供审查
tofu plan -out=tfplan
tofu show tfplan
# 仅显示特定资源类型
tofu plan | grep "aws_instance"
问题:超时错误
# 配置长时间运行操作的超时
resource "aws_db_instance" "main" {
# ...
timeouts {
create = "60m"
update = "60m"
delete = "60m"
}
}
10. 测试和验证
应用前验证:
# 格式化代码
tofu fmt -recursive
# 验证配置
tofu validate
# 安全扫描(使用tfsec)
tfsec .
# 成本估算(使用infracost)
infracost breakdown --path .
# 策略检查(使用OPA)
terraform-compliance -f compliance/ -p .
计划审查清单:
- 所有更改都是预期的
- 没有资源被意外标记为删除
- 敏感数据未在输出中暴露
- 标签应用到所有资源
- 在需要的地方启用加密
- IAM策略遵循最小权限
- 网络安全组具有限制性
最佳实践总结
- 模块设计:创建具有清晰接口的可重用、版本化模块
- 远程状态:始终使用带锁定的加密远程状态
- 变量:参数化所有内容,添加验证规则
- 工作区/环境:根据隔离需求选择策略
- 格式化:提交前始终运行
tofu fmt - 验证:使用
tofu validate和静态分析工具 - 计划审查:应用前始终审查计划输出
- 安全:启用加密,使用Secrets Manager,遵循最小权限
- 标记:标记所有资源以进行成本分配和管理
- 文档:记录模块、变量、输出和架构决策
- 版本约束:固定提供程序版本以确保可重现性
- 状态操作:谨慎使用状态命令,理解影响
- 测试:在生产使用前隔离测试模块
- 漂移检测:定期使用
tofu plan -refresh-only检查漂移
OpenTofu与Terraform命令对比
| Terraform | OpenTofu(首选) |
|---|---|
terraform init |
tofu init |
terraform plan |
tofu plan |
terraform apply |
tofu apply |
terraform destroy |
tofu destroy |
terraform fmt |
tofu fmt |
terraform validate |
tofu validate |
terraform state |
tofu state |
terraform import |
tofu import |
terraform output |
tofu output |
terraform workspace |
tofu workspace |
示例
示例1:AWS VPC模块
# modules/vpc/main.tf
terraform {
required_version = ">= 1.0"
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 5.0"
}
}
}
variable "name" {
description = "资源名称前缀"
type = string
}
variable "cidr_block" {
description = "VPC的CIDR块"
type = string
default = "10.0.0.0/16"
}
variable "availability_zones" {
description = "可用区列表"
type = list(string)
}
variable "tags" {
description = "应用到资源的标签"
type = map(string)
default = {}
}
locals {
common_tags = merge(var.tags, {
Terraform = "true"
Module = "vpc"
})
}
resource "aws_vpc" "main" {
cidr_block = var.cidr_block
enable_dns_hostnames = true
enable_dns_support = true
tags = merge(local.common_tags, {
Name = "${var.name}-vpc"
})
}
resource "aws_internet_gateway" "main" {
vpc_id = aws_vpc.main.id
tags = merge(local.common_tags, {
Name = "${var.name}-igw"
})
}
resource "aws_subnet" "public" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
cidr_block = cidrsubnet(var.cidr_block, 4, count.index)
availability_zone = var.availability_zones[count.index]
map_public_ip_on_launch = true
tags = merge(local.common_tags, {
Name = "${var.name}-public-${var.availability_zones[count.index]}"
Tier = "public"
})
}
resource "aws_subnet" "private" {
count = length(var.availability_zones)
vpc_id = aws_vpc.main.id
cidr_block = cidrsubnet(var.cidr_block, 4, count.index + length(var.availability_zones))
availability_zone = var.availability_zones[count.index]
tags = merge(local.common_tags, {
Name = "${var.name}-private-${var.availability_zones[count.index]}"
Tier = "private"
})
}
resource "aws_eip" "nat" {
count = length(var.availability_zones)
domain = "vpc"
tags = merge(local.common_tags, {
Name = "${var.name}-nat-eip-${count.index}"
})
}
resource "aws_nat_gateway" "main" {
count = length(var.availability_zones)
allocation_id = aws_eip.nat[count.index].id
subnet_id = aws_subnet.public[count.index].id
tags = merge(local.common_tags, {
Name = "${var.name}-nat-${count.index}"
})
depends_on = [aws_internet_gateway.main]
}
output "vpc_id" {
description = "VPC ID"
value = aws_vpc.main.id
}
output "public_subnet_ids" {
description = "公共子网ID"
value = aws_subnet.public[*].id
}
output "private_subnet_ids" {
description = "私有子网ID"
value = aws_subnet.private[*].id
}
示例2:EKS集群配置
# main.tf
terraform {
required_version = ">= 1.0"
backend "s3" {
bucket = "my-terraform-state"
key = "eks/terraform.tfstate"
region = "us-west-2"
encrypt = true
dynamodb_table = "terraform-locks"
}
}
provider "aws" {
region = var.region
default_tags {
tags = {
Environment = var.environment
Project = var.project
ManagedBy = "terraform"
}
}
}
module "vpc" {
source = "./modules/vpc"
name = "${var.project}-${var.environment}"
cidr_block = var.vpc_cidr
availability_zones = var.availability_zones
tags = var.tags
}
module "eks" {
source = "terraform-aws-modules/eks/aws"
version = "~> 19.0"
cluster_name = "${var.project}-${var.environment}"
cluster_version = "1.28"
vpc_id = module.vpc.vpc_id
subnet_ids = module.vpc.private_subnet_ids
cluster_endpoint_public_access = true
eks_managed_node_groups = {
general = {
desired_size = 2
min_size = 1
max_size = 5
instance_types = ["t3.medium"]
capacity_type = "ON_DEMAND"
labels = {
role = "general"
}
}
}
tags = var.tags
}
示例3:变量和输出
# variables.tf
variable "region" {
description = "AWS区域"
type = string
default = "us-west-2"
}
variable "environment" {
description = "环境名称"
type = string
validation {
condition = contains(["dev", "staging", "prod"], var.environment)
error_message = "环境必须是dev、staging或prod。"
}
}
variable "project" {
description = "项目名称"
type = string
}
variable "vpc_cidr" {
description = "VPC CIDR块"
type = string
default = "10.0.0.0/16"
}
variable "availability_zones" {
description = "可用区列表"
type = list(string)
default = ["us-west-2a", "us-west-2b", "us-west-2c"]
}
variable "tags" {
description = "附加标签"
type = map(string)
default = {}
}
# outputs.tf
output "vpc_id" {
description = "VPC ID"
value = module.vpc.vpc_id
}
output "eks_cluster_endpoint" {
description = "EKS集群端点"
value = module.eks.cluster_endpoint
sensitive = true
}
output "eks_cluster_name" {
description = "EKS集群名称"
value = module.eks.cluster_name
}
示例4:使用Terraform工作区的多环境配置
# main.tf
locals {
environment = terraform.workspace
# 环境特定配置
config = {
dev = {
instance_type = "t3.small"
min_size = 1
max_size = 3
enable_backups = false
}
staging = {
instance_type = "t3.medium"
min_size = 2
max_size = 5
enable_backups = true
}
prod = {
instance_type = "m5.large"
min_size = 3
max_size = 10
enable_backups = true
}
}
current_config = local.config[local.environment]
}
resource "aws_autoscaling_group" "app" {
name = "${var.project}-${local.environment}-asg"
min_size = local.current_config.min_size
max_size = local.current_config.max_size
desired_capacity = local.current_config.min_size
launch_template {
id = aws_launch_template.app.id
version = "$Latest"
}
}