Skip to content

₿⛓️🧱 An AI-powered security platform for anomaly detection in blockchain transactions. Built with Terraform for AWS infrastructure, this project integrates AI agents, Go microservices, containerized deployments, and CI/CD pipelines to deliver a robust DevSecOps solution.

License

Notifications You must be signed in to change notification settings

meleksabit/blockchain-ai-security-platform-terraform-aws

Repository files navigation

Platform Badge

PR Title Check GitHub Release

An ֎🇦🇮-powered security platform for detecting anomalies in blockchain transactions, built with Terraform Terraform for AWS AWS infrastructure, Helm for Kubernetes deployments, and a CI/CD pipeline. The platform integrates AI agents , Go Go microservices, RDS, and containerized deployments for a robust DevSecOps solution.

Table of Contents

💡🚀Implementation Overview

  • Purpose: Monitors blockchain transactions (Ethereum testnet) for anomalies using AI-driven microservices, a dashboard, and RDS for data persistence.
  • Testing:
    • Local: Tested on a local Kubernetes cluster via kubectl (e.g., Docker Desktop, Kind).
    • AWS: Deployed via Terraform Cloud.
  • Components:
    • Python AI Agent: Core anomaly detection service (port 8000).
    • Go Microservices:
      • blockchain-monitor: Tracks transactions (port 8081).
      • anomaly-detector: Analyzes anomalies (port 8082).
      • dashboard: Visualizes transaction data (port 8083, LoadBalancer).
    • Vault: HashiCorp Vault for secret management (port 8200).
    • RDS: Relational database for storing transaction metadata.
  • Secrets: Sensitive variables (e.g., Infura API key, RDS credentials) stored in HashiCorp Vault (vault.vault.svc.cluster.local).
  • Ethereum Blockchain: Anomalies implemented using Ethereum testnet (via Infura).
  • Helm Charts:
    • Located in helm/ (ai-agent, go-microservices/blockchain-monitor, go-microservices/anomaly-detector, go-microservices/dashboard).
    • Configures replicas, image tags, service types, and environment variables.
  • Health Checks (Kubernetes):
    • Liveness Probe: Ensures pods are running (e.g., /health for ai-agent, /dashboard for dashboard).
    • Readiness Probe: Confirms pods are ready to serve traffic.
    • Startup Probe: Allows initial pod startup (e.g., 30 retries for dashboard).
    • Example (dashboard):
        probes:
      liveness:
        enabled: true
        path: "/dashboard"
        initialDelaySeconds: 10
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
      readiness:
        enabled: true
        path: "/dashboard"
        initialDelaySeconds: 5
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
      startup:
        enabled: true
        path: "/dashboard"
        initialDelaySeconds: 5
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 30
  • Database: AWS RDS (configured in terraform/modules/rds/) for persistent storage, with IAM role (rds-service-role) for enhanced monitoring and S3 backups.
  • Jenkins CI/CD: Jenkins pipeline (Jenkinsfile) builds/pushes Docker images to ECR and deploys to EKS via Helm.

📝✅Prerequisites

  1. AWS AWS Account:

    • Active account with IAM user access keys (EKS, EC2, ELB, ECR, IAM, S3, RDS permissions).
    • Region: eu-central-1.
  2. Tools:

    • Terraform: Install version 1.11.4:
      # Linux (Ubuntu/Debian)
      sudo apt-get update && sudo apt-get install -y gnupg software-properties-common
      wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
      echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
      sudo apt-get update && sudo apt-get install terraform=1.11.4
      Alternatively, use tfenv for version management:
      tfenv install 1.11.4
      tfenv use 1.11.4
    • AWS CLI: Configured via ~/.aws/config and ~/.aws/credentials:
      # Example ~/.aws/credentials
      [default]
      aws_access_key_id = <your-access-key>
      aws_secret_access_key = <your-secret-key>
      
      # Example ~/.aws/config
      [default]
      region = eu-central-1
      output = json
      Alternatively, use:
      aws configure
    • Helm: For deploying application charts.
    • Kubernetes kubectl: For Kubernetes interaction.
    • Docker Docker: For local testing (e.g., Docker Desktop).
    • GitHub Git: Clone the repository:
      git clone https://github.com/meleksabit/blockchain-ai-security-platform-terraform-aws.git
      cd blockchain-ai-security-platform-terraform-aws
  3. Vault HashiCorp Vault:

    • Local testing: Run via docker-compose.yml or Helm (see below).
    • EKS: Deployed via Helm and terraform/modules/vault.

🌲🗀️Project Structure: Directory Overview

This section outlines the repository’s directory structure to help you navigate the project’s key components, including Terraform modules, Helm charts, Go microservices, and CI/CD configuration.

.
├── ai-agent
│   ├── ai_agent.py
│   ├── Dockerfile
│   └── requirements.txt
├── docker-compose.yml
├── go-services
│   ├── anomaly-detector
│   │   ├── Dockerfile
│   │   ├── go.mod
│   │   ├── go.sum
│   │   └── main.go
│   ├── blockchain-monitor
│   │   ├── Dockerfile
│   │   ├── go.mod
│   │   ├── go.sum
│   │   └── main.go
│   └── dashboard
│       ├── Dockerfile
│       ├── go.mod
│       ├── go.sum
│       ├── main.go
│       └── templates
│           └── dashboard.tmpl
├── helm
│   ├── ai-agent
│   │   ├── Chart.yaml
│   │   ├── templates
│   │   │   ├── deployment.yaml
│   │   │   ├── hpa.yaml
│   │   │   ├── rbac.yaml
│   │   │   ├── rolebinding.yaml
│   │   │   ├── serviceaccount.yaml
│   │   │   └── service.yaml
│   │   └── values.yaml
│   ├── go-microservices
│   │   ├── anomaly-detector
│   │   │   ├── Chart.yaml
│   │   │   ├── templates
│   │   │   │   ├── deployment.yaml
│   │   │   │   ├── hpa.yaml
│   │   │   │   ├── rolebinding.yaml
│   │   │   │   ├── role.yaml
│   │   │   │   ├── serviceaccount.yaml
│   │   │   │   └── service.yaml
│   │   │   └── values.yaml
│   │   ├── blockchain-monitor
│   │   │   ├── Chart.yaml
│   │   │   ├── templates
│   │   │   │   ├── deployment.yaml
│   │   │   │   ├── hpa.yaml
│   │   │   │   ├── rbac.yaml
│   │   │   │   ├── rolebinding.yaml
│   │   │   │   ├── role.yaml
│   │   │   │   ├── serviceaccount.yaml
│   │   │   │   └── service.yaml
│   │   │   └── values.yaml
│   │   └── dashboard
│   │       ├── Chart.yaml
│   │       ├── templates
│   │       │   ├── deployment.yaml
│   │       │   ├── hpa.yaml
│   │       │   ├── rolebinding.yaml
│   │       │   ├── role.yaml
│   │       │   ├── serviceaccount.yaml
│   │       │   └── service.yaml
│   │       └── values.yaml
│   └── vault
│       ├── Chart.yaml
│       ├── templates
│       │   ├── deployment.yaml
│       │   └── service.yaml
│       └── values.yaml
├── Jenkinsfile
├── LICENSE
├── README.md
└── terraform
    ├── backend.tf
    ├── LICENSE.txt
    ├── main.tf
    ├── modules
    │   ├── alb
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── ecr
    │   │   ├── main.tf
    │   │   └── outputs.tf
    │   ├── eks
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   ├── security_groups.tf
    │   │   └── variables.tf
    │   ├── iam
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   ├── network
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   ├── security_groups.tf
    │   │   ├── variables.tf
    │   │   └── variables.tfvars.disabled
    │   ├── rds
    │   │   ├── data.tf.disabled
    │   │   ├── locals.tf.disabled
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   ├── security_groups.tf
    │   │   └── variables.tf
    │   ├── s3
    │   │   ├── main.tf
    │   │   ├── outputs.tf
    │   │   └── variables.tf
    │   └── vault
    │       ├── main.tf
    │       ├── outputs.tf
    │       └── variables.tf
    ├── outputs.tf
    ├── provider.tf
    ├── terraform.tfvars.disabled
    ├── variables.tf
    └── versions.tf

29 directories, 94 files
  • Key Directories:
    • terraform/: Manages AWS infrastructure (EKS, RDS, S3, ALB, IAM, Vault).
    • helm/: Contains Helm charts for deploying services to Kubernetes.
    • ai-agent/: Source code for the AI agent service, responsible for anomaly detection.
    • go-services/: Source code for Go microservices (blockchain-monitor, anomaly-detector, dashboard).
    • Jenkinsfile: Defines the CI/CD pipeline for building and deploying to EKS.
    • .terraformignore: Optimizes Terraform Cloud uploads by excluding unnecessary files.

🔧🔩🔛Setup

  1. Configure Terraform Cloud:

    • Edit terraform/backend.tf to enable Terraform Cloud backend (configured via UI or CLI).
     terraform {
         cloud {
             organization = "your-organization-name"
    
             workspaces {
                 name = "your-workspace-name"
             }
         }
     }
    • Set variables in Terraform Cloud > Variables:
      • Environment Variables:
        • AWS_ACCESS_KEY_ID: Valid access key.
        • AWS_SECRET_ACCESS_KEY: Corresponding secret key.
        • AWS_DEFAULT_REGION: Your deployment region.
      • Terraform Variables:
        • aws_role_arn: arn:aws:iam::<your-account-id>:role/TerraformCloudRole.
        • allowed_ssh_ip: ["<your-ip>/32"] (find via curl ifconfig.me).
  2. Configure Vault:

    • Option 1: Docker Compose (Local):
      docker-compose up -d
      vault operator init
    • Option 2: Helm (Local or EKS):
      • Add HashiCorp Helm repository:
        helm repo add hashicorp https://helm.releases.hashicorp.com
        helm repo update
      • Install Vault locally (e.g., Docker Desktop):
        helm install vault hashicorp/vault --namespace default --set "server.dev.enabled=true"
      • For EKS, deploy with production-ready settings (after Terraform applies terraform/modules/vault):
        helm install vault hashicorp/vault --namespace default --set "server.ha.enabled=true" --set "server.ha.replicas=3"
      • Initialize and unseal Vault:
        kubectl exec -it vault-0 -- vault operator init
        kubectl exec -it vault-0 -- vault operator unseal <unseal-key-1>
        kubectl exec -it vault-0 -- vault operator unseal <unseal-key-2>
        kubectl exec -it vault-0 -- vault operator unseal <unseal-key-3>
    • Store secrets (e.g., Infura API key, RDS credentials):
      vault kv put secret/infura api_key=<your-infura-api-key>
      vault kv put secret/rds username=<rds-username> password=<rds-password>

Warning

This project uses Vault’s kv-v2 secrets engine, so secrets are stored internally at secret/data/<path> (e.g., secret/data/infura). Use vault kv get secret/infura to retrieve the latest version, as the CLI automatically maps secret/<path> to secret/data/<path>. To access a specific version, use vault kv get -version=<version> secret/infura. If the Infura API key is rotated, update it with vault kv put secret/infura api_key=<new-key> and verify with vault kv get secret/infura. Ensure services (e.g., Helm charts) are configured to fetch secrets from secret/data/<path> when using Vault’s API or secret injection. The shorthand CLI syntax (secret/<path>) is correct but may cause confusion since it doesn’t reflect the full internal path (secret/data/<path>). For details, see the official Vault CLI tutorial: https://developer.hashicorp.com/vault/tutorials/get-started/learn-cli.

  • Option 3: AWS Secrets Manager:
    • Use AWS Secrets Manager to store secrets as an alternative to Vault. Ensure the IAM role (TerraformCloudRole) has secretsmanager:CreateSecret, secretsmanager:PutSecretValue, and secretsmanager:GetSecretValue permissions.
    • Store secrets using AWS CLI or Console:
      # Infura API key
      aws secretsmanager create-secret --name infura --secret-string '{"api_key":"<your-infura-api-key>"}' --region eu-central-1
      # RDS credentials
      aws secretsmanager create-secret --name rds --secret-string '{"username":"<rds-username>","password":"<rds-password>"}' --region eu-central-1
    • Retrieve secrets for verification:
      aws secretsmanager get-secret-value --secret-id infura --region eu-central-1
      aws secretsmanager get-secret-value --secret-id rds --region eu-central-1
    • Configure services (e.g., Helm charts) to access secrets via AWS SDK or Kubernetes secrets:
      • Option A: IAM Role: Assign an IAM role to EKS pods (e.g., via IRSA) with secretsmanager:GetSecretValue permissions. Update Helm charts (e.g., helm/go-microservices/blockchain-monitor/values.yaml) to use AWS SDK to fetch secrets.
      • Option B: Kubernetes Secrets: Use the AWS Secrets Manager CSI Driver to mount secrets as Kubernetes secrets. Install the driver:
        helm repo add secrets-store-csi-driver https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts
        helm install csi-secrets-store secrets-store-csi-driver/secrets-store-csi-driver --namespace kube-system
        Create a SecretProviderClass and update Helm charts to mount secrets. Example (SecretProviderClass):
        apiVersion: secrets-store.csi.x-k8s.io/v1
        kind: SecretProviderClass
        metadata:
          name: aws-secrets
          namespace: default
        spec:
          provider: aws
          parameters:
            objects: |
              - objectName: "infura"
                objectType: "secretsmanager"
                objectAlias: "infura.json"
              - objectName: "rds"
                objectType: "secretsmanager"
                objectAlias: "rds.json"
        Update Helm chart (e.g., values.yaml):
        volumes:
          - name: secrets
            csi:
              driver: secrets-store.csi.k8s.io
              readOnly: true
              volumeAttributes:
                secretProviderClass: "aws-secrets"
        volumeMounts:
          - name: secrets
            mountPath: "/mnt/secrets"
            readOnly: true
    • For more details, see the AWS Secrets Manager documentation: https://docs.aws.amazon.com/secretsmanager/latest/userguide/intro.html.
  1. Configure Blockchain Network: Set the blockchain network for the blockchain-monitor and ai-agent services using the NETWORK environment variable. Supported networks are:
  • Mainnet: Ethereum Mainnet (production, requires Infura API key with Mainnet access).
  • Sepolia: Sepolia testnet (default, recommended for testing).
  • Holesky: Holesky testnet (alternative testnet for validator and staking tests).
  • Hoodi: Hoodi testnet (new testnet for Pectra upgrade testing).
  • local: Local Ethereum node (e.g., Hardhat, Ganache) for development.
⚠️ⓘ❗ NOTE
Obtain an Infura API key by creating an account at infura.io (MetaMask wallet login supported). Avoid using MetaMask’s default Infura key due to rate limits, as it is shared and heavily restricted. Using Mainnet incurs higher Infura API costs and interacts with real Ethereum transactions. Ensure your Infura API key supports Mainnet and testnet access and use cautiously in production environments.

To configure the network:

  1. Set the NETWORK environment variable:
    • For local testing:
    export NETWORK=<mainnet|sepolia|holesky|hoodi|local>
    • For EKS deployment, update Helm values:
    helm upgrade --install blockchain-monitor ./helm/go-microservices/blockchain-monitor --set env.NETWORK=<mainnet|sepolia|holesky|hoodi|local>
    helm upgrade --install ai-agent ./helm/ai-agent --set env.NETWORK=<mainnet|sepolia|holesky|hoodi|local>
  2. Verify network configuration:
    • Check the /health endpoint for each service:
    curl http://<blockchain-monitor-load-balancer>:8081/health
    curl http://<ai-agent-load-balancer>:8000/health

  - Ensure the `network` field matches the configured value.
  1. IAM Role (TerraformCloudRole):

    • Ensure trust policy allows Terraform Cloud user:
      {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": { "AWS": "<terraform-cloud-user-arn>" },
            "Action": "sts:AssumeRole"
          }
        ]
      }
    • Permissions: EKS, EC2, ELB, ECR, IAM, S3, RDS.
  2. Optimize Uploads:

    • To optimize uploads in Terraform Cloud, it’s recommended to create a .terraformignore file in the project root to exclude unnecessary files, reducing upload size and speeding up Terraform runs. Create the file with the following content:
       # .terraformignore
       # Terraform state and backup files
       *.tfstate
       *.tfstate.backup
       *.tfplan
      
       # Terraform internal directories
       .terraform/
      
       # Logs and debug artifacts
       *.log
       crash.log
       *.out
       *.err
      
       # AWS credentials and sensitive files
       .aws/
       *.pem
       *.key
       *.crt
       *.env
       .env.*
       credentials.json
       secrets/
       **/credentials.tfvars
      
       # Python artifacts
       __pycache__/
       *.pyc
       *.pyo
       *.pyd
       *.egg-info/
       *.dist-info/
       *.ipynb_checkpoints/
       *.sqlite3
       *.db
       .venv/
       venv/
       env/
      
       # Docker build cache and local overrides
       Dockerfile.*
       docker-compose.override.yml
       *.tar
       *.img
      
       # Node-related (if present)
       node_modules/
       npm-debug.log
       yarn-error.log
      
       # Editor/OS/CI-related
       .idea/
       .vscode/
       *.swp
       *.swo
       *.bak
       *.tmp
       .DS_Store
       Thumbs.db
      
       # Git
       .git/
       .gitignore
       .gitattributes
      
       # Tests & coverage (optional)
       coverage/
       tests/__pycache__/
       test-results/
      
       # Model cache (if using Hugging Face Transformers)
       model_cache/
       transformers_cache/
       huggingface/
    • This excludes Terraform state, logs, credentials, Python artifacts, Docker files, editor files, Git, and non-essential Helm files while including critical Helm templates and configurations.

💻➡️🟢Deploy Infrastructure and Application

🐳☸Local Testing (Kubernetes)

  1. Start a local Kubernetes cluster (e.g., Docker Desktop):
    kubectl cluster-info
  2. Deploy Vault (via Helm or Docker Compose, see Setup).
  3. Install Helm Charts:
    helm install ai-agent ./helm/ai-agent --set image.repository=<local-ai-agent-image>
    helm install blockchain-monitor ./helm/go-microservices/blockchain-monitor --set image.repository=<local-blockchain-monitor-image>
    helm install anomaly-detector ./helm/go-microservices/anomaly-detector --set image.repository=<local-anomaly-detector-image>
    helm install dashboard ./helm/go-microservices/dashboard --set image.repository=<local-dashboard-image>
    • Replace <local-*-image> with the locally built Docker image tags for each service (e.g., ai-agent:latest, blockchain-monitor:latest, etc.).
  4. Verify pods and health checks:
    kubectl get pods
    kubectl describe pod <dashboard-pod>

☁️📄🚀AWS Deployment (Terraform Cloud)

  1. Initialize Terraform:

    cd terraform
    terraform init
  2. Format and Validate:

    terraform fmt
    terraform validate
    • Ensure no errors in the configuration files.
  3. Plan and Apply:

    terraform plan
    terraform apply
    • Deploys EKS (blockchain_eks), ALB, S3 (blockchain-ai-models-prod, blockchain-logs-prod), RDS, IAM roles, ECR, VPC.
  4. Configure EKS:

    aws eks update-kubeconfig --region eu-central-1 --name blockchain_eks
  5. Deploy Vault (via Helm, see Setup).

  6. Deploy Application via Helm:

    helm upgrade --install ai-agent ./helm/ai-agent --namespace default --set image.repository=<ecr-repo>/ai-agent
    helm upgrade --install blockchain-monitor ./helm/go-microservices/blockchain-monitor --namespace default --set image.repository=<ecr-repo>/blockchain-monitor
    helm upgrade --install anomaly-detector ./helm/go-microservices/anomaly-detector --namespace default --set image.repository=<ecr-repo>/anomaly-detector
    helm upgrade --install dashboard ./helm/go-microservices/dashboard --namespace default --set image.repository=<ecr-repo>/dashboard
    • Replace <ecr-repo> with your ECR registry URI (from Jenkinsfile).
    • Alternatively, use the provided Jenkinsfile to automate building, pushing Docker images to ECR, and deploying these Helm charts to EKS. Run the Jenkins pipeline after configuring Jenkins with aws-credentials and ecr-registry-uri credentials.
  7. Access Dashboard:

    kubectl get svc dashboard --namespace default
    • Use the LoadBalancer URL (port 8083).

🏗️🧱📐Infrastructure Details

Infrastructure is managed in the terraform/ folder:

  • Modules: eks, alb, s3, iam, network, vault, rds, vault.
  • Resources:
    • EKS cluster (blockchain_eks) with Spot instances.
    • ALB with target groups (blockchain-monitor:8081, anomaly-detector:8082, dashboard:8083, ai-agent:8000, vault:8200).
    • S3 buckets with lifecycle policies (180 days for models, 90 days for logs).
    • RDS instance for transaction metadata, with IAM role for monitoring.
    • Vault service for secret management, with infrastructure provisioned by Terraform and deployed via Helm.
    • VPC with public/private subnets.
  • Deploy via Terraform Cloud (see AWS Deployment (Terraform Cloud)).

⚠️🩺🔧Troubleshooting

  • Credentials Error:
    • Verify ~/.aws/credentials:
      aws sts get-caller-identity
  • Vault Secrets:
    • Check connectivity:
      vault kv get secret/infura
  • Helm Deployment:
    • Inspect pod logs:
      kubectl logs <pod-name>
  • RDS Issues:
    • Verify RDS endpoint and credentials in Vault:
      vault kv get secret/rds
    • Check security group allows EKS access (port 3306 for MySQL, 5432 for PostgreSQL).

Important

  • Ethereum Networks: Supports Mainnet, Sepolia (default), Holesky, Hoodi, and local networks. Set NETWORK environment variable to configure (see Setup > Configure Blockchain Network).
  • CI/CD: Jenkins pipeline builds/pushes images to ECR and deploys to EKS.
  • Health Checks: Ensure probes are configured per service.
  • Region: eu-central-1 (Frankfurt) is the default region for all AWS resources (EKS, RDS, S3, Secrets Manager). To use a different region, update AWS_DEFAULT_REGION in Terraform Cloud variables or terraform/backend.tf. Ensure consistency across resources to avoid cross-region latency or costs.
  • Secret Rotation: Regularly rotate sensitive secrets (e.g., Infura API key, RDS credentials) in Vault or AWS Secrets Manager to maintain security, and update dependent services accordingly.
  • Monitoring: Configure monitoring tools (e.g., AWS CloudWatch, Prometheus) to track EKS cluster metrics and application health, leveraging logs in S3 (blockchain-logs-prod).
  • Cost Management: Monitor AWS resource usage (e.g., EKS Spot instances, RDS, S3) to optimize costs, especially when using Terraform Cloud and Jenkins pipelines

For issues, check Terraform Cloud logs or review kubectl outputs.

⬆️

About

₿⛓️🧱 An AI-powered security platform for anomaly detection in blockchain transactions. Built with Terraform for AWS infrastructure, this project integrates AI agents, Go microservices, containerized deployments, and CI/CD pipelines to deliver a robust DevSecOps solution.

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Contributors 2

  •  
  •