DevOps & Infrastructure
K8s · CI/CD · Monitoring

Thiết kế hạ tầng vận hành cho hệ thống ERP/MES — Kubernetes cluster, CI/CD pipeline tự động, monitoring stack Prometheus/Grafana, và chiến lược bảo mật.

Cập nhật: 22/07/2026

Platform: Kubernetes 1.32+

CI/CD: GitHub Actions

01 — Kubernetes

Kubernetes Architecture

Toàn bộ hệ thống chạy trên Kubernetes với namespace phân tách theo domain. Mỗi module của Modular Monolith được deploy như một Deployment riêng, chia sẻ cùng codebase nhưng có thể scale độc lập.

flowchart TB subgraph Ingress["Ingress Layer"] IG["NGINX Ingress Controller\nTLS Termination · Rate Limiting"] end subgraph AppNS["Namespace: erp-mes-app"] API["api-gateway\nDeployment · 2-4 replicas\nHPA: CPU 70%"] CORE["core-service\nDeployment · 2-6 replicas\nHPA: CPU + RPS"] PROJ["projection-worker\nDeployment · 2-4 replicas\nKafka consumer group"] SAGA["saga-orchestrator\nDeployment · 2 replicas\nStatefulSet-like"] end subgraph DataNS["Namespace: erp-mes-data"] PG["postgresql-primary\nStatefulSet · 1 replica\n+ read-replica × 2"] KFK["kafka-cluster\nStatefulSet · 3 brokers\nKRaft mode"] RDS["redis-cluster\nStatefulSet · 3 nodes"] ES["elasticsearch\nStatefulSet · 3 nodes"] end subgraph MonNS["Namespace: monitoring"] PROM["Prometheus"] GRAF["Grafana"] ALERT["AlertManager"] LOKI["Loki\nLog aggregation"] end IG --> API API --> CORE CORE --> PG CORE --> KFK CORE --> RDS KFK --> PROJ PROJ --> PG PROJ --> ES PROJ --> TS PROM -.->|"scrape metrics"| CORE LOKI -.->|"collect logs"| AppNS style Ingress fill:#0C2340,stroke:#06B6D4,color:#E2E8F0 style AppNS fill:#1E1B4B,stroke:#6366F1,color:#E2E8F0 style DataNS fill:#052E16,stroke:#10B981,color:#E2E8F0 style MonNS fill:#1A1200,stroke:#F59E0B,color:#E2E8F0

🖱️ Kéo để di chuyển · Cuộn chuột để zoom

Resource Quotas per Namespace

Namespace	CPU Request	CPU Limit	Memory	PVC Storage
`erp-mes-app`	4 cores	16 cores	8–32 Gi	50 Gi
`erp-mes-data`	8 cores	24 cores	32–64 Gi	2 Ti
`monitoring`	2 cores	4 cores	4–8 Gi	200 Gi

Horizontal Pod Autoscaler (HPA)

YAML — HPA cho core-service

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: core-service-hpa
  namespace: erp-mes-app
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: core-service
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300   # Không scale down quá nhanh
    scaleUp:
      stabilizationWindowSeconds: 30

02 — CI/CD

CI/CD Pipeline — GitHub Actions

flowchart LR PR["Pull Request\nto main"] --> CI subgraph CI["CI — On PR"] L["Lint\n+ Format check"] T["Unit Tests\n+ Integration Tests"] S["Security Scan\nTrivy + Semgrep"] B["Docker Build\nmulti-stage"] L --> T --> S --> B end CI -->|"PR merged"| CD subgraph CD["CD — On merge to main"] TAG["Tag image\ngit-sha + semver"] PUSH["Push to\nContainer Registry\nGHCR / ECR"] DEV["Deploy to\nDEV (auto)"] STA["Deploy to\nSTAGING (auto)"] GATE["Manual approval\nTech Lead"] PROD["Deploy to\nPROD (blue-green)"] TAG --> PUSH --> DEV --> STA --> GATE --> PROD end style CI fill:#1E1B4B,stroke:#6366F1,color:#E2E8F0 style CD fill:#052E16,stroke:#10B981,color:#E2E8F0

🖱️ Kéo để di chuyển · Cuộn chuột để zoom

Blue-Green Deployment cho Production

YAML — Blue-Green với Kubernetes Service

# Service trỏ vào active slot (blue hoặc green)
apiVersion: v1
kind: Service
metadata:
  name: core-service
  namespace: erp-mes-app
spec:
  selector:
    app: core-service
    slot: blue           # Đổi thành "green" khi swap
  ports:
    - port: 8080
      targetPort: 8080

---
# Deployment GREEN (mới deploy, chưa live)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: core-service-green
  namespace: erp-mes-app
spec:
  replicas: 3
  selector:
    matchLabels:
      app: core-service
      slot: green
  template:
    metadata:
      labels:
        app: core-service
        slot: green
    spec:
      containers:
        - name: core-service
          image: ghcr.io/org/erp-mes-core:v1.2.0
          # Sau khi verify green healthy → patch Service selector ke green
          # Sau 15 phút → delete blue deployment

Zero-downtime deployment

Blue-Green đảm bảo zero downtime. Rollback chỉ cần 1 lệnh: patch Service selector từ green về blue. Toàn bộ traffic chuyển ngay lập tức mà không restart pod nào.

03 — Monitoring

Monitoring & Observability Stack

Ba trụ cột của Observability: Metrics (Prometheus + Grafana), Logs (Loki), Traces (Tempo / Jaeger). Tất cả được visualize trên Grafana.

flowchart LR App["ERP/MES\nServices"] -->|"expose /metrics"| Prom App -->|"structured logs"| Loki App -->|"trace spans\nOpenTelemetry"| Tempo Prom["Prometheus\nMetrics scrape\n15s interval"] Loki["Loki\nLog aggregation"] Tempo["Tempo\nDistributed Tracing"] AM["AlertManager\nPagerDuty · Slack"] Prom --> Grafana Loki --> Grafana Tempo --> Grafana Prom -->|"alert rules"| AM Grafana["Grafana\nDashboards\nUnified View"] style Grafana fill:#1A1200,stroke:#F59E0B,color:#E2E8F0

🖱️ Kéo để di chuyển · Cuộn chuột để zoom

Key Metrics cần monitor

Metric	Tool	Alert threshold	Dashboard
API latency P99	Prometheus	> 500ms	API Performance
Event Store write latency	Prometheus	> 100ms	Event Store Health
Kafka consumer lag	Prometheus (JMX)	> 10,000 messages	Kafka Dashboard
Outbox pending count	Prometheus (custom)	> 1,000 rows	Outbox Monitor
PostgreSQL connections	Prometheus (pg_exporter)	> 80% pool size	Database Health
Redis memory usage	Prometheus (redis_exporter)	> 85%	Cache Dashboard
OEE per machine	oee_daily_snapshots → Grafana	< 65%	MES Operations
Error rate (5xx)	Prometheus (NGINX)	> 1%	API Performance

Structured Logging Format

JSON Log — chuẩn OpenTelemetry

{
  "timestamp":    "2026-06-29T09:15:00.123Z",
  "level":        "INFO",
  "service":      "core-service",
  "module":       "production",
  "traceId":      "abc123def456",
  "spanId":       "789xyz",
  "tenantId":     "tenant_abc",
  "userId":       "uuid",
  "correlationId":"uuid",
  "message":      "ProductionOrder started successfully",
  "orderId":      "uuid",
  "machineId":    "uuid",
  "durationMs":   42
}

04 — Security

Security & Secrets Management

flowchart LR User["User / Client"] -->|"HTTPS + JWT"| Ingress Ingress -->|"mTLS"| API["API Gateway"] API -->|"RBAC check"| Core["Core Service"] subgraph Secrets["Secrets Management"] Vault["HashiCorp Vault\nor AWS Secrets Manager"] K8sSecret["K8s Secrets\n(encrypted at rest)"] Vault -->|"inject at pod start"| K8sSecret end Core -->|"read secrets"| K8sSecret subgraph NetPol["Network Policy"] NP["Deny all ingress by default\nAllow only declared routes"] end style Secrets fill:#1A0A10,stroke:#EF4444,color:#E2E8F0 style NetPol fill:#1E1B4B,stroke:#6366F1,color:#E2E8F0

🖱️ Kéo để di chuyển · Cuộn chuột để zoom

Checklist bảo mật

TLS everywhere: HTTPS tại Ingress, mTLS giữa các service (Istio hoặc Linkerd)
JWT + RBAC: Token-based auth, role-based access per module và action
Secrets không trong code: Vault hoặc AWS Secrets Manager, inject qua K8s Secret
Container scanning: Trivy quét image mỗi CI build — fail build nếu có CVE critical
Network Policy: Deny all by default, whitelist theo namespace và pod selector
Pod Security Standards: restricted profile — no root, no privileged
Audit logging: K8s audit log + application audit trail vào Elasticsearch
Rate limiting: NGINX Ingress + Redis (sliding window, per user/IP)

05 — Infrastructure as Code

Infrastructure as Code

Layer	Tool	Mô tả
Cloud infra (VPC, EKS, RDS)	Terraform	Provisioning cloud resources — version controlled, plan/apply workflow
K8s manifests	Helm Charts	Package K8s resources, values per environment
GitOps / CD	ArgoCD	Sync K8s state từ Git repo — drift detection tự động
Config management	ConfigMaps	Non-secret config per environment, mounted vào pods
Secret management	Vault + ESO	External Secrets Operator sync Vault secrets → K8s Secrets

Helm Chart Structure

Shell — Helm chart layout

charts/erp-mes/
├── Chart.yaml
├── values.yaml           # default values
├── values-dev.yaml       # dev overrides
├── values-staging.yaml   # staging overrides
├── values-prod.yaml      # prod overrides
└── templates/
    ├── deployment.yaml
    ├── service.yaml
    ├── hpa.yaml
    ├── ingress.yaml
    ├── configmap.yaml
    ├── network-policy.yaml
    └── serviceaccount.yaml

# Deploy to prod:
helm upgrade --install erp-mes ./charts/erp-mes \
  -f values-prod.yaml \
  --namespace erp-mes-app \
  --atomic \
  --timeout 5m

06 — Environments

Environment Strategy

Environment	Branch	Deploy trigger	DB	Kafka
DEV	`feature/*` → `develop`	Auto khi merge to develop	Shared PostgreSQL (dev schema)	Single broker
STAGING	`develop` → `main`	Auto khi merge to main	Isolated PostgreSQL (prod-like)	3-broker cluster
PRODUCTION	`main`	Manual approval → Blue-Green	PostgreSQL HA (primary + replicas)	3-broker cluster + replication

Database Migration Strategy

Dùng Flyway hoặc Liquibase cho schema migration. Nguyên tắc: Expand-then-Contract — thêm column mới (nullable) trước, deploy app mới, rồi mới xóa column cũ sau vài deploy cycle. Không bao giờ drop column hay rename trong 1 deploy.

07 — Checklist

Production Readiness Checklist

flowchart LR Dev["Code Ready"] --> G1 G1{"CI Gates\nTests pass?\nNo critical CVE?"} -->|Pass| G2 G1 -->|Fail| Fix["Fix Issues"] Fix --> Dev G2{"Staging Gates\nSmoke tests?\nMetrics normal?"} -->|Pass| G3 G2 -->|Fail| Fix G3{"Manual Gate\nTech Lead approval"} -->|Approve| PROD G3 -->|Reject| Fix PROD["Blue-Green\nDeploy to PROD"] --> G4 G4{"Health Check\n5 min observation"} -->|Healthy| Done["Done"] G4 -->|Unhealthy| Rollback["Auto Rollback\nSwitch to Blue"] style Done fill:#052E16,stroke:#10B981,color:#E2E8F0 style Rollback fill:#1A0A10,stroke:#EF4444,color:#E2E8F0

🖱️ Kéo để di chuyển · Cuộn chuột để zoom

Checklist trước mỗi Production deploy

Hạng mục	Kiểm tra	Owner
Database migration	Flyway migration scripts đã review, không có destructive change	Tech Lead
Event schema compatibility	New events backward-compatible với old consumers	Backend Dev
Kafka topic setup	New topics đã tạo với partition count phù hợp	DevOps
Feature flags	New features ẩn sau flag, không bật mặc định	Dev
Rollback plan	Có documented rollback steps, test trên Staging	Tech Lead
Monitoring alerts	Alert rules cho feature mới đã setup	DevOps
Load test	Staging đã chạy load test với 2x expected traffic	QA
Security scan	Trivy scan pass, không có Critical CVE	DevSecOps

Hướng A hoàn thành — Tiếp theo

Hướng B — Phân tích chi tiết từng Bounded Context

← Về trang chủ

DevOps & InfrastructureK8s · CI/CD · Monitoring

Kubernetes Architecture

Resource Quotas per Namespace

Horizontal Pod Autoscaler (HPA)

CI/CD Pipeline — GitHub Actions

Blue-Green Deployment cho Production

Monitoring & Observability Stack

Key Metrics cần monitor

Structured Logging Format

Security & Secrets Management

Checklist bảo mật

Infrastructure as Code

Helm Chart Structure

Environment Strategy

Production Readiness Checklist

Checklist trước mỗi Production deploy

DevOps & Infrastructure
K8s · CI/CD · Monitoring