Shared VPC
Originally
ADR-0002-SHARED-VPC (v7) · Source on Confluence ↗Shared VPC Architecture
Context and Problem Statement
Decision for host projects and subnet design.
Networks must not overlap.
Every league will have its own service project, its own GKE clusters.
- The secondary address ranges for Pods, Services and nodes must not overlap
- Ranges for nodes can be expanded or added onto
- Ranges for pods can be expanded with additional node pools
- Ranges for services cannot be expanded.
GCP Quota for subnetwork ranges per VPC: 300
VLAN attachments for Cato Cloud Interconnect can only be assigned to one VPC
Decision Drivers: scale, complexity
Decision for Network Firewall
Network firewalls are important for access control when considering east-west (intra-VPC) and north-south (Internet ingress and egress) traffic flows.
Placement of firewalls can impact the shared VPC traffic flow.
GKE cluster in a service project cannot create and manage the firewall resources in the host project
- Manually configure firewall rules that would usually be created automatically
Decision Drivers: scale, management complexity, technical complexity, operational complexity, resiliency and high availability, NIST 800-53 compliance
Decision for Network Intrusion Detection System (IDS)/Intrusion Prevention System (IPS)
- NIST 800-53 Control SI-4(4) requires detection of attacks and indicators of potential attacks.
- IDS analyzes all traffic and generates events out of band.
- IPS must be deployed in the path of traffic, allowing it to break the connection.
- Decision Drivers: scale, management complexity, technical complexity, operational complexity, resiliency, NIST 800-53 compliance
Considered Options
Host project and subnet design
- Single or Multiple Host Projects
- Per project subnets or Large, shared subnets per environment
Network Firewall
East-West (intra-GCP)
- GCP Cloud Firewall
- Third-Party Firewall Appliances
North-South (Ingress/Egress)
- GCP Cloud Firewall Standard
- Third-Party Firewall Appliances
Network IDS/IPS
Packet Mirroring powered IDS
- Google Cloud IDS
- Google Packet Mirroring partner provider
Inline IPS
- Next-Generation Firewall with IPS capability
- Open Source IPS
Decision Outcome
Single Host Project, with per project subnets per environment
Network Firewall
- GCP Cloud Firewall for East-West traffic, with IAM-governed tags and Firewall Insights.
- GCP Cloud Firewall Standard for North-South traffic.
Google Cloud IDS
Consequences
Single Host Project, with per project subnets per environment
Diagram- Good, simpler management
- Good, one set of VLAN attachments required per region for Cato connectivity
- Good, Google’s recommended practice
- Good, security between Prod and Dev VPC has an extra layer builtin since VPC peering does not support transitive routing.
- Bad, Future number of projects could cause a VPC to reach the 300 subnet range limit, however, additional shared VPCs can be created.
Network Firewall
GCP Cloud Firewall Essentials for East-West traffic, with IAM-governed Tags and Firewall Insights.
- Good, no additional infrastructure to deploy and maintain. Network firewall policies are global by default and apply to all regions.
- Good, Follows Google’s new recommendations
- Good, increased scale and operational resiliency and reduces chances that high bandwidth use of one project affects other flows
- Good, simpler management with single global firewall policy to update with project add/remove/changes
- Good, Micro-segmentation enabled with combination of firewall policies and Identity and Access Management (IAM)-governed Tags
- Good, Firewall Insights provides visibility into firewall rule usage and detects firewall rule configuration issues
- Bad, does not provide additional Next-Generation Firewall capabilities like inline Network Intrusion Prevention, antivirus, anti-malware, or Data Loss Prevention (DLP)
GCP Cloud Firewall for North-South traffic.
- Good, no additional infrastructure management required
- Bad, limited security features available, but does not reduce compliance
- Bad, additional cost for data volume of a flow including both request and response traffic evaluated by Standard rules
- Good, Policy objects, such as Google Cloud Threat Intelligence lists, domain name (FQDN) objects, and geo-location objects, provide advanced protection for firewall rules. These objects are curated by Google, constantly updated, and automatically applied in firewall rules that call them.
More information and example provided.
Google Cloud IDS
- Good, non-intrusive way to monitor East-West and North-South traffic for Google Compute Engine and Google Kubernetes Engine (GKE)
- Good, does not require any additional software on the clusters or VMs
- Good, fully distributed across each mirrored VM and cluster
- Good, traffic duplication happens transparently at the SDN layer. The collectors see network traffic at Layer 3 and above the same way that the mirrored resources see the traffic. This includes any NATing and/or SSL decryption that may occur at a higher layer within Google Cloud. SSL decryption requires mirrored sources to be behind Load Balancer
- Good, threat detection capabilities are powered by the following Palo Alto Networks threat prevention technologies.
- Good, Google managed infrastructure
- Good, only needs to be created once per region in Shared VPC host projects
- Good, GCP has default signature sets to get started
- Good, GCP Cloud Logging to Datadog integration can be used for alerting
- Bad, increased cost for Cloud Logging
- Bad, time and expertise required to tune rules and alerts
- Bad, does not block in real-time, however events can trigger remediation via a webhook to create a ticket in a service desk system or a security orchestration and response tool which can quarantine by shutting down the instance or lock down the firewall rule.
Pros and Cons of the Options
Multiple Host Projects
- Bad, additional VLAN attachments required for Cato access increases complexity and cost
- Bad, increased management and operational complexity with separate project policies (especially firewall)
- Good, separate bandwidth provided for Prod and Dev to Cato
Large, shared subnets in each VPC
- Good, much smaller number of subnets makes it harder to reach VPC quota
- Good, do not need to request and create new subnets for a new project
- Bad, not Google’s recommended configuration for GKE because one cluster could use up IP space, causing another cluster to not allocate required resources
Inline Firewall appliances
Good, offers IPS/malware/protocol inspection between projects if traffic is allowed
Good, vendors have automation tools available for auditing and orchestration for remediation
Bad, additional infrastructure (Google Compute Engine VMs) required to deploy and maintain
Bad, higher technical complexity for routing and operational management
- default route can hinder Google APIs and services if Google APIs and services do not use a Private Service Connect endpoint
Bad, cannot be used with Shared VPC design for East-West traffic because of GCP routing does not allow subnet system routes to be overridden
- Bad, same VPC peering limitations presented in the dedicated VPC network design
Bad, shared firewall resources can present scale limitations - high bandwidth from one project can affect other traffic flows within the deployment
Bad, as bandwidth requirements increase, larger VM sizes or number of autoscaled VMs can significantly increase cost
Network IDS with Google Packet Mirroring partner providers
Same pros and cons as Google Cloud IDS except:
- Bad, additional infrastructure deployment, maintenance, and associated costs
- Bad, in-house expertise may be be available for open source options
Inline IPS
- Good, actively blocks malicious traffic
- Good, enforces protocol standards
- Bad, potential to impact critical traffic
- Bad, cannot be used with Shared VPC design for East-West traffic because of GCP routing does not allow subnet system routes to be overridden
- Bad, as bandwidth requirements increase, larger VM sizes or number of autoscaled VMs can significantly increase cost
- Bad, can hinder Google APIs and services if Google APIs and services do not use a Private Service Connect endpoint
- Bad, time and expertise required to actively monitor and tune blocks and alerts
More Information
Setting up clusters with Shared VPC