Scaling Network Identity Attestation

Network Identity AttestorNetwork Identity Attestor: Network Identity Attestor is an Aembit Edge component deployed in VMware vSphere environments that verifies VM identity through the vCenter API and issues signed attestation documents for workload authentication.Learn more has a unique network relationship with the Client WorkloadsClient Workload: Client Workloads represent software applications, scripts, or automated processes that initiate access requests to Server Workloads, operating autonomously without direct user interaction.Learn more that rely on it. This relationship shapes how you scale out your Network Identity Attestor deployment.

Before you scale, make sure you understand how Network Identity Attestation works and have set up Network Identity Attestor.

Layer 2 requirement

Consider this simplified deployment diagram for a single network segmentL2 Network Segment: A Layer 2 (L2) network segment is a portion of a network where devices communicate using MAC addresses at the data link layer. Devices on the same L2 segment can directly reach each other without routing. MAC addresses must be unique within this boundary for the network to be fully functional.Learn more(opens in new tab).

Each Network Identity Attestor provides cryptographically signed identity documents to any Agent Proxy deployed within the same network segment. Network Identity Attestor rejects requests coming from other network segments because it can’t rely on the MAC addressMAC Address: A Media Access Control (MAC) address is a unique hardware identifier assigned to a network interface card (NIC). In virtualized environments, the hypervisor assigns virtual MAC addresses to VMs, which Network Identity Attestor uses to verify VM identity.Learn more(opens in new tab) for those connections.

For a full production deployment, deploy at least two Network Identity Attestor instances to each Layer 2 network segment. Configure each Agent Proxy with the full list of Network Identity Attestor URLs. This facilitates rotation of Network Identity Attestor credentials without downtime for your Client Workloads.

Consider this reformulation of the previous diagram, this time with multiple network segments.

Each Agent Proxy within each network segment spreads requests for identity documents across the Network Identity Attestor instances it’s configured with.

Note

The HTTP load balancer in this diagram sits in front of the Agent Controllers, not the Network Identity Attestor instances. Never place a load balancer or HTTP reverse proxy in front of Network Identity Attestor. Doing so masks the Client Workload’s MAC address and breaks attestation. For details, see About Network Identity Attestation.

Size your Layer 2 network segments

There’s no absolute limit to the number of Client Workloads you can deploy alongside Network Identity Attestor. However, the number of virtual machines in a network segment affects the reliability and speed of attestation. Every Client Workload virtual machine must connect to Network Identity Attestor. The Linux kernel on the Network Identity Attestor host limits how many of these connections it tracks at once.

How segment size affects attestation:

• Fewer than 512 virtual machines: The default kernel thresholds are sufficient, so you can skip tuning.
• 512 to 1023 virtual machines: Problems are unlikely. They occur only when more than 512 virtual machines establish connections within the same 5-second window, such as after many virtual machines reboot at once. If that happens, a small fraction of Client Workloads time out and retry, adding at least 2 seconds to their attestation. Tune the neighbor tables to prevent it.
• 1024 or more virtual machines: Each additional virtual machine increases both the chance of connection problems and the time Client Workloads take to attest, because more connections must time out and retry, each with its own timeout. Tune the neighbor tables and consider deploying additional Network Identity Attestor instances.

Deploying more Network Identity Attestor instances to a segment spreads these connections across more hosts, which drastically reduces the chance of problems at every segment size.

The Linux kernel tracks the hosts it has recently communicated with in its neighbor table, also known as the Address Resolution Protocol (ARP) cache. Because every Client Workload virtual machine in the segment connects to Network Identity Attestor, the Network Identity Attestor host fills this table far faster than a typical virtual machine does. When the table fills, the kernel ignores additional incoming connections until it evicts older entries. This is why problems appear during a connection surge, such as a mass reboot, rather than during steady-state operation. Agent Proxy caches its identity document for far longer than a surge lasts.

Size your network neighbor tables

For segments of 512 or more virtual machines, raise the neighbor table thresholds on each Network Identity Attestor host.

By default, the Linux kernel neighbor table has a hard maximum of 1024 entries, and the kernel prunes it every 60 seconds. Three garbage collection thresholds control this behavior:

Parameter	Default	Behavior
net.ipv4.neigh.default.gc_thresh3	1024	Hard maximum. The kernel can’t allocate new entries beyond this.
net.ipv4.neigh.default.gc_thresh2	512	Above this, garbage collection becomes aggressive (5-second eviction of stale entries).
net.ipv4.neigh.default.gc_thresh1	128	Below this, no garbage collection runs.

Set gc_thresh3 to at least twice the maximum number of unique client IPs each Network Identity Attestor host sees concurrently, and maintain the approximate ratio of 1:2:4 across the three thresholds.

Apply the thresholds with a sysctl configuration file:

/etc/sysctl.d/99-arp-cache.conf

net.ipv4.neigh.default.gc_thresh1 = 512
net.ipv4.neigh.default.gc_thresh2 = 1024
net.ipv4.neigh.default.gc_thresh3 = 2048

sudo sysctl -p /etc/sysctl.d/99-arp-cache.conf

Verify the applied values:

sysctl net.ipv4.neigh.default.gc_thresh1
sysctl net.ipv4.neigh.default.gc_thresh2
sysctl net.ipv4.neigh.default.gc_thresh3

Virtual machine sizing

When provisioning virtual machines for your Network Identity Attestor instances, consider the number of Client Workloads that rely on Network Identity Attestor within the network segment. Plan for approximately 2 virtual CPUs and 2 GB of RAM per 256 Client Workloads, with a minimum of 2 virtual CPUs and 2 GB of RAM.

Number of Client Workloads	Minimum virtual CPUs	Minimum RAM
up to 256	2	2 GB
up to 512	4	4 GB
up to 1024	8	8 GB

These values are approximations based on averages. Depending on the characteristics of your workloads, you may require more resources.

Network Identity Attestor integrates with the Linux kernel’s netlink subsystem. If you see intermittent 403 responses despite VMs being on the correct Layer 2 segment, first check your logs:

sudo journalctl -u aembit_netid_attestor --namespace=aembit_netid_attestor | grep -i -E "(overrun|netlink)"

If you see messages reporting buffer overruns or other netlink synchronization warnings, increase the virtual CPU allocation for the Network Identity Attestor virtual machine. For more ways to inspect Network Identity Attestor logs and metrics, see Monitor Network Identity Attestor.

vCenter API throughput

Network Identity Attestor makes multiple API calls to vCentervCenter: VMware vCenter Server is a centralized management platform for VMware vSphere environments that provides VM lifecycle management, monitoring, and APIs for querying VM metadata such as UUIDs and MAC addresses.Learn more(opens in new tab) to confirm the identity of each Client Workload virtual machine that requests an identity document. It limits vCenter API usage by caching some API results, and serves repeat requests from the same virtual machine from a short-term local cache. However, because vCenter is the source of truth for the virtual machine’s identity, Network Identity Attestor must make frequent requests:

• Minimum: Plan for the vCenter API user to make at least four API calls per Client Workload per hour.
• Maximum: In disaster recovery scenarios, where many virtual machines boot simultaneously, you could see as many as four requests per Client Workload per 30-second interval.

Additionally, each Network Identity Attestor checks once per minute whether its authentication session needs refreshing.

To make Network Identity Attestor check its session less frequently or cache vCenter data for longer, update the systemd unit environment and restart the service:

sudo systemctl edit aembit_netid_attestor.service

Add the following in the editor:

[Service]
Environment=AEMBIT_VCENTER_SESSION_REFRESH_SECS=900
Environment=AEMBIT_VCENTER_HTTP_CACHE_EXPIRATION_SECS=60

Then reload and restart:

sudo systemctl daemon-reload
sudo systemctl restart aembit_netid_attestor.service

Operational limits

The following table lists the operational limits for Network Identity Attestor:

Parameter	Value	Configurable
API server port	443	Yes
Metrics port	9099	Yes
vCenter session timeout	30 minutes	Yes
vCenter HTTP timeout	5 seconds	Yes
Cache TTL	30 seconds	Yes
Cache max entries	1,000	No
Max idle connections to vCenter	6	No
Session refresh check interval	60s (normal), 1s (on failure)	Yes (normal interval only)
Proactive refresh threshold	80% of session lifetime	No
Journal max file size	100 MB	Via journald config
Journal max files	12	Via journald config
TLS minimum version	1.3	No

For details on the environment variables that control these settings, see the Network Identity Attestor reference.