Home

CGroup Practical Experiments

A cgroup (control group) is a Linux kernel feature that organises processes into hierarchical groups, enabling the management (limiting, accounting and isolating) of system resources such as CPU, memory, disk I/O, and network bandwidth.

Each cgroup is associated with a set of limits or parameters, which are enforced by the kernel through subsystems (sometimes also known as resource controllers, or simply controllers).

All cgroup functionalities are accessed through the cgroup filesystem (/sys/fs/cgroup). This is a virtual filesystem with special files that act as the interface for creating, removing, or altering cgroups. To see available cgroup subsystems on your Linux OS:

❯ cat /proc/cgroups
#subsys_name	hierarchy	num_cgroups	enabled
cpuset	0	50	1
cpu	0	50	1
cpuacct	0	50	1
blkio	0	50	1
memory	0	50	1
devices	0	50	1
freezer	0	50	1
net_cls	0	50	1
perf_event	0	50	1
net_prio	0	50	1
hugetlb	0	50	1
pids	0	50	1
rdma	0	50	1
misc	0	50	1

Experiments

Prerequisites

Before starting experiments, install following tools:

apt install -y cgroup-tools iperf3

Experiment 1: Memory Limitation

Let's create a simple program that continuously allocates memory:

#include <stdlib.h>
#include <stdio.h>
#include <unistd.h>

int main() {
  // Allocated times
  size_t times = 0;
  // 1 MiB
  size_t blockSize = 1024 * 1024;

  while (1) {
    int *ptr = (int *) malloc(blockSize);

    // This is necessary or the actual memory won't be allocated.
    for (int j = 0; j < blockSize/sizeof(int); ++j) {
      ptr[j] = j;
    }

    ++times;
    printf("\rMemory allocated: %ldMiB", times);
    fflush(stdout);
    usleep(2000);
  }
  return 0;
}

Compile this program and run it under a memory-limited cgroup:

# Create the memory cgroup
cgcreate -g memory:/my_mem

# The unit is actually 500MiB
echo 500M > /sys/fs/cgroup/my_mem/memory.max

# Run the program within the cgroup
cgexec -g memory:/my_mem ./a.out

The final output:

Memory allocated: 496 MiBzsh: killed     cgexec -g memory:/my_mem ./a.out

Experiment 2: Network Throttling with net_cls

Kernel module net_cls (Network Classifier CGroup) may not be enabled by default. To enable it:

mkdir /sys/fs/cgroup/net_cls
mount -t cgroup -onet_cls net_cls /sys/fs/cgroup/net_cls

Egress Limiting

Steps:

  1. Use net_cls to tag network packages
  2. Use tc to control the traffic

Limitations for the following method:

First create a new net_cls cgroup:

cgcreate -g net_cls:limited_bw

Assign a class ID to the cgroup (class ID format is 0xAAAABBBB, where AAAA is the major number and BBBB is the minor number):

# Set a a 10:1 handle
cgset -r net_cls.classid=0x00100001 limited_bw
# Alternatively
echo 0x00100001 > /sys/fs/cgroup/net_cls/limited_bw/net_cls.classid

# Check the value, 0x00100001 equals 1048577
cat /sys/fs/cgroup/net_cls/limited_bw/net_cls.classid
1048577

Then control traffic using tc:

  1. Add a queueing discipline HTB (Hierarchy Token Bucket) to shape outbound traffic:

    tc qdisc add dev enp1s0 root handle 10: htb
    • qdisc: queueing discipline
    • 10: is the qdisc-id in format <major>: (also called handle), this id should be consistent with the cgroup class ID major number

  2. Create a traffic class with limit:

    tc class add dev enp1s0 parent 10: classid 10:1 htb rate 1mbps
    • 10:1 is the class-id in format <major>:<minor>, the major number must be consistent with parent handle, the minor number must be consistent with minor number of the cgroup
    • 1mbps means 1 mega-bytes per second

  3. Add a filter to associate traffic from the cgroup with this class

    tc filter add dev enp1s0 parent 10: handle 1: cgroup
    • 1: is the filter-id, don't need to be consistent with the cgroup class ID minor number (tested)

Starting a new process to test:

cgexec -g net_cls:limited_bw iperf3 -s

# Need to test on another machine
iperf3 -c <server_ip> -R

Alternatively, apply to an existing process:

cgclassify -g net_cls:limited_bw ${pid}

For convenience, here is a helper script.

References