StoneWork: a high-performance, all-(CNFs)-an in-one network solution

The stoneWork is used by PANTHEON.tech to integrate its CNFs on top of a single shared
FD.io VPP data-plane instance to achieve the best possible resource
utilization. This network appliance, however, is not a step back from
distributed chained/meshed microservices to the monolith architecture. Instead,
the integration is dynamic, based on container orchestration, CNF discovery,
sharing of network namespaces and re-use of data paths for packet punting
between CNFs.

Features

High-performance VPP-based data-plane
Management agent build on top of Ligato VPP-Agent
Suitable for both cloud and bare-metal deployments
Can be deployed as either multiple interconnected instances (service function
chaining), or a set of control/management plane microservices that use a
single VPP instance for data plane (this is a trade-off between flexibility
and resource utilization)
Northbound APIs are modeled with protobuf and accessible over gRPC, REST,
K8s CRDor through a key-value DB (etcd, redis, …)
Wide-range of networking features natively implemented in VPP, e.g.:
- High-performance device drivers (DPDK, RDMA, virtio)
- Routing, switching
- Tunneling (VXLAN, GRE, IP-IP)
- ACL-based filtering and routing
- NAT44, NAT64
- Segment routing
- VPN (wireguard, IPSec)
- Bridge domains, VRFs (multi-tenancy)
Management of features provided by the Linux network stack:
- Routes, ARPs
- iptables
- Namespaces, VRFs (multi-tenancy)
Dynamically (at run-time) extensible with additional features provided by
CNFs from our offering

Examples:

Before using StoneWork we recommend reading in this README and related
documentation in the StoneWork distribution folder about how to install and
configure StoneWork. If you are new to StoneWork, it may be easier to first
explore and run the provided examples, rather than trying to create deployment
manifests from scratch.

Examples of deployment manifests and configurations for various use-cases can be
found under the examples sub-directory.\ The [Getting Started]
a getting-started example will guide you through your first StoneWork
deployment.

Configuration:

Configuration for StoneWork is twofold:

1. VPP Startup Configuration:

The VPP Startup Configuration comprises configuration options which are set
before VPP is started. They cannot be changed at the run-time, either by a
management plane API or the VPP CLI). For StoneWork, the default VPP startup
configuration file is packaged in the image under /etc/vpp/vpp.conf. Some of
the examples override the default configuration with a customized
version of vpp.confmounted into the container using volumes. Typically, the
only configuration section that may require customization is the dpdk stanza,
where PCI addresses of NICs to be used by VPP should be listed. Run the
lshw-class network -businfo command to obtain the available network devices
and their respective PCI addresses. For example, if the PCI addresses of
interfaces were 0000:00:08.0 and 0000:00:09.0 (e.g. inside and outside
network), then the dpdk configuration would be:

dpdk {
    dev 0000:00:08.0 {
        name eth0
    }
    dev 0000:00:09.0 {
        name eth1
    }
}

Interface names can be selected arbitrarily, for example as eth0 and eth1‘
in the above example.

More information about attaching physical interfaces into VPP can be found
here.

2. Protobuf-modeled Network Configuration

StoneWork’s network configuration (VPP, Linux, CNFs) is modeled using
Google Protocol Buffers. A summary of all configuration items and their
attributes with descriptions can be found here (in markdown; also
available as a single PDF document). Provided is also a
JSON Schema that can be used to validate input
configuration before it is submitted. Some text editors, for example
VS Code, can even load the Schema and provide
autocomplete suggestions based on it, thus making the process of preparing
input configuration a lot easier. The original protobuf files from which the
documentation and schema were generated can be found in the /api folder inside
the StoneWork distribution. There is also the /api/models.spec.yaml file,
which contains one yaml document with metadata for every configuration model.
These metadata are used to associate configuration model with corresponding
protobuf definitions.

Network configuration is submitted into the control-plane agent either via
CLI (yaml formatted), written into a key-value datastore (e.g.
etcd; JSON-formatted) or applied programmatically over gRPC (serialized by
protobuf) or REST (JSON) APIs. The initial configuration that should be applied
immediately after StoneWork starts up can be mounted into the container under
/etc/stonework/config/day0-config.yaml (YAML formatted).

Each of the attached examples has a sub-directory named config,
where you can find configuration stanzas to learn from. Each example contains
the startup configuration day0-config.yaml. Additional attached *.yaml files
are used to show how run-time configuration can be modified over CLI. Please
refer to each example’s EXAMPLE.md file for more information.

Installation:

The following steps will guide you through the StoneWork installation process.
The distribution package contains the StoneWork docker image (stonework.image),
documentation (*.md) and some examples to get you started.

Requirements:

StoneWork requires a Ubuntu VM or a bare-metal server running Ubuntu,
preferably version 18.04 (Bionic Beaver).
Next, Docker and Docker-compose must be installed.Install with:$ apt-get install docker.io docker-compose
[For DPDK only]
Depending on the type of NICs that VPP of StoneWork should bind to, you may
have to install/enable the corresponding drivers. For example, in a VM
environment, the Virtual Function I/O (VFIO) is preferred over the
UIO framework for better performance and more security. In order to load VFIO
driver, run:$ modprobe vfio-pci $ echo "vfio-pci" > /etc/modules-load.d/vfio.conf Check with:$ lsmod | grep vfio_pci vfio_pci 45056 0 More information about Linux network I/O drivers that are compatible with
DPDK (used by VPP), can be found here.
[For DPDK only]
Make sure that the network interfaces are not already used by the Linux
kernel, or else VPP/DPDK will not be able to grab them. Run ip link set dev {device} down for each device to un-configure it from Linux. Preferably
disable the interfaces using configuration files to make the changes
persistent (e.g. inside /etc/network/interfaces).
[For DPDK only]
In order to optimize memory access, VPP/DPDK uses Huge Pages,
which have to be allocated before deploying StoneWork.
For example, to allocate 512 Huge Pages (1024MiB memory for default 2M
hugepage size), run:$ echo "vm.nr_hugepages=512" >> /etc/sysctl.conf $ sysctl -p Detailed recommendations on allocations of Huge Pages for VPP can be found
here.
Finally, the StoneWork image has to be loaded so that
Docker/DockerCompose/K8s is able to provision a container instance. Run:$ docker load <./stonework.image

Deployment:

StoneWork is deployed using Docker Compose version 3.3 or
newer. StoneWork itself is only a single container (with VPP and StoneWork agent
inside), but every CNF that is deployed alongside it runs in a separate
container, hence the use of Compose. The following is a template of
docker-compose.yaml file, used to describe deployment in the language of
Docker Compose. The template contains detailed comments that explain the meaning
of attributes contained in the template and how they relate to StoneWork. Angle
brackets are used to mark placeholders that have to be replaced with appropriate
actual values in the target deployment.

version: '3.3'

# Volume shared between StoneWork and every CNF deployed alongside it.
# CNFs and StoneWork use it to discover each other.
volumes:
  runtime_data: {}

services:
  stonework:
    container_name: stonework
    image: "ghcr.io/pantheontech/stonework"
    # StoneWork runs in the privileged mode to be able to perform administrative network operations.
    privileged: true
    # StoneWork runs in the PID namespace of the host so that it can read PIDs of CNF processes.
    pid: "host"
    environment:
      # Set log level (i.e. only log entries with that severity or anything above it will be printed).
      # Supported values: Trace, Debug, Info, Warning, Error, Fatal and Panic.
      INITIAL_LOGLVL: "debug"
      # MICROSERVICE_LABEL is used to mark container with StoneWork.
      MICROSERVICE_LABEL: "stonework"
      # By default etcd datastore is used as the source of the configuration.
      # Env. variable ETCD_CONFIG with empty value is used to disable etcd
      # and use CLI (agentctl) or gRPC as the primary source of the configuration.
      ETCD_CONFIG: ""
    ports:
      # Expose HTTP and gRPC APIs.
      - "9111:9111"
      - "9191:9191"
    volumes:
      # /run/stonework must be shared between StoneWork and every CNF.
      - runtime_data:/run/stonework
      # /sys/bus/pci and /dev are mounted for StoneWork to be able to access PCI devices over DPDK.
      - /sys/bus/pci:/sys/bus/pci
      - /dev:/dev
      # Docker socket is mounted so that StoneWork can obtain container metadata for every CNF.
      - /run/docker.sock:/run/docker.sock
      # To customize vpp startup configuration, create your own version of vpp.conf (here called vpp-startup.conf),
      # put it next to this docker-compose.yaml and mount it under /etc/vpp/vpp.conf.
      # Otherwise remove this mount.
      - ./vpp-startup.conf:/etc/vpp/vpp.conf
      # To start StoneWork with some initial configuration, create day0-config.yaml under the config
      # sub-directory, placed next to this docker-compose.yaml and mount it under /etc/stonework/config
      # Otherwise remove this mount.
      - ./config:/etc/stonework/config

  # Multiple CNFs may share the same Linux network namespace. This is in some case needed
  # if CNFs are to work together (e.g. BGP peering established over OSPF-learned routes).
  # The common network namespace is represented by a separate container (similar to the
  # sandbox container of a K8s Pod).
  router-ns:
    container_name: router-ns
    image: "busybox:1.29.3"
    command: tail -f /dev/null

  # CNF running alongside StoneWork (i.e. using the VPP of StoneWork as data-plane).
  # Name the container such that it is clear what services CNF provides (e.g. "cnf-dhcp").
  <cnf-name>:
    container_name: <cnf-name>
    image: "<cnf-image-name>"
    depends_on:
      - stonework
    # CNFs typically require privileges to perform administrative network operations.
    privileged: true
    # <cnf-name>-license.env is file that is obtained when the license of CNF is purchased.
    # Put <cnf-name>-license.env into the same directory as docker-compose.yaml.
    # It contains single line:
    # LICENSE=<signed license content>
    env_file:
      - <cnf-name>-license.env
    volumes:
      # /run/stonework must be shared between StoneWork and every CNF.
      - runtime_data:/run/stonework
    environment:
      INITIAL_LOGLVL: "debug"
      # MICROSERVICE_LABEL is effectively used to mark the container with CNF name.
      # StoneWork is then able to identify the CNF container among all containers.
      MICROSERVICE_LABEL: "<cnf-name>"
      ETCD_CONFIG: ""
      # If CNF runs alongside StoneWork (and not standalone), env. variable "CNF_MODE"
      # must be defined with value "STONEWORK_MODULE".
      CNF_MODE: "STONEWORK_MODULE"
    # Multiple CNFs may share the same Linux network namespace.
    # Use network_mode and point a group of CNFs to the same container (acting just like sandbox
    # container of a K8s Pod).
    network_mode: "service:router-ns"

  # here list other CNFs...

Development:

Build instruction for StoneWork can be found here.
The architecture of StoneWork is described in detail here.
A guide on how to make CNF compatible with StoneWork can be found here.
Documentation about StoneWork GNS3 VM development is here.