BWCTL Version 1.3
(Bandwidth Control)
http://e2epi.internet2.edu/bwctl/
$Date: 2009-02-23 08:08:04 -0500 (Mon, 23 Feb 2009) $
BWCTL is a command line client application and a scheduling and policy
daemon that wraps the throughput testing tools Iperf, Thrulay, and Nuttcp. These tests can measure
maximum TCP bandwidth, with various tuning options available, or, by doing a
UDP test, the delay, jitter, and datagram loss of a network.
The bwctl client application works by contacting a bwctld process
on the two test endpoint systems. BWCTL will work as a 3-party application.
The client can arrange a test between two servers on two different systems. If
the local system is intended to be one of the endpoints of the test, bwctl
will detect whether a local bwctld is running and will handle the required server
functionality if needed. bwctld manages and schedules the resources of the host
on which it runs. A more detailed description of the architecture is available
in the architecture documentation.
The bwctl client is used to request the type of throughput
test wanted. Furthermore, it requests when the test should be executed. bwctld
either responds with a tentative reservation or a test denied message.
Once bwctl is able to get a matching reservation from both bwctld
processes (one for each host involved in the test), it confirms the reservation.
Then, the bwctld processes run the test and return the results.
The results are returned to the client from both sides of the test. Additionally,
the bwctld processes share the results from their respective sides
of the test with each other.
BWCTL is used to enable non-specific throughput tests to a host without
having to give full user accounts on the given system. Users want the ability
to run throughput tests to determine the achievable or available bandwidth
between a pair of hosts. It is often useful to test to multiple points
along a network path to determine the network characteristics along that
path. Typically, users who want to do this path decomposition have to contact
the network/system administrators that control the hosts along the path
directly. The administrator needs to either run half of the test for the
user or provide a user account on the host. Also, network paths of interest
usually are controlled by multiple administrators. These hurdles have made
this kind of testing difficult in practice.
BWCTL was designed to help with this problem. It allows an administrator
to configure a given host as an Iperf, Thrulay or Nuttcp
endpoint that can be shared by multiple users without concern that those users
will interfere with each other. Specific policy limits can be applied to
specific users, and individual tests are scheduled so they will not interfere
with each other. BWCTL allows the administrator to classify incoming
connections based upon a user name and AES key (generated by a pass phrase)
combination or, alternatively, based upon an IP/netmask. Once the connection is
classified, the bwctld can determine the exact type and intensities of
throughput tests that will be allowed. (More details on the policy controls
can be found in the bwctld.limits(8)
manual page.)
What's Changed Since Version 1.2a
- Allows for different throughput testing tools. Currently supported are Iperf, Thrulay and Nuttcp. Use the -T option on bwctl to specify which throughput testing tool to use. Use bwctl -h to see which are supported.
- NOTE: Some of the throughput test options are not available for some of the testers. BWCTL does not support UDP tests, changing the output format or changing the output units with either Nuttcp or Thrulay.
- Allows throughput tests to use multiple, parallel streams. This can be specified with the -P option in the bwctl client.
- Allows for UDP tests with bandwidth limits of higher than 4.3Gbps. This only works with 1.3 clients and servers.
- Allows for comma-separated value output for Iperf. Add "-y c" to the bwctl command line client to use it.
- Allows administrators to define scripts to run at the completion of each test, allowing for results to be stored in a database, accounting tracking or any number of other tasks.
- Improves robustness of time/scheduling.
- Improves error handling of Iperf children processes.
- Improves error messages for failure conditions.
What's Changed Since Version 1.1b
- Added -f option to bwctld to allow it to run with root permissions.
- Added better usage() messages and more information to several user prompts.
- Fixed bug that caused all server side errors to be reported incorrectly be the client.
- Fixed bwctld bug that disabled srcnode option.
- Ported to Solaris.
- Ported to OS X.
- Removed NTP system call dependency. Added allow_unsync options.
- Increased default -L value for bwctl.
- Added messages to data output to indicate if Iperf was killed.
- Modified all config options to have '_' between words.
Features
-
Supports version 2.0.x of Iperf.
-
Supports version 5.3.1 of Nuttcp.
- NOTE: BWCTL does not support UDP tests, changing the output format or changing the output units with Nuttcp.
-
Includes built-in support for Thrulay.
- NOTE: BWCTL does not support UDP tests, changing the output format or changing the output units with Thrulay.
-
Full IPv6 support. No options needed. If the target of a test is specified
by a DNS hostname, and that name has both an IPv4 and an IPv6 address defined,
the bwctl command line application prefers the IPv6 address.
-
Data from both sides of the test is returned so that sending rate can be
compared with receiving rate.
-
3-Party communication is supported. The client does not have to be on one
of the test endpoint hosts.
-
A local bwctld is not required. bwctl will detect if there
is a local bwctld and use it if it is there; bwctl will mimic the
missing functionality if needed.
-
Port ranges for the bwctld peer connections can be specified to
allow bwctld to work in firewall environments.
-
Port ranges for the throughput tests can be specified to allow passive
monitoring to characterize the test streams more easily.
-
Limits the types of throughput tests that can be run. Sending a set amount of
data is not supported because BWCTL needs to know how long the test will take
before the test is started.
-
Supports dynamic TCP window size determination. (Currently, this is limited
because the bottleneck capacity is specified as a constant in the config
file. In the future, bottleneck capacity could potentially be dynamically
determined using techniques such as packet dispersion.)
Requirements
- BWCTL includes the Thrulay tester, but it would be best if Iperf 2.0.x is installed on the system.
-
The bwctld daemon prefers an NTP synchronized system clock to ensure the
two endpoints of the test are actually agreeing to the same scheduled time
window for test execution.
Therefore, BWCTL requires that NTP be running to synchronize the
system clock. (See the allow_unsync configuration option
described in the bwctld.conf(8) manual page
to see how to by-pass this requirement.)
NTP should be setup correctly on the system so that bwctld
can actually fetch reasonable estimates of the time error from NTP.
For the NTP algorithms to work correctly,
NTP MUST
be configured with no fewer than 4 clocks. (See http://twiki.ntp.org/bin/view/Support/SelectingOffsiteNTPServers
for more details.)
-
gnumake may be required for the build process (see
Building the Application).
-
To get good results, the endpoints will need to be well
tuned. See http://www.psc.edu/networking/projects/tcptune/
and http://www-didc.lbl.gov/TCP-tuning/buffers.html
for some excellent tips in that regard.
Supported Systems
BWCTL has been tested most extensively on Red Hat Linux 4 and 5 systems, but
most recent versions of UNIX should work. The code cleanly compiles on FreeBSD,
Linux, and OS X.
BWCTL compiles on Solaris. However, Thrulay does not work on that
platform so is not compiled in. If anyone has any interest in updating thrulay
to work on Solaris, let us know.
Recommended Hardware
BWCTL does not have any strict hardware requirements. The limits imposed
by the specific throughput tests are more taxing to hardware than the
scheduling and policy functions added by BWCTL. The hardware requirements
are therefore a function of the specific throughput tests that are desired.
Internet2
had good luck using the following hardware to perform 1Gbps tests on the
Abilene network:
-
Intel SCB2 motherboard
-
2 x 1.266 GHz PIII, 512 KB L2 cache, 133 MHz FSB
-
2 x 512 MB ECC registered RAM (one/slot to enable interleaving)
-
2 x Seagate 18 GB SCSI (ST318406LC)
-
SysConnect Gigabit Ethernet SK-9843 SX
Building the Application
The BWCTL software uses the gnu autoconf tools to configure the
build process. BWCTL is distributed with a pre-built configure script
so the actual autoconf tools should not be needed on the target
system. (Although, gnumake may be required.) The
configure
script can be run with the --help option to determine the full set
of configurable options.
A basic build procedure would be:
% gzip -cd bwctl-$VERS.tar.gz | tar xf -
% cd bwctl-$VERS
# --prefix is only needed if you don't like the default
# (/usr/local on most systems)
% ./configure --prefix=/install/root
% make
% make install
Please report any build problems to bwctl-users@internet2.edu.
Configuring bwctld
The basic procedure to configure bwctld is to create a
bwctld.conf
and, optionally, a bwctld.limits file
and a
bwctld.keys file. These files
need to be installed in the same directory that is specified with the -c
option to bwctld. The recommended directory is /install/root/etc.
(The etc directory below your install root.) There are examples
of these files in the bwctl-$VERS/conf sub directory of the
distribution.
bwctld.conf:
Used to configure basic operation of the daemon, such as server listening
port, the path for Iperf, and error logging. For a detailed description
of the options available, see the bwctld.conf(8)
manual page.
bwctld.limits:
Used to configure the policy limits for the daemon. There are two parts
to this policy: 1) authentication and 2) authorization. The authentication
is done by assigning a class to each new connection. Each class has a set
of limits associated with it. The classes are hierarchical, so a connection
must pass the limit restrictions of the assigned class as well as all parent
classes. For a detailed description of the options available, see the
bwctld.limits(8)
manual page.
bwctl.keys:
Used to associate identities with AES keys. These identities are then
mapped to a class by the bwctld.limits file. For a more detailed
description, see the
bwctld.keys(8)
manual page.
Configuring firewalls
If a firewall is enabled on the machine running bwctld, it will need to be
configured to allow incoming clients and tests. Use
this set of directions to configure the
firewall.
Configuring NTP
To ensure that tests can be run, the machine running bwctld should have its
clock synchronized using NTP. Use this set of
directions to configure NTP.
Running bwctld
The normal command line to start the daemon is:
% bwctld -c /install/root/etc
It is possible to run the daemon without a config file directory if enough
command line options are specified, but it is easier to use a config file.
To see all the available options:
% bwctld -h
More details on running the daemon, as well as a complete description of
the command line options, is available from the bwctld
manual page.
Running bwctl
The basic command line for the client is:
% bwctl [options] [-c catchhost] [-s sendhost]
The -c option indicates that the catchhost should be an
Iperf
server (think packet catcher). The -s option indicates the
sendhost
should be an Iperf client (think packet sender). At least
one of the -c/-s options must be specified. If only one of these options
is specified, the local host is assumed to be the other test endpoint.
To see a list of available options:
% bwctl -h
More details on running the client application, with a complete description
of all command line options, is available from the bwctl
manual page.
Caveats
Thrulay support is deprecated. If there is enough desire, or if someone is
willing to maintain thrulay, the functionality could be undeprecated.
Mailing Lists
There are two email lists to support this software:
-
bwctl-users
-
This is a general discussion list for users to discuss problems.
This list is monitored as bug reports should be sent here.
-
bwctl-announce
-
This list is used to announce new versions or significant developments
with regard to the software.
Information about these lists, including links to subscribe, is at
https://mail.internet2.edu/wws/lists/engineering.
Author
Jeff Boote
boote@internet2.edu
Internet2
Aaron Brown
aaron@internet2.edu
Internet2
Acknowledgments
The following individuals have greatly aided in the development, testing, or
debugging of BWCTL:
- Shumon Huque (UPenn)
- Paul Hyder (NOAA)
- Warren Matthews (GaTech)
- Sean Peisert (SDSC)
- Chris Tracy (MAX)
$Id: index.html 530 2009-02-23 13:08:04Z aaron $