Orbit Experiment Management (Design & Analysis)
Experiment Initation
Flowchart
An experiment is created by a “batchexp” from a GUI interface whether it is a batch or interactive experiment. The experiment flow is as follows.
- The user creates an experiment through the web based GUI and provides an ns file to run. This ns file can be created by the user from scratch or created using the web based experiment builder.
- The ns script is parsed and node information and static routes between them are generated. This virtual state of the network is stored in the database.
- When the experiment is executed, a mapping is created between the virtual and physical topologies and the corresponding resources are allocated.
- The serial console access to the nodes is setup, NSF mount for experimenter’s home directory is exported at Ops; the corresponding DNS names and event system are setup.
The test nodes are rebooted and the new experiment images are loaded onto the test nodes (see node bootup for details). The experimenter’s home directory is mounted. VLAN and delay nodes for experiments are also set up.
Experiment information/control flow between entities
- The user creates a new experiment through the web interface and submits to BOSS.
- A NS file is submitted with this experiment. The ns source file is sent to OPS for pre-run operations.
- On OPS, this file is parsed and the topology specified in the file is stored in the database. An experiment is represented by a set of tables in the database.
- When the experiment is swapped to active for run, it is mapped to the available physical nodes, links and other resources such as delay nodes and traffic generators. The program used for doing this mapping is called assign; which is described in detail in the following paper: “A Solver for the Network Testbed Mapping Problem” (http://www.cs.utah.edu/flux/papers/assign-ccr03.pdf).
- At this point, OPS updates the database with the mapping information.
- The information about the nfs mounts and serial console setup is sent to OPS. The project directory will be mounted on every chosen node, after bringing it up.
- All the selected nodes are now rebooted remotely. This can be done via an addressable power strip or issuing a remote reboot command (death of ping) to the node directly. After rebooting the nodes are re-imaged and made ready for experiment.
- The VLAN is setup on the switch using snmp commands.
- Delay nodes are set up. Delay nodes are nodes that are designated to artificially create link delays in the experimental network as per user specification.
1. Starting an Experiment
1.1 File list:
|
File
name |
Type |
Location |
Function |
|
beginexp.php3 |
Php |
/usr/testbed/www |
Web interface for experiment start
process |
|
Webbatchexp |
Perl |
/usr/testbed/libexec |
Just a dummy file |
|
Mkexpdir |
perl |
/usr/testbed/libexec |
Create exp directories |
|
parse-ns |
perl |
/usr/testbed/libexec |
Parse-ns files, check Tcl syntax |
|
assign_wrapper |
perl |
/usr/testbed/libexec |
Mapping virtual topology to physical
nodes |
|
nalloc |
Perl |
/usr/testbed/bin |
Allocate free nodes to exp as reserved |
|
assign |
C |
/usr/testbed/libexec |
Execute assign algorithm |
|
Batchexp |
Perl |
/usr/testbed/bin |
|
|
Startexp |
Perl |
/usr/testbed/bin |
Call “tbswap in” |
|
tbswap |
perl |
/usr/testbed/bin |
Swap in the experiment, maintain
exp-state change, call “assign_wrapper” |
|
os_setup |
Perl |
/usr/testbed/bin |
Setup a new OS in nodes |
|
libdb.pm |
Perl lib |
/usr/testbed/lib |
All database-query support |
|
Libtestbed.pm |
Perl lib |
/usr/testbed/lib |
Define relations and access directories
about testbed |
|
Snmpit |
Perl |
/usr/testbed/bin |
Create Vlans through SNMP |
|
Snmpit_lib |
Perl lib |
/usr/testbed/lib |
SNMP routines for all Cisco or Intel
switches. |
|
nfree |
perl |
/usr/testbed/bin |
Free nodes from the experiment |
|
Stated |
Perl |
/usr/testbed/sbin |
A daemon to monitor state change and
update database |
|
os_load |
Perl |
/usr/testbed/bin |
Load an OS in the node |
|
os_select |
Perl |
/usr/testbed/bin |
Set imageid on node |
|
node_reboot |
Perl |
/usr/testbed/bin |
Manual reboot with power controller |
|
stated.log |
log |
/usr/testbed/log |
Output of stated daemon |
1.2 Database Tables used
|
Table name |
Major fields |
Usage |
Comments |
|
users |
uid |
Find user email |
|
|
experiments |
pid,eid |
find the experiment information |
|
|
reserved |
|
Find nodes in the experiment; mark nodes reserved for this eid |
|
|
nodes |
node_id node_type def_boot_osid |
Find os_info, pxe_boot_path |
Nodes are going to reboot with
def_boot_osid. |
|
node_types |
|
|
|
|
delays |
|
Add exp delays |
|
|
interfaces |
ip_addr |
Remove ip and ip alias |
|
|
Interface_types |
|
|
|
|
os_info |
os_id |
Get the detail information about image |
|
|
portmap |
|
|
|
|
vlans |
|
|
Used by assign |
|
virt_nodes |
|
|
Used by assign |
|
virt_lans |
|
|
Used by assign |
|
virt_vtypes |
|
|
Used by assign |
|
virt_routes |
|
|
Used by assign |
|
virt_agents |
|
|
|
|
virt_trafgen |
|
|
|
|
virt_node_desires |
|
|
|
|
wires |
|
For Vlan port information |
Connection between interfaces and
switches |
|
Switch_stackes |
|
SNMP-related search |
We have only 1 switch in each stack |
|
Event_objecttypes |
|
|
|
1.3 Flow chart
2. Terminating an Experiment
2.1 Relating file list:
|
File
name |
Type |
Location |
Function |
|
endexp.php3 |
Php |
/usr/testbed/www |
Web interface for experiment termination
process |
|
webendexp |
Perl |
/usr/testbed/libexec |
Just a dummy file |
|
endexp |
Perl |
/usr/testbed/bin |
Send email to experiment head, and Call tbswap out and tbend |
|
tbswap |
perl |
/usr/testbed/bin |
Swap out the experiment, maintain
exp-state change |
|
tbend |
Perl |
/usr/testbed/bin |
Clear virtual nodes… |
|
libdb.pm |
Perl lib |
/usr/testbed/lib |
All database-query support |
|
Libtestbed.pm |
Perl lib |
/usr/testbed/lib |
|
|
Snmpit |
Perl |
/usr/testbed/bin |
Removing Vlans through Snmp |
|
Snmpit_lib |
Perl lib |
/usr/testbed/lib |
Snmp routines for all cisco or intel
switches. |
|
nfree |
perl |
/usr/testbed/bin |
Free nodes from the experiment |
|
reload_daemon |
Perl |
/usr/testbed/sbin |
|
|
stated |
Perl |
/usr/testbed/sbin |
A daemon to monitor state change and
update database |
|
os_load |
Perl |
/usr/testbed/bin |
Load an OS in the node |
|
os_select |
|
|
Set imageid on node |
|
Node_reboot |
|
|
Manual reboot with power controller |
|
reloadlog |
log |
/usr/testbed/log |
Output of reload_daemon |
|
stated.log |
log |
/usr/testbed/log |
Output of stated daemon |
2.2 Related Database items:
|
Table name |
Major fields |
Usage |
Comments |
|
users |
uid |
Find user email |
|
|
experiments |
pid,eid |
find the experiment information |
|
|
reserved |
|
Find nodes in the experiment; put nodes reserved for reloading experiment….. |
|
|
nodes |
node_id node_type def_boot_osid |
Find os_info, pxe_boot_path |
Nodes are going to reboot with
def_boot_osid. |
|
delays |
|
Remove all delays |
|
|
vlans |
|
Remove all vlans |
|
|
interfaces |
Ip_addr |
Remove ip and ip alias |
|
|
os_info |
|
|
|
|
Scheduled_reloads |
|
Info queried by reload_daemon |
If empty, nodes will be loaded with
def_osid |
|
wires |
|
For Vlan port information |
|
|
Switch_stackes |
|
SNMP-related search |
We have only 1 switch in each stack |
2.3 Procedure Flow chart
3. Analysis:
The experiment start-up process is designed
in a very complicated way. To reuse Emulab program for Orbit, we need to remove
the “assign” process and implement a new interface. It requires some work.. It has been found that the assign process is being called
by the “tbswap in” call. Most essential routines and functions are enclosed in
the “assign_wrapper” program. It is tightly coupled with the usage of many
tables in the database. To replace this “assign” module with our Orbit “assign”
procedure, we need to first clarify the goals of “assign” program:
l
Map virtual nodes
to physical nodes.
l
Reserve those
nodes in reserved table of database
l
Setting up delays
and VLANs in database
l
Establish VLAN
through SNMP
It is better to separate the last three
parts to independent modules. For this purpose, we can ignore the
virtual-related tables from the database. The input to the new program should
be a set of parameters regarding to the physical topology of testbed nodes,
such as location and status, number of available interfaces.
The experiment termination process is
relatively simple and could be re-used by orbit without many modifications.