CS284 TA Page: FAQs
Project
The QualNet Programmer's Guide is available at
http://www.cs.ucsb.edu/~gayatri/ta/ProgrammersGuide-3.7.pdf
Homework2
1. Buffering:
Ques:
If I send a msg and wait for the ack, I still have a copy of the msg sent
in case I have to retransmit it in the case of a failed receive (no ack). So
does this message count as part of the 20 (max) messages buffered? (i.e. Does a
message that was sent count as buffered until I receive an ack?, e.g. I have 5
messages in the buffer, I send one, but haven't received an ack yet. Do I have 5
buffered, or only 4?)
Ans:
Yes, the un-acked message counts as part of the 20. So, in your
example, you have 5 buffered messages until the ack arrives.
2. Throughput:
Ques:
What is throughput exactly? I looked up some
definitions online and I am confused as to what it is exactly. Also, what metric
would you like to
have it in and in general how did you calculate it?
Ans:
Throughput is often defined differently by different researchers. The most
common definition is the amount of data that gets through the network per
unit time, i.e. data bytes delivered to their destinations per second. Packet
delivery ratio is also sometimes referred to as throughput.
Regarding calculation of throughput for the assignment, please see the mail
regarding metric calculation that I had sent to the class list
earlier (it is posted on the TA FAQ page).
3. Test Simulation #2:
Ques:
In the test simulation #2, it says that we have to have a 4x4 grid of nodes, and
that we send from 1-6, 2-7, &c.
So what is the numbering scheme for this? Do we put the senders and receivers as
far as possible from each other? Or do we just go
1,2,3,4;
5,6,7,8;
9,10,11,12;
13,14,15,16;
Ans:
That's a good question. Surprisingly, nobody has brought it up before. I think
you should go with 1, 2, 3, 4; 5, 6, 7, 8; and so on.
4. Error in fileio.cpp:
---------- Forwarded message
----------
Date: Wed, 2 Feb 2005 12:38:53 -0800
From: Firat Kart <firatkart@gmail.com>
Ques:
I have one small question... I added all my code to UDP, everything goes fine (ACKs
get send etc...) but at the end of the simulation
qualnet terminates with:
---snip---
Error in file ../main/fileio.cpp:6326
Invalid network type.
Aborted
---snap---
I could not find "fileio.cpp", so my guess is that they only provide the objects
for this.
Ans:
To help that guy. I have realized that when we send message and the ack is lost.
The data packet send to client is deleted when it is
delivered to application layer. So, when the server tries to retransmit the
packet again, if we dont have a copy, the object is no
more existing. I suggest, before transmitting any packet, make a local copy of
that packet in case it may be deleted when it will be necessary later.
I spent 2 days to figure this out.. Maybe useful for others too..
5. Random Distribution of Nodes:
Ques:
The assignment problem statement mentions "RANDOM distribution of nodes". Does
that mean that the node placement distribution parameter (NODE PLACEMENT) in the
.config file should be changed to "RANDOM" from "UNIFORM" (used in the previous
homework)? The reason for confusion is that even for the uniform model, the
nodes are placed randomly in within each cell.
Ans:
The parameter should be set to RANDOM.
6.
Modifying UDP for UDP:
Some students raised the concern that
directly modifying UDP for HW2 (as opposed to creating a new protocol) affects
the operation of
AODV. From what I can see, the Qualnet AODV implementation does not use UDP at
all (it directly passes control messages to IP), so AODV should not
be affected by changes made to UDP. I would like you guys to verify this.
As long as AODV is not affected, you can go ahead and directly modify UDP for
HW2, or you could create a new protocol, whichever you prefer. If AODV
is affected, then UDP should not be directly modified.
7. Simulation Time
Ques:
I am confused about the simulation time for
homework 2. The assignment says "Run each simulation for 600 seconds". But I
don't know if that means the whole simulation, or each one of the CBR sessions.
If it is the whole simulation, then what should be the duration of the CBR
sessions? Same as in hw1 (200seconds)?
Ans:
Each session is supposed to send 1000 packets at 2 pkts/sec, so the session run
time will be 500 seconds each. The entire
simulation runs for 600 seconds.
8. Error in fileio.cpp
Ques:
I have one small question... I added all my
code to UDP, everything goes fine (ACKs get send etc...) but at the end of the
simulation qualnet
terminates with:
---snip---
Error in file ../main/fileio.cpp:6326
Invalid network type.
Aborted
---snap---
I could not find "fileio.cpp", so my guess is that they only provide the
objects for this.
Ans:
I would suggest you write the Qualnet support people about this. They usually
respond within a day.
9.
Source/Destination Pairs
Ques:
I have a question regarding the
procedure for testing our protocol modification:
For tests 2-4, where it says that the source nodes are 1-5, and the destination
nodes are 6-10, am I correct in assuming that the source and destination nodes
are respective, as in the last homework? (i.e. 1->6, 2->7, 3->8, 4->9, 5->10?)
Ans:
Yes, you are correct. They are
respective source and destination pairs like in HW1.
10.
Creating UDP ack packets
Some students have questions about creating UDP
Ack packets for HW2, such as what parameters to supply to the MESSAGE_Alloc
function etc. Although the TCP code is the logical place to reference, it is
rather complex and confusing. I would recommend looking at the Qualnet AODV
implementation (I guess you guys know what AODV is by now) for examples of how a
new
packet type is created and processed. For example, look at how an AODV RREP is
created and sent, and how it is received and processed. This is not very
different from what you need to do for UDP Acks, except the appropriate
modifications for operating at the Transport layer instead of the Network layer.
11. Overview of HW2:
Ques:
I was wondering if you could give me a brief overview of what is needed to be
done for hw2. I was going through the tutorial for using the developer gui and I
am wondering whether it is better to modify C++ code or make a state diagram for
our new protocol model. I'd like to know what you recommend, modifying C++ code
or using the gui.
I am not sure if it is possible to make the reliable UDP model using the gui or
not. Also, if I am to go through the C++ code I am confident that I can modify
and extend all of the features that are necessary. However, is there some way to
easily make qualnet recognize this as a new protocol so I don't have to run the
protocol as udp and keep track of my changes in case I want to use regular udp
later? And how would I compile it and link it to qualnet?
I know that I am not really making myself clear but I am not sure where to go
with this assignment.
Ans:
I personally prefer modifying the C++ code directly. Although I don't have much
experience with the GUI, what I understand is that the GUI is useful for doing
stuff like creating a new protocol, in which case it will automatically generate
a code skeleton for the new protocol and update necessary makefiles etc. After a
point, you will probably need to manually edit the C++ code. You could use this
technique to create a new UDP version, copy the UDP code into it and then modify
it as needed. Alternatively, you could simply make a copy of the udp files, give
it a new name and manually figure out which makefiles and other Qualnet files
need to be modified to include the new protocol. One way to do this is to look
for what files have been modified to include UDP itself - it shouldn't be too
hard.
Since this is a grad class, you are encouraged to explore Qualnet more on your
own and figure out what is the best way to get things done.
12. Protocol Designer:
Ques:
I would like to know if for the HW2, we have to use the protocol designer
included with Qualnet (ie the GUI tool). That is because during the
presentation of Qualnet in class, I remember that it was mentioned as an useful
tool. Is the HW the point to learn to use Qualnet (and its GUI) or to read cpp
code? If we can use the GUI, after having created my transport protocol, I would
like to import the files from UDP. Is it possible, or am I just dreaming?
Ans:
It is entirely upto you how you want to do the homework. You could use the GUI
or directly modify the C++ code - whichever you prefer. I don't have much
experience with the Qualnet GUI. If you want to use it, you should experiment to
figure out what is/is not possible. Refer to the manuals - they are a good
resource. An important objective of the assignment is for you to play around
with the software on your own and learn to use it for your own purposes.
13.
TCP Timers:
Ques:
In the second assignment we have to change the udp code of Qualnet to make it
reliable. We have to use timers in order to do that. The assignment says:
The best way to understand timers in QualNet and how to generate
acknowledgements is by going through the TCP code included in the QualNet
package.
I have gone through the tcp code during 2 days, but I have found it really
difficult to read, so could I implement my own timer or could you give me any
clue about this?
Ans:
An important objective of this assignment is for you to figure out how timers
are done in Qualnet. You do not have to manage your timers exactly
the way TCP does. However, you will have to use the same underlying timer
mechanism provided by Qualnet, which is used by TCP as well as all other
protocols that need timers. You can figure out what this mechanism is by looking
at the TCP code (you don't have to understand all of the code,
just see timer-related methods in tcp.cpp) or by going through the Qualnet
manual.
14. Details on Metric Calculation:
Average throughput
Method:
Take an average of the throughput values reported by the CBR Servers (not
Clients).
Reasoning:
A throughput value is reported by each CBR client and server. The throughput
reported by the client is the 'sending' throughput - it only
depends on the configured packet size and rate, and has nothing to do with
network performance. The throughput reported by the server, on the other
hand, is the 'receiving' throughput, and will vary depending on network
conditions. So, when calculating average throughput for analyzing network
performance, it makes sense to average ONLY the throughput values reported by
the CBR Servers.
Packet delivery ratio
Method:
i. For each simulation run, sum up the total number of packets sent and the
total number of packets received across ALL data sessions for that
run.
ii. Calculate the packet delivery ratio for each run as:
ratio = (total packets received over all sessions)/(total packets
sent over all sessions)
iii. When executing multiple runs with different seed values (tests 3 and 4),
take an average of the ratios obtained for each run.
Reasoning:
There are different ways to calculate the packet delivery ratio. One way is to
take a ratio for each session separately and then average these
ratios (this was the way I had recommended earlier for HW1). The other way is
the one described above. Each method has its merits and demerits, and
there's no one correct answer really. However, the numbers resulting from each
method can be very different. So, to avoid confusion and ease
grading, everyone should use the method described above for HW2.
End-to-end delay
Method:
Simply take an average of the end-to-end delays reported by the CBR servers.
No reasoning required for this - pretty straightforward.
Setting the transmission range
We noticed that the transmission power required for obtaining a 250m
transmission range may be different depending upon whether you're using
the commandline or GUI to run your simulations. The reason is that some
parameters have different default values set through the GUI as compared
to default.config. To get a 250m transmission range when using default.config,
the transmission power needs to be set to 8.1441. However, through the GUI,
you may need to set it to 9.6931.
In either case, make sure you verify the transmission range using the
radio_range program. This program takes your .config file as parameter.
Homework1
1.
Setting the
transmission range:
There are a bunch of different ways to set the
transmission range
of the
nodes to 250m. One could modify the transmission power of the sender, or
the sensitivity of the receiver, or antenna characteristics, etc. Each
of
these ways is likely to have different side effects and can produce
different results.
So, for HW 1, you are required to change the transmission range by
modifying the transmission power (TX-POWER) ONLY. Play around with
different values until the transmission range is set to EXACTLY 250m for
the desired bandwidth. Remember that the 802.11b physical layer is to be
used.
2. Calculating the AODV
control overhead:
Here's how the AODV control overhead is to be calculated:
To find total number of RREQs, sum the following values across all nodes
in the simulation:
Number of RREQ Initiated
Number of RREQ Retried
Number of RREQ Forwarded
Number of RREQ Initiated for local repair
Number of RREQ sent for alternate route
Similarly for RREPs, sum up
Number of RREP Initiated as Dest
Number of RREP Initiated as intermediate node
Number of RREP Forwarded
Number of Gratuitous RREP sent
and for RERRs:
Number of RERR Initiated
Number of RERR Initiated with N flag
Number of RERR forwarded
Number of RERR forwarded with N flag
The totals should be divided by the number of data sessions to get the
average number per session. This is then to be averaged across all
simulation runs.
3. Seed Value and
Node Deployment:
Ques:
I ran the Qualnet simulation using batch experiment with 5 different
seed values, but the initial node locaitons are the same in each of the
run. Shouldn't the seed value change the initial deployment? Or I have
to redeploy those nodes in each simulation?
Ans:
As long as you're getting different stat files by changing the seed,
its fine. Since we're using the Uniform model of initial node placement,
the initial node positions will not be drastically different with
different seed values.
4. Single Script:
Ques:
In the final question of
Part1, I need to write
scripts to determine the values. Could I use "grep" to get the
information I want to a new file first, and then write a script to
parse the new file?
Ans:
You should have a single script that does all the
necessary processing.
5. End to End Delay:
Ques:
I'm writing the script to calculate
the end to end delay of UDP traffic, but the only end to end delay
information I have from my default.stat file
(after running default.config) is from CBR servers. Is this what the
professor meant?
Ans:
CBR traffic runs on top of UDP, so the end-to-end CBR
delay is the same as
the end-to-end UDP delay.
6. CBR Sessions, Total Bytes etc:
1) To clarify,
when you say sum the following values do you mean
total number of RREQs = (Number of RREQ + Initiated + Number of
RREQ Retried + Number of RREQ Forwarded + Number of RREQ Initiated for
local
repair + Number of RREQ sent for alternate route)
Correct? Also, I'm not sure what the definition of 'session' is? Does
session mean 'each node'? Can you please clarify?
------> That is correct. By 'session' I mean CBR session. There are
5 CBR sessions
in every simulation run for part 2.
2) I've ran the default simulation for Question 1. I've inspected the
default.stat file and I see where the CBR Total Bytes Received value is
defined but how do you calculate the CBR Total Bytes Sent value?
------> That value is also present in the .stat file. Look for the
statistics printed by the CBR Client.
3) How do you calculate the end-to-end delay for the UDP traffic? There
is no data in the .stat file that informs you of UDP traffic.
The only UDP traffic data states how many packets from and to the
Application Layer.
------> CBR traffic runs on top of UDP. So the end-to-end delay of
UDP packets is
essentially the end-to-end delay of the CBR packets.
7. Scripting:
Ques:
I am not very familiar with script writing, and I cant figure out how I
will be able
to write scripts for the first part that I will use in the second part
of the homework.
Ans:
You need to write scripts to parse the .stat file and extract the
information you need. You could use any scripting language for this,
such
as perl or python. If you are not familiar with these languages, you
could also write a C or Java program to achieve the same thing.
Basically you
need to read in the file, parse the lines and extract the information
you need to compute the metrics.
>> Do we look for the transmissio range by running
bin/radio_range *.config.
Yes, that is correct.
8. GUI vs. command-line:
Some students tell me that they were able to successfully run the
simulations for HW1 part 2 from the GUI without having to change the
scheduler type. There were concerns that the configuration parameters
might be set differently from the GUI and that the results might be
different.
I haven't yet had a chance to check out the GUI and make sure the results
are consistent. Anyway, no matter whether you're using the GUI or
command-line, you should ensure that the SPLAYTREE scheduler is being
used, and that all the other parameters are correctly configured. As long
as that is done, I see no reason why the results should be different by
the two methods.
I suggest that in your homework turn-in you mention what method you
used to run the simulations (GUI or command-line). So in case there are
any discrepancies, we can take care of them while grading.
9. Skipping nodes?
Ques:
I would like to ask, when finding AODV, I have 50 values for each run, I mean,
would I just look at the values of the nodes that have
traffic. Because in my stat file, I see nodes after node id 10, having non-zero
values for AODV parameters. Should I skip these?
Ans:
No, you don't skip any nodes. You are supposed to total
all 50 values to
get one resulting value for each run. Then you find the average and
standard deviation for the set of 5 values (one from each run).
10. More on metric calculation:
Add a 'Method' section to your homework turn-in and detail how you ran the
simulations (GUI vs. commandline) and exactly how you
calculated each metric. This is very important - different people have used
different methods and unless you describe your method clearly it
would be hard for us to do the grading.
As long as your method is reasonable, and your results are correct, your grade
will NOT be affected.
Now just for the sake of information, and for future homeworks/projects,
here's how I think the metrics should be calculated:
1. For packet delivery fraction, the metric should be
calculated per data session first, and then averaged across sessions and runs.
For this homework, this would result in 25 packet delivery fraction values. Same
applies to the end-to-end delay.
2. For AODV overhead, you can get only one value per simulation run, since its
not possible to figure out which packet was sent for what session. So, for the
AODV overhead metrics for this homework, you would get a set of 5 values for
each kind of packet.
3. When averaging the metric, you can either average per simulation run and then
across runs, or you can average all 25 values together - it would
give the same result.
4. Standard deviation is more confusing. Here's how I see it: for each metric,
you are ultimately going to report ONE average value across all
runs. That is the point of doing multiple runs with different seeds in the first
place - that you can get a good average. The purpose of the standard
deviation is to indicate how much your ORIGINAL data points deviate from the
average you have calculated. So, it makes sense to do the standard deviation
calculation ONCE across the entire set of 25 (or 5 for AODV overhead) values.
11. Standard Deviation Formula:
Ques:
It has been awhile since I calculated standard deviation
and I was wondering if we should divide by N or N-1. In other words, are we
calculating sample standard deviation or population standard deviation.
http://infinity.sequoias.cc.ca.us/faculty/woodbury/Stats/Tutorial/Disp_SD.htm
Ans:
Again, since this was not explicitly specified in the assignment, you can use either one and specify what formula you've used.
12. Sessions?
Ques:
The assignment says: "... These values should be an average of each session
within a simulation and then should be averaged over all seed
values." What is a session? The data transmission between two nodes (e.g. 1-6)?
or all transmissions of a simulation?
------> A session refers to one CBR session between two nodes (e.g 1 and 6). There
are 5 data sessions in every simulation run.
The stat file only contains the number of send and the number of received
packets for a session(?) so the average of a simulation is the
average of the 5 sessions?
------> That is correct.
For my understanding: I have to calculate 12 values per metric (6 averages and 6
standard deviations), right? Of these 6, 5 are for the
sessions and 1 for all of them together.
------> You have to ultimately report only one average and one standard deviation per
metric. To get the average, you first take the average of all sessions
in each simulation run and then average these values across all 5 runs. For the
standard deviation, compute just one value across all the sessions
in all the runs.
QualNet Installation
1. Quotas increased:2. Downloading the Tarball:
Many
of you are finding that, on downloading Qualnet, the size of the
tarball is only 25MB when it is supposed to be around 88MB. The reason
for
this is an upper limit on file size set by the operating system
(typically 25MB). With this limit, even though your quota has
been increased, each
file can still be only 25MB at the maximum and the download gets
terminated on reaching 25MB.
If you're facing this problem, here's how to fix it:
1. At the shell prompt, type the following command:
>> ulimit -f unlimited
This will remove the upper limit on the file size
(**make
sure to do this in the bash shell**)
2. Start your browser from the same shell prompt (I'm not sure if this
is
necessary, but I just want to be safe).
>> mozilla
3. Now download the file - it should get downloaded completely.
4. Check the size of the downloaded file and make sure it is around 88MB
4. QualNet on
Windows:
---------- Forwarded message ----------
I could run the simulations on my windows machine without any problems.
Ofcourse i downloaded the two week evaluation copy from the
scalable-networks site.
5. QualNet Crashing Problem:
The default.config
file in Qualnet 3.8 is set up to use the CALENDAR scheduler which is
still
in beta and therefore buggy. So, we should all move to the default
SPLAYTREE scheduler - this will fix the crashing problem.
To use the SPLAYTREE scheduler, change the line
'SCHEDULER-QUEUE-TYPE CALENDAR'
in default.config to
'SCHEDULER-QUEUE-TYPE SPLAYTREE'
(or simply comment out the line and Qualnet will default to SPLAYTREE).