After using the Wondershaper script (available from LARTC) and merging the ideas from the scripts found in the QoS-Connection-Tuning-HOWTO and ADSL-Bandwidth-Management-HOWTO, I thought all I needed from Linux and the QoS scheduler in it was over; oh how wrong I could of been.
Whilst staying at the BPL I grew to love the 2Mbit download/256kbit upload we had, until all eight members of the flat started using the connection. Two of whom were permanent P2P users, others gaming, others USENET downloading. Whilst all I wanted was to be able to stream SomaFM's Groove Salad and Monkey Radio, play XPilot and Enemy Territory plus connecting to some remote servers (and the other way round when outside the flat). “Dear God, was it too much to ask for?” Apparently so.....then did I decided it was time to get serious with Linux's implementation of QoS and throw together this script.
This script is really only suitable for situations where you want to fairly share and prioritize the traffic for a group of machines on a per IP basis. The script relies on ESFQ to take all the hard work in fair sharing the bandwidth, and uses IMQ for IP-Masquerading and traffic generated by the traffic shaping box (when it is not being an Ethernet bridge). This makes the script perfect for home and small office setups; even for large offices I would imagine where you want to throttle a large number of machines evenly and identically. Every machine under the scripts ‘domain’ will have the bandwidth fairly shared and their packets prioritized.
Although the Wondershaper script taught me a lot it did rely on the people who used the script having to know about how to use ‘tc’ and ‘ip’ to tune it to their needs rather than the much simpler ‘iptables’ MARKing idea that the QOS-Connection-Tunning-HOWTO and ADSL-Bandwidth-Management-HOWTO introduced me to. So I designed a script so that 95% of users will just need to fill in their upload/download speed and how their box is connected on the network, 4% of the users would also change the ‘iptables’ rules at the end of the script, and the remaining 1% would say “nice idea” and then write their own version ;)
I feel this approach opens up QoS to the masses, and gives a Bill-Gates-touchy-feely-tickbox that the M$ Whores have come to love.....only in this case it actually works and does something.
....yeah I know.....I will fill this in at some stage..... If you are really curious have a look at the script :)
Download the regular ‘jdg-qos-script’ and the ‘binaries’ tarball and extract the files in the tarball (‘tar -jxf binaries-blar’). Make the ‘jdg-qos-script’ executable by ‘chmod +x jdg-qos-script’ and then edit the values at the top of the file to suit your situation.
These parameters simply describe which network/ppp interface is internet facing and which one points to the internal network. The speed values are all in kilobits. This script has really only been tested in a PPPoA/sat-link/cable environment and we have found 80% of the actual bandwidth is a good starting point to play with; as PPPoE sucks more than a milking machine you will have to probably lower your bandwidth further still.
DWIF=eth1 # LAN facing interface DWIFLIMIT=408 # Download Speed (2048*0.8) UPIF=eth0 # WAN facing interface UPIFLIMIT=204 # Upload Speed (256*0.8)
These tells the script if you have a NAT setup and what the IP address regions are of local addresses for what traffic should not be shaped by the script (this is space seperated).
NAT=1 # 1 || 0 (yes || no) LOCAL_TRAFFIC="10.0.0.0/24 30.23.43.16/29" # 'local' LAN address
If the script is running on a machine that is acting as a transparent Ethernet bridge then you should turn on this parameter, as it removes the need for IMQ.
ETHERNET_BRIDGE=0 # 1 || 0 (yes || no)
Now this variable is useful if you are running anything that needs a guaranteed amount of bandwidth such as VoIP. If you set this to how much bandwidth you need (non-zero) then the script creates a special bucket that will always get this amount of bandwidth which is taken away as needed. By setting this you will need to add some IPTables rules to make use of the bucket, the rules need to be inserted (notice the ‘-I’ and not ‘-A’ that is used in the example IPTables rules in the script) at the top of the SHAPER-IN and SHAPER-OUT chains before everything else and after marking the packet you must RETURN the same rule. If this is not done you will not get your special bucket to work.
REALTIME_BANDWIDTH=0 # value in kbit
If you have to force a couple of TCP port numbers to be in the low priority bucket you can do so here by simply putting them in the variable FORCE_LOW_PRIORITY space separated.
FORCE_LOW_PRIORITY="4662" # low priority traffic (edonkey: 4662)
If you have compiled support for IPP2P and CONNMARK then you may turn on this variable to gain much better support for identifying P2P traffic and throwing it into the lowest priority bucket.
NETFILTER_P2P=0 # netfilter p2p matching support
Type ‘iptables -v’ and remember your version number for iptables. You now need to type ‘cp binaries/binaries/libipt_<feature>.so-<version> /lib/iptables/libipt_<feature>.so’ to give iptables support for any of IMQ, IPP2P or CONNMARK (please remember that IPP2P needs CONNMARK). Then ‘cp binaries/binaries/tc /usr/local/sbin/’ to give you one ready to use patched ‘tc’ binary. If you do not trust my binaries, you probably know what needs doing anyway and can find the ESFQ/HTB/IMQ patches for iptables and iproute2. Remember to make all the above files ‘chown root:root’ and ‘chmod 755’.....
Now you need to patch your kernel (typing from within /usr/src/linux-<version> ‘patch -p1 < ../path/to/binaries/patches/kernel/v2.x/0x_blar-blar.diff’, you probably will need to fix very minor failed hunks) with all the patches found in ‘binaries/patches/kernel/v2.x/’. After this make sure you compile the following into your kernel, you probably have most of these on already; these are just what my script depends upon for kernel 2.4.x (emphasized parts are optional and what they are needed to provide support for are given within the ‘[...]’):
If kernel 2.4.x does not float your boat then there is always 2.6.x :)
Now reboot your machine with your new kernel, and hopefully all you need to do is type ‘./jdg-qos-script start‘ and it should work. Since version 031207 the script can now store the variable values in an external file, if you would like your configuration values to remain the same even when you place the script, copy them straight from the script into a textfile and point the script to this textfile, the default is ‘/etc/default/jdg-qos-script’ (which suits Debian perfectly).
The script has a stack of helpful parameters that you can call it with, it also has been designed to hopefully just slot in and out of your current setup regardless of what your iptables rules might currently be:
| ./jdg-qos-script <parameter> | What it does |
|---|---|
| start | Removes its-self from your machine and puts in a fresh copy |
| stop | Removes its-self from your machine, effectively disabling traffic shaping |
| restart | It actually does exactly the same as ‘start’ |
| status | A single shot printout of the iptables ‘hits’ and how ‘tc’ is getting along |
| pollbuckets | Prints the results of what ‘tc’ is up to every second, great to see if things are actually working in realtime |
| pollrules | Prints the results of what the iptables MARKing is up to every second. Really only useful to those wanting to tweaking the iptables rules at the end of the script |
‘Advanced’ users can tweak the iptables section at the end of the script. For upload you have ten ‘buckets’ (numbered 20 to 29, where 29 is low priority) to pick from. You work out a chain of iptables rules to decide what type of packet goes into which bucket. Treat the beginning of the chain to set a base 'MARK', and then later iptables rules adjust the packet so when you leave the chain the packet has the desired MARKing (and so priority). For downloading, you have only two buckets, number 20 where traffic cannot be dropped, such as ICMP and UDP traffic (and also high priority); whilst number 21 is for bulk traffic which can be dropped safely with knowledge that it is not going to generate more traffic, such as FTP data traffic (however not TCP ACK packets).
| [yyyy/mm/dd] | stuff happened....... |
|---|---|
| [2003/12/07] (binaries-031207) | Lots of major features have been added for your pleasure. Apart from the general tidy up and re-arrangements I have decide to removed the QOS_GATEWAY_TRAFFIC as I could never imagine a situation where this would be turned off; if people scream and rant about its removal I will put it back. RED has now been configured more sensibly (hopefully) after carefully reading through (read as used their example) the Differentiated Service on Linux HOWTO and in the process removed what I found to be the now defunct DWIFSPEED variable. Three major features have been added to the script, one of which is that you can now use an external configuration file if you want, so you do not have to keep replacing all those values each time you get an update or replacement. The second feature is support for IPP2P which is a really nice Netfilter patch that lets you identity P2P traffic very easily; after the kernel patching. As IPP2P really can only pick out the first few packets of a P2P stream you have to use it in combination with CONNMARK to mark the rest of the stream. The third toy is that the edonkey ‘support’ has been replaced with a generic system where you simply list the TCP port numbers you wish to be forced to a low priority in the ‘FORCE_LOW_PRIORITY’ variable, its a space separated list. The final surprise (I know I said three, but this is better than a feature) I have, with the help of my flatmate, ported IMQ, ESFQ, IPP2P and CONNMARK to the 2.6.x kernel series as it seems everyone else on the Internet seemed far to lazy to do it; so we did it. :) The upshot of this is that my script, and many others, now run fine under 2.6.x. Chatting by e-mail to Andy Furniss, he explained that I needed a patch to get IMQ, NAT and ESFQ working as expected together; this is now included in the binaries package. |
| [2003/11/16] | The major changes in this release is the added support for multiple local subnets and also a bucket which guarantees a particular amount of bandwidth to whatever is associated with it. A few reports from friends (not the Internet community though) that edonkey breaks things and also VNC through SSH involves great badness. The only fixes for edonkey is to hard code in the hooks to make a particular port low priority. I also have tweaked the IPTables rules at the end to hopefully make things behave better. Decided to turn on the ECN marking part of RED after some thought; ECN is a much better idea however broken routers drop the packets thinking they are bad, this is why I left it off before however this is what we want anyway. |
| [2003/08/17] (binaries-030817) | Noticed that whilst SCPing to some remote machine and SSHing to a different remote machine made the SSH session very laggy. After battling with iptables for ages, I found that although it can set TOS to 0x0a (Maximum-Throughput and Minimum-Cost) it is unable to check for it; which is what SCP does to its bulk traffic. So I decided to be cunning and check for port 22 and when the Minimize-Delay TOS flag is not set; the problem is now fixed. Tweaked the regular TCP control flag priorities, the ACK flags are still dealt with as before, however everything else is (RST, FIN, etc) is made high priority; this reduces four iptables checks to just one. Updated the installation instructions to correspond to the new IMQ patches, rather than the old ones. Found my FTP client (LFTP) seems to set all its traffic TOS flags to Minimize-Cost so I added the extra iptables rule to classify bulk traffic and removed the FTP catchall I was using before. PLEASE NOTE that the binaries-030817 was ‘fixed’ on the 030818 as it was missing a patch (netfilter-imq-patch-2.4.21.diff which is based on pom-20030625.diff) |
| [2003/08/05] | Tweaked the P2P identification a bit better. Now copes with large numbers of ACK packets from P2P clients without killing everyone else. This was only noticed when Bob returned back to the BPL and Lobley was having some downloading fun also. This was originally in some of the older scripts, but I removed it thinking I knew better, goes to show :) |
| [2003/07/27] (binaries-030725) | Tweaked the iptables rules to account for FIN/RST traffic and removed some of the pointless ones. Properly word-wrapped the script to look nicer. Changed ‘prio $[$LOOP+1]’ to ‘prio $LOOP’ which makes no overall difference except cleanups. Once I have created this website for the script and actually receive feedback from more than two out of ten people this version probably will become the first public release. Added better commenting in the script. (FIRST RELEASE TO THE GENERAL PUBLIC) |
| [2003/07/19] | Worked out why local destined traffic was working for a bridge but not in a gateway setup, so added a new parameter (ETHERNET_BRIDGE), and at the same time added NAT support, I think. I think pretty much this is it, the dependencies have been kept low. (BETA RELEASE TO ANYONE UPON E-MAIL REQUEST) |
| [2003/07/17] | QoS on locally generated traffic in both directions now seems to be working nicely.....finally. The NAT stuff is completely fscked, never really worked, so the code removed and will be worked on in the near future. |
| [2003/07/15] | Cleaned up (read as fixed bugs) a few things LOCAL_TRAFFIC should now work as advertised, unlike before where pings to the gateway where higher than to the outside world! :-/ |
| [2003/07/12] | Made silent the HTB debugging, change with HTB_DEBUG variable. Tweaked the iptables rules to be better, or so I think :) Improved, parameters to ‘qos’ script. |
| [2003/07/11] | Fixed a few NAT issues with LOCAL_TRAFFIC preventing uploads being bucketed; do to over use of iptables :-/ Looked into the solution for fair sharing uploading bit under NAT and it looks like IMQ will have to become a dependency for our NAT friends. Next version of the script will come with hopefully this support added. |
| [2003/07/08] | NAT/IPMasq support, which was more or less there, I found after consulting was already present. Not 100% convinced though as LOCAL_TRAFFIC might not work with IPMasq/NAT when thinking about it. Changed ‘tc’ to be ‘$TC’, same for ‘iptables’ to ‘$IPTABLES’. Used ‘for’ loops to make script clearer to read. Made sure the ‘stop’ commands completely tidies up, usually.... Adjusted the RED probability to 0.02 instead of the 1.00, due to me mis-reading things.... ;) Still think it needs tweaking.... |
| [2003/07/06] | tweaked values, convinced myself that 15% bandwidth was lost due to header overhead and trimmed a further 5% to clear buffers. Tweaked the download link, as ESFQ was set to ‘src’ instead of ‘dst’ and made RED use the $(DW/IP)IFLIMIT values. result in non-fluctuating ping and RED is now working. Also added support to exclude ‘local’ traffic. Fixed various bugs where traffic was not going to their correct queues. Note that if you have no ‘private’ addresses (I have Linux running as a Ethernet bridge and so its on a private IP) you can leave the LOCAL_TRAFFIC section alone |
| [2003/04/29] | added some RED usage (downloading TCP traffic), migration to use iptables MANGLE rather than using ‘tc’ filter by using the QoS-Connection-Tuning-HOWTO and ADSL-Bandwidth-Management-HOWTO for reference migration to not use ingress |
| [2003/04/28] | migrated using SFQ to ESFQ (for source IP hashing) |
| [2003/04/28] | Started with Wondershaper-1.1a script |
The latest version of the script is dated the 7th of December 2003 (aka 031207 [yy/mm/dd]). To obtain older versions look at the Changelog above and get it that way.
As well as the actual script you will more than likely need the kernel patches and binaries available in this downloadable package. The binaries give ‘iptables’ access to IMQ whilst the ‘tc’ binary has support for HTB3, ESFQ and IMQ in it. If you do not trust me then you can find the required patches at:
The ‘iptables’ binaries where originally for just version 1.2.6a (as supplied on the website) except that I used a hexeditor to change 1.2.6a to 1.2.7a, 1.2.8 and 1.2.9 to permit it to work; probably have broken stuff due to it but it works for me. All the binaries I have supplied just came from the URL's above, feel free to check the MD5 sums ;)
All the IPP2P and CONNMARK patches may be found for both the kernel and userspace are in the binaries package, along with the binaries for iptables which should work, the 1.2.9 ones do, as they received some hex editing loving also. Again if you do not trust me then you can find the required patches at:
Also in the binaries files is the all the patches you would need to put this script into full functionality mode; this is the preferably way, easy if you follow my instructions but make sure you have a beer to hand to occupy your time.
Well as no one else could be bothered and here in the flat we needed to upgrade to Linux Kernel 2.6.x my flatmate and I ported IMQ and ESFQ to Linux Kernel 2.6.x. The patch contains both ESFQ and IMQ and they work for us. For the techies out there the only real changes we had to make were the cosmetics that handle the new Kconfig style help and its kernel dependency malarkey plus also we had to very slightly adjust some of the netfilter structures that had only slightly changed between kernel's 2.4.x and 2.6.x. The patches we used were based on the latest 2.4.x patches and applied to kernel version linux-2.6.0-test11; however they should work with more-or-less any 2.6.x kernel.
If you have any problems then do get in touch, we are happy to accept bug reports for this. ESFQ and IMQ seem to have been outcasted by their parents so I guess my flatmate and I can adopt them for the time being until someone better comes along; which is not too difficult.
As for IPP2P and CONNMARK, we detected plenty of the regular lazyness in the Linux community (lazier than ourselves.... :) so I ported these over too.
Here are the kernel patches, please remember that you will still need to patch ‘iptables’ and ‘tc’ apprioately. All patches I have pre-prepared and dropped into the binaries package and may be found there instead; the links below really are only for those people not interested in my script (damn you.....) and just want the 2.6.x ports of them we have done.
Userspace wise you will need ‘tc’, ‘ip’ and ‘iptables’. For Debian users this is a case of ‘apt-get install iproute iptables’. Of course kernel support for RED, ESFQ, iptables mangle table and mangle MARK, plus support for HTB3, which is available in kernels later and including 2.4.18.
Deciding if you need IMQ is simple, for IP-Masquerading (aka NAT) you must have IMQ. If you want to shape traffic generated by the local box then you must have IMQ. You do not need IMQ if you linux traffic shaping box is a (transparent IP-less) Ethernet bridge; however you will need the EBTables patch to do this. If you do not need IMQ then you can leave out the iptables IMQ support and of course the kernel patches for it.
Mainly so far to Jeeves, Lentin, the BPL and Chijioke for being useful alpha and beta testers. Thanks to Andy Furniss and his e-mails with suggestions, including pointing out IMQ really needs a patch to make it work with NAT and ESFQ which is now included in the binaries package. To the rest who sent no comments or complaints “Screw you guys....I'm going home” ;) Thanks to Reeler (#lartc) for suggestions to improve/explain a bit clearer why and what is going on in the iptables section at the end; so the ‘advanced’ users do not dump my script. Finally thanks of course to the people who produced the references I used, and the OSS community, hopefully this can be part of my payment back into the OSS pool.
My name is Alexander Clouter (pronounced ‘Clue-ter’ as opposed to ‘Clow-ter’, its a ‘bou-quet’/‘buck-et’ thing you know ;) You can get in touch with me by e-mail, remember to remove the ‘junk-this’ bit in the e-mail address. Or if you want via instant messaging:
|
|
Web Hosting very kindly provided by
|
|---|---|