Page 1 of 1

www.newlisp.org site server problems

Posted: Sun Jul 27, 2014 7:31 pm
by Lutz
Since two to three days used bandwidth on www.newlisp.org (newlisp.nfshost.com) has grown 10 times, this allthough page hits, downloads and other statistics in the logs seem to be the same. I am already using the following .htaccess on Apache for a few months:

Code: Select all

RewriteCond %{HTTP_USER_AGENT} MJ12bot
order allow,deny
deny from 220.181.157.164
deny from 220.181.157.165
deny from 220.181.157.166
deny from 220.181.157.196
deny from 220.181.157.197
deny from 220.181.55.55
deny from 218.30.117.160 
allow from all
.. as I have multiple downloads requests daily for all of the IPs above everyday for all downloads on the site. This has brought downloads and pages statistics down, but in the last 3 days bandwidth suddenly is 10 fold and I don’t know why. Download and page hit numbers stay low, but bandwidth is going through the roof as if .htaccess is not working correctly anymore? Does anybody have suggestions? Is there a more efficient way to suppress access from certain IPs ?

Re: www.newlisp.org site server problems

Posted: Sun Jul 27, 2014 7:47 pm
by TedWalther
Lutz, are you operating a virtual server? Perhaps the bandwidth usage isn't coming in over the web, but is a ping flood, or traffic going to some other internet port on your server? I'm willing to log in and look around, contact me in private with details.

How are you measuring the bandwidth? Google analytics? Webalyzer?

Re: www.newlisp.org site server problems

Posted: Sun Jul 27, 2014 10:27 pm
by ralph.ronnquist
how about iptables rules at the OS level? Well, assuming linux or mac, and that you "own" the host.

Re: www.newlisp.org site server problems

Posted: Sun Jul 27, 2014 10:48 pm
by Lutz
This is not my own host but a site at the hosting company https://www.nearlyfreespeech.net . I like them because they charge purely by bandwidth usage. Before opening a paid support ticket with them, I am asking for help here at the forum, as there are so many system admin people among newLISP users.

It's a FreeBSD 9.2-RELEASE-p3 site running virtual servers and Apache - don't know which version. Tools installed are limited for users of virtual servers. I don't think, I have permissions to filter at the OS level. The .htaccess files where working fine, but suddenly bandwidth usage is 10-12 times what it was before with similar content of log files, generated with software by http://www.awstats.org, configuration of which cannot be done by me.

Re: www.newlisp.org site server problems

Posted: Sun Jul 27, 2014 11:48 pm
by TedWalther
Thank you. Does awstats have an option so you can see which IP and which file is using the most bandwidth? Google analytics does.

If it is $30 or less, I'd open a support ticket.

I remember I colocated a server years ago with a large colo company. I was watching my logs, at the web server level, and also at the packet level. Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't. I protested, and they held my server hostage. Ever since, I try to host my own servers.

Re: www.newlisp.org site server problems

Posted: Mon Jul 28, 2014 12:45 am
by rickyboy
Lutz wrote:Before opening a paid support ticket with them, I am asking for help here at the forum, ...
Please check all the "Support Options" (https://members.nearlyfreespeech.net/support/request), before you pay the $30 -- unless you are in a hurry. One of the options is to post on the NFSN Member Forum. Good luck!

Re: www.newlisp.org site server problems

Posted: Mon Jul 28, 2014 12:47 am
by rickyboy
TedWalther wrote:I remember I colocated a server years ago with a large colo company. I was watching my logs, at the web server level, and also at the packet level. Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't. I protested, and they held my server hostage. Ever since, I try to host my own servers.
I agree with that. I like NFSN, but a $5/mo Digital Ocean box looks really good. :)

Re: www.newlisp.org site server problems

Posted: Mon Jul 28, 2014 7:55 am
by ralph.ronnquist
I haven't used them myself, but there are bandwidth limiting and throttling modules for apache; maybe something like the "mod_cband" would serve the purpose. I suppose you need something that would limit the amount of data a single user can get in a short time period, ideally without affecting a "normal" user.

Re: www.newlisp.org site server problems

Posted: Mon Jul 28, 2014 3:08 pm
by Lutz
Thanks to everybody for all the good advice. I posted more details about the case on the nearlyfreespeech.net discussion forum.

All the bandwidth increase is reported for CGI pages, which is strange because the number of hits on these pages is the same as before and scripts haven't changed. Hopefully I will get some answers soon. Today the bandwidth increase is 20-fold, yesterday about 10-fold !?!?

Re: www.newlisp.org site server problems

Posted: Mon Jul 28, 2014 6:59 pm
by TedWalther
Has the awstats software been updated? How does it tell the size of a CGI transaction? Is your CGI script accurately reporting the size of the data it is returning, in the MIME header?

Re: www.newlisp.org site server problems

Posted: Tue Jul 29, 2014 2:09 am
by Lutz
Looks like the problem is solved :)

One of the nfs guys on the nfshost.com user forum gave me a command to extract file sizes from the access log. That helped to identify a big text file in the newLISP wiki's pages directory and from more and more sources over time. It had nothing to do with the IPs locked out by .htaccess, which is still working well.

Thanks again to everybody for helping to shed light on this.