www.newlisp.org site server problems

Notices and updates

www.newlisp.org site server problems

Postby Lutz » Sun Jul 27, 2014 7:31 pm

Since two to three days used bandwidth on www.newlisp.org (newlisp.nfshost.com) has grown 10 times, this allthough page hits, downloads and other statistics in the logs seem to be the same. I am already using the following .htaccess on Apache for a few months:

Code: Select all
RewriteCond %{HTTP_USER_AGENT} MJ12bot
order allow,deny
deny from 220.181.157.164
deny from 220.181.157.165
deny from 220.181.157.166
deny from 220.181.157.196
deny from 220.181.157.197
deny from 220.181.55.55
deny from 218.30.117.160
allow from all


.. as I have multiple downloads requests daily for all of the IPs above everyday for all downloads on the site. This has brought downloads and pages statistics down, but in the last 3 days bandwidth suddenly is 10 fold and I don’t know why. Download and page hit numbers stay low, but bandwidth is going through the roof as if .htaccess is not working correctly anymore? Does anybody have suggestions? Is there a more efficient way to suppress access from certain IPs ?
Lutz
 
Posts: 5258
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: www.newlisp.org site server problems

Postby TedWalther » Sun Jul 27, 2014 7:47 pm

Lutz, are you operating a virtual server? Perhaps the bandwidth usage isn't coming in over the web, but is a ping flood, or traffic going to some other internet port on your server? I'm willing to log in and look around, contact me in private with details.

How are you measuring the bandwidth? Google analytics? Webalyzer?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 602
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: www.newlisp.org site server problems

Postby ralph.ronnquist » Sun Jul 27, 2014 10:27 pm

how about iptables rules at the OS level? Well, assuming linux or mac, and that you "own" the host.
ralph.ronnquist
 
Posts: 178
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: www.newlisp.org site server problems

Postby Lutz » Sun Jul 27, 2014 10:48 pm

This is not my own host but a site at the hosting company https://www.nearlyfreespeech.net . I like them because they charge purely by bandwidth usage. Before opening a paid support ticket with them, I am asking for help here at the forum, as there are so many system admin people among newLISP users.

It's a FreeBSD 9.2-RELEASE-p3 site running virtual servers and Apache - don't know which version. Tools installed are limited for users of virtual servers. I don't think, I have permissions to filter at the OS level. The .htaccess files where working fine, but suddenly bandwidth usage is 10-12 times what it was before with similar content of log files, generated with software by http://www.awstats.org, configuration of which cannot be done by me.
Lutz
 
Posts: 5258
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: www.newlisp.org site server problems

Postby TedWalther » Sun Jul 27, 2014 11:48 pm

Thank you. Does awstats have an option so you can see which IP and which file is using the most bandwidth? Google analytics does.

If it is $30 or less, I'd open a support ticket.

I remember I colocated a server years ago with a large colo company. I was watching my logs, at the web server level, and also at the packet level. Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't. I protested, and they held my server hostage. Ever since, I try to host my own servers.
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 602
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: www.newlisp.org site server problems

Postby rickyboy » Mon Jul 28, 2014 12:45 am

Lutz wrote:Before opening a paid support ticket with them, I am asking for help here at the forum, ...

Please check all the "Support Options" (https://members.nearlyfreespeech.net/support/request), before you pay the $30 -- unless you are in a hurry. One of the options is to post on the NFSN Member Forum. Good luck!
(λx. x x) (λx. x x)
rickyboy
 
Posts: 554
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: www.newlisp.org site server problems

Postby rickyboy » Mon Jul 28, 2014 12:47 am

TedWalther wrote:I remember I colocated a server years ago with a large colo company. I was watching my logs, at the web server level, and also at the packet level. Suddenly the colo said bandwidth usage jumped, but my own logs said it didn't. I protested, and they held my server hostage. Ever since, I try to host my own servers.

I agree with that. I like NFSN, but a $5/mo Digital Ocean box looks really good. :)
(λx. x x) (λx. x x)
rickyboy
 
Posts: 554
Joined: Fri Apr 08, 2005 7:13 pm
Location: Front Royal, Virginia

Re: www.newlisp.org site server problems

Postby ralph.ronnquist » Mon Jul 28, 2014 7:55 am

I haven't used them myself, but there are bandwidth limiting and throttling modules for apache; maybe something like the "mod_cband" would serve the purpose. I suppose you need something that would limit the amount of data a single user can get in a short time period, ideally without affecting a "normal" user.
ralph.ronnquist
 
Posts: 178
Joined: Mon Jun 02, 2014 1:40 am
Location: Melbourne, Australia

Re: www.newlisp.org site server problems

Postby Lutz » Mon Jul 28, 2014 3:08 pm

Thanks to everybody for all the good advice. I posted more details about the case on the nearlyfreespeech.net discussion forum.

All the bandwidth increase is reported for CGI pages, which is strange because the number of hits on these pages is the same as before and scripts haven't changed. Hopefully I will get some answers soon. Today the bandwidth increase is 20-fold, yesterday about 10-fold !?!?
Lutz
 
Posts: 5258
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California

Re: www.newlisp.org site server problems

Postby TedWalther » Mon Jul 28, 2014 6:59 pm

Has the awstats software been updated? How does it tell the size of a CGI transaction? Is your CGI script accurately reporting the size of the data it is returning, in the MIME header?
Cavemen in bearskins invaded the ivory towers of Artificial Intelligence. Nine months later, they left with a baby named newLISP. The women of the ivory towers wept and wailed. "Abomination!" they cried.
TedWalther
 
Posts: 602
Joined: Mon Feb 05, 2007 1:04 am
Location: Abbotsford, BC

Re: www.newlisp.org site server problems

Postby Lutz » Tue Jul 29, 2014 2:09 am

Looks like the problem is solved :)

One of the nfs guys on the nfshost.com user forum gave me a command to extract file sizes from the access log. That helped to identify a big text file in the newLISP wiki's pages directory and from more and more sources over time. It had nothing to do with the IPs locked out by .htaccess, which is still working well.

Thanks again to everybody for helping to shed light on this.
Lutz
 
Posts: 5258
Joined: Thu Sep 26, 2002 4:45 pm
Location: Pasadena, California


Return to newLISP newS

Who is online

Users browsing this forum: No registered users and 1 guest

cron