User specific bandwidth monitor

Portal Home > Knowledgebase > Articles Database > User specific bandwidth monitor

Posted by fqaisrani, 02-17-2010, 08:24 PM
HI folks, I am in process of build a SaaS system, and we need to monitor bandwidth of each user account. The users will be able to use web base normal services which will be browsing, downloading, posting stuff, etcFile uploading, and it does include partial/failed uploads. for example if a user is upload a large file and in middle of the process he closes the browser, the partial upload will be counted.FTP upload, download...(FTP features are not included in initial stages, but will be added on later stages)user storage usage. Right now the solution I have in mind are.. To calc each page/web-request served for first point.Perl based uploading so we can monitor the raw upload same as used to display upload progress. -- no idea--after a file is uploaded, its size will be saved in DB, and the counter will increase/decrease of uploading/deleting of files. But if its possible to somehow customize and use any apache mod, I think that will be alot better and efficient solution. Any other ideas or any improvements to the above solutions are welcome. thanks in advance
Posted by SwordfishHostingLtd, 02-21-2010, 12:12 PM
Hi, Well taking webalizer / awstats as an example of how to do it they both work off analysing the logs for a site. So they look through mail logs / www logs etc.. to work all this out - that way most of the hard work is done for you. However if it has to be a more general system which can work on any platform then i suppose that rules that out. So in terms of a solution you create then if you we're to preform an insert statement for every page view / download it's going to have a fairly noticeable performance overhead... If it's a linux environment you are after would it not be possible to modify the way logs are dealt with so instead of storing it in the usual apache method you store it in anotherway more accessible to your needs?
Posted by mattle, 02-21-2010, 01:11 PM
QUOTE=fqaisrani;6655419]HI folks, I am in process of build a SaaS system, and we need to monitor bandwidth of each user account. The users will be able to use web base normal services which will be browsing, downloading, posting stuff, etcFile uploading, and it does include partial/failed uploads. for example if a user is upload a large file and in middle of the process he closes the browser, the partial upload will be counted.FTP upload, download...(FTP features are not included in initial stages, but will be added on later stages)user storage usage. [/QOUTE] Here are my suggestions: I'd use regular the regular apache access_log for this. If you can track users based on part of the path, for example, you can use this to determine bandwidth by user. If each user has a virtual host, you could set up custom logging so they each have their own log file.I would log that with the script that is performing the upload. Alternatively, it looks like there might be some methods for having apache also log post data size.FTP will probably come down to the server you're using...which you didn't specifyI'd probably just go with a daily du of their file directory.
Posted by tim2718281, 02-21-2010, 04:18 PM
To track usage by user, you need some way to identify users. For example, when a person accesses a web page, the Apache logs lists how much data was transferred, but not who transferred it. The Apache log will list the requesting IP address though. So I guess you could have a log in process to identify the users. Code the log in process so it itself requests a page, including the userid as part of the URL. Similarly for logout. Now you can process the Apache logs, and the login/logout requests appearing in the logs will allow you to associate an IP address with a user. Once a user logs in, all datra transfer from the IP address used will be attributed to them until they log out.