Linux: monitor my own internet usage page by page

Question

I have a monthly broadband subscription which rolls over on the 23rd of each month. I had arrived near the end of my usage so I was carefully checking that I didn't go over my limit - they crucify you for excess usage!

So, when my usage didn't roll over on the 23rd, I rang them to check what was up (turned out that it was due to their end-user web interface not being entirely not up to date - they're quick enough to bill though!).

Now to the question - the agent told me that I had used 2.5MB (or similar figures) PER PAGE just to check my usage (the only pages I had accessed on the two days in question). To say I was shocked is an understatement. These are not fancy pages - v. little graphics &c.

So, now I want to check up my internet usage (per page). Can anyone point me to useful Linux tools in this domain? I have Googled but have found that everything is either Windows and/or doesn't give results by page. I don't want something that works in real time, but rather keeps a record, preferably something I can put in a database or at least a spreadsheet.

This is not a shopping list question - I just want tool names - not "but blah1 is better than blah2..." type debates (I can research this myself).

NOTE: in response to Warren Young's answer.

1) Don't want all data usage (well, I do, but on a URL by URL (page by page) basis)

2) As mentioned in my original question, my ISP's net usage tracker is about as useful as tits on a bull - days behind, and anyway, has no granularity whatsoever.

3) As I said to @peterph, I am thinking about something browser based (I use Chrome) for ad-hoc analyses, but I would like a daemon to do this for me automatically.

4) As I said to @SailorCire, I'm more of a DBA guy and have enough on my plate without going down a development route - I'm not being lazy, I also think I don't really have the skills.

Finally, I was thinking of doing something with Wireshark - do you think this might be a good route? Or even Snort?

Answer 1

There are many ways to skin that cat:

• If you want a graph of all data usage over a given period plus other summary statistics, the standard tools for that on Linux are MRTG and Cacti.

• Your ISP may offer a web API that lets Net Usage Item periodically check your usage and report it with a "fuel gauge" in the browser. It works in Firefox and Chrome. Pulling the data this way is probably less expensive in terms of data transfer than going to an interactive web page.

• If you want to see if your ISP's support people are spouting nonsense when they tell you that those simple pages weigh 2.5 MiB each, then you already have what you need to answer the question: your browser's development tools.

  In the past 2-3 years, all the major browser providers have added Firebug-like tool sets to their browsers. Firefox includes its own developer tools now, too, so that Firebug itself is increasingly unnecessary.

  The keystroke for this is usually F12 or Ctrl-Shift-I.

  What you're looking for is the Network tab. (Firefox, Chrome) When you open a page with this panel open, it gives you all kinds of details about what happens on the network when that page loads, including its page weight.

• If you're asking for something that will let you dig back into history and perform long-term analyses that let you answer questions like "What is the median page size I downloaded during the month?" then I don't know of anything currently in existence. If it doesn't in fact exist, you could build it by accessing the data collected by MRTG or Cacti.