Known Limitations Of PmGraph « Aptivate – International I.T. Development

Known Limitations of pmGraph

Local Subnets

pmGraph needs to know exactly which machines you consider to be on your local subnet, for two reasons:

The database does not record the direction of traffic (upload or download), but pmGraph needs to figure this out in order to draw upload and download traffic in opposite directions on the graph. As you have separate upload and download limits from your ISP, it doesn’t make sense to combine the traffic in both directions.

Local traffic (to and from local hosts) sometimes gets recorded in the database (e.g. connections to the monitoring machine) and should not be drawn on the graph, as it doesn’t contribute to bandwidth use. pmGraph automatically excludes local-local and remote-remote traffic, as it doesn’t know where to draw it (it’s not either an upload or a download).

The current local subnet definition only allows a single prefix, e.g. 10. or 10.0.156. Many networks have local subnets which are not so easily expressed. Also, in some cases you would want to monitor inside your proxy server and treat the proxy server as external to your network, so that traffic to and from the proxy counts as external traffic (uploads and downloads). If the proxy is in the same range as the rest of your network, this is not currently possible, but would not be hard to add.

Proxy Servers

Interaction with proxy servers is difficult. If you monitor your network inside the proxy server, you only see all clients connecting to the proxy, and inaccurate byte counts due to caching. If you monitor outside the proxy, you see all connections coming from the proxy.

This is basically how proxies work. The client asks the proxy to act on its behalf, and this is what you see on the network. pmacct is recording exactly what is happening. Ideally, it would be possible to link the proxy logs to the pmacct database, to identify the requesting client for each proxy request, and graph the traffic as having come from that host. However, this feature is not implemented yet, and the necessary features to implement it are not available in Squid, the most common free proxy server.

In terms of what already exists, a trade-off decision must be made:

If you’re happy to overestimate your bandwidth usage, and you don’t care to log remote IP addresses, then monitor inside the proxy and accept that you’ll see “bandwidth use” by proxied content, which doesn’t actually use your external connection.

If you’re happy to overestimate your bandwidth usage, but you do care to log remote IP addresses, and you don’t use proxy authentication, you could switch to using a transparent proxy and turn off the explicit proxy setting on users’ desktops, and monitor inside the proxy, with the same caveats as above.

If you want byte accuracy, then monitor outside the proxy and accept that you, and the rest of the world, will only see the proxy’s public IP address as the source of connections. Perhaps combine this with a separate tool to analyse proxy request logs, allowing you to dig further into periods of heavy web activity.

Host Groups

Several users have asked for the ability to classify hosts into groups, for example departments or customers. It is possible to do this in the database, using pmacct’s networks file feature, but this removes the ability to view individual hosts as well, except by logging to a separate table.

Database Maintenance

Disk space used by the pmacct database grows without bound. We recommend that users implement their own data retention policy, e.g. running the following SQL query automatically as a cron job, to delete records older than 1 week:

DELETE FROM acct_v6 WHERE stamp_inserted < DATE_SUB(NOW(), INTERVAL 1 WEEK);

Performance

Drawing graphs over long time periods takes a long time, because the database has to read and sum millions of flow records to generate these graphs. We are working in supporting longer time periods more efficiently with less detailed flow records.

It’s also possible to massively reduce database size and increase performance (about 100 times) at the expense of exact accuracy, by excluding the smallest flows (e.g. under 1000 bytes per minute) from being logged at all. Under normal circumstances, these records would not even be visible on the graph, or only at the highest zoom levels. However, a virus-infected host may generate enormous numbers of such flows, and use enormous amounts of bandwidth, which would therefore not be logged at all.