ipfixcol-profilestats-inter.dbk

<?xml version="1.0" encoding="utf-8"?>
<refentry
		xmlns:db="http://docbook.org/ns/docbook"
		xmlns:xlink="http://www.w3.org/1999/xlink"
		xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
		xsi:schemaLocation="http://www.w3.org/1999/xlink http://docbook.org/xml/5.0/xsd/xlink.xsd
			http://docbook.org/ns/docbook http://docbook.org/xml/5.0/xsd/docbook.xsd"
		version="5.0" xml:lang="en">
	<info>
		<copyright>
			<year>2017</year>
			<holder>CESNET, z.s.p.o.</holder>
		</copyright>
		<date>29 August 2017</date>
		<authorgroup>
			<author>
				<personname>
					<firstname>Michal</firstname>
					<surname>Kozubik</surname>
				</personname>
				<email>kozubik@cesnet.cz</email>
				<contrib>developer</contrib>
			</author>
			<author>
				<personname>
					<firstname>Lukas</firstname>
					<surname>Hutak</surname>
				</personname>
				<email>lukas.hutak@cesnet.cz</email>
				<contrib>developer</contrib>
			</author>
		</authorgroup>
		<orgname>The Liberouter Project</orgname>
	</info>

	<refmeta>
		<refentrytitle>ipfixcol-profilestats-inter</refentrytitle>
		<manvolnum>1</manvolnum>
		<refmiscinfo otherclass="manual" class="manual">ProfileStats plugin for IPFIXcol.</refmiscinfo>
	</refmeta>

	<refnamediv>
		<refname>ipfixcol-profilestats-inter</refname>
		<refpurpose>ProfileStats plugin for IPFIXcol.</refpurpose>
	</refnamediv>

	<refsect1>
		<title>Description</title>
		<simpara>
		The <command>ipfix-profilestats-inter</command> is intermediate plugin for IPFIXcol (IPFIX collector).
		</simpara>
		<simpara>
		The plugin creates and updates RRD statistics per profile and channel. Based on a configuration of profiles maintained by <citerefentry><refentrytitle>ipfixcol-profiler-inter</refentrytitle><manvolnum>1</manvolnum></citerefentry> plugin, corresponding databases are updated every interval. If RRD databases already exist, old ones will be used for updates and a previous content will be preserved. The databases store high quality i.e. short term records (defined by a size of update interval) for last 3 months and low quality i.e. long term records (one record per day) for 5 years. Each RRD consists of following Data Sources:
		</simpara>
		<para>
		<segmentedlist>
			<segtitle>Name</segtitle>
			<segtitle>Description</segtitle>

			<!-- Flows -->
			<seglistitem>
				<seg>flows</seg>
				<seg>Number of all flows per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>flows_tcp</seg>
				<seg>Number of TCP flows per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>flows_udp</seg>
				<seg>Number of UDP flows per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>flows_icmp</seg>
				<seg>Number of ICMP flows per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>flows_other</seg>
				<seg>Number of other flows (i.e. not TCP/UDP/ICMP) per second</seg>
			</seglistitem>

			<seglistitem>
				<!-- Blank line -->
			</seglistitem>

			<!-- Packets -->
			<seglistitem>
				<seg>packets</seg>
				<seg>Number of all packets per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_tcp</seg>
				<seg>Number of TCP packets per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_udp</seg>
				<seg>Number of UDP packets per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_icmp</seg>
				<seg>Number of ICMP packets per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_other</seg>
				<seg>Number of other packets (i.e. not TCP/UDP/ICMP) per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_max</seg>
				<seg>Maximum packets from one flow per interval</seg>
			</seglistitem>
			<seglistitem>
				<seg>packets_avg</seg>
			<seg>Average number of packets per flow and per interval</seg>
			</seglistitem>

			<seglistitem>
				<!-- Blank line -->
			</seglistitem>

			<!-- Traffic -->
			<seglistitem>
				<seg>traffic</seg>
				<seg>Number of all bytes per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_tcp</seg>
				<seg>Number of TCP bytes per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_udp</seg>
				<seg>Number of UDP bytes per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_icmp</seg>
				<seg>Number of ICMP bytes per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_other</seg>
				<seg>Number of other bytes (i.e. not TCP/UDP/ICMP) per second</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_max</seg>
				<seg>Maximum bytes from one flow per interval</seg>
			</seglistitem>
			<seglistitem>
				<seg>traffic_avg</seg>
				<seg>Average number of bytes per flow and per interval</seg>
			</seglistitem>
		</segmentedlist>
		</para>

		<simpara>
			Each source have consolidation functions "MAX" and "AVERAGE". Consolidation functions consilidate primary data points (in this case, a record per interval) via an agregation functions. This is necessary, for example, for storing long term statistics, because one long term RRD recond is generated from multiple data points (even several thousands). "AVERAGE" is an average of the data points and "MAX" is the larges of the data points. For short term statistics (for example displaying statistics since last hour) consolidation functions usuallly generate same results.
		</simpara>

		<simpara>
		An RRD database for a profile is created in "/rrd/" subdirectory of a main directory of the profile. For each channel of the profile is created a separated RRD database in "/rrd/channels/" subdirectory. For example, for profile "live" and its channels "ch1" and "ch2", the plugin will create databases: &lt;profile_dir&gt;/rrd/live.rrd, &lt;profile_dir&gt;/rrd/channels/ch1.rrd and &lt;profile_dir&gt;/rrd/channels/ch2.rrd.
		</simpara>
	</refsect1>

	<refsect1>
		<title>Configuration</title>
		<simpara>The collector must be configured to use ProfileStats plugin in startup.xml configuration. The profiler plugin (<citerefentry><refentrytitle>ipfixcol-profiler-inter</refentrytitle><manvolnum>1</manvolnum></citerefentry>) <emphasis>must</emphasis> be placed before ProfileStats in the IPFIXcol pipeline. Otherwise no profiles description will be available and no databases will be created/updated.
		</simpara>
		<simpara>
		The configuration specifies which plugins are used by the collector to process data and provides configuration for the plugins themselves.
		</simpara>
		<simpara><filename>startup.xml</filename> profilestats example</simpara>
		<programlisting>
	<![CDATA[
    <profilestats>
        <interval>300</interval>
        <align>true</align>
        <baseDir></baseDir>
    </profilestats>
	]]>
		</programlisting>

		<para>
			<variablelist>
				<varlistentry>
					<term><command>interval</command></term>
					<listitem>
						<simpara>Update interval (in seconds).  Size of the interval significantly influence size of databases. [min: 5, max: 3600, default: 300]</simpara>
					</listitem>
				</varlistentry>

				<varlistentry>
					<term><command>align</command></term>
					<listitem>
						<simpara>Align update interval. For example, if interval is 5 min, perform database update at 0, 5, 10, etc. wall clock minutes. [default: true]</simpara>
					</listitem>
				</varlistentry>

				<varlistentry>
					<term><command>baseDir</command></term>
					<listitem>
						<simpara>
						Expected storage for databases. Profiles with storage directories outside of the path are omitted. This allows you to make sure that files will be stored only into specified location.
						This directory must already exist in your system, otherwise all data will be lost. If the directory is not specified or the path is empty, no directory check will be performed.
						Path may contain special character sequences, each of which is introduced by a "%" character and terminated by some other character. Each of this sequences is substituted by its value. Currently supported special characters: %h = hostname.
						</simpara>
					</listitem>
				</varlistentry>
			</variablelist>
		</para>
	</refsect1>

	<refsect1>
		<title>How to generate a graph (with RRD tools)</title>
		<simpara>
		For example, let us consider a profile with two channels, "ch1" and "ch2".
		The channel "ch1" accepts all records with IPv4 addresses and channel
		"ch2" accepts all records with IPv6 address. We want to create a graph of
		a number of TCP flows per second since last 2 hours (i.e. -7200 seconds).
		</simpara>
		<simpara>
		First, we have to fetch data from both RRD files (one file per channel).
		This function fetch data from an RRD file:
		<command>
		DEF:&lt;vname&gt;=&lt;file&gt;:&lt;source&gt;:&lt;cf&gt;
		</command> ("vname" is variable name of a time series, "file" is path
		to an RRD file, "source" is a name of a Data Source in the RRD file
		and "cf" is a Consolidation function).
		To create the time series called "IP4" with the number of TCP
		flows (Data Series "flows_tcp") from the file
		"ch1.rrd" use command "DEF:IP4=ch1.rrd:flows_tcp:AVERAGE". Similarly
		for the IP6 time series.
		</simpara>
		<simpara>
		Finally, any time series can displayed, for example, as area using the command:
		<command>AREA:&lt;vname&gt;:&lt;#color&gt;:&lt;legend&gt;</command>
		("vname" is variable name of a time series, "color" is an RGB(A) color,
		"legend" is a legend of the series).
		</simpara>
		<simpara>
		Example rrdtool command:
		</simpara>
		<programlisting>
	<![CDATA[
    rrdtool graph traffic.png              \
      --full-size-mode -w 640 -h 480       \
      --start -7200 --end now              \
      --title "TCP flows"                  \
      --lower-limit 0 --alt-y-grid         \
      --vertical-label "flows per second"  \
      DEF:IP4=ch1.rrd:flows_tcp:AVERAGE    \
      DEF:IP6=ch2.rrd:flows_tcp:AVERAGE    \
      AREA:IP4#66CC00:"IPv4"               \
      AREA:IP6#FF0000:"IPv6":STACK
	]]>
		</programlisting>
		<simpara>
		Tip: Say you want to display a traffic speed in bits per seconds (instead of bytes
		per seconds	as stored in the database). You have to define a calculation on a variable.
		<command>CDEF:&lt;vname&gt;=&lt;expression&gt;</command>. For example:
		</simpara>
		<programlisting>
	<![CDATA[
      ...
      DEF:bytes=ch1.rrd:traffic:AVERAGE    \
      CDEF:bits=bytes,8,*                  \
      AREA:bits#FF0000:"bits per second"   \
      ...
	]]>
		</programlisting>
	</refsect1>

	<refsect1>
		<title>Notes</title>
		<simpara>
		It is highly recommended to use only one instance of the plugin in the configuration of IPFIXcol, because external RRD library is not very thread-safety.
		</simpara>
	</refsect1>

	<refsect1>
		<title>See Also</title>
		<para></para>
		<para>
			<variablelist>
				<varlistentry>
					<term>
						<citerefentry><refentrytitle>ipfixcol</refentrytitle><manvolnum>1</manvolnum></citerefentry>
					</term>
					<listitem>
						<simpara>Man pages</simpara>
					</listitem>
				</varlistentry>
				<varlistentry>
					<term>
						<link xlink:href="http://www.liberouter.org/technologies/ipfixcol/">http://www.liberouter.org/technologies/ipfixcol/</link>
					</term>
					<listitem>
						<para>IPFIXcol Project Homepage</para>
					</listitem>
				</varlistentry>
				<varlistentry>
					<term>
						<link xlink:href="http://www.liberouter.org">http://www.liberouter.org</link>
					</term>
					<listitem>
						<para>Liberouter web page</para>
					</listitem>
				</varlistentry>
				<varlistentry>
					<term>
						<email>tmc-support@cesnet.cz</email>
					</term>
					<listitem>
						<para>Support mailing list</para>
					</listitem>
				</varlistentry>
			</variablelist>
		</para>
	</refsect1>
</refentry>