From: Florian Forster Date: Sat, 1 May 2010 08:49:45 +0000 (+0200) Subject: Merge branch 'ka/pinba' X-Git-Tag: collectd-4.10.0~7 X-Git-Url: https://git.octo.it/?p=collectd.git;a=commitdiff_plain;h=b34e15480992d9433a0e3fda47b5f6d387bfda5e;hp=bd2d62708d3d284baa45011457f624b0e5d7cdb4 Merge branch 'ka/pinba' --- diff --git a/ChangeLog b/ChangeLog index 74c5ccba..caf7db03 100644 --- a/ChangeLog +++ b/ChangeLog @@ -1,3 +1,94 @@ +2010-05-01, Version 4.10.0 + * collectd: JSON output now includes the "dstypes" and "dsnames" + fields. This makes it easier for external applications to interpret + the data. Thanks to Chris Buben for his work. + * collectd: The new "Timeout" option can be used to specify a + "timeout" for missing values. This is used in the threshold checking + code to detect missing values. Thanks to Andrés J. Díaz for the + patch. + * apache plugin: Support for "IdleWorkers" (Apache 1.*: "IdleServers") + has been added. + * curl plugin: The new "ExcludeRegex" allows to easily exclude certain + lines from the match. + * curl_xml plugin: This new plugin allows to read XML files using cURL + and extract metrics included in the files. Thanks to Amit Gupta for + his work. + * filecount plugin: The new "IncludeHidden" option allows to include + "hidden" files and directories in the statistics. Thanks to Vaclav + Malek for the patch. + * logfile plugin: The new "PrintSeverity" option allows to include the + severity of a message in the output. Thanks to Clément Stenac for + his patch. + * memcachec plugin: The new "ExcludeRegex" allows to easily exclude + certain lines from the match. + * modbus plugin: This new plugin allows to read registers from + Modbus-TCP enabled devices. + * network plugin: The new "Interface" option allows to set the + interface to be used for multicast and, if supported, unicast + traffic. Thanks to Max Henkel for his work. + * openvpn plugin: The "CollectUserCount" and "CollectIndividualUsers" + options allow more detailed control over how to report sessions of + multiple users. Thanks to Fabian Schuh for his work. + * ping plugin: The new "MaxMissed" allows to re-resolve a hosts + address when it doesn't reply to a number of ping requests. Thanks + to Stefan Völkel for the patch. + * postgresql plugin: The "Interval" config option has been added. The + plugin has been relicensed under the 2-clause BSD license. Thanks to + Sebastian Harl for his work. + * processes plugin: Support for "code" and "data" virtual memory sizes + has been added. Thanks to Clément Stenac for his patch. + * python plugin: Support for Python 3 has been implemented. Thanks to + Sven Trenkel for his work. + * routeros plugin: Support for collecting CPU load, memory usage, used + and free disk space, sectors written and number of bad blocks from + MikroTik devices has been added. + * swap plugin: Support for Linux < 2.6 has been added. Thanks to Lorin + Scraba for his patch. + * tail plugin: The new "ExcludeRegex" allows to easily exclude certain + lines from the match. Thanks to Peter Warasin for his patch. + * write_http plugin: The "StoreRates" option has been added. Thanks to + Paul Sadauskas for his patch. + * regex match: The "Invert" option has been added. Thanks to Julien + Ammous for his patch. + +2010-04-22, Version 4.9.2 + * Build system, various plugins: Fixes for AIX compatibility have been + added. Thanks to Manuel Sanmartin for his patches. + * Build system: Checking for "nanosleep" on old Solaris machines has + been fixed. Thanks to Vincent McIntyre and Sebastian Harl for + figuring out a way to make this work. + * collectd: Append a newline to messages written to STDERR. + * collectd: Serialization of NANs in JSON format has been fixed. + Thanks to Chris Buben for pointing out the resulting syntax error. + * collectd: Checks whether a "sleep" returned early have been added; + the cases are now handled correctly. Thanks to Michael Stapelberg + for the patch. + * collectd: Continue reading files in a directory when parsing one + file fails. + * apache plugin: Collection of the number of active connections has + been fixed for Apache 2.*. + * contextswitch plugin: Handle large counter/derive values correctly. + Thanks to Martin Merkel for reporting the bug. + * exec plugin: Error messages have been improved. The "running" flag + is now cleared correctly when forking a child fails. + * iptables plugin: Fix a violation of aliasing rules. This resolves a + warning / error with new GCC versions. Thanks to Jan Engelhardt for + the work-around. + * java plugin: The Java API files are now packaged into a .jar file. + Thanks to Amit Gupta for his patch. + * network plugin: Fix a segmentation fault when receiving packets with + an unknown data source type. + * network plugin: A memory leak when receiving encrypted network + packets has been fixed. + * openvpn plugin: Fix naming schema when reading "MULTI1" type status + files. + * oracle plugin: Fix checking for lost connections and reconnect in + this case. Thanks to Sven Trenkel for pointing out the problem. + * unixsock plugin: A memory leak in the "LISTVAL" command has been + fixed. Thanks to Peter Warasin for pointing it out. + * write_http plugin: Use the "any" authentication schema. This used to + be "digest". Thanks to Paul Sadauskas for the patch. + 2010-01-14, Version 4.9.1 * Documentation: Some manpage fixes. * Default config: Added sample configuration for missing plugins. @@ -52,6 +143,42 @@ * scale target: This target to scale (multiply) values by an arbitrary value has been added. +2010-04-22, Version 4.8.5 + * collectd: Append a newline to messages written to STDERR. + * network plugin: Fix a segmentation fault when receiving packets with + an unknown data source type. + +2010-04-07, Version 4.8.4 + * Build system, various plugins: Fixes for AIX compatibility have been + added. Thanks to Manuel Sanmartin for his patches. + * Build system: Checking for "nanosleep" on old Solaris machines has + been fixed. Thanks to Vincent McIntyre and Sebastian Harl for + figuring out a way to make this work. + * collectd: Serialization of NANs in JSON format has been fixed. + Thanks to Chris Buben for pointing out the resulting syntax error. + * collectd: Checks whether a "sleep" returned early have been added; + the cases are now handled correctly. Thanks to Michael Stapelberg + for the patch. + * collectd: Continue reading files in a directory when parsing one + file fails. + * apache plugin: Collection of the number of active connections has + been fixed for Apache 2.*. + * exec plugin: Error messages have been improved. The "running" flag + is now cleared correctly when forking a child fails. + * iptables plugin: Fix a violation of aliasing rules. This resolves a + warning / error with new GCC versions. Thanks to Jan Engelhardt for + the work-around. + * java plugin: The Java API files are now packaged into a .jar file. + Thanks to Amit Gupta for his patch. + * network plugin: A memory leak when receiving encrypted network + packets has been fixed. + * oracle plugin: Fix checking for lost connections and reconnect in + this case. Thanks to Sven Trenkel for pointing out the problem. + * unixsock plugin: A memory leak in the "LISTVAL" command has been + fixed. Thanks to Peter Warasin for pointing it out. + * write_http plugin: Use the "any" authentication schema. This used to + be "digest". Thanks to Paul Sadauskas for the patch. + 2010-01-14, Version 4.8.3 * Documentation: Some manpage fixes. * rrdtool plugin: Fix a bug with random write timeouts. Due to an diff --git a/configure.in b/configure.in index 84cccba7..804cdfc3 100644 --- a/configure.in +++ b/configure.in @@ -118,7 +118,7 @@ AC_HEADER_SYS_WAIT AC_HEADER_DIRENT AC_HEADER_STDBOOL -AC_CHECK_HEADERS(stdio.h stdint.h errno.h math.h stdarg.h syslog.h fcntl.h signal.h assert.h sys/types.h sys/socket.h sys/select.h poll.h netdb.h arpa/inet.h sys/resource.h sys/param.h kstat.h regex.h sys/ioctl.h endian.h sys/isa_defs.h) +AC_CHECK_HEADERS(stdio.h errno.h math.h stdarg.h syslog.h fcntl.h signal.h assert.h sys/types.h sys/socket.h sys/select.h poll.h netdb.h arpa/inet.h sys/resource.h sys/param.h kstat.h regex.h sys/ioctl.h endian.h sys/isa_defs.h) # For ping library AC_CHECK_HEADERS(netinet/in_systm.h, [], [], @@ -480,7 +480,7 @@ AC_HEADER_TIME # Checks for library functions. # AC_PROG_GCC_TRADITIONAL -AC_CHECK_FUNCS(gettimeofday select strdup strtol getaddrinfo getnameinfo strchr memcpy strstr strcmp strncmp strncpy strlen strncasecmp strcasecmp openlog closelog sysconf if_indextoname) +AC_CHECK_FUNCS(gettimeofday select strdup strtol getaddrinfo getnameinfo strchr memcpy strstr strcmp strncmp strncpy strlen strncasecmp strcasecmp openlog closelog sysconf setenv if_indextoname) AC_FUNC_STRERROR_R @@ -718,6 +718,9 @@ if test "x$fp_layout_type" = "xunknown"; then #if HAVE_STDINT_H # include #endif +#if HAVE_INTTYPES_H +# include +#endif #if HAVE_STDBOOL_H # include #endif @@ -762,6 +765,9 @@ if test "x$fp_layout_type" = "xunknown"; then #if HAVE_STDINT_H # include #endif +#if HAVE_INTTYPES_H +# include +#endif #if HAVE_STDBOOL_H # include #endif @@ -814,6 +820,9 @@ if test "x$fp_layout_type" = "xunknown"; then #if HAVE_STDINT_H # include #endif +#if HAVE_INTTYPES_H +# include +#endif #if HAVE_STDBOOL_H # include #endif @@ -950,8 +959,8 @@ AC_MSG_CHECKING([if have htonll defined]) AC_LANG_PROGRAM([ #include #include -#ifdef HAVE_INTTYPES_H -#include +#if HAVE_INTTYPES_H +# include #endif ], [ return htonll(0); diff --git a/contrib/exec-nagios.px b/contrib/exec-nagios.px index a9f46633..938721fc 100755 --- a/contrib/exec-nagios.px +++ b/contrib/exec-nagios.px @@ -88,6 +88,8 @@ with the C). =back +=back + =cut sub handle_config_addtype @@ -293,7 +295,7 @@ sub execute_script if ($perfdata) { - push (@serviceperfdata, split (' ', $perfdata)); + push (@serviceperfdata, split (' ', $perfdata)); } $state = 1; @@ -308,8 +310,8 @@ sub execute_script if ($perfdata) { - push (@serviceperfdata, split (' ', $perfdata)); - $state = 2; + push (@serviceperfdata, split (' ', $perfdata)); + $state = 2; } } else # ($state == 2) @@ -347,7 +349,7 @@ sub execute_script for (@serviceperfdata) { handle_performance_data ($host, 'nagios', $pinst, $script->{'type'}, - $time, $_); + $time, $_); } } } # execute_script diff --git a/contrib/python/getsigchld.py b/contrib/python/getsigchld.py new file mode 100644 index 00000000..557adc09 --- /dev/null +++ b/contrib/python/getsigchld.py @@ -0,0 +1,21 @@ +#!/usr/bin/python + +############################################################################### +# WARNING! Importing this script will break the exec plugin! # +############################################################################### +# Use this if you want to create new processes from your python scripts. # +# Normally you will get a OSError exception when the new process terminates # +# because collectd will ignore the SIGCHLD python is waiting for. # +# This script will restore the default SIGCHLD behavior so python scripts can # +# create new processes without errors. # +############################################################################### +# WARNING! Importing this script will break the exec plugin! # +############################################################################### + +import signal +import collectd + +def init(): + signal.signal(signal.SIGCHLD, signal.SIG_DFL) + +collectd.register_init(init) diff --git a/src/collectd-nagios.c b/src/collectd-nagios.c index 29e34a4c..45162bd3 100644 --- a/src/collectd-nagios.c +++ b/src/collectd-nagios.c @@ -316,7 +316,7 @@ static int do_check_con_none (size_t values_num, { printf (" |"); for (i = 0; i < values_num; i++) - printf (" %s=%g;;;;", values_names[i], values[i]); + printf (" %s=%f;;;;", values_names[i], values[i]); } printf ("\n"); @@ -370,7 +370,7 @@ static int do_check_con_average (size_t values_num, printf ("%s: %g average |", status_str, average); for (i = 0; i < values_num; i++) - printf (" %s=%g;;;;", values_names[i], values[i]); + printf (" %s=%f;;;;", values_names[i], values[i]); printf ("\n"); return (status_code); @@ -420,7 +420,7 @@ static int do_check_con_sum (size_t values_num, printf ("%s: %g sum |", status_str, total); for (i = 0; i < values_num; i++) - printf (" %s=%g;;;;", values_names[i], values[i]); + printf (" %s=%f;;;;", values_names[i], values[i]); printf ("\n"); return (status_code); diff --git a/src/collectd-python.pod b/src/collectd-python.pod index 335f6a91..36d2be33 100644 --- a/src/collectd-python.pod +++ b/src/collectd-python.pod @@ -105,6 +105,15 @@ environment variable, e.g. I before starting collectd. Depending on your version of python this might or might not result in an B exception which can be ignored. +If you really need to spawn new processes from python you can register an init +callback and reset the action for SIGCHLD to the default behavior. Please note +that this I break the exec plugin. Do not even load the exec plugin if +you intend to do this! + +There is an example script located in B to do +this. If you import this from I SIGCHLD will be handled +normally and spawning processes from python will work as intended. + =back =item EB IE block @@ -231,6 +240,28 @@ collectd you're done. The following complex types are used to pass values between the Python plugin and collectd: +=head2 Signed + +The Signed class is just a long. It has all its methods and behaves exactly +like any other long object. It is used to indicate if an integer was or should +be stored as a signed or unsigned integer object. + + class Signed(long) + +This is a long by another name. Use it in meta data dicts +to choose the way it is stored in the meta data. + +=head2 Unsigned + +The Unsigned class is just a long. It has all its methods and behaves exactly +like any other long object. It is used to indicate if an integer was or should +be stored as a signed or unsigned integer object. + + class Unsigned(long) + +This is a long by another name. Use it in meta data dicts +to choose the way it is stored in the meta data. + =head2 Config The Config class is an object which keeps the information provided in the @@ -394,6 +425,15 @@ If the sequence does not have the correct size upon dispatch a I exception will be raised. If the content of the sequence is not a number, a I exception will be raised. +=item meta +These are the meta data for this Value object. +It has to be a dictionary of numbers, strings or bools. All keys must be +strings. I and objects will be dispatched as signed integers unless +they are between 2**63 and 2**64-1, which will result in a unsigned integer. +You can force one of these storage classes by using the classes +B and B. A meta object received by a write +callback will always contain B or B objects. + =back =head2 Notification diff --git a/src/collectd.c b/src/collectd.c index abab10f9..277d3b0d 100644 --- a/src/collectd.c +++ b/src/collectd.c @@ -41,6 +41,7 @@ */ char hostname_g[DATA_MAX_NAME_LEN]; int interval_g; +int timeout_g; #if HAVE_LIBKSTAT kstat_ctl_t *kc; #endif /* HAVE_LIBKSTAT */ @@ -148,6 +149,18 @@ static int init_global_variables (void) } DEBUG ("interval_g = %i;", interval_g); + str = global_option_get ("Timeout"); + if (str == NULL) + str = "2"; + timeout_g = atoi (str); + if (timeout_g <= 1) + { + fprintf (stderr, "Cannot set the timeout to a correct value.\n" + "Please check your settings.\n"); + return (-1); + } + DEBUG ("timeout_g = %i;", timeout_g); + if (init_hostname () != 0) return (-1); DEBUG ("hostname_g = %s;", hostname_g); diff --git a/src/collectd.conf.pod b/src/collectd.conf.pod index 3c8b26c4..773fc36c 100644 --- a/src/collectd.conf.pod +++ b/src/collectd.conf.pod @@ -102,6 +102,16 @@ Configures the interval in which to query the read plugins. Obviously smaller values lead to a higher system load produced by collectd, while higher values lead to more coarse statistics. +=item B I + +Consider a value list "missing" when no update has been read or received for +I iterations. By default, I considers a value list +missing when no update has been received for twice the update interval. Since +this setting uses iterations, the maximum allowed time without update depends +on the I information contained in each value list. This is used in +the I configuration to dispatch notifications about missing values, +see L<"THRESHOLD CONFIGURATION"> below. + =item B I Number of threads to start for reading plugins. The default value is B<5>, but @@ -4446,10 +4456,12 @@ as a moving average or similar - at least not now. Also, all values that match a threshold are considered to be relevant or "interesting". As a consequence collectd will issue a notification if they are -not received for twice the last timeout of the values. If, for example, some -hosts sends it's CPU statistics to the server every 60 seconds, a notification -will be dispatched after about 120 seconds. It may take a little longer because -the timeout is checked only once each B on the server. +not received for B iterations. The B configuration option is +explained in section L<"GLOBAL OPTIONS">. If, for example, B is set to +"2" (the default) and some hosts sends it's CPU statistics to the server every +60 seconds, a notification will be dispatched after about 120 seconds. It may +take a little longer because the timeout is checked only once each B +on the server. When a value comes within range again or is received after it was missing, an "OKAY-notification" is dispatched. diff --git a/src/collectd.h b/src/collectd.h index 957654bc..8849b30b 100644 --- a/src/collectd.h +++ b/src/collectd.h @@ -300,5 +300,6 @@ typedef bool _Bool; extern char hostname_g[]; extern int interval_g; +extern int timeout_g; #endif /* COLLECTD_H */ diff --git a/src/configfile.c b/src/configfile.c index fe2bce3c..787ad0ea 100644 --- a/src/configfile.c +++ b/src/configfile.c @@ -99,6 +99,7 @@ static cf_global_option_t cf_global_options[] = {"FQDNLookup", NULL, "false"}, {"Interval", NULL, "10"}, {"ReadThreads", NULL, "5"}, + {"Timeout", NULL, "2"}, {"PreCacheChain", NULL, "PreCache"}, {"PostCacheChain", NULL, "PostCache"} }; diff --git a/src/cpython.h b/src/cpython.h index 3e80cb0c..2a14ce07 100644 --- a/src/cpython.h +++ b/src/cpython.h @@ -24,6 +24,10 @@ * Sven Trenkel **/ +/* Some python versions don't include this by default. */ + +#include + /* These two macros are basicly Py_BEGIN_ALLOW_THREADS and Py_BEGIN_ALLOW_THREADS * from the other direction. If a Python thread calls a C function * Py_BEGIN_ALLOW_THREADS is used to allow other python threads to run because @@ -155,7 +159,9 @@ static inline PyObject *cpy_string_to_unicode_or_bytes(const char *buf) { #endif } - /* Python object declarations. */ +void cpy_log_exception(const char *context); + +/* Python object declarations. */ typedef struct { PyObject_HEAD /* No semicolon! */ @@ -164,7 +170,6 @@ typedef struct { PyObject *values; /* Sequence */ PyObject *children; /* Sequence */ } Config; - PyTypeObject ConfigType; typedef struct { @@ -176,15 +181,14 @@ typedef struct { char type[DATA_MAX_NAME_LEN]; char type_instance[DATA_MAX_NAME_LEN]; } PluginData; - PyTypeObject PluginDataType; typedef struct { PluginData data; PyObject *values; /* Sequence */ + PyObject *meta; /* dict */ int interval; } Values; - PyTypeObject ValuesType; typedef struct { @@ -192,5 +196,10 @@ typedef struct { int severity; char message[NOTIF_MAX_MSG_LEN]; } Notification; - PyTypeObject NotificationType; + +typedef PyLongObject Signed; +PyTypeObject SignedType; + +typedef PyLongObject Unsigned; +PyTypeObject UnsignedType; diff --git a/src/exec.c b/src/exec.c index 681b94d6..c64f949f 100644 --- a/src/exec.c +++ b/src/exec.c @@ -269,11 +269,23 @@ static void set_environment (void) /* {{{ */ { char buffer[1024]; +#ifdef HAVE_SETENV ssnprintf (buffer, sizeof (buffer), "%i", interval_g); setenv ("COLLECTD_INTERVAL", buffer, /* overwrite = */ 1); ssnprintf (buffer, sizeof (buffer), "%s", hostname_g); setenv ("COLLECTD_HOSTNAME", buffer, /* overwrite = */ 1); +#else + ssnprintf (buffer, sizeof (buffer), "COLLECTD_INTERVAL=%i", interval_g); + putenv (buffer); + + ssnprintf (buffer, sizeof (buffer), "COLLECTD_HOSTNAME=%s", hostname_g); + putenv (buffer); +#endif + +#ifdef HAVE_SETENV +#else +#endif } /* }}} void set_environment */ __attribute__((noreturn)) diff --git a/src/libcollectdclient/client.h b/src/libcollectdclient/client.h index b0d092d0..11e7b13c 100644 --- a/src/libcollectdclient/client.h +++ b/src/libcollectdclient/client.h @@ -27,7 +27,9 @@ /* * Includes (for data types) */ -#include +#if HAVE_STDINT_H +# include +#endif #include #include diff --git a/src/modbus.c b/src/modbus.c index 5dc958f6..adab0d0f 100644 --- a/src/modbus.c +++ b/src/modbus.c @@ -810,9 +810,8 @@ static int mb_config_add_host (oconfig_item_t *ci) /* {{{ */ else interval.tv_sec = 0; - plugin_register_complex_read (name, mb_read, - (interval.tv_sec > 0) ? &interval : NULL, - &ud); + plugin_register_complex_read (/* group = */ NULL, name, + mb_read, (interval.tv_sec > 0) ? &interval : NULL, &ud); } else { diff --git a/src/network.c b/src/network.c index 74325a29..457637be 100644 --- a/src/network.c +++ b/src/network.c @@ -350,7 +350,8 @@ static _Bool check_send_okay (const value_list_t *vl) /* {{{ */ return (!received); } /* }}} _Bool check_send_okay */ -static int network_dispatch_values (value_list_t *vl) /* {{{ */ +static int network_dispatch_values (value_list_t *vl, /* {{{ */ + const char *username) { int status; @@ -391,6 +392,18 @@ static int network_dispatch_values (value_list_t *vl) /* {{{ */ return (status); } + if (username != NULL) + { + status = meta_data_add_string (vl->meta, "network:username", username); + if (status != 0) + { + ERROR ("network plugin: meta_data_add_string failed."); + meta_data_destroy (vl->meta); + vl->meta = NULL; + return (status); + } + } + plugin_dispatch_values (vl); stats_values_dispatched++; @@ -744,11 +757,11 @@ static int parse_part_values (void **ret_buffer, size_t *ret_buffer_len, break; default: - sfree (pkg_types); - sfree (pkg_values); NOTICE ("network plugin: parse_part_values: " "Don't know how to handle data source type %"PRIu8, pkg_types[i]); + sfree (pkg_types); + sfree (pkg_values); return (-1); } /* switch (pkg_types[i]) */ } @@ -892,7 +905,8 @@ static int parse_part_string (void **ret_buffer, size_t *ret_buffer_len, #define PP_SIGNED 0x01 #define PP_ENCRYPTED 0x02 static int parse_packet (sockent_t *se, - void *buffer, size_t buffer_size, int flags); + void *buffer, size_t buffer_size, int flags, + const char *username); #define BUFFER_READ(p,s) do { \ memcpy ((p), buffer + buffer_offset, (s)); \ @@ -987,6 +1001,8 @@ static int parse_part_sign_sha256 (sockent_t *se, /* {{{ */ { ERROR ("network plugin: gcry_md_setkey failed: %s", gcry_strerror (err)); gcry_md_close (hd); + sfree (secret); + sfree (pss.username); return (-1); } @@ -1008,9 +1024,6 @@ static int parse_part_sign_sha256 (sockent_t *se, /* {{{ */ gcry_md_close (hd); hd = NULL; - sfree (secret); - sfree (pss.username); - if (memcmp (pss.hash, hash, sizeof (pss.hash)) != 0) { WARNING ("network plugin: Verifying HMAC-SHA-256 signature failed: " @@ -1019,9 +1032,12 @@ static int parse_part_sign_sha256 (sockent_t *se, /* {{{ */ else { parse_packet (se, buffer + buffer_offset, buffer_len - buffer_offset, - flags | PP_SIGNED); + flags | PP_SIGNED, pss.username); } + sfree (secret); + sfree (pss.username); + *ret_buffer = buffer + buffer_len; *ret_buffer_len = 0; @@ -1065,7 +1081,8 @@ static int parse_part_sign_sha256 (sockent_t *se, /* {{{ */ warning_has_been_printed = 1; } - parse_packet (se, buffer + part_len, buffer_size - part_len, flags); + parse_packet (se, buffer + part_len, buffer_size - part_len, flags, + /* username = */ NULL); *ret_buffer = buffer + buffer_size; *ret_buffer_size = 0; @@ -1184,7 +1201,9 @@ static int parse_part_encr_aes256 (sockent_t *se, /* {{{ */ } parse_packet (se, buffer + buffer_offset, payload_len, - flags | PP_ENCRYPTED); + flags | PP_ENCRYPTED, pea.username); + + /* XXX: Free pea.username?!? */ /* Update return values */ *ret_buffer = buffer + part_size; @@ -1246,7 +1265,8 @@ static int parse_part_encr_aes256 (sockent_t *se, /* {{{ */ #undef BUFFER_READ static int parse_packet (sockent_t *se, /* {{{ */ - void *buffer, size_t buffer_size, int flags) + void *buffer, size_t buffer_size, int flags, + const char *username) { int status; @@ -1346,7 +1366,7 @@ static int parse_packet (sockent_t *se, /* {{{ */ if (status != 0) break; - network_dispatch_values (&vl); + network_dispatch_values (&vl, username); sfree (vl.values); } @@ -2180,7 +2200,8 @@ static void *dispatch_thread (void __attribute__((unused)) *arg) /* {{{ */ continue; } - parse_packet (se, ent->data, ent->data_len, /* flags = */ 0); + parse_packet (se, ent->data, ent->data_len, /* flags = */ 0, + /* username = */ NULL); sfree (ent->data); sfree (ent); } /* while (42) */ diff --git a/src/plugin.c b/src/plugin.c index a134446b..3682fa5b 100644 --- a/src/plugin.c +++ b/src/plugin.c @@ -1547,14 +1547,6 @@ void plugin_log (int level, const char *format, ...) va_list ap; llentry_t *le; - if (list_log == NULL) - { - va_start (ap, format); - vfprintf (stderr, format, ap); - va_end (ap); - return; - } - #if !COLLECT_DEBUG if (level >= LOG_DEBUG) return; @@ -1565,6 +1557,12 @@ void plugin_log (int level, const char *format, ...) msg[sizeof (msg) - 1] = '\0'; va_end (ap); + if (list_log == NULL) + { + fprintf (stderr, "%s\n", msg); + return; + } + le = llist_head (list_log); while (le != NULL) { diff --git a/src/python.c b/src/python.c index 2f4f01e1..00516fb2 100644 --- a/src/python.c +++ b/src/python.c @@ -259,7 +259,7 @@ static void cpy_build_name(char *buf, size_t size, PyObject *callback, const cha PyErr_Clear(); } -static void cpy_log_exception(const char *context) { +void cpy_log_exception(const char *context) { int l = 0, i; const char *typename = NULL, *message = NULL; PyObject *type, *value, *traceback, *tn, *m, *list; @@ -335,7 +335,7 @@ static int cpy_read_callback(user_data_t *data) { static int cpy_write_callback(const data_set_t *ds, const value_list_t *value_list, user_data_t *data) { int i; cpy_callback_t *c = data->data; - PyObject *ret, *list; + PyObject *ret, *list, *temp, *dict = NULL; Values *v; CPY_LOCK_THREADS @@ -374,6 +374,59 @@ static int cpy_write_callback(const data_set_t *ds, const value_list_t *value_li CPY_RETURN_FROM_THREADS 0; } } + dict = PyDict_New(); + if (value_list->meta) { + int i, num; + char **table; + meta_data_t *meta = value_list->meta; + + num = meta_data_toc(meta, &table); + for (i = 0; i < num; ++i) { + int type; + char *string; + int64_t si; + uint64_t ui; + double d; + _Bool b; + + type = meta_data_type(meta, table[i]); + if (type == MD_TYPE_STRING) { + if (meta_data_get_string(meta, table[i], &string)) + continue; + temp = cpy_string_to_unicode_or_bytes(string); + free(string); + PyDict_SetItemString(dict, table[i], temp); + Py_XDECREF(temp); + } else if (type == MD_TYPE_SIGNED_INT) { + if (meta_data_get_signed_int(meta, table[i], &si)) + continue; + temp = PyObject_CallFunctionObjArgs((void *) &SignedType, PyLong_FromLongLong(si), (void *) 0); + PyDict_SetItemString(dict, table[i], temp); + Py_XDECREF(temp); + } else if (type == MD_TYPE_UNSIGNED_INT) { + if (meta_data_get_unsigned_int(meta, table[i], &ui)) + continue; + temp = PyObject_CallFunctionObjArgs((void *) &UnsignedType, PyLong_FromUnsignedLongLong(ui), (void *) 0); + PyDict_SetItemString(dict, table[i], temp); + Py_XDECREF(temp); + } else if (type == MD_TYPE_DOUBLE) { + if (meta_data_get_double(meta, table[i], &d)) + continue; + temp = PyFloat_FromDouble(d); + PyDict_SetItemString(dict, table[i], temp); + Py_XDECREF(temp); + } else if (type == MD_TYPE_BOOLEAN) { + if (meta_data_get_boolean(meta, table[i], &b)) + continue; + if (b) + PyDict_SetItemString(dict, table[i], Py_True); + else + PyDict_SetItemString(dict, table[i], Py_False); + } + free(table[i]); + } + free(table); + } v = PyObject_New(Values, (void *) &ValuesType); sstrncpy(v->data.host, value_list->host, sizeof(v->data.host)); sstrncpy(v->data.type, value_list->type, sizeof(v->data.type)); @@ -383,6 +436,7 @@ static int cpy_write_callback(const data_set_t *ds, const value_list_t *value_li v->data.time = value_list->time; v->interval = value_list->interval; v->values = list; + v->meta = dict; ret = PyObject_CallFunctionObjArgs(c->callback, v, c->data, (void *) 0); /* New reference. */ if (ret == NULL) { cpy_log_exception("write callback"); @@ -936,6 +990,10 @@ static int cpy_config(oconfig_item_t *ci) { PyType_Ready(&ValuesType); NotificationType.tp_base = &PluginDataType; PyType_Ready(&NotificationType); + SignedType.tp_base = &PyLong_Type; + PyType_Ready(&SignedType); + UnsignedType.tp_base = &PyLong_Type; + PyType_Ready(&UnsignedType); sys = PyImport_ImportModule("sys"); /* New reference. */ if (sys == NULL) { cpy_log_exception("python initialization"); @@ -955,6 +1013,8 @@ static int cpy_config(oconfig_item_t *ci) { PyModule_AddObject(module, "Config", (void *) &ConfigType); /* Steals a reference. */ PyModule_AddObject(module, "Values", (void *) &ValuesType); /* Steals a reference. */ PyModule_AddObject(module, "Notification", (void *) &NotificationType); /* Steals a reference. */ + PyModule_AddObject(module, "Signed", (void *) &SignedType); /* Steals a reference. */ + PyModule_AddObject(module, "Unsigned", (void *) &UnsignedType); /* Steals a reference. */ PyModule_AddIntConstant(module, "LOG_DEBUG", LOG_DEBUG); PyModule_AddIntConstant(module, "LOG_INFO", LOG_INFO); PyModule_AddIntConstant(module, "LOG_NOTICE", LOG_NOTICE); diff --git a/src/pyvalues.c b/src/pyvalues.c index a632dc16..60462adb 100644 --- a/src/pyvalues.c +++ b/src/pyvalues.c @@ -308,6 +308,14 @@ static char values_doc[] = "These are the actual values that get dispatched to c "exception will be raised. If the content of the sequence is not a number,\n" "a TypeError exception will be raised."; +static char meta_doc[] = "These are the meta data for this Value object.\n" + "It has to be a dictionary of numbers, strings or bools. All keys must be\n" + "strings. int and long objects will be dispatched as signed integers unless\n" + "they are between 2**63 and 2**64-1, which will result in a unsigned integer.\n" + "You can force one of these storage classes by using the classes\n" + "collectd.Signed and collectd.Unsigned. A meta object received by a write\n" + "callback will always contain Signed or Unsigned objects."; + static char dispatch_doc[] = "dispatch([type][, values][, plugin_instance][, type_instance]" "[, plugin][, host][, time][, interval]) -> None. Dispatch a value list.\n" "\n" @@ -336,6 +344,7 @@ static PyObject *Values_new(PyTypeObject *type, PyObject *args, PyObject *kwds) return NULL; self->values = PyList_New(0); + self->meta = PyDict_New(); self->interval = 0; return (PyObject *) self; } @@ -344,14 +353,14 @@ static int Values_init(PyObject *s, PyObject *args, PyObject *kwds) { Values *self = (Values *) s; int interval = 0; double time = 0; - PyObject *values = NULL, *tmp; + PyObject *values = NULL, *meta = NULL, *tmp; const char *type = "", *plugin_instance = "", *type_instance = "", *plugin = "", *host = ""; static char *kwlist[] = {"type", "values", "plugin_instance", "type_instance", - "plugin", "host", "time", "interval", NULL}; + "plugin", "host", "time", "interval", "meta", NULL}; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdi", kwlist, + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdiO", kwlist, NULL, &type, &values, NULL, &plugin_instance, NULL, &type_instance, - NULL, &plugin, NULL, &host, &time, &interval)) + NULL, &plugin, NULL, &host, &time, &interval, &meta)) return -1; if (type[0] != 0 && plugin_get_ds(type) == NULL) { @@ -373,21 +382,109 @@ static int Values_init(PyObject *s, PyObject *args, PyObject *kwds) { Py_INCREF(values); } + if (meta == NULL) { + meta = PyDict_New(); + PyErr_Clear(); + } else { + Py_INCREF(meta); + } + tmp = self->values; self->values = values; Py_XDECREF(tmp); + tmp = self->meta; + self->meta = meta; + Py_XDECREF(tmp); + self->interval = interval; return 0; } +static meta_data_t *cpy_build_meta(PyObject *meta) { + int i, s; + meta_data_t *m = NULL; + PyObject *l; + + if (!meta) + return NULL; + + m = meta_data_create(); + l = PyDict_Items(meta); + s = PyList_Size(l); + for (i = 0; i < s; ++i) { + const char *string, *keystring; + PyObject *key, *value, *item, *tmp; + + item = PyList_GET_ITEM(l, i); + key = PyTuple_GET_ITEM(item, 0); + Py_INCREF(key); + keystring = cpy_unicode_or_bytes_to_string(&key); + if (!keystring) { + PyErr_Clear(); + Py_XDECREF(key); + continue; + } + value = PyTuple_GET_ITEM(item, 1); + Py_INCREF(value); + if (value == Py_True) { + meta_data_add_boolean(m, keystring, 1); + } else if (value == Py_False) { + meta_data_add_boolean(m, keystring, 0); + } else if (PyFloat_Check(value)) { + meta_data_add_double(m, keystring, PyFloat_AsDouble(value)); + } else if (PyObject_TypeCheck(value, &SignedType)) { + long long int lli; + lli = PyLong_AsLongLong(value); + if (!PyErr_Occurred() && (lli == (int64_t) lli)) + meta_data_add_signed_int(m, keystring, lli); + } else if (PyObject_TypeCheck(value, &UnsignedType)) { + long long unsigned llu; + llu = PyLong_AsUnsignedLongLong(value); + if (!PyErr_Occurred() && (llu == (uint64_t) llu)) + meta_data_add_unsigned_int(m, keystring, llu); + } else if (PyNumber_Check(value)) { + long long int lli; + long long unsigned llu; + tmp = PyNumber_Long(value); + lli = PyLong_AsLongLong(tmp); + if (!PyErr_Occurred() && (lli == (int64_t) lli)) { + meta_data_add_signed_int(m, keystring, lli); + } else { + PyErr_Clear(); + llu = PyLong_AsUnsignedLongLong(tmp); + if (!PyErr_Occurred() && (llu == (uint64_t) llu)) + meta_data_add_unsigned_int(m, keystring, llu); + } + Py_XDECREF(tmp); + } else { + string = cpy_unicode_or_bytes_to_string(&value); + if (string) { + meta_data_add_string(m, keystring, string); + } else { + PyErr_Clear(); + tmp = PyObject_Str(value); + string = cpy_unicode_or_bytes_to_string(&tmp); + if (string) + meta_data_add_string(m, keystring, string); + Py_XDECREF(tmp); + } + } + if (PyErr_Occurred()) + cpy_log_exception("building meta data"); + Py_XDECREF(value); + Py_DECREF(key); + } + return m; +} + static PyObject *Values_dispatch(Values *self, PyObject *args, PyObject *kwds) { int i, ret; const data_set_t *ds; int size; value_t *value; value_list_t value_list = VALUE_LIST_INIT; - PyObject *values = self->values; + PyObject *values = self->values, *meta = self->meta; double time = self->data.time; int interval = self->interval; const char *host = self->data.host; @@ -397,10 +494,10 @@ static PyObject *Values_dispatch(Values *self, PyObject *args, PyObject *kwds) { const char *type_instance = self->data.type_instance; static char *kwlist[] = {"type", "values", "plugin_instance", "type_instance", - "plugin", "host", "time", "interval", NULL}; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdi", kwlist, + "plugin", "host", "time", "interval", "meta", NULL}; + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdiO", kwlist, NULL, &type, &values, NULL, &plugin_instance, NULL, &type_instance, - NULL, &plugin, NULL, &host, &time, &interval)) + NULL, &plugin, NULL, &host, &time, &interval, &meta)) return NULL; if (type[0] == 0) { @@ -416,6 +513,10 @@ static PyObject *Values_dispatch(Values *self, PyObject *args, PyObject *kwds) { PyErr_Format(PyExc_TypeError, "values must be list or tuple"); return NULL; } + if (meta != NULL && meta != Py_None && !PyDict_Check(meta)) { + PyErr_Format(PyExc_TypeError, "meta must be a dict"); + return NULL; + } size = (int) PySequence_Length(values); if (size != ds->ds_num) { PyErr_Format(PyExc_RuntimeError, "type %s needs %d values, got %i", type, ds->ds_num, size); @@ -456,6 +557,7 @@ static PyObject *Values_dispatch(Values *self, PyObject *args, PyObject *kwds) { } } value_list.values = value; + value_list.meta = cpy_build_meta(meta); value_list.values_len = size; value_list.time = time; value_list.interval = interval; @@ -464,7 +566,6 @@ static PyObject *Values_dispatch(Values *self, PyObject *args, PyObject *kwds) { sstrncpy(value_list.plugin_instance, plugin_instance, sizeof(value_list.plugin_instance)); sstrncpy(value_list.type, type, sizeof(value_list.type)); sstrncpy(value_list.type_instance, type_instance, sizeof(value_list.type_instance)); - value_list.meta = NULL; if (value_list.host[0] == 0) sstrncpy(value_list.host, hostname_g, sizeof(value_list.host)); if (value_list.plugin[0] == 0) @@ -486,7 +587,7 @@ static PyObject *Values_write(Values *self, PyObject *args, PyObject *kwds) { int size; value_t *value; value_list_t value_list = VALUE_LIST_INIT; - PyObject *values = self->values; + PyObject *values = self->values, *meta = self->meta; double time = self->data.time; int interval = self->interval; const char *host = self->data.host; @@ -497,10 +598,10 @@ static PyObject *Values_write(Values *self, PyObject *args, PyObject *kwds) { const char *dest = NULL; static char *kwlist[] = {"destination", "type", "values", "plugin_instance", "type_instance", - "plugin", "host", "time", "interval", NULL}; - if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdi", kwlist, + "plugin", "host", "time", "interval", "meta", NULL}; + if (!PyArg_ParseTupleAndKeywords(args, kwds, "|etOetetetetdiO", kwlist, NULL, &type, &values, NULL, &plugin_instance, NULL, &type_instance, - NULL, &plugin, NULL, &host, &time, &interval)) + NULL, &plugin, NULL, &host, &time, &interval, &meta)) return NULL; if (type[0] == 0) { @@ -564,7 +665,7 @@ static PyObject *Values_write(Values *self, PyObject *args, PyObject *kwds) { sstrncpy(value_list.plugin_instance, plugin_instance, sizeof(value_list.plugin_instance)); sstrncpy(value_list.type, type, sizeof(value_list.type)); sstrncpy(value_list.type_instance, type_instance, sizeof(value_list.type_instance)); - value_list.meta = NULL; + value_list.meta = cpy_build_meta(meta);; if (value_list.host[0] == 0) sstrncpy(value_list.host, hostname_g, sizeof(value_list.host)); if (value_list.plugin[0] == 0) @@ -582,17 +683,19 @@ static PyObject *Values_write(Values *self, PyObject *args, PyObject *kwds) { static PyObject *Values_repr(PyObject *s) { PyObject *ret, *tmp; - static PyObject *l_interval = NULL, *l_values = NULL, *l_closing = NULL; + static PyObject *l_interval = NULL, *l_values = NULL, *l_meta = NULL, *l_closing = NULL; Values *self = (Values *) s; if (l_interval == NULL) l_interval = cpy_string_to_unicode_or_bytes(",interval="); if (l_values == NULL) l_values = cpy_string_to_unicode_or_bytes(",values="); + if (l_meta == NULL) + l_meta = cpy_string_to_unicode_or_bytes(",meta="); if (l_closing == NULL) l_closing = cpy_string_to_unicode_or_bytes(")"); - if (l_interval == NULL || l_values == NULL || l_closing == NULL) + if (l_interval == NULL || l_values == NULL || l_meta == NULL || l_closing == NULL) return NULL; ret = cpy_common_repr(s); @@ -602,11 +705,16 @@ static PyObject *Values_repr(PyObject *s) { CPY_SUBSTITUTE(PyObject_Repr, tmp, tmp); CPY_STRCAT_AND_DEL(&ret, tmp); } - if (self->values != NULL && PySequence_Length(self->values) > 0) { + if (self->values && (!PyList_Check(self->values) || PySequence_Length(self->values) > 0)) { CPY_STRCAT(&ret, l_values); tmp = PyObject_Repr(self->values); CPY_STRCAT_AND_DEL(&ret, tmp); } + if (self->meta && (!PyDict_Check(self->meta) || PyDict_Size(self->meta) > 0)) { + CPY_STRCAT(&ret, l_meta); + tmp = PyObject_Repr(self->meta); + CPY_STRCAT_AND_DEL(&ret, tmp); + } CPY_STRCAT(&ret, l_closing); return ret; } @@ -614,12 +722,14 @@ static PyObject *Values_repr(PyObject *s) { static int Values_traverse(PyObject *self, visitproc visit, void *arg) { Values *v = (Values *) self; Py_VISIT(v->values); + Py_VISIT(v->meta); return 0; } static int Values_clear(PyObject *self) { Values *v = (Values *) self; Py_CLEAR(v->values); + Py_CLEAR(v->meta); return 0; } @@ -631,6 +741,7 @@ static void Values_dealloc(PyObject *self) { static PyMemberDef Values_members[] = { {"interval", T_INT, offsetof(Values, interval), 0, interval_doc}, {"values", T_OBJECT_EX, offsetof(Values, values), 0, values_doc}, + {"meta", T_OBJECT_EX, offsetof(Values, meta), 0, meta_doc}, {NULL} }; @@ -897,3 +1008,57 @@ PyTypeObject NotificationType = { 0, /* tp_alloc */ Notification_new /* tp_new */ }; + +static const char Signed_doc[] = "This is a long by another name. Use it in meta data dicts\n" + "to choose the way it is stored in the meta data."; + +PyTypeObject SignedType = { + CPY_INIT_TYPE + "collectd.Signed", /* tp_name */ + sizeof(Signed), /* tp_basicsize */ + 0, /* Will be filled in later */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ + Signed_doc /* tp_doc */ +}; + +static const char Unsigned_doc[] = "This is a long by another name. Use it in meta data dicts\n" + "to choose the way it is stored in the meta data."; + +PyTypeObject UnsignedType = { + CPY_INIT_TYPE + "collectd.Unsigned", /* tp_name */ + sizeof(Unsigned), /* tp_basicsize */ + 0, /* Will be filled in later */ + 0, /* tp_dealloc */ + 0, /* tp_print */ + 0, /* tp_getattr */ + 0, /* tp_setattr */ + 0, /* tp_compare */ + 0, /* tp_repr */ + 0, /* tp_as_number */ + 0, /* tp_as_sequence */ + 0, /* tp_as_mapping */ + 0, /* tp_hash */ + 0, /* tp_call */ + 0, /* tp_str */ + 0, /* tp_getattro */ + 0, /* tp_setattro */ + 0, /* tp_as_buffer */ + Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE, /*tp_flags*/ + Unsigned_doc /* tp_doc */ +}; diff --git a/src/routeros.c b/src/routeros.c index a020692e..ff8789ed 100644 --- a/src/routeros.c +++ b/src/routeros.c @@ -125,7 +125,7 @@ static void cr_submit_counter (cr_data_t *rd, const char *type, /* {{{ */ vl.values = values; vl.values_len = STATIC_ARRAY_SIZE (values); - sstrncpy (vl.host, rd->node, sizeof (vl.host)); /* FIXME */ + sstrncpy (vl.host, rd->node, sizeof (vl.host)); sstrncpy (vl.plugin, "routeros", sizeof (vl.plugin)); sstrncpy (vl.type, type, sizeof (vl.type)); sstrncpy (vl.type_instance, type_instance, sizeof (vl.type_instance)); @@ -182,7 +182,7 @@ static int handle_regtable (__attribute__((unused)) ros_connection_t *c, /* {{{ return (0); } /* }}} int handle_regtable */ -#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) /* FIXME */ +#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) static int handle_system_resource (__attribute__((unused)) ros_connection_t *c, /* {{{ */ const ros_system_resource_t *r, __attribute__((unused)) void *user_data) @@ -276,7 +276,7 @@ static int cr_read (user_data_t *user_data) /* {{{ */ } } -#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) /* FIXME */ +#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) if (rd->collect_cpu_load || rd->collect_memory || rd->collect_df @@ -350,7 +350,7 @@ static int cr_config_router (oconfig_item_t *ci) /* {{{ */ cf_util_get_boolean (child, &router_data->collect_interface); else if (strcasecmp ("CollectRegistrationTable", child->key) == 0) cf_util_get_boolean (child, &router_data->collect_regtable); -#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) /* FIXME */ +#if ROS_VERSION >= ROS_VERSION_ENCODE(1, 1, 0) else if (strcasecmp ("CollectCPULoad", child->key) == 0) cf_util_get_boolean (child, &router_data->collect_cpu_load); else if (strcasecmp ("CollectMemory", child->key) == 0) diff --git a/src/utils_cache.c b/src/utils_cache.c index 648c54de..69ea864b 100644 --- a/src/utils_cache.c +++ b/src/utils_cache.c @@ -319,7 +319,7 @@ int uc_check_timeout (void) while (c_avl_iterator_next (iter, (void *) &key, (void *) &ce) == 0) { /* If entry has not been updated, add to `keys' array */ - if ((now - ce->last_update) >= (2 * ce->interval)) + if ((now - ce->last_update) >= (timeout_g * ce->interval)) { char **tmp; diff --git a/version-gen.sh b/version-gen.sh index bd87bbb5..2a93ab7b 100755 --- a/version-gen.sh +++ b/version-gen.sh @@ -1,6 +1,6 @@ #!/bin/sh -DEFAULT_VERSION="4.8.2.git" +DEFAULT_VERSION="4.9.2.git" VERSION="`git describe 2> /dev/null | sed -e 's/^collectd-//'`"