Kea: Kea Legal Log Hooks Library

| Kea 3.2.0-git

Kea Legal Log Hooks Library

Welcome to Kea Legal Log Hooks Library.

This documentation is addressed at developers who are interested in internal operation of the libdhcp_legal_log library. This file provides information needed to understand and perhaps extend this library.

This documentation is stand-alone: you should have read and understood Kea Developer's Guide and in particular its section about hooks: Hooks Developer's Guide.

# How To Use Libdhcp_legal_log

## Introduction

libdhcp_legal_log is a hooks library which customizes the DHCP lease management provided by Kea DHCP server modules (kea-dhcp4 and kea-dhcp6). Specifically it implements recording a detailed log of lease assignments and renewals as might be required for legal purposes. It is also described in the Kea Administrator Reference Manual instead in the section titled Forensic Logging Hooks.

This library is designed to help with the requirement that in many legal jurisdictions companies, especially ISPs, must record information about the addresses they have leased to DHCP clients. If the information that it records is sufficient it may be used directly. If your jurisdiction requires that you save a different set of information you may use it as a template or example and create your own custom logging hooks.

## Log Files

Legal logs are appended into a text file implemented by the class `isc::legal_log::RotatingFile`. By default, it is rotated daily, but this can be configured via the time-unit and count parameters.

Additional actions can be performed just before closing the old file and after opening the new file. These actions must point to an external executable or script and are configured by setting prerotate and postrotate parameters. Kea will instantiate the prerotate action with the name of the file that will be closed, but does not wait for the process to finish. Kea will instantiate the postrotate action with the name of the file that had been opened, but does not wait for the process to finish.

Legal file names, if using day, month or year as time unit have the form:

    <path>/<base-name>.<CCYYMMDD>.txt

where CC represents century, YY represents current year, MM represents current month and DD represents current day.

Legal file names, if using second as time unit have the form:

    <path>/<base-name>.<TXXXXXXXXXXXXXXXXXXXX>.txt

where XXXXXXXXXXXXXXXXXXXX represents time in seconds since epoch.

To note that when using second as time unit, the file will be rotated when the count number of seconds pass. In contrast, when using day, month or year as time unit, the file will be rotated whenever the countth day, month or year starts respectively.

## DHCPv4 Log Detailed Structure

For DHCPv4, the library creates entries based on DHCPREQUEST, DHCPDECLINE, DHCPRELEASE messages et.al. and their responses. The resulting packets and leases are taken into account, intercepted through the following hook points: pkt4_receive leases4_committed pkt4_send lease4_release lease4_decline This is necessary to get the subnet logging information. Since Kea 1.9.8, the logging action is performed only on the pkt4_send hook (for request and renew replies), on the lease4_release hook (for release replies) and on the lease4_decline hook (for declined replies). This guarantees that nothing is logged if the server or any other hook decides to drop the packet between lease update and packet send operations.

An entry is a single string with no embedded end-of-line markers, a prepended timestamp and has the following sections:

<timestamp><address><duration><device-id>{client-info}{relay-info}{user-context}

Where:

- timestamp - the current date and time the log entry was written in "%Y-%m-%d %H:%M:%S %Z" strftime format.
- address - the leased IPv4 address given out and whether it was assigned, renewed or released.
- duration - the lease lifetime expressed in days (if present), hours, minutes and seconds. A lease lifetime of 0xFFFFFFFF will be denoted with the text "infinite duration". This information is not given when the lease is released.
- device-id - the client's hardware address shown as numerical type and hex digit string.
- client-info - the DHCP client id option (61) if present, shown as hex digit string. When the id is printable it is added.
- relay-info - for relayed packets the giaddr and the RAI circuit id and remote id options (option 82 sub options 1 and 2) if present.
- user-context - the optional user context.

For instance:

    2018-01-06 01:02:03 CET Address: 192.2.1.100 has been renewed for 1 hrs 52 min 15 secs to a device with hardware address: hwtype=1 08:00:2b:02:3f:4e, client-id: 17:34:e2:ff:09:92:54 connected via relay at address: 192.2.16.33, identified by circuit-id: 68:6f:77:64:79 (howdy) and remote-id: 87:f6:79:77:ef

or for a release:

    2018-01-06 01:02:03 CET Address: 192.2.1.100 has been released from a device with hardware address: hwtype=1 08:00:2b:02:3f:4e, client-id: 17:34:e2:ff:09:92:54 connected via relay at address: 192.2.16.33, identified by circuit-id: 68:6f:77:64:79 (howdy) and remote-id: 87:f6:79:77:ef

The request-parser-format and response-parser-format can be used to extract and log data from the incoming packet and server response packet respectively. The configured value is an evaluated parsed expression returning a string. This can be used to generate custom log entries. Use with caution as this might affect server performance. If any of them is configured, the default logging format is not used. If both of them are configured, the resulting log message is constructed by concatenating the logged data extracted from the request and the logged data extracted from the response.

The timestamp-format parameter can be used to change the timestamp logged at the beginning of each line. Permissible formatting is the one supported by strftime plus the 'Q' extra format which adds the microseconds subunits. The default is: "%Y-%m-%d %H:%M:%S %Z". This parameter has no effect for the database backends, where the timestamp is defined at the schema level.

In addition to logging lease activity driven by DHCPv4 client traffic, it also logs entries for the following lease management control channel commands: lease4-add, lease4-update, and lease4-del. Each entry is a single string with no embedded end-of-line markers and they will typically have the following forms:

lease4-add:

<timestamp> Administrator added a lease of address: <address> to a device with hardware address: <device-id>

Dependent on the arguments of the add command, it may also include the client-id and duration.

Example:

    2018-01-06 01:02:03 CET Administrator added a lease of address: 192.0.2.202 to a device with hardware address: 1a:1b:1c:1d:1e:1f for 1 days 0 hrs 0 mins 0 secs

lease4-update:

<timestamp> Administrator updated information on the lease of address: <address> to a device with hardware address: <device-id>

Dependent on the arguments of the update command, it may also include the client-id and lease duration.

Example:

    2018-01-06 01:02:03 CET Administrator updated information on the lease of address: 192.0.2.202 to a device with hardware address: 1a:1b:1c:1d:1e:1f, client-id: 1234567890

lease4-del: Deletes have two forms, one by address and one by identifier and identifier type:

<timestamp> Administrator deleted the lease for address: <address>

or

<timestamp> Administrator deleted a lease for a device identified by: <identifier-type> of <identifier>

Currently only a type of hw-address (hardware address) is supported.

Examples:

    2018-01-06 01:02:03 CET Administrator deleted the lease for address: 192.0.2.202

    2018-01-06 01:02:12 CET Administrator deleted a lease for a device identified by: hw-address of 1a:1b:1c:1d:1e:1f

If High availability module is enabled, the partner will periodically send lease commands which have a similar format, the only difference is that the issuer of the command is 'HA partner' instead of 'Administrator'.

Example:

       *timestamp* HA partner added ...

or

       *timestamp* HA partner updated ...

or

       *timestamp* HA partner deleted ...

## DHCPv6 Log Detailed Structure

For DHCPv6, the library creates entries based on REQUEST, RENEW, RELEASE, DECLINE messages et.al. and their responses. The resulting packets and leases are taken into account, intercepted through the following hook points: pkt6_receive leases6_committed pkt6_send lease6_release lease6_decline This is necessary to get the subnet logging information. Since Kea 1.9.8, the logging action is performed only on the pkt6_send hook (for request, renew, rebind, release and declined replies). This guarantees that nothing is logged if the server or any other hook decides to drop the packet between lease update and packet send operations.

Note:

When using custom log format, the lease address and other parameters must be extracted from the information stored in the client request or server response. In the case of IPv6, the packets can contain multiple IA_NA (3) or IA_PD (25) options, each containing multiple options, including OPTION_IAADDR (5) or OPTION_IAPREFIX (25) suboptions. To be able to print the current lease associated with the log entry, the forensic log hook library internally isolates the corresponding IA_NA or IA_PD option and respective suboptions matching the current lease. The hook library will iterate over all new allocated addresses (stored in the leases6 hook argument/context) and all deleted addresses (stored in the deleted_lease6 hook argument/context), making each address available for logging as the current lease for the respective logged entry. It is recommended to extract the data from the server response (if available) because the server can allocate other leases if the ones requested by the client are not available. In this case the server must respond and update lifetimes to 0 for such leases, and they can be filtered out by the parser expression.

An entry is a single string with no embedded end-of-line markers, a prepended timestamp and has the following sections:

<timestamp><address><duration><device-id>{relay-info}*{user-context}*

Where:

- timestamp - the current date and time the log entry was written in "%Y-%m-%d %H:%M:%S %Z" strftime format.
- address - the leased IPv6 address or prefix given out and whether it was assigned, renewed or released.
- duration - the lease lifetime expressed in days (if present), hours, minutes and seconds. A lease lifetime of 0xFFFFFFFF will be denoted with the text "infinite duration". This information is not given when the lease is released.
- device-id - the client's DUID and hardware address (if present).
- relay-info - for relayed packets the content of relay agent messages, remote id and subscriber id options (x and xx) if present.
- user-context - the optional user context.

For instance:

    2018-01-06 01:02:03 PST Address:2001:db8:1:: has been assigned for 0 hrs 11 mins 53 secs to a device with DUID: 17:34:e2:ff:09:92:54 and hardware address: hwtype=1 08:00:2b:02:3f:4e (from Raw Socket) connected via relay at address: fe80::abcd for client on link address: 3001::1, hop count: 1, identified by remote-id: 01:02:03:04:0a:0b:0c:0d:0e:0f and subscriber-id: 1a:2b:3c:4d:5e:6f

or for a release:

    2018-01-06 01:02:03 PST Address:2001:db8:1:: has been released from a device with DUID: 17:34:e2:ff:09:92:54 and hardware address: hwtype=1 08:00:2b:02:3f:4e (from Raw Socket) connected via relay at address: fe80::abcd for client on link address: 3001::1, hop count: 1, identified by remote-id: 01:02:03:04:0a:0b:0c:0d:0e:0f and subscriber-id: 1a:2b:3c:4d:5e:6f

In addition to logging lease activity driven by DHCPv6 client traffic, it also logs entries for the following lease management control channel commands: lease6-add, lease6-update, and lease6-del. Each entry is a single string with no embedded end-of-line markers and they will typically have the following forms:

lease6-add:

<timestamp> Administrator added a lease of address: <address> to a device with DUID: <DUID>
DUID
Holds DUID (DHCPv6 Unique Identifier).
Definition duid.h:142

Dependent on the arguments of the add command, it may also include the hardware address and duration.

Example:

    2018-01-06 01:02:03 PST Administrator added a lease of address: 2001:db8::3 to a device with DUID: 1a:1b:1c:1d:1e:1f:20:21:22:23:24 for 1 days 0 hrs 0 mins 0 secs

lease6-update:

<timestamp> Administrator updated information on the lease of address: <address> to a device with DUID: <DUID>

Dependent on the arguments of the update command, it may also include the hardware address and lease duration.

Example:

    2018-01-06 01:02:03 PST Administrator updated information on the lease of address: 2001:db8::3 to a device with DUID: 1a:1b:1c:1d:1e:1f:20:21:22:23:24, hardware address: 1a:1b:1c:1d:1e:1f

lease6-del: Deletes have two forms, one by address and one by identifier and identifier type:

<timestamp> Administrator deleted the lease for address: <address>

or

<timestamp> Administrator deleted a lease for a device identified by: <identifier-type> of <identifier>

Currently only a type of DUID is supported.

Examples:

    2018-01-06 01:02:03 PST Administrator deleted the lease for address: 2001:db8::3

    2018-01-06 01:02:11 PST Administrator deleted a lease for a device identified by: duid of 1a:1b:1c:1d:1e:1f:20:21:22:23:24

If High availability module is enabled, the partner will periodically send lease commands which have a similar format, the only difference is that the issuer of the command is 'HA partner' instead of 'Administrator'.

Example:

       *timestamp* HA partner added ...

or

       *timestamp* HA partner updated ...

or

       *timestamp* HA partner deleted ...

## Configuring the DHCP Modules

It must be configured as a hook library for the desired DHCP server modules. Note that the legal_log library is installed alongside the Kea libraries in "<install-dir>/lib" where <install-dir> is determined by the –prefix meson setup option. It defaults to "/usr/local". Assuming the default value then, configuring kea-dhcp4 to load the legal_log library could be done with the following Kea4 configuration:

"Dhcp4": {
"hooks-libraries": [
{ "library": "libdhcp_legal_log.so",
"parameters": {
"path": "/var/log/kea",
"base-name": "kea-legal4" } },
...
]
}

To configure it for kea-dhcp6, the commands are simply as shown below:

"Dhcp6": {
"hooks-libraries": [
{ "library": "libdhcp_legal_log.so",
"parameters": {
"path": "/var/log/kea",
"base-name": "kea-legal6" } },
...
]
}

Two string Hook Library Parameters are supported:

- path - Directory in which the legal file(s) will be written. The default value is "<prefix>/var/log/kea". The directory must exist.
- base-name - An arbitrary value which is used in conjunction with the current system date to form the current legal file name. It defaults to "kea-legal".

When running Kea servers for both DHCPv4 and DHCPv6 the log names must be distinct, either the path or the base-name (or both) should be different for DHCP4 vs DHCP6.

# Commented Design for Libdhcp_legal_log

## Introduction

Some countries require ISPs to log the connection meta-data for legal purposes. A legal log hook library demonstrating how to log this information is then useful as both an example explaining how to program with hooks and for direct use.

As explained in the tutorial of the Hook Developer's Guide Tutorial a hook library is a Dynamic Shared Object with known entry points for the framework functions and hooks. The DSO is (when not static linked) dynamically loaded by a Kea server module. Some parameters can be passed to configure it and DSO entry points with a well known hook names are auto-magically registered.

The Kea server module invokes the registered callouts during processing and passes some context arguments. The returned status is used to control further processing, e.g., by dropping a message to cancel the current action.

## Framework

The file version.cc defines the required framework version() function which protects against Hooks API changes. The file load_unload.cc defines a load() function which processes the configuration parameters and creates and opens the legal log file (implemented by class `isc::legal_log::RotatingFile` in rotating_file.h and rotating_file.cc). The unload() function destroys the object which by side effect closes the legal log file. Note the load() function does not register callouts nor a logger: the Hooks Library Manager takes care of this (triggered for the logger by legal_log_log.h, legal_log_log.cc and legal_log_messages.h files).

## Callouts

For DHCPv4 pkt4_receive(), leases4_committed(), pkt4_send(), lease4_release() and lease4_decline() hook callouts are defined in lease4_callouts.cc file. For DHCPv6 pkt6_receive(), leases6_committed(), pkt6_send(), lease6_release() and lease6_decline() hook callouts are defined in lease6_callouts.cc file.

The legalLog4Handler() or legalLog6Handler() is called with the callout handle (to get the parsed query message and the lease) and an action type which can either be ASSIGN or RELEASE. This requires the receive packet callouts (pkt4_receive and pkt6_receive) to initialize an empty context which can be used in all following hooks.

When something goes wrong callouts return 1 (next step skip) which tells the server to skip processing the next step. So in the server any legal log error is converted to an internal error and the client is denied service.

The hook callout, command_processed() is defined to log lease management control channel commands (lease<4/6>-add, -update, -del). Entries are only generated for commands that execute successfully.

## Log Generation

For DHCPv4 the only difficulty is to get the relay agent information option and its two interesting (i.e., to log) circuit ID and remote ID sub-options for any queries that were relayed.

For DHCPv6 the hardware address is not built-in but can be recovered and recorded by kea-dhcp6 in many cases. A DHCPv6 query can be relayed more than once but the current code logs only the first (closest to the client) relay information with first found remote ID and subscriber ID relay options.

The lease client context can only be printed using the default format, as this information is not directly stored in the incoming packet or in the server response packet.

## Database

As an alternative to files a database backend is available. The code is based on the lease backend with an automatic timestamp and log text columns in a new logs table.

The configuration reuses the lease database one with for backward (i.e. file) compatibility either no "type" entry or "logfile" value.

Since Kea 1.9.6, the libdhcp_legal_log hooks library supports database connection recovery which can be enabled by setting the on-fail parameter. If not specified, the on-fail parameter defaults to serve-retry-continue as opposed to the case of lease manager, host manager and config backend where it defaults to stop-retry-exit. In this case, the server will continue serving clients and it will not shut down even if the recovery mechanism fails. If the on-fail is set to serve-retry-exit, the server will shut down if the connection to the database backend is not restored according to the max-reconnect-tries and reconnect-wait-time parameters, but it will continue serving clients while this mechanism is activated. During server startup, the inability to connect to any of the configured backends is considered fatal only if retry-on-startup is set to false (the default). A fatal error is logged and the server exits, based on the idea that the configuration should be valid at startup. Exiting to the operating system allows nanny scripts to detect the problem. If retry-on-startup is set to true, the server will start reconnection attempts even at server startup or on reconfigure events, and will honor the action specified in the on-fail parameter. The retry-on-startup parameter is supported since Kea 2.5.5.

Design notes:

- the timestamp should be set by the database server (vs. DHCP server / database client). According to documentation it refers to the beginning of the transaction.
- database systems manage time zones: the timestamp is stored in UTC and is displayed following options given to query tools.
- the address (or prefix) column can be null / empty. This allows to insert no address related log entries.

## Syslog

Log entries can be inserted into syslog by setting the "type" to "syslog". When syslog type is configured, the "pattern" parameter specifies the details that are used for logging. If not configured, it defaults to: "%-5p [%c.%t] %m\n".

The "facility" parameter specifies the syslog facility and it defaults to "local0".

# Multi-Threading Compatibility

The libdhcp_legal_log hooks library is compatible with multi-threading.