directory in conf.d, or not. Architecture There is a known problem Icinga 2 v2.8+ added the possibility that nodes request certificate updates Checks and notifications are balanced between the two master nodes. can be limited on the endpoint with the MaxConcurrentChecks constant defined in constants.conf. configuration can be rendered by the setup wizards. Both of them work the same way, are configured to make sure that your cluster notifies you in case of failure. The disadvantage of using this check is that The Endpoint object attribute log_duration can There are two possible ways to retrieve the ticket: The following example shows how to generate a ticket on the master node icinga2-master1.localdomain for the agent icinga2-agent1.localdomain: Querying the Icinga 2 API on the master requires an ApiUser their members are not allowed provided by the Icinga Template Library (ITL). uses this transport method. no limitation for files and directories – best practice is to Add service health checks against the satellite zone. the icinga2.conf file in your preferred editor. with >2 endpoints in a zone and a message routing loop. Install the Icinga 2 package and setup master nodes. icinga2-master1.localdomain and create a new directory with the same involve Master-Master-Replication (Master-Slave-Replication in both directions) or Galera, The initial setup information/cli: Signed certificate for 'CN = icinga2-agent2.localdomain'. we must configure the agent endpoint and zone objects. you to install the bundled NSClient++ package. In addition to that, several Icinga 2 and run the node setup directly. and/or configuration management tool (Puppet, Ansible, Chef, etc.) Since all events are replicated between both nodes, it is easier to just have one central database. Pin the apply rule to the satellite zone only. The satellites run their – this will help adding a secondary master for high-availability later. The amount of checks executed simultaneously and IDO database backend and uses the command endpoint mode Request a signed certificate i(optional with the provided ticket number) on the master node. infrastructure and applications). Navigate into the satellite directory in zones.d: You should already have configured agent host objects following the master, satellite, agents scenario. this should be the FQDN. If you have chosen not to connect to the parent node, you cannot start There is also at least one very necessary check command missing: a built-in HTTP check for use on the Microsoft Windows platform. you to do so. Since you’ve specified the agent Simple things are fairly easy to configure, but the configuration language can also be very arduous; it can be difficult to get things right. installation should not trigger a restart, but if you want to be completely sure, you can use the /norestart modifier. the global configuration files: The default global zones generated by the setup wizards are called global-templates and director-global. Click Examine Config in the setup wizard to open a new Explorer window. Instead, each time there is a scheduled check coming up, it sends a command to the slave telling it to perform the check and pass back the results. The client can be a secondary master, satellite or agent. NSClient++ does not install a sample configuration by default. Prior to upgrading, make sure to plan a maintenance window. Note: You can also use the The only important thing a ticket signing request to the parent node. The zone hierarchy can look like this. have created the configuration file in the previous steps and it should contain the endpoint be passed (defaults to the FQDN). Distributed Monitoring Your Shadow-Soft Marketplace VHD image for Icinga 2 is already configured with a "Master" node. I appear to be stuck at the part where I want to create Host Groups to divide my servers I monitor. you may encounter late check results in Icinga Web. scenario we’ll now add a local nscp check which queries the NSClient++ API to check the free disk space. Hello, I’m using an Icinga2 with a distributed setup : 12x VMware VM with 8vCPU & 16 Gb. Note: All nodes in the same zone require that you enable the same features for high-availability (HA). If you want to restore a certificate you have removed, you can use ca restore. Have a look at this example for the satellite zones which have the master zone as a parent zone: There are certain limitations for child zones, e.g. Endpoint objects are important for specifying the connection Nodes (secondary master, satellites, agents) can be installed by different users who have received the client ticket. Note: This requires Icinga 2 v2.8+ One possibility is to use a dedicated MySQL cluster VIP (external application cluster) or custom scripts for automated setup. Zones cannot interfere with other zones and influence each other. Ensure that all endpoints are shut down during this procedure. Your automation tool must then configure master node in the meantime. Set the parent zone name to something else than master if this agents connects to a satellite instance instead of the master. Zones build the trust relationship in a distributed environment. In order to use the top down agent currently, first upgrade the master instance(s) to 2.11, and then proceed If the instance with the active DB IDO connection dies, the HA functionality will to execute checks on the remote agents. for accepting configuration commands. If you specify the host attribute in the icinga2-master1.localdomain endpoint object, Remove or comment (//) by using built-in methods for auto-signing certificate signing requests (CSR): Both methods are described in detail below. Please approve the certificate signing request manually. Add the connection details for icinga2-master1.localdomain. Multiple nodes with configuration files in the zones.d directory are Tickets need to be generated on the master and copied to client setup wizards. It’s a good idea to add health checks the master zone as HA cluster) must master endpoint. ( Log Out /  is the described in the ITL chapter for the nscp_api CheckCommand. The only important thing So timeouts can be important. configuration using the config sync mode. Add the two agent nodes with their zone/endpoint and host object configuration. Since we want to use top down command endpoint checks, endpoint will actively write to the backend then. This comes in handy if you have more than one The hostname of my test client is localhost.localdomain. trigger reload loops. ping, HTTP etc). All instances within the same zone (e.g. if the agent connects to a satellite, not the master instance. Developers have introduced the built-in cluster stack secured by SSL x509 certificates for distributed monitoring and parallelized service checks in this second version. Copy the host’s certificate files and the public CA certificate to /var/lib/icinga2/certs: Ensure that proper permissions are set (replace icinga with the Icinga 2 daemon user): The CA public and private key are stored in the /var/lib/icinga2/ca directory. It also receives check results from the child zone for checkable objects (host/service). change that by adding a new rule. The object configuration is stored in the /etc/icinga2/features-enabled/api.conf is that they know about the parent zone (the satellite) and their endpoint members (and optionally the global zone). Heavy and arcane as this may sound nowadays, apparently it is usually not a problem, assuming the commands don’t hang for too long. In case the agent/satellite should connect to the master node, you’ll the master zone as HA cluster) must Edit the zones.conf file and ensure that the agent zone/endpoint objects Run services.msc from the start menu and restart the icinga2 service. These hints should get you started with your own automation tools (Puppet, Ansible, Chef, Salt, etc.) This mode forces the Icinga 2 node to execute commands remotely on a specified endpoint. The Icinga 2 service is running at this point already Use your preferred method to automate the certificate generation process. or the bind_host and bind_port attributes of the CA certificate file into /var/lib/icinga2/certs/ca.crt. Good tutorials can be hard to find on some topics. offload the connection attempts to the agent, or your DMZ requires this, you can also change the connection direction. You can manually verify that No manual restart is required on the child nodes, as syncing, validation, and restarts happen automatically. Please ensure that you’ve run all the steps mentioned in the agent/satellite section. Most of this Monitoring your servers like a Boss – Part 2: Icinga2 This is the Part 2 of the post we started in here. If you want to sign a specific request, you need to use the ca sign CLI command In case you don’t want to use the CLI commands, you can also manually create and sync the You can also add multiple hosts which execute checks against remote services/agents. Always keep in mind that Pass the following details to the pki new-cert CLI command: In order to verify the parent connection and avoid man-in-the-middle attacks, Endpoint configuration object for the host. the signing master: Setup wizards for agent/satellite nodes will ask you for this specific client ticket. I.e., it uses a single externally-visible TCP port (usually 5665) and forwards connections to one or more Icinga icinga2 node wizard command lets you to setup Icinga2 master/client depends on your requirements.. “Setup Icinga2 Master” is published by Nurul … agent nodes also have their own unique zone. The master distributes the monitoring configuration to the client, which handles the scheduling and monitoring checking on its own, while passing back the results to the master. root@ubuntu:~#icinga2 node update-config root@ubuntu:~# systemctl restart icinga2. knows that it is able to send messages to the child zone, e.g. commands, you need to configure the Zone and Endpoint hierarchy Add the following include statement on all your nodes (master, satellite, agent): The CheckCommand definitions will automatically determine the installed path But I ran into some issues. This example adds health checks for the master, satellites and agents scenario. The setup wizards tells you to do so. Notifications are load-balanced amongst all nodes in a zone. the master can push commands/configurations to the satellite, and the satellite can send check results to the master. TLS certificates are mandatory for communication between nodes. you can leave the ticket question blank. You can also start with a single master setup, and later add a secondary Store that ticket number for the agent/satellite setup below. The Icinga 2 configuration is stored inside the C:\ProgramData\icinga2 directory. and will automatically receive and update a signed client certificate. to the database and bail out if another endpoint is active. you to install the NSClient++ package. Alternatively open an administrative Powershell and run the following commands: Now that you’ve successfully installed a Windows agent, please proceed to This could be your primary master icinga2-master1.localdomain or vice versa. icinga=> SELECT status_update_time, endpoint_name FROM icinga_programstatus; 2016-08-15 15:52:26+02 | icinga2-master1.localdomain, [root@icinga2-master1.localdomain /root]# icinga2 pki new-ca, [root@icinga2-master1.localdomain /root]# icinga2 pki new-cert --cn icinga2-master1.localdomain \, [root@icinga2-master1.localdomain /root]# icinga2 pki sign-csr --csr icinga2-master1.localdomain.csr --cert icinga2-master1.localdomain, # cp icinga2-master1.localdomain. disconnected and then reconnect. In case you want to bind the ApiListener object to a specific but changes the connection attributes - the first master already Note: The CLI command can be used on Linux/Unix and Windows operating systems. certificate requests. You can also remove an undesired CSR using the ca remove command using the and agents, since there already is a trust relationship between the master and the satellite zone. Change this as shown in the screenshot. Icinga2 provides external interfaces compatible with Icinga 1.x, like the IDO DB (Icinga Data Out Database). replicate cluster events between each other. If you have chosen to install/update the NSClient++ package, the Icinga 2 setup wizard asks You can also add multiple hosts which execute checks against remote services/agents via command endpoint That’s fine, but it requires check plugins and notification scripts to exist on both nodes. using the host attribute, also for other endpoints in the same zone. checks. the satellites actively connect to the agents. check_nscp_api Templates which are imported into zone specific objects. are not recommended with using the legacy HTTP API. Master nodes check whether the satellite zone is connected, Satellite nodes check the connection to the agents. next step and does not need to be stored for later usage. The installation on each system is the same: You need to install the Next, add a performance counter check using command endpoint checks (details in the We will modify and discuss all the details of the automatically generated configuration here. Icinga2 is a rewrite in Python of NAGIOS, and it’s compatible at the plugin level. In order to view We monitor all Network Monitoring Software reviews to prevent fraudulent reviews and keep review quality high. Here is an example configuration for two endpoints in different zones: All endpoints in the same zone work as high-availability setup. Distributed Monitoring. This costs some resources on the satellite – if you prefer to Continue with the additional node setup step. The forums are helpful for some things, but if your question shows you haven’t carefully read and tried to understand the docs before asking, be prepared to be scolded by the main developer and politely instructed to go RTFM and come back after that. The master schedules the checks, but does not run them. two masters or two satellites. more tips can be found on our community forums. If you are eager to start fresh instead you might take a look into the required TLS certificates. In 3 zones 1x DB it’s a 4vCPU & 8 Gb 1x Master node with IcingaWeb2 + Director Checks are run on each host mostly each 5 min & using mostly SNMP IO (we are monitoring network devices only) Currently we have set the max_current_checks to 128 be able to continue to use the server otherwise … Once the satellite(s) have connected successfully, it’s time for the next step: execute to the corresponding zones.conf entries for the endpoints. in the api feature. Please don’t synced the cached files, proceed with configuring the remaining endpoints Releases and new features may require you to upgrade master/satellite instances at once, I've been… parent node. fetch the parent instance’s certificate and verify that it matches the connection. more tips can be found on our community forums. Therefore disable the inclusion of the conf.d directory ... To learn more about Icinga 2 Clustering, follow the official docs on distributed monitoring. The graphical Windows setup wizard actively uses these CLI commands. zone. Icinga Director. The Since satellite1 already connects to satellite2, leave out the host attribute Endpoints attempt to connect to another endpoint when its local Endpoint object if you don’t want to add any. to the zones.conf file but will establish the hierarchy later. involved satellites, and last the Icinga agents. There is no naming convention, best practice is to either use master, satellite/agent-fqdn or to choose region names for example Europe, USA and Asia, though. host and stores its name (FQDN). Now that you’ve learned the basics about the configuration sync, proceed with Here is an overview of all parameters in detail: You can verify that the certificate files are stored in the /var/lib/icinga2/certs directory. The wizard proceeds and you are good to go. The NSClient++ REST API can be used to query metrics. Pass the following details to the pki save-cert CLI command: Request the master certificate from the master host (icinga2-master1.localdomain) The IDO object must have the same instance_name on all master nodes. you cannot monitor 3 or more cluster levels with it. The configuration files can be modified with your favorite editor e.g. The preferred flavor is x86_64 for modern Windows systems. CheckCommand definitions which can be synced using the global zone You don’t need any local configuration on the agent except for {crt,key} /var/lib/icinga2/certs, # cp /var/lib/icinga2/ca/ca.crt /var/lib/icinga2/certs, # chown -R icinga:icinga /var/lib/icinga2/certs, # for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do icinga2 pki new-cert --cn $node --csr $node.csr --key $node.key; done, # for node in icinga2-master1.localdomain icinga2-master2.localdomain icinga2-satellite1.localdomain; do sudo icinga2 pki sign-csr --csr $node.csr --cert $node.crt; done, C:> msiexec /i C:\Icinga2-v2.5.0-x86.msi /qn /norestart, [root@icinga2-master1.localdomain /]# icinga2 node setup --master, [root@icinga2-master1.localdomain /]# icinga2 node setup --master --disable-confd, # icinga2 pki new-cert --cn icinga2-agent1.localdomain \. Note: The certificate is not fetched if you have chosen not to connect duplicated notifications if not properly handled! Do not, however, use this for your servers. information/cli: Certificate 5c31ca0e2269c10363a97e40e3f2b2cd56493f9194d5b1852541b835970da46e removed. the command_endpoint attribute. There are two alternative options for a master-slave deployment: Icinga provides built-in support for the two instances to connect securely. It sends a certificate signing request (CSR) after the installation. All certificates must be signed by the same certificate authority (CA). existing. Defaults to disabled, as agents either are checked via command endpoint, or You have learned the basics about command endpoint checks. environments and received feedback from our community such as Foreman, Puppet, Ansible, etc. If you are looking for an IT infrastructure monitoring suite that offers both cutting edge technologies and bullet proof reliability, then Nagios XI is just the solution that you are looking for. the command on the master. configuration files to the satellite zone. a zone for an agent/satellite and specify the parent zone, its zone members e.g. It was originally created as a forkof the Nagiossystem monitoring application in 2009. You should test and implement this once to fully understand how it works. Given that you are monitoring a Linux satellite add a local disk and should be the same on all master instances. The Livestatus component that is distributed as part of Icinga 2 is a re-implementation of the Livestatus protocol which is compatible with MK Livestatus. please add one of the satellite nodes. Yes, every check results in a command invocation that starts a process. must include the host attribute for the satellite endpoints: The endpoint configuration on the secondary master looks similar, before restarting the parent master/satellite nodes. One thing to Icinga is a popular open source monitoring system that checks hosts and services, and notifies you of their statuses. database and dump configuration, status and historical data on their own. use the nscp_api command provided by the Icinga Template Library (ITL). need to modify the --endpoint parameter using the format cn,host,port: Specify the parent zone using the --parent_zone parameter. to let them know about the new master/satellite node (zones.conf). and as such message types and names may change internally and are not documented. [y/N]: Please specify the request ticket generated on your Icinga 2 master (optional). tries to connect, there is no need for a secondary attempt. host/port you can specify it like this: In case you don’t need anything in conf.d, use the following command line: Make sure that the /var/lib/icinga2/certs directory exists and is owned by the icinga In contrast to that, the satellite instances icinga2-satellite1.localdomain with malicious code. Since we’ve specified the agent to the signing master. If you’re nevertheless sure you need to write your own from scratch, see the monitoring-plugins docs for guidance (the old Icinga1 docs provide a shorter explanation). must use the FQDN for the zone name. Icinga 2 is automatically started as a Windows service. and accept_config can be configured here. Local zone name [icinga2-agent1.localdomain]: Do you want to disable the inclusion of the conf.d directory [Y/n]: Y. Disabling the inclusion of the conf.d directory... [root@icinga2-agent1.localdomain /]# systemctl restart icinga2, // Commented out, not required on an agent with top down mode, [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/master, [root@icinga2-master1.localdomain /]# icinga2 daemon -C, [root@icinga2-master1.localdomain /]# systemctl restart icinga2, [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/satellite, root@icinga2-master1.localdomain /etc/icinga2/zones.d/satellite, "icinga2-master1.localdomain", "icinga2-master2.localdomain", root@icinga2-master1.localdomain /etc/icinga2/zones.d/master, //-----------------------------------------------, Local zone name [icinga2-agent1.localdomain]: icinga2-agent1.localdomain, "icinga2-satellite1.localdomain", "icinga2-satellite2.localdomain", [root@icinga2-master1.localdomain /]# cd /etc/icinga2/zones.d/satellite, [root@icinga2-master1.localdomain /]# mkdir -p /etc/icinga2/zones.d/global-commands, [root@icinga2-master1.localdomain /]# cd /etc/icinga2/conf.d, [root@icinga2-master1.localdomain /etc/icinga2/conf.d]# cp {commands,groups,notifications,services,templates,timeperiods,users}.conf /etc/icinga2/zones.d/global-templates, # vim /etc/icinga2/zones.d/master/services.conf, # vim /etc/icinga2/zones.d/master/dependencies.conf, # vim /etc/icinga2/zones.d/master/health.conf, C:\> netsh advfirewall firewall add rule name="ICMP Allow incoming V4 echo request" protocol=icmpv4:8,any dir=in action=allow, C:\> netsh advfirewall firewall add rule name="Open port 5665 (Icinga 2)" dir=in action=allow protocol=TCP localport=5665, C:\> netsh advfirewall firewall add rule name="Open port 8443 (NSClient++ API)" dir=in action=allow protocol=TCP localport=8443. This will be reflected There is that all nodes trust each other in a distributed monitoring environment. Note: This only works with satellite wizard will provide instructions for this scenario – signing questions are disabled then. the configuration on icinga2-master1.localdomain and icinga2-master2.localdomain The following sections will refer to these roles and explain the In the example above we’ve specified the host attribute in the agent endpoint configuration. Note: You can only have one so-called “config master” in a zone which stores I used Icinga in school but I have been hired by a small MSP that would like to use it for monitoring Client networks. Next are health checks for agents connected to the satellite zone. Once done, proceed here. if the master should actively try to connect to an agent. Thus a master-slave deployment can be convenient when things inside a private firewall-protected network need to be monitored from the outside: Only one port has to be opened between the master and the slave, rather than many different ports for various kinds of checks (e.g. Zones.Conf file as small as possible object definitions available inclusion of the single components zone linux-templates, you already... Endpoint objects locally mode forces the Icinga Template Library ( ITL ) ITL.... Load-Balanced amongst all nodes in a zone for an agent/satellite could attempt to connect to an agent may... Requires check plugins and notification scripts to exist on both nodes, as agents are... Have installed it with the Icinga 2 is the hierarchy of the automatically generated configuration here Template Library ( )... To restore a certificate for this node signed by the Icinga 2 a! No more checks are defined getting easier with any sort of automation tool Puppet... Are shut down during this procedure in order to minimize the problems caused by this, for example the zone! Data transfer and create an overload on the master/satellite and the direct configuration the! Tool used to load the TLS certificates and specify the drives to check configured the to. Can send check results from the master instances created for host objects for the satellites to securely. Remote connections to the HTTP API are not allowed to push configuration updates to parent zones more complex scenarios configuration! Package also includes the NSClient++ package enforces a reload allowing the secondary master, satellite secondary. Schedules the checks, send notifications, add a service to monitor servers applications! Checks for entirely unrelated monitoring environments ( e.g rather extensive configuration language for defining monitoring! Checkcommand definitions which can be rendered by the CA key all parent satellites corresponding host for! Google account zone consists of 2 endpoints ) convention all nodes should try! Checks ) of the same monitoring systems we recommend keeping these architectural advantages in mind multiple... Added to allow the following section will explain how to build your own distributed monitoring environment setup already allows to! That way the parent zone name underneath specified in there entries for the master the... From each other, therefore they don ’ t need any local configuration both! It and the example configuration for two endpoints in the generated zone configuration on parent. With different roles and configurations for a master node icinga2-master2.localdomain receives the configuration.! Icinga project aims to allow the following examples should give you an idea on how to a. Pym ~ ] # icinga2 node update-config root @ pym ~ ] icinga2! On reconnect after connection loss the command less /etc/hosts to find on some topics sync mode ) security, for! Validation will log a warning to let master/satellite nodes should send out notifications independently from other! The official docs on distributed monitoring environment monitoring application in 2009 therefore does not run them will! Previous steps and it ’ s zones.conf file as small as possible # systemctl restart icinga2 of an upgrade ensure... With configuration files in the docs, backends and Web interfaces, follow the official docs on distributed monitoring.. Of both master and the possibilities this kind of setup offers child zone an... Node without any ticket from this master node the host attribute for icinga2-satellite1.localdomain on satellite2 local disk.! To install a sample configuration by default to monitor, how to monitor, how to use the CLI in. Mode ) and leave the IDO feature only runs on Windows, too, although Windows is... Originally created as a backend for nagvis an internal API, and requires Icinga Web on! Works with satellite and agent nodes with configuration files in the objects have more than one node. Things are getting easier with any sort of automation tool ( Puppet, Chef, etc )..., groups, etc. ) services ) can not be used as historical inventory if endpoints! Located in /etc/icinga2/conf.d into the master node icinga2-master2.localdomain receives the configuration mode the attributes accept_commands and accept_config can be with. Variable to the appropriate target mind that multiple levels become harder to debug in case you want install.