How can I install Splunk Enterprise and Splunk Forwarder on Bright 8.2?

Contents

By following the procedure outlined here:

Installing Splunk-Forwarder and building a new software image:

The method used here is to use a working regular node, with splunk-forwarder installed and configured on it. This will then be used to create a new software image using the grabimage command.

The following Splunk documentation is used as a guide for creating the image:

https://docs.splunk.com/Documentation/Forwarder/7.2.5/Forwarder/Makeauniversalforwarderpartofahostimage

The steps:

Use a working/clean node — node001 in our case — to install and configure Splunk-forwarder.

Download the Splunk-Forwarder from:
https://www.splunk.com/en_us/download/universal-forwarder.html#tabs/linux

Install it on the regular node:

[root@node001 ~]# yum localinstall splunkforwarder-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm

Start the forwarder for the first time, accept the license and create an admin account:

[root@node001 ~]# cd /opt/splunkforwarder/bin/
[root@node001 bin]# ./splunk start --accept-license
This appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username:
WARN: You entered nothing, using the default 'admin' username.
Password must contain at least:
  * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Splunk> See your world.  Maybe wish you hadn't.
Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-7.2.5.1-962d9a8e1586-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...  
Done
                                                  [ OK ]

Enable auto-start

[root@node001 bin]# ./splunk enable boot-start
Init script installed at /etc/systemd/system/.
Init script is configured to run at boot.

Configure the forwarder to send the logs to your Splunk indexer:

[root@node001 bin]# ./splunk add forward-server 10.141.255.254:9997 -auth admin:<YOUR SPLUNK-FORWARDER PASSWORD CONFIGURED IN SETP 2>
Added forwarding to: 10.141.255.254:9997.

Add a monitor(s) that you want. These are monitored by the forwarder, and forwarded to the indexer:

[root@node001 bin]# ./splunk add monitor /var/log/maillog
Added monitor of '/var/log/maillog'.

Make sure everything is configured correctly

[root@node001 bin]# ./splunk list monitor
Monitored Directories:
$SPLUNK_HOME/var/log/splunk
/opt/splunkforwarder/var/log/splunk/audit.log
[...]
Monitored Files:
$SPLUNK_HOME/etc/splunk.version
/var/log/maillog
[root@node001 bin]# ./splunk list forward-server
Active forwards:
None
Configured but inactive forwards:
10.141.255.254:9997

Stop the forwarder:

[root@node001 bin]# ./splunk stop
Stopping splunkd...
Shutting down.  Please wait, as this may take a few minutes.
...........................................................[  OK ]....................................................
Stopping splunk helpers...
                                                          [ OK ]
Done.

Clear the node specific configuration – preparing for grabimage:

[root@node001 bin]# ./splunk clone-prep-clear-config
Erased key "serverName" from server.conf; contained "node001"
Erased key "guid" from instance.cfg; contained "EB49B792-EF31-4E4E-8D49-C8CBFF12A9AC"
Erased key "host" from inputs.conf; contained "node001"

We will clone the original image used for node001, and then use grabimage to rsync the changes only to the new image. This will save us time and network bandwidth, instead of transferring the full image from node001:

[test14->softwareimage]% clone default-image splunk-image
[test14->softwareimage*[splunk-image*]]% commit
[test14->softwareimage[splunk-image]]%
Tue Apr 16 10:52:15 2019 [notice] test14: Started to copy: /cm/images/default-image -> /cm/images/splunk-image (184)
[test14->softwareimage[splunk-image]]%
Tue Apr 16 10:55:54 2019 [notice] test14: Copied: /cm/images/default-image -> /cm/images/splunk-image (196)
[test14->softwareimage[splunk-image]]%
Tue Apr 16 10:55:54 2019 [notice] test14: Initial ramdisk for image splunk-image is being generated
[test14->softwareimage[splunk-image]]%
Tue Apr 16 10:56:21 2019 [notice] test14: Initial ramdisk for image splunk-image was generated successfully

Grab the changes to the new image “splunk-image”

[test14->softwareimage]% device
[test14->device]% use node001
[test14->device[node001]]% grabimage -w -i splunk-image
[test14->device[node001]]%
Tue Apr 16 10:58:49 2019 [notice] test14: Provisioning started: sending node001:/ to test14:/cm/images/splunk-image, mode GRABNEW, dry run = no
[test14->device[node001]]%
Tue Apr 16 10:59:20 2019 [notice] test14: Provisioning completed: sent node001:/ to test14:/cm/images/splunk-image, mode GRABNEW, dry run = no
grabimage -w -i splunk-image [ COMPLETED ]

We will need to set our exclude lists correctly to avoid overwriting node-specific configuration every time it is rebooted. You can set the exclude lists at category level or at node level. We are going to use category level here:

[test14->softwareimage[splunk-image]]% category
[test14->category]% use default
[test14->category[default]]% set excludelistsyncinstall 
[...]
#Splunk forwarder
- /opt/splunkforwarder/var/*
- /opt/splunkforwarder/etc/system/local/*
- /opt/splunkforwarder/etc/*.cfg
- /opt/splunkforwarder/etc/*.conf
- /opt/splunkforwarder/etc/passwd
- /opt/splunkforwarder/etc/auth/*
- /opt/splunkforwarder/etc/myinstall/*
- /opt/splunkforwarder/etc/system/local/*
no-new-files: - /opt/splunkforwarder/ftr
no-new-files: - /usr/share/*splunk*
no-new-files: - /etc/systemd/system/SplunkForwarder.service
no-new-files: - /etc/systemd/system/multi-user.target.wants/SplunkForwarder.service

Do the same for excludelistupdate and commit:
[test14->category[default]]% set excludelistupdate
[test14->category*[default*]]% commit

Set the new software image to be used for your nodes, category level or node level:
[test14]% category use default
[test14->category[default]]% set softwareimage splunk-image
[test14->category*[default*]]% commit

Reboot the nodes to use the new software image with splunk-forwarder installed and configured

Install Splunk Enterprise on your Bright cluster, if it is not already available in your environment:

We are installing it on the head node, just for testing/demo purposes. Usually you should not install any additional 3rd party software on the head node.

The following Splunk documents can be looked up for details on how to install and configure Splunk:
https://docs.splunk.com/Documentation

[root@test14 ~]# yum localinstall splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64.rpm

Start splunk server, accept the license and create the admin account as we did with the forwarder:

[root@test14 ~]# /opt/splunk/bin/splunk start
This appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username:
WARN: You entered nothing, using the default 'admin' username.
Password must contain at least:
  * 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 2048 bit long modulus
..........+++++
.......+++++
e is 65537 (0x10001)
writing RSA key
Generating RSA private key, 2048 bit long modulus
....................+++++
..........+++++
e is 65537 (0x10001)
writing RSA key
Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
Splunk> See your world.  Maybe wish you hadn't.
Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration...  Done.
Creating: /opt/splunk/var/lib/splunk
Creating: /opt/splunk/var/run/splunk
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunk/var/run/splunk/upload
Creating: /opt/splunk/var/spool/splunk
Creating: /opt/splunk/var/spool/dirmoncache
Creating: /opt/splunk/var/lib/splunk/authDb
Creating: /opt/splunk/var/lib/splunk/hashDb
New certs have been generated in '/opt/splunk/etc/auth'.
Checking critical directories...Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility...  Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunk-7.2.5.1-962d9a8e1586-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...  
Generating a 2048 bit RSA private key
.+++++
.......................+++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=test14/O=SplunkUser
Getting CA Private Key
writing RSA key
Done
                                                          [ OK ]
Waiting for web server at http://127.0.0.1:8000 to be available.... Done
If you get stuck, we're here to help.  
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://test14:8000

The admin interface is on port 8000. To allow it through the Shorewall firewall, the following rule is added in “/etc/shorewall/rules” before the last line.

ACCEPT   net      fw tcp     8000
#LAST LINE -- ADD YOUR ENTRIES BEFORE THIS ONE -- DO NOT REMOVE
    The shorewall service is then restarted:
[root@test14 ~]# systemctl restart shorewall

Now, Splunk needs to accept forwarded logs on the port that was specified earlier on, when the forwarder “9997” was configured.
This can be done under the path: Settings > Forwarding and receiving: