Our blog

How To Set Up And Start Using Splunk Lookups

using lookup splunk featured image

If you work with data, chances are you’ve heard of Splunk – a powerful tool for indexing, searching, and visualizing machine-generated data. One of the key features of Splunk is Lookups, which allows you to augment your data with information from external sources.

In this guide, we’ll walk you through the process of setting up and using Lookups in Splunk. We’ll cover the different types of Lookups available in Splunk, show you how to create Lookups using CSV files or external scripts and demonstrate how to use Lookups in search queries, alerts, reports, and dashboards.

By the end of this guide, you’ll have a solid understanding of how to use Lookups in Splunk to enhance your data analysis and gain new insights from your data.

So let’s get started!

What are Splunk Lookups and Why Use Them?

Splunk Lookups are a powerful feature that allows you to enrich your data with additional information from external sources. In simple terms, Lookups allow you to add new fields to your data that are not present in the original events. These additional fields can be used to perform more advanced analysis, create reports and dashboards, and gain new insights into your data.

There are several types of Lookups available in Splunk, including:

  • CSV Lookups: These are the most common type of Lookups in Splunk, and involve creating a CSV file containing the additional information you want to add to your data.
  • KV Store Lookups: These Lookups use the Splunk KV Store, which is a NoSQL database that allows you to store and retrieve key-value pairs.
  • External Lookups: These Lookups allow you to execute an external script or program that generates the additional information you want to add to your data.

So why use Lookups in Splunk? Here are a few reasons:

  • Enrichment: Lookups allow you to enrich your data with additional information that is not present in the original events. This can help you gain new insights into your data, identify trends and patterns, and create more informative reports and dashboards.
  • Reusability: Once you’ve created a Lookup, you can use it across multiple searches, reports, and dashboards. This can save you time and effort in the long run, as you don’t need to recreate the Lookup every time you need to use it.
  • Flexibility: Lookups are highly customizable and can be tailored to meet your specific data analysis needs. You can create Lookups using CSV files, KV Store tables, or external scripts, depending on your requirements.

Overall, Splunk Lookups are a powerful tool for enhancing your data analysis and gaining new insights into your data. In the next section, we’ll dive into the different types of Lookups available in Splunk and show you how to set them up.

Types of Lookups in Splunk

Splunk provides several types of Lookups that you can use to enrich your data with additional information. 

Each type of Lookup has its own strengths and use cases, and choosing the right type depends on your specific data analysis needs. Here are the main types of Lookups available in Splunk:

1. CSV Lookups

CSV Lookups are the most commonly used type of Lookups in Splunk. They involve creating a CSV file that contains the additional information you want to add to your data. The CSV file can be stored on the Splunk server or on an external location.

CSV Lookups are easy to create and maintain, and can be used for a wide range of use cases. For example, you can use a CSV Lookup to add geographic information to your data, such as the city or country of origin for an IP address.

Creating a CSV Lookup in Splunk

CSV Lookups are a common way to enrich your data with additional information in Splunk. Here’s how to create a CSV Lookup in Splunk:

Prepare your CSV file

Create a CSV file that contains the additional information you want to add to your data. The CSV file should have a header row that defines the field names, and each subsequent row should contain the field values for a specific lookup key.

For example, if you want to add geographic information to your data based on the IP address, your CSV file might look something like this:

ip_address,city,country 192.168.0.1,San Francisco,United States 10.0.0.1,London,United Kingdom

Save the CSV file on the Splunk server or on an external location that is accessible to Splunk.

Create the Lookup definition

In Splunk, navigate to the Settings menu and select Lookups. Click on the Add New button and select CSV File as the Lookup type.

Give your Lookup a name and a description, and select the CSV file you created in Step 1. Specify the key field that will be used to match the Lookup data to the original events. In the example above, the key field would be “ip_address”.

Test the Lookup

Once you’ve created the Lookup definition, you can test it by running a search that includes the Lookup. For example, if you want to add the city and country information to a search that looks for failed login attempts, you could use the following search command:

index=security sourcetype=auth fail* | lookup geo_lookup ip_address OUTPUT city country

This command will search for events in the security index with a sourcetype of auth that contain the string “fail”, and then use the Lookup to add the city and country information based on the IP address.

Use the Lookup in your reports and dashboards

Once you’ve confirmed that the Lookup is working correctly, you can use it in your reports and dashboards. Simply add the Lookup fields to your search commands or visualizations to display the additional information.

2. KV Store Lookups

KV Store Lookups use the Splunk KV Store, which is a NoSQL database that allows you to store and retrieve key-value pairs. KV Store Lookups are more scalable and flexible than CSV Lookups, and can handle larger data volumes and more complex data structures.

KV Store Lookups are ideal for use cases where you need to perform frequent updates to the Lookup data or where you have a large number of Lookup tables. For example, you can use a KV Store Lookup to store a list of customer names and addresses, and use it to enrich your sales data with customer information.

Creating a KV Store Lookup

Here’s how to create a KV Store Lookup in Splunk:

Prepare your data

Before you can create a KV Store Lookup, you’ll need to prepare your data. In this example, we’ll assume that we want to add user contact information to our data based on the user ID. 

We’ll create a new KV Store collection that contains this information.

To start, create a CSV file with your user contact information, including a field for the user ID. Here’s an example:

user_id, email,phone_number

jdoe,jdoe@example.com,555-1234

asmith,asmith@example.com,555-5678

Save this file to the Splunk server or to an external location that is accessible to Splunk.

Create the KV Store collection

In Splunk, navigate to the Settings menu and select Lookups. Click on the Add New button and select KV Store Collection as the Lookup type.

Give your Lookup a name and a description, and then define the fields for your collection. In this example, we’ll create a collection called “user_contacts” with fields for user_id, email, and phone_number.

Next, select the “Import data from a CSV file” option and choose the CSV file you created in Step 1. Map the fields in your CSV file to the fields in your KV Store collection.

Define the Lookup

Once you’ve created your KV Store collection, you can define the Lookup that will use it. Click on the Add New button again and select KV Store Lookup as the Lookup type.

Give your Lookup a name and a description, and then define the fields that you want to add to your data. In this example, we’ll create a Lookup called “user_contacts_lookup” that adds email and phone_number fields based on the user_id field.

Test the Lookup

To test your KV Store Lookup, you can run a search that includes the Lookup. For example, if you want to add user contact information to a search that looks for login events, you could use the following search command:

index=security sourcetype=auth | lookup user_contacts_lookup user_id OUTPUT email phone_number

This command will search for events in the security index with a sourcetype of auth, and then use the Lookup to add the email and phone_number fields based on the user_id field.

Use the Lookup in your reports and dashboards

Once you’ve confirmed that your Lookup is working correctly, you can use it in your reports and dashboards. Simply add the Lookup fields to your search commands or visualizations to display the additional information.

3. External Lookups

External Lookups allow you to execute an external script or program that generates the additional information you want to add to your data. External Lookups are highly customizable and can be used for complex data analysis tasks that cannot be handled by CSV or KV Store Lookups.

External Lookups can be written in any language that can read and write to standard input and output. For example, you can use an External Lookup to perform sentiment analysis on tweets or to extract data from a REST API.

Creating an External Lookup in Splunk

External lookups can be a powerful tool for enriching your Splunk data with additional context and information. By following these steps, you can create and use external lookups to enhance your data analysis in Splunk.

Create a lookup file  

First, you need to create a CSV file that contains the data you want to use for the external lookup. The first column of the CSV file should contain the field you want to use as the key for the lookup. The subsequent columns should contain the additional data you want to add to your Splunk events. Save this file to a location that is accessible by the Splunk server.

Define the external lookup in Splunk: 

Next, you need to define the external lookup in Splunk. To do this, go to Settings > Lookups > Lookup Definitions in the Splunk web interface. Click “New” to create a new lookup definition. Give your lookup a name and select “External” as the lookup type.

Configure the external lookup

In the lookup definition configuration page, you need to specify the path to your lookup file and the field names that correspond to the key field and the additional fields in the lookup file. You can also specify any options or arguments to be passed to the lookup script.

Create a lookup table

Once you have defined your external lookup, you need to create a lookup table that will use this lookup definition. To do this, go to Settings > Lookups > Lookup Tables in the Splunk web interface. Click “New” to create a new lookup table. Give your lookup table a name and select the lookup definition you created in the previous step.

Use the lookup in a search 

Now that you have created the lookup table, you can use it in your searches. To use the lookup, simply use the “lookup” command in your search and specify the name of the lookup table and the field to use as the key. The lookup command will add the additional fields from the lookup file to the events that match the key field.

Using Lookups in Search Queries

To use a lookup in Splunk, you first need to define it. To do this, go to Settings > Lookups in the Splunk web interface. Here, you can create new lookups and manage existing ones.

Once you have defined a lookup, you can use it in your search queries. The syntax for using a lookup in a search is as follows: [| inputlookup <lookup-name>]

Replace <lookup-name> with the name of the lookup you want to use. You can then use the fields from the lookup in your search query, just as you would any other fields.

For example, let’s say you have a lookup table containing a list of IP addresses and their associated countries. You can use this lookup to add a “country” field to your events based on the IP address in the event. Here’s an example search query that does this:

index=web_logs [| inputlookup ip_country_lookup.csv] | lookup ip_country_lookup.csv ip_address OUTPUT country | stats count by country

This query searches the web_logs index and uses the ip_country_lookup.csv lookup to add a “country” field to each event based on the IP address in the event. It then uses the stats command to count the number of events by country.

In addition to adding fields to your events, lookups can also be used to filter events based on external criteria. For example, you can use a lookup to filter events based on a list of known malicious IP addresses, or to exclude events from known internal IP ranges.

Using Lookups in Alerts

Alerts in Splunk allow you to proactively monitor your data and notify you when certain conditions are met. Lookups can be a powerful tool in creating alerts that are more sophisticated and targeted.

By using lookups in alerts, you can add additional context to your alert notifications, such as the name of the server or the application associated with the event. You can also use lookups to filter out events that are not relevant to your alert conditions, making your alerts more precise and reducing false positives.

To use a lookup in an alert, you first need to define the lookup and make it available to the alerting system. This can be done by creating a lookup definition in the Splunk web interface and configuring it to use an external file or database as the data source.

Once you have defined your lookup, you can use it in your alert condition by adding a sub-search that references the lookup. The syntax for using a lookup in an alert condition is similar to using a lookup in a search query

php

Copy code

[| inputlookup <lookup-name> | fields <field-list> | format]

Replace <lookup-name> with the name of the lookup you want to use, and <field-list> with a comma-separated list of fields to include in the output of the sub-search. The format command is used to format the output of the sub-search as a table, which can then be used in the alert condition.

For example, let’s say you have a lookup table containing a list of servers and their associated owners. You can use this lookup to add an “owner” field to your alert notifications based on the server name in the event. Here’s an example alert condition that does this:

index=web_logs status=404 [| inputlookup server_owner_lookup.csv | fields server, owner | format]

This alert condition searches the web_logs index for events with a status field of 404. It then uses the inputlookup command to add an “owner” field to the alert notification based on the server name in the event. The fields command is used to limit the output of the sub-search to just the server and owner fields, and the format command is used to format the output as a table.

Using lookups in alerts can help you create more targeted and actionable alert notifications. By adding context and filtering out irrelevant events, you can reduce alert fatigue and improve the overall effectiveness of your monitoring strategy.

Using Lookups in Reports and Dashboards

By using lookups in reports and dashboards, you can add additional information to your data that’s not already indexed in Splunk. For example, you might have a lookup table that contains customer information, and you want to include that information in your reports to understand which customers are experiencing issues.

To use a lookup in a report or dashboard, you’ll first need to define the lookup and make it available to the report or dashboard. You can do this by creating a lookup definition in the Splunk web interface and configuring it to use an external file or database as the data source.

Once you’ve defined your lookup, you can use it in your report or dashboard by adding a sub-search that references the lookup. The sub-search will use the lookup to add additional fields to your data, which can then be used in your report or dashboard visualizations.

For example, let’s say you have a lookup table that contains customer information, and you want to include that information in a dashboard that shows the number of support tickets by customer. You can use the lookup to add a “customer name” field to your data, which can then be used in a table visualization.

Using lookups in reports and dashboards can help you gain a deeper understanding of your data and make more informed decisions. 

By adding context and additional information, you can create visualizations that are more meaningful and impactful. Additionally, lookups can help you filter out irrelevant data and focus on the most important insights. Overall, lookups are a powerful tool for creating more insightful and effective reports and dashboards in Splunk.

Best Practices for Using Lookups in Splunk

When it comes to using lookups in Splunk, there are a few best practices you should keep in mind to ensure that your lookups are effective and efficient.

  • First, you should be careful when using lookups with large datasets. Lookups can be slow and resource-intensive, so if you’re working with a large dataset, you may want to consider using a database instead. Additionally, you should avoid using lookups in search queries that run frequently or across large time ranges, as this can cause performance issues.
  • Another best practice is to make sure your lookup table is properly formatted and indexed. Splunk indexes the fields in a lookup table, so if you have a large number of fields or complex data types, it can slow down performance. To avoid this, you should only include the fields you need in your lookup table, and make sure they are formatted correctly.
  • It’s also important to keep your lookup tables up-to-date. If your lookup table contains data that is frequently changing, such as IP address mappings or user information, you should make sure to update the lookup table regularly. You can use scheduled searches or scripts to update the lookup table automatically, or you can manually update the table as needed.
  • Finally, you should be careful when using lookups in distributed environments. If you’re using lookups across multiple search heads or indexers, you need to make sure that the lookup table is available in all instances. 
  • You can use a distributed search to ensure that the lookup table is available to all search heads, or you can replicate the lookup table across all indexers.

By following these best practices, you can ensure that your lookups are efficient, effective, and up-to-date. Lookups can be a powerful tool in Splunk, but it’s important to use them correctly and responsibly to get the most out of your data.

Setup and leverage Splunk Lookups seamlessly with BitsIO

While setting up Splunk Lookups is quite simple, there is a lot of room for error if not paid attention to. 

To avoid these errors and ensure that Splunk Lookups works seamlessly with your data, we highly recommend working with Splunk professionals like BitsIO to gain the most out of it.

As Certified Splunk Professionals, our team of experts work to ensure that your Splunk environment is properly architected, correctly configured, and fully secured. 

That includes making sure that Splunk Lookups is set up in a way that works optimally for your business.

Contact us to learn more about how we can help.