Eric Partington

Log - Parsing Cisco Umbrella

Blog Post created by Eric Partington Employee on Jan 12, 2018

Interesting question from an internal resource about how to parse CSV files that contain information pulled from Cisco Umbrella S3 Buckets to a local filestore and how to get them into RSA NetWitness Logs.  As a learning process I have documented the steps that were used to get this working, hopefully you find it useful. 

Thanks to Dave Glover for the help with the parser framework and Nir Oz for the original question.

 

At high level we will assume that the logs are pulled down by an external script to a location that either has the SFTP agent installed or can be placed in the correct directory on the log collector for the parsing pipeline to take over in NetWitness.

 

Review the typespec framework as we will create a new one for the file collection method.

https://community.rsa.com/docs/DOC-54570

 

Create the typespec file for file collection

 

<?xml version="1.0" encoding="UTF-8"?>
<typespec>
 
   <name>cisco_umbrella</name>
   <type>file</type>
   <prettyName>cisco_umbrella</prettyName>
   <version>1.0</version>
   <author>eric_partington</author>
   <description>FileCollection specification for eventsource type "Cisco Umbrella" using file handler type "cisco_umbrella"</description>
 
        <device>
                <name>cisco_umbrella</name>
                <parser>cisco_umbrella</parser>
        </device>
 
        <configuration>
        </configuration>
 
        <collection>
                <file>
                <parserId>file.cisco_umbrella</parserId>
                <processorType>generic</processorType>
                <dataStartLine>1</dataStartLine>
                        <fieldDelim>,</fieldDelim>
                        <idField></idField>
                        <lineDelim>\n</lineDelim>
                        <transformPrefixTag>cisco_umbrella_logs</transformPrefixTag>
                        <transformReplaceFieldDelim>0</transformReplaceFieldDelim>
                        <transformPrefixFilename>0</transformPrefixFilename>
                        <transformMultipleDelimiterAsOne>0</transformMultipleDelimiterAsOne>
                        <transformReplacementFieldDelim></transformReplacementFieldDelim>
                </file>
        </collection>
</typespec>

 

Upload the typespec file to the log collector

/etc/netwitness/ng/logcollection/content/collection/file

set the name as cisco_umbrella.xml

make sure permissions are set right (same as the other files in the directory)

 

Restart the log collector service

 

Create Event Source

In the log collector UI Set up collection

Log collector > Config > Event Sources > File > New Event Category

call it cisco_umbrella

Create a new source

File Directory is cisco_umbrella (this is the directory inside the uploads directory where files will be placed for this collection)

Address - this set the device.ip of this collection so set it to the IP of the cloud system potentially so you have a good record in device.ip of where logs 'came from' originally

 

Review filesystem for uploads directory created for this collection

/var/netwitness/logcollector/upload/

 

Start file collection mechanism on log collector

 

Upload test file

review that the collection mechanism works

place the csv in this directory for collection to take place

/var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella

 

Review Logs

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkGrp[1]:64008] [getWork:651] [cisco_umbrella.cisco_umbrella] [idle] Work Unit Given to Work Manager: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1073] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File processed successfully: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv

Jan  8 20:53:49 nw11ldecoder NwLogCollector[62493]: [FileCollection] [info] [file:WrkUnit[2]:64010] [postWork:1104] [cisco_umbrella.cisco_umbrella] [processing] [generic:2017-12-13-00-10-dcd8.csv] [processing success] File deleted: /var/netwitness/logcollector/upload/cisco_umbrella/cisco_umbrella/work/2017-12-13-00-10-dcd8.csv

 

View investigator

 

Default Parsing with no parser

 

New features added in NW11.0 are now included in the log decoders which are the lua parsers below...these help in the best effort parsing of messages to try to locate useful information from logs and parse them out.

 

 

With no parser created pulls out some information from these logs by default (new feature in NW11.0)

ip.addr, alias.host, filename, sld and tld.

 

 

Create Log Parser

Category for the logs will be Web Logs

Define the header with the LPT1.0 tool (or notepad++)

this is the raw log data as the decoder sees it

%cisco_umbrella-4: "2017-12-13 00:08:01","DC

 

Define the additional fields to parse out according to the columns in the CSV and the need for the data.

 

<?xml version="1.0" encoding="UTF-8"?>
<DEVICEMESSAGES
                 name="cisco_umbrella"
                 displayname="Cisco Umbrella:custom"
                 group="Web Logs"
                 type="7104">
<VERSION
                 xml="1"
                 revision="1"
                 device="2.0"/>
<TAGVALMAP/>
<HEADER
                 id1="HDR1"
                 id2="HDR1"
                 content="%cisco_umbrella_&lt;messageid&gt;-4:&lt;!payload&gt;"/>

<MESSAGE
                 id1="logs"
                 id2="logs"         
     eventcategory="1612000000"                content="&quot;&lt;event_time_string&gt;&quot;,&quot;&lt;rulename&gt;&quot;,&quot;&lt;saddr&gt;&quot;,&quot;&lt;stransaddr&gt;&quot;,&quot;&lt;daddr&gt;&quot;,&quot;&lt;content_type&gt;&quot;,&quot;&lt;action&gt;&quot;,&quot;&lt;url&gt;&quot;,&quot;&lt;web_referer&gt;&quot;,&quot;&lt;user_agent&gt;&quot;,&quot;&lt;resultcode&gt;&quot;,&quot;&lt;fld1&gt;&quot;,&quot;&lt;fld2&gt;&quot;,&quot;&lt;fld3&gt;&quot;,&quot;&lt;uid&gt;&quot;,&quot;&lt;fld4&gt;&quot;,&quot;&lt;fld5&gt;&quot;,&quot;&lt;fld6&gt;&quot;,&quot;&lt;fld7&gt;&quot;,&quot;&lt;fld8&gt;&quot;,&quot;&lt;fld9&gt;&quot;,&quot;&lt;group&gt;&quot;"/>
</DEVICEMESSAGES>

 

Verified in the LPT1.0 tool

 

save the file as cisco_umbrellamsg.xml

create the cisco_umbrella.ini for his device as well

 

DatabaseName=cisco_umbrella
DisplayName=cisco_umbrella
DeviceGroup=Web Logs
DeviceType=7104

 

save it in this directory structure for easy upload to the log decoder

/etc/devices/cisco_umbrella/

place the xml and ini in this directory

 

zip archive the structure and rename the etc.zip as cisco_umbrella.envision

 

Upload the parser

Log Decoder > Config > Parsers

upload

check the filesystem to make sure the permissions are set right on the folder and files (same as the other files in the directory structure

/etc/netwitness/ng/envision/etc/devices/cisco_umbrella/

 

Reload the Parsers

Log Decoder > Explore menu

Decoder > Parsers - right click - select properties

reload - submit

Review the logs to ensure the parser was loaded with no errors

Cat /var/log/messages | grep –i cisco_umbrella

Should show up in the log decoder parsers list when enabled correctly

 

Upload new file to test collection and new parsing

place in same folder location as before

 

Review Parsing

 

parsing looks good!

review the data that was parsed and what is visible on the decoder/concentrator ( keys like referer and url may not be indexed by default depending on what other customizations have been done those may need to be added to table-map.-custom.xml and index-concentrator-custom.xml)

Outcomes