Log Streaming record format

Updated 4 months ago by admin

Web Security - Web Hits

[
{
"visit_id": 30158838,
"utc_timestamp": "2021-06-17 09:08:37.334087",
"tz_offset": -60,
"url_domain": "the.earth.li",
"samaccountname": "james.smith",
"web_categories": "10460",
"url_categories": null,
"keyword_categories": null,
"keywords": [
"putty"
],
"final_action": "Allowed",
"log_level": "High",
"device_type": "Desktop",
"ip_address": "192.168.5.136",
"mac_address": "08:71:90:F6:8D:5D",
"tag": "Managed",
"dst_ip_address": "93.93.131.124",
"http_method": "GET",
"url_scheme": "https",
"url_path": "/~sgtatham/putty/latest/w64/putty-64bit-0.75-installer.msi",
"url_query": "dc=8738743",
"icap_agent": "WindowsPC 4.3.20.5596",
"feature_control_name": null,
"filter_rule_name": "Block Unsafe Files",
"netbios_domain": "DESKTOP-1ME50IR",
"browser": "Chrome 91.0.4472.77",
"operating_system": "Windows 10",
"module_name": null,
"category_name": null,
"pattern": null,
"response_action": null,
"threshold": null,
"score": null,
"web_category_names": [
"Technology - Other"
],
"url_category_names": [],
"keyword_category_names": [],
"hostname": false,
"count": 1288
}
]

Schema Description

Field

Type

Length

Description

visit_id

int

64

Reference to grouped hits that form part of this visit. See Terminology.

utc_timestamp

timestamp

Date and time the request was received, in UTC.

tz_offset

int

32

The offset in minutes from UTC when the request was received (can be positive or negative). Used to generate a local timestamp for the request.

netbios_domain

string

256

The domain part of the captured username string. Depending on the authentication mode, this will either be the NetBIOS name or the domain part of the UPN. If no domain is found, this will default to the device hostname.

samaccountname

string

256

The captured username that originated the request. Depending on the authentication mode, this will either be the SAM account name or the first part of the UPN.

tag

string

64

An arbitrary text string applied to the request through the use of Tags.

ip_address

string

64

The local IPv4 address that generated the request.

mac_address

string

32

The local MAC address that generated the request.

dst_ip_address

string

64

The remote IPv4 address of the url_domain.

http_method

string

32

The HTTP method of the request. Possible values: GET, PUT, POST, DELETE, OPTIONS, PATCH

url_scheme

string

32

The HTTP protocol scheme. Possible options: http, https

url_domain

string

1024

The bare domain from the request, without any protocol, path or query string.

url_path

string

2048

The path part of the requested URL.

url_query

string

2048

The query string part of the requested URL.

browser

string

64

The browser name and version that generated the request e.g. Chrome 91.0.4472.101 (determined from User Agent)

operating_system

string

64

The operating system name that generated the request e.g. Windows 10 (determined from User Agent)

device_type

string

64

The type of device that generated the request e.g. Windows 10 (determined from User Agent)

xhr_flag

int

32

Determines if the request contains an XHR (AJAX) header. Possible values: 0, 1

icap_agent

string

32

The version string of the installed agent or gateway, e.g. WindowsPC 4.3.20.5596, Gateway 1.2.45

feature_control_name

string

128

The name of the Feature Control rule that matched the request.

filter_rule_name

string

128

The name of the Filter Rule that matched the request.

final_action

string

32

The final action that was applied to the request. Possible values: Allowed, Blocked, Warn, Redirect, Quota

See Rules Engine Concepts.

log_level

int

32

The Log Level that was applied to the request. Possible values: Low, Normal, High, Do Not Log

web_categories

Deprecated.

web_category_names

array

A list of Web Category names for the requested URL.

url_categories

Deprecated.

url_category_names

array

A list of URL Category names for the requested URL.

keyword_categories

Deprecated.

keyword_category_names

array

A list of Keyword Category names for the requested URL.

module_name

text

The name of the scanner that scanned the response content. Possible values: MIME Type, Image Control, Anti Virus(subject to agent/gateway support)

category_name

text

The categorisation from the scanner that scanned the response content e.g. Executable Binary, Adult

pattern

text

If available, the pattern that triggered the response scanner e.g. application/pdf, CoinMiner, EICAR-Test-File

threshold

int

32

Reserved for use by response scanner modules.

score

int

32

Reserved for use by response scanner modules.

hostname

string

128

The hostname of the device if the ip_address or mac_address has a corresponding entry in the Devices list.

count

int

64

Internal use only. The count of the rows returned in the log streaming response.

CASB - Inline Event

[
{
"visit_id": 37090874,
"utc_timestamp": "2019-11-19 14:36:21.472163",
"tz_offset": 0,
"netbios_domain": null,
"samaccountname": null,
"tag": "Test Automation",
"ip_address": "10.0.0.207",
"mac_address": "8C:16:45:60:26:8D",
"dst_ip_address": "216.58.213.100",
"http_method": "GET",
"url_scheme": "https",
"url_domain": "www.google.com",
"url_path": "\/search",
"url_query": "ei=zP3TXd6DEK-E1fAPm72lgAQ&q=censornet&oq=censornet&gs_l=psy-ab.12...0.0..23541...0.0..0.0.0.......0......gws-wiz.xQ9MTpfR4hk&ved=0ahUKEwjemZjbv_blAhUvQhUIHZteCUAQ4dUDCAo",
"browser": "Firefox",
"operating_system": "Windows 10",
"device_type": "Desktop",
"icap_agent": "USS Gateway",
"feature_control_name": null,
"filter_rule_name": "app capture",
"final_action": "Allowed",
"log_level": "High",
"web_categories": "10313",
"url_categories": null,
"app_class": "Search Engine",
"app_name": "Google Search Engine",
"app_action": "Searched on the web",
"app_data": "q=censornet&",
"keyword_categories": "65321 22213",
"keywords": [
"google,censornet",
"search"
],
"web_category_names": [
"Search Engines"
],
"url_category_names": [],
"keyword_category_names": [
"Unknown",
"tomtest"
],
"hostname": null,
"count": 3
}
]
Schema Description

The schema is the same as Web Security with the addition of:

Field

Type

Length

Description

app_class

string

128

The category of Cloud Application that the requested matched, e.g. Cloud Storage

app_name

string

128

The name of the Cloud Application that the request matched, e.g. Dropbox

app_action

string

128

The name of the action performed in the Cloud Application e.g. Upload File

app_data

string

4096

The captured meta data from the Cloud Application action e.g. document1.docx. The type of meta data is dependent on the action.

CASB - API Event

[
{
"event_id": 95978,
"utc_timestamp": "2019-10-04 09:40:35",
"action": "add",
"action_value": "427029_10.jpg",
"user_id": "uZGJpZDpBQURBNUwzQTcxX3FXQkFxdF9vS0xEM3laWmtiVkdZNlI5UQ==",
"object_type": "file",
"path": "\/CASB\/IA Test Images\/427029_10.jpg",
"parent_name": "IA Test Images",
"mime_type": "image\/jpeg",
"ip_address": "2.26.223.28",
"user_name": "Trevor Leeds",
"user_email": "tl@domaintest.com",
"service_name": "dropbox",
"threats": [
{
"threat_type": "Adult",
"threat_description": null,
"filename": "427029_10.jpg",
"archive_path": "427029_10.jpg",
"threat_detail": "100% certainty",
"scanner": "ia"
}
]
}
]
Schema Description

Field

Type

Length

Description

event_id

int

64

A unique reference for this CASB API event.

utc_timestamp

timestamp

Date and time the event was received, in UTC.

action

string

256

The type of action carried out, possibile values: add, delete, share_link, update, move, view, download, login, preview, group_add, group_add_user, collaboration_add, collaboration_remove, rename, restore, copy, clean, permission_create, unknown

action_value

string

256

The captured meta data for the action e.g. document1.docx

user_id

string

256

The unique user ID of the user.

object_type

string

256

The type of object that the action was performed on. Possible values: file, folder, Unknown

path

string

256

The path to the object on the Cloud Storage service e.g. /CASB/IA Test Images/427029_10.jpg

parent_name

string

256

The name of the immediate parent for the object, e.g. IA Test Images

mime_type

string

256

The MIME type of the object e.g. image/jpeg

ip_address

string

256

The IPv4 address of the user that performed the action.

user_name

string

256

The real name / display name of the user that performed the action.

user_email

string

256

The email address of the user that performed the action.

service_name

string

256

The short name for the CASB API connector in use. Possible values: dropbox, box, gdrive, onedrivebiz, sharepoint

threats

array

See Threat Types Array below.

Threat Types Array

Field

Type

Length

Description

threat_type

string

256

The type of threat detected on the file object. Possible values: clean, infected, spyware app, adware app, dialer app, potentially dangerous app, e-mail, dictionary, ssn/* (where * is 2 letter ISO country code), credit card, address/* (where * is 2 letter ISO country code), adult, drugs, gore, swimunder, terrorism, weapons

threat_description

text

Any available meta data about the threat type e.g. EICAR-Test-Virus

filename

text

The filename of the object with the threat e.g document1.docx

archive_path

text

Deprecated.

threat_detail

text

Any available meta data about the cause of the threat, generally used for DLP scanning e.g. bob@acme.com (found 2 times)

scanner

string

32

The scanner that detected the threat. Possible values: av, dlp, ia

Email Security - Email Message

Please ensure your SQS maximum message size is set large enough to receive all of the meta data for each email message.
[{
"direction": "incoming",
"delivery_state": "spam_quarantine",
"email_guid": "322fbd0e-c5be-49bb-acac-2a31e87f71b4",
"email_size": "38737",
"originating_ip": "209.85.215.201",
"from_address": "172-GOP-811.0.1650.0.0.8792.9.4585932@google.com",
"to_addresses": [{
"to_address": "james.clerkmaxwell@sssclient.com"
}],
"cc_addresses": [],
"bcc_addresses": [],
"received_date": "2019-12-09 16:30:10+00",
"subject": "Just Published - The 2019 Data Review",
"attachments": [],
"verdict": "spam",
"verdict_details": "Score is 206",
"urls": [{
"url": "https:\/\/fonts.googleapis.com\/css?family=Roboto"
},
{
"url": "https:\/\/fonts.googleapis.com\/css?family=Roboto:100,100i,300,300i,400,400i,500,500i,700,700i&display=swap"
},
{
"url": "https:\/\/fonts.googleapis.com\/css?family=Google+Sans"
}
],
"rules": [{
"rule_data": "DKIM Pass or SPF Pass",
"recipients": [
"ALL"
],
"rule_action": "DMARC Verifying Pass",
"final_action": 0,
"rule_data_name": "DKIM Pass or SPF Pass",
"rule_description": "(Default) Signature Verification"
},
{
"rule_data": "Authentication-Results: in-stage-01; spf=pass smtp.mailfrom=172-GOP-811.0.1650.0.0.8792.9.4585932@google.com; dkim=pass header.i=@google.com; dmarc=pass action=reject header.from=google.com;",
"recipients": [
"ALL"
],
"rule_action": "Add Message Header",
"final_action": 0,
"rule_data_name": "DKIMVereficationHeader",
"rule_description": "(Default) Signature Verification"
},
{
"rule_data": "105",
"recipients": [
"ALL"
],
"rule_action": "Add to Spam Score",
"final_action": 0,
"rule_data_name": "105",
"rule_description": "(Default) CoreService Suspect"
},
{
"rule_data": "101",
"recipients": [
"ALL"
],
"rule_action": "Add to Spam Score",
"final_action": 0,
"rule_data_name": "101",
"rule_description": "Customers - Newsletter Spam"
},
{
"rule_data": "[Marketing Medium]",
"recipients": [
"ALL"
],
"rule_action": "Prefix Text to Subject",
"final_action": 0,
"rule_data_name": "[Marketing Medium]",
"rule_description": "Medium Reputation Marketing"
},
{
"rule_data": "12046",
"recipients": [
"ALL"
],
"rule_action": "Quarantine",
"final_action": 1,
"rule_data_name": "Spam",
"rule_description": "Possible Spam"
}
],
"email_headers": [{
"header_name": "Authentication-Results",
"header_value": " in-stage-01; spf=pass smtp.mailfrom=172-GOP-811.0.1650.0.0.8792.9.4585932@google.com; dkim=pass header.i=@google.com; dmarc=pass action=reject header.from=google.com;"
},
{
"header_name": "Received",
"header_value": "from mail-pg1-f201.google.com ([209.85.215.201]) by IN-STAGE-01 over TLS secured channel with Microsoft SMTPSVC(8.5.9600.16384);\t Mon, 9 Dec 2019 16:30:09 +0000"
},
{
"header_name": "Received",
"header_value": "by mail-pg1-f201.google.com with SMTP id g20so7667944pgb.18 for <james.clerkmaxwell@sssclient.com>; Mon, 09 Dec 2019 08:30:09 -0800 (PST)"
},
{
"header_name": "DKIM-Signature",
"header_value": "v=1; a=rsa-sha256; c=relaxed\/relaxed; d=google.com; s=20161025; h=date:from:reply-to:to:message-id:subject:mime-version :list-unsubscribe; bh=BiFc3n2znjncUk2yyyg+H0w9Y+UzqJ+xJ4bVSoZErDc=; b=pCnJBuv+Gef5Z0DnH21J4jxAnTMFR8a2XD6K6OwKuam3+aFlBqw+\/J2Z5B8ZAnoAgc J0YKWGC68J0I1C3Kto6s4orXZP5onkoEU\/IszjfG7WAebgacTIsStY7tZ3H2c1SqZxih Kx2C8fa7oac1MZIHGzq\/knbEIeXjSGCOpWlwfjSLxjOZCiSJvlNMZfGYY1oTGJgFoi3F IPIpvGkcog2CkDs6btPhXb8Z4pU03wenedbNHcn2dd\/v9EtvgbgV2Dfa4bsPrhr1pxVm i7ykV+sBZqZdywpTAwGfpFLgRH88rJvdUtRtBI0EGfppmLH0b8MdGSkTeDJliJSjCrrw mlsQ=="
},
{
"header_name": "X-Google-DKIM-Signature",
"header_value": "v=1; a=rsa-sha256; c=relaxed\/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:reply-to:to:message-id:subject :mime-version:list-unsubscribe; bh=BiFc3n2znjncUk2yyyg+H0w9Y+UzqJ+xJ4bVSoZErDc=; b=W657VAMoar8hF2PTEzZDL4JGpJq5z2ZId7dAlkICrs8pkDqJbd1yQz8CbXvMfcYn3r QevJXpN2oi9X42kCJsR+8LDr\/DPTdgST1gGhHqKYu2gtIPNI6P3Eh3pTwKPjMQrVyC\/2 kJGvSA+fgildp+3PLbj0X+TREWb84KyfHHEsoKAkOUpw51gEnZ\/kqlsZ85QSxWjphyCD ph34dmLquUASaPo6l4YgjiiYENe65so4vtMWUTGm2hNNeBf9xF6U5QlyZNIeot5EaNte G62zjXyFf6vF0yzdE8otlinM9CIPbEe+\/ykoEE5kLFLtzJcrP5NWoDLYrxcKjW7YMitZ Fl1w=="
},
{
"header_name": "X-Gm-Message-State",
"header_value": "APjAAAVexFyO6bzpo5o+m1Ph7treFyUyr2lQFgPxK6gybD6t9+RmZPcX\tUXLyf5STMCo6EzHWCkoows1cBw=="
},
{
"header_name": "X-Google-Smtp-Source",
"header_value": "APXvYqzy5Bk9ka7Wibr2qMe5dBBhmhuyjUI2qPXmE5GmuVhWlwtWCHqrKUSrG81JVSG9pCDuBQFjx7yZQayljyA="
},
{
"header_name": "X-Received",
"header_value": "by 2002:a17:90a:2469:: with SMTP id h96mr33443573pje.121.1575909008343; Mon, 09 Dec 2019 08:30:08 -0800 (PST)"
},
{
"header_name": "Return-Path",
"header_value": "<172-GOP-811.0.1650.0.0.8792.9.4585932@google.com>"
},
{
"header_name": "Received",
"header_value": "from gopher.mktdns.com (gopher.mktdns.com. [199.15.215.164]) by gmr-mx.google.com with ESMTPS id j2si5136pfi.1.2019.12.09.08.30.08 for <james.clerkmaxwell@sssclient.com> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128\/128); Mon, 09 Dec 2019 08:30:08 -0800 (PST)"
},
{
"header_name": "Return-Path",
"header_value": "<thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "X-MSFBL",
"header_value": "zePA5fetFyFr4kRw005VALnGAHrxqlJkX\/NqGK19mnQ=|eyJ1IjoiMTcyLUdPUC0\t4MTE6NjQ1OTo1MDczOjExODY5OjA6ODc5Mjo5OjE2NTA6NDU4NTkzMiIsImIiOiJ\tkdnAtMTk5LTE1LTIxNS0xNjQiLCJyIjoiamFtZXMuY2xlcmttYXh3ZWxsQHNzc2N\tsaWVudC5jb20iLCJnIjoiYmctc2pyLTA2In0="
},
{
"header_name": "Received",
"header_value": "from [10.0.15.43] ([10.0.15.43:38376] helo=sjmas03.marketo.org)\tby sjmta02.marketo.org (envelope-from <thinkwithgoogle-noreply@google.com>)\t(ecelerity 4.2.38.62370 r(:)) with ESMTP\tid 08\/94-14989-A567EED5; Mon, 09 Dec 2019 10:29:14 -0600"
},
{
"header_name": "Date",
"header_value": "Mon, 9 Dec 2019 10:29:14 -0600"
},
{
"header_name": "From",
"header_value": "Think with Google <thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "Reply-To",
"header_value": "<thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "To",
"header_value": "<james.clerkmaxwell@sssclient.com>"
},
{
"header_name": "Message-ID",
"header_value": "<1802064205.-995280616.1575908954104.JavaMail.mktmail@sjmas03.marketo.org>"
},
{
"header_name": "Subject",
"header_value": "[Marketing Medium] Just Published - The 2019 Data Review"
},
{
"header_name": "MIME-Version",
"header_value": "1.0"
},
{
"header_name": "Content-Type",
"header_value": "multipart\/alternative"
},
{
"header_name": "X-PVIQ",
"header_value": "mkto-172GOP811-000001-000000-001650"
},
{
"header_name": "X-Binding",
"header_value": "bg-sjr-06"
},
{
"header_name": "X-MarketoID",
"header_value": "172-GOP-811:6459:5073:11869:0:8792:9:1650:4585932"
},
{
"header_name": "X-MktArchive",
"header_value": "false"
},
{
"header_name": "List-Unsubscribe",
"header_value": "<mailto:NRKWE5JUOVAXQUJUKZKVA4LQOF2DMLLDGRIT2PI.1650.8792.9@unsub-sj.mktomail.com>"
},
{
"header_name": "X-Mailfrom",
"header_value": "172-GOP-811.0.1650.0.0.8792.9.4585932@google.com"
},
{
"header_name": "X-MSYS-API",
"header_value": "{\"options\":{\"open_tracking\":false,\"click_tracking\":false}}"
},
{
"header_name": "X-OriginalArrivalTime",
"header_value": "09 Dec 2019 16:30:09.0637 (UTC) FILETIME=[EC6BF150:01D5AEAD]"
}
],
"smtp_conversation": null
}]
Schema Description

Field

Type

Length

Description

direction

string

32

The direction of the email message. Possible values: incoming, outgoing

delivery_state

string

32

The delivery status for the message. Possible values: delivered, spam_quarantine, virus_quarantine, reject, delivery_error, dropped, rejected, delivery_delay

email_guid

string

64

A unique identifier for the email message e.g. 1cd4f6fc-2fce-4a36-a1c7-a8fb3037e95e

email_size

float

24

The size of the email message in bytes.

originating_ip

string

64

The IPv4 address of the sender.

from_address

text

The email address of the sender.

to_address

text

A JSON array of objects in the format {"to_address": "email@domain.com"}

cc_address

text

A JSON array of objects in the format {"cc_address": "email@domain.com"}

bcc_addresses

text

A JSON array of objects in the format {"bcc_address": "email@domain.com"}

received_date

timestamp

The timestamp with timezone that the server received the email message for processing, e.g. 2021-06-17 18:58:45+01

subject

text

The subject of the email message.

attachments

text

See Attachments Array below.

verdict

string

32

Deprecated. Possible values: clean, spam, virus, reject, enduser

verdict_details

text

Deprecated.

urls

text

A JSON array of objects in the format {"url": "https://domain.com/path?query"}

rules

text

See Rules Array below.

email_headers

text

See Email Headers Array below.

smtp_conversation

text

See SMTP Conversation Array below.

Attachments Array

Field

Type

Length

Description

attachment_name

string

The filename of the attachment e.g. document1.docx

attachment_digest

string

The SHA256 hash of the file attachment.

attachment_mimetype

string

The MIME type for the attachment e.g. image/png

Rules Array

Field

Type

Length

Description

rule_data

string

The meta data captured by the rule, e.g. 5.0.1 user unknown

recipients

array

An array of email address strings.

rule_action

string

The rule action that was triggered e.g. Permanent Reject Error

final_action

int

32

Indicates whether the rule was a final action. Possible values: 0, 1

rule_data_name

string

Internal name for the rule_data.

rule_description

string

The name of the rule. See Connection Rules and Message Rules.

Email Headers Array

Field

Type

Length

Description

header_name

string

The header name e.g. Date, Received, Content-Type

header_value

string

The value of the header.

SMTP Conversation Array

Field

Type

Length

Description

date_time

string

Date and Time of the server message in the format DD/MM/YYYY g:i:s A

servery_reply

string

The server message text, e.g. RECEIVE:ehlo sonic313-20.consmr.mail.ir2.yahoo.com

destination_server

string

The IPv4 address of the delivery email server.

Admin Audit - Event

[
{
"utc_timestamp": "2019-11-19 14:35:45",
"username": "tom.moreton@emaildomain.com",
"ip_address": "10.0.0.207",
"model": "KeywordCategory",
"url": "https:\/\/api2.clouduss.com\/keyword_categories",
"method": "PUT",
"payload": {
"original": {
"id": 65321,
"name": "tomtest2",
"description": "",
"match_all_patterns": 0,
"match_url_content": 1,
"match_appdata_content": 1,
"match_dlp_content": 1,
"dlp_enabled": 0
},
"changed": {
"id": 65321,
"name": "jankyqa",
"description": "",
"match_all_patterns": 0,
"match_url_content": 1,
"match_appdata_content": 1,
"match_dlp_content": 1,
"dlp_enabled": 0
}
}
}
]
Schema Description

Field

Type

Length

Description

utc_timestamp

timestamp

Date and time the audit event was received, in UTC.

username

string

256

The authenticated username that performed the audited action.

ip_address

string

128

The IPv4 address of the user that performed the audited action.

model

string

128

A string representing a name for the related API request.

url

string

512

The URL of the API request that relates to the audited action.

method

string

16

The HTTP method of the related API request. Possible values: GET, PUT, POST, DELETE, OPTIONS, PATCH

payload

object

An object containing two further objects: original and changed which contain the API request payload before and after the audited action.


How did we do?