Log Streaming record format
Web Security - Web Hits
[
{
"visit_id": 30158838,
"utc_timestamp": "2021-06-17 09:08:37.334087",
"tz_offset": -60,
"url_domain": "the.earth.li",
"samaccountname": "james.smith",
"web_categories": "10460",
"url_categories": null,
"keyword_categories": null,
"keywords": [
"putty"
],
"final_action": "Allowed",
"log_level": "High",
"device_type": "Desktop",
"ip_address": "192.168.5.136",
"mac_address": "08:71:90:F6:8D:5D",
"tag": "Managed",
"dst_ip_address": "93.93.131.124",
"http_method": "GET",
"url_scheme": "https",
"url_path": "/~sgtatham/putty/latest/w64/putty-64bit-0.75-installer.msi",
"url_query": "dc=8738743",
"icap_agent": "WindowsPC 4.3.20.5596",
"feature_control_name": null,
"filter_rule_name": "Block Unsafe Files",
"netbios_domain": "DESKTOP-1ME50IR",
"browser": "Chrome 91.0.4472.77",
"operating_system": "Windows 10",
"module_name": null,
"category_name": null,
"pattern": null,
"response_action": null,
"threshold": null,
"score": null,
"web_category_names": [
"Technology - Other"
],
"url_category_names": [],
"keyword_category_names": [],
"hostname": false,
"count": 1288
}
]
Schema Description
Field | Type | Length | Description |
visit_id | int | 64 | Reference to grouped hits that form part of this visit. See Terminology. |
utc_timestamp | timestamp | Date and time the request was received, in UTC. | |
tz_offset | int | 32 | The offset in minutes from UTC when the request was received (can be positive or negative). Used to generate a local timestamp for the request. |
netbios_domain | string | 256 | The domain part of the captured username string. Depending on the authentication mode, this will either be the NetBIOS name or the domain part of the UPN. If no domain is found, this will default to the device hostname. |
samaccountname | string | 256 | The captured username that originated the request. Depending on the authentication mode, this will either be the SAM account name or the first part of the UPN. |
tag | string | 64 | An arbitrary text string applied to the request through the use of Tags. |
ip_address | string | 64 | The local IPv4 address that generated the request. |
mac_address | string | 32 | The local MAC address that generated the request. |
dst_ip_address | string | 64 | The remote IPv4 address of the url_domain. |
http_method | string | 32 | The HTTP method of the request. Possible values: |
url_scheme | string | 32 | The HTTP protocol scheme. Possible options: |
url_domain | string | 1024 | The bare domain from the request, without any protocol, path or query string. |
url_path | string | 2048 | The path part of the requested URL. |
url_query | string | 2048 | The query string part of the requested URL. |
browser | string | 64 | The browser name and version that generated the request e.g. Chrome 91.0.4472.101 (determined from User Agent) |
operating_system | string | 64 | The operating system name that generated the request e.g. Windows 10 (determined from User Agent) |
device_type | string | 64 | The type of device that generated the request e.g. Windows 10 (determined from User Agent) |
xhr_flag | int | 32 | Determines if the request contains an XHR (AJAX) header. Possible values: |
icap_agent | string | 32 | The version string of the installed agent or gateway, e.g. WindowsPC 4.3.20.5596, Gateway 1.2.45 |
feature_control_name | string | 128 | The name of the Feature Control rule that matched the request. |
filter_rule_name | string | 128 | The name of the Filter Rule that matched the request. |
final_action | string | 32 | The final action that was applied to the request. Possible values: |
log_level | int | 32 | The Log Level that was applied to the request. Possible values: |
web_categories | Deprecated. | ||
web_category_names | array | A list of Web Category names for the requested URL. | |
url_categories | Deprecated. | ||
url_category_names | array | A list of URL Category names for the requested URL. | |
keyword_categories | Deprecated. | ||
keyword_category_names | array | A list of Keyword Category names for the requested URL. | |
module_name | text | The name of the scanner that scanned the response content. Possible values: | |
category_name | text | The categorisation from the scanner that scanned the response content e.g. Executable Binary, Adult | |
pattern | text | If available, the pattern that triggered the response scanner e.g. application/pdf, CoinMiner, EICAR-Test-File | |
threshold | int | 32 | Reserved for use by response scanner modules. |
score | int | 32 | Reserved for use by response scanner modules. |
hostname | string | 128 | The hostname of the device if the ip_address or mac_address has a corresponding entry in the Devices list. |
count | int | 64 | Internal use only. The count of the rows returned in the log streaming response. |
CASB - Inline Event
[
{
"visit_id": 37090874,
"utc_timestamp": "2019-11-19 14:36:21.472163",
"tz_offset": 0,
"netbios_domain": null,
"samaccountname": null,
"tag": "Test Automation",
"ip_address": "10.0.0.207",
"mac_address": "8C:16:45:60:26:8D",
"dst_ip_address": "216.58.213.100",
"http_method": "GET",
"url_scheme": "https",
"url_domain": "www.google.com",
"url_path": "\/search",
"url_query": "ei=zP3TXd6DEK-E1fAPm72lgAQ&q=censornet&oq=censornet&gs_l=psy-ab.12...0.0..23541...0.0..0.0.0.......0......gws-wiz.xQ9MTpfR4hk&ved=0ahUKEwjemZjbv_blAhUvQhUIHZteCUAQ4dUDCAo",
"browser": "Firefox",
"operating_system": "Windows 10",
"device_type": "Desktop",
"icap_agent": "USS Gateway",
"feature_control_name": null,
"filter_rule_name": "app capture",
"final_action": "Allowed",
"log_level": "High",
"web_categories": "10313",
"url_categories": null,
"app_class": "Search Engine",
"app_name": "Google Search Engine",
"app_action": "Searched on the web",
"app_data": "q=censornet&",
"keyword_categories": "65321 22213",
"keywords": [
"google,censornet",
"search"
],
"web_category_names": [
"Search Engines"
],
"url_category_names": [],
"keyword_category_names": [
"Unknown",
"tomtest"
],
"hostname": null,
"count": 3
}
]
Schema Description
The schema is the same as Web Security with the addition of:
Field | Type | Length | Description |
app_class | string | 128 | The category of Cloud Application that the requested matched, e.g. Cloud Storage |
app_name | string | 128 | The name of the Cloud Application that the request matched, e.g. Dropbox |
app_action | string | 128 | The name of the action performed in the Cloud Application e.g. Upload File |
app_data | string | 4096 | The captured meta data from the Cloud Application action e.g. document1.docx. The type of meta data is dependent on the action. |
CASB - API Event
[
{
"event_id": 95978,
"utc_timestamp": "2019-10-04 09:40:35",
"action": "add",
"action_value": "427029_10.jpg",
"user_id": "uZGJpZDpBQURBNUwzQTcxX3FXQkFxdF9vS0xEM3laWmtiVkdZNlI5UQ==",
"object_type": "file",
"path": "\/CASB\/IA Test Images\/427029_10.jpg",
"parent_name": "IA Test Images",
"mime_type": "image\/jpeg",
"ip_address": "2.26.223.28",
"user_name": "Trevor Leeds",
"user_email": "tl@domaintest.com",
"service_name": "dropbox",
"threats": [
{
"threat_type": "Adult",
"threat_description": null,
"filename": "427029_10.jpg",
"archive_path": "427029_10.jpg",
"threat_detail": "100% certainty",
"scanner": "ia"
}
]
}
]
Schema Description
Field | Type | Length | Description |
event_id | int | 64 | A unique reference for this CASB API event. |
utc_timestamp | timestamp | Date and time the event was received, in UTC. | |
action | string | 256 | The type of action carried out, possibile values: |
action_value | string | 256 | The captured meta data for the action e.g. document1.docx |
user_id | string | 256 | The unique user ID of the user. |
object_type | string | 256 | The type of object that the action was performed on. Possible values: |
path | string | 256 | The path to the object on the Cloud Storage service e.g. /CASB/IA Test Images/427029_10.jpg |
parent_name | string | 256 | The name of the immediate parent for the object, e.g. IA Test Images |
mime_type | string | 256 | The MIME type of the object e.g. image/jpeg |
ip_address | string | 256 | The IPv4 address of the user that performed the action. |
user_name | string | 256 | The real name / display name of the user that performed the action. |
user_email | string | 256 | The email address of the user that performed the action. |
service_name | string | 256 | The short name for the CASB API connector in use. Possible values: |
threats | array | See Threat Types Array below. |
Threat Types Array
Field | Type | Length | Description |
threat_type | string | 256 | The type of threat detected on the file object. Possible values: |
threat_description | text | Any available meta data about the threat type e.g. EICAR-Test-Virus | |
filename | text | The filename of the object with the threat e.g document1.docx | |
archive_path | text | Deprecated. | |
threat_detail | text | Any available meta data about the cause of the threat, generally used for DLP scanning e.g. bob@acme.com (found 2 times) | |
scanner | string | 32 | The scanner that detected the threat. Possible values: |
Email Security - Email Message
[{
"direction": "incoming",
"delivery_state": "spam_quarantine",
"email_guid": "322fbd0e-c5be-49bb-acac-2a31e87f71b4",
"email_size": "38737",
"originating_ip": "209.85.215.201",
"from_address": "172-GOP-811.0.1650.0.0.8792.9.4585932@google.com",
"to_addresses": [{
"to_address": "james.clerkmaxwell@sssclient.com"
}],
"cc_addresses": [],
"bcc_addresses": [],
"received_date": "2019-12-09 16:30:10+00",
"subject": "Just Published - The 2019 Data Review",
"attachments": [],
"verdict": "spam",
"verdict_details": "Score is 206",
"urls": [{
"url": "https:\/\/fonts.googleapis.com\/css?family=Roboto"
},
{
"url": "https:\/\/fonts.googleapis.com\/css?family=Roboto:100,100i,300,300i,400,400i,500,500i,700,700i&display=swap"
},
{
"url": "https:\/\/fonts.googleapis.com\/css?family=Google+Sans"
}
],
"rules": [{
"rule_data": "DKIM Pass or SPF Pass",
"recipients": [
"ALL"
],
"rule_action": "DMARC Verifying Pass",
"final_action": 0,
"rule_data_name": "DKIM Pass or SPF Pass",
"rule_description": "(Default) Signature Verification"
},
{
"rule_data": "Authentication-Results: in-stage-01; spf=pass smtp.mailfrom=172-GOP-811.0.1650.0.0.8792.9.4585932@google.com; dkim=pass header.i=@google.com; dmarc=pass action=reject header.from=google.com;",
"recipients": [
"ALL"
],
"rule_action": "Add Message Header",
"final_action": 0,
"rule_data_name": "DKIMVereficationHeader",
"rule_description": "(Default) Signature Verification"
},
{
"rule_data": "105",
"recipients": [
"ALL"
],
"rule_action": "Add to Spam Score",
"final_action": 0,
"rule_data_name": "105",
"rule_description": "(Default) CoreService Suspect"
},
{
"rule_data": "101",
"recipients": [
"ALL"
],
"rule_action": "Add to Spam Score",
"final_action": 0,
"rule_data_name": "101",
"rule_description": "Customers - Newsletter Spam"
},
{
"rule_data": "[Marketing Medium]",
"recipients": [
"ALL"
],
"rule_action": "Prefix Text to Subject",
"final_action": 0,
"rule_data_name": "[Marketing Medium]",
"rule_description": "Medium Reputation Marketing"
},
{
"rule_data": "12046",
"recipients": [
"ALL"
],
"rule_action": "Quarantine",
"final_action": 1,
"rule_data_name": "Spam",
"rule_description": "Possible Spam"
}
],
"email_headers": [{
"header_name": "Authentication-Results",
"header_value": " in-stage-01; spf=pass smtp.mailfrom=172-GOP-811.0.1650.0.0.8792.9.4585932@google.com; dkim=pass header.i=@google.com; dmarc=pass action=reject header.from=google.com;"
},
{
"header_name": "Received",
"header_value": "from mail-pg1-f201.google.com ([209.85.215.201]) by IN-STAGE-01 over TLS secured channel with Microsoft SMTPSVC(8.5.9600.16384);\t Mon, 9 Dec 2019 16:30:09 +0000"
},
{
"header_name": "Received",
"header_value": "by mail-pg1-f201.google.com with SMTP id g20so7667944pgb.18 for <james.clerkmaxwell@sssclient.com>; Mon, 09 Dec 2019 08:30:09 -0800 (PST)"
},
{
"header_name": "DKIM-Signature",
"header_value": "v=1; a=rsa-sha256; c=relaxed\/relaxed; d=google.com; s=20161025; h=date:from:reply-to:to:message-id:subject:mime-version :list-unsubscribe; bh=BiFc3n2znjncUk2yyyg+H0w9Y+UzqJ+xJ4bVSoZErDc=; b=pCnJBuv+Gef5Z0DnH21J4jxAnTMFR8a2XD6K6OwKuam3+aFlBqw+\/J2Z5B8ZAnoAgc J0YKWGC68J0I1C3Kto6s4orXZP5onkoEU\/IszjfG7WAebgacTIsStY7tZ3H2c1SqZxih Kx2C8fa7oac1MZIHGzq\/knbEIeXjSGCOpWlwfjSLxjOZCiSJvlNMZfGYY1oTGJgFoi3F IPIpvGkcog2CkDs6btPhXb8Z4pU03wenedbNHcn2dd\/v9EtvgbgV2Dfa4bsPrhr1pxVm i7ykV+sBZqZdywpTAwGfpFLgRH88rJvdUtRtBI0EGfppmLH0b8MdGSkTeDJliJSjCrrw mlsQ=="
},
{
"header_name": "X-Google-DKIM-Signature",
"header_value": "v=1; a=rsa-sha256; c=relaxed\/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:reply-to:to:message-id:subject :mime-version:list-unsubscribe; bh=BiFc3n2znjncUk2yyyg+H0w9Y+UzqJ+xJ4bVSoZErDc=; b=W657VAMoar8hF2PTEzZDL4JGpJq5z2ZId7dAlkICrs8pkDqJbd1yQz8CbXvMfcYn3r QevJXpN2oi9X42kCJsR+8LDr\/DPTdgST1gGhHqKYu2gtIPNI6P3Eh3pTwKPjMQrVyC\/2 kJGvSA+fgildp+3PLbj0X+TREWb84KyfHHEsoKAkOUpw51gEnZ\/kqlsZ85QSxWjphyCD ph34dmLquUASaPo6l4YgjiiYENe65so4vtMWUTGm2hNNeBf9xF6U5QlyZNIeot5EaNte G62zjXyFf6vF0yzdE8otlinM9CIPbEe+\/ykoEE5kLFLtzJcrP5NWoDLYrxcKjW7YMitZ Fl1w=="
},
{
"header_name": "X-Gm-Message-State",
"header_value": "APjAAAVexFyO6bzpo5o+m1Ph7treFyUyr2lQFgPxK6gybD6t9+RmZPcX\tUXLyf5STMCo6EzHWCkoows1cBw=="
},
{
"header_name": "X-Google-Smtp-Source",
"header_value": "APXvYqzy5Bk9ka7Wibr2qMe5dBBhmhuyjUI2qPXmE5GmuVhWlwtWCHqrKUSrG81JVSG9pCDuBQFjx7yZQayljyA="
},
{
"header_name": "X-Received",
"header_value": "by 2002:a17:90a:2469:: with SMTP id h96mr33443573pje.121.1575909008343; Mon, 09 Dec 2019 08:30:08 -0800 (PST)"
},
{
"header_name": "Return-Path",
"header_value": "<172-GOP-811.0.1650.0.0.8792.9.4585932@google.com>"
},
{
"header_name": "Received",
"header_value": "from gopher.mktdns.com (gopher.mktdns.com. [199.15.215.164]) by gmr-mx.google.com with ESMTPS id j2si5136pfi.1.2019.12.09.08.30.08 for <james.clerkmaxwell@sssclient.com> (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128\/128); Mon, 09 Dec 2019 08:30:08 -0800 (PST)"
},
{
"header_name": "Return-Path",
"header_value": "<thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "X-MSFBL",
"header_value": "zePA5fetFyFr4kRw005VALnGAHrxqlJkX\/NqGK19mnQ=|eyJ1IjoiMTcyLUdPUC0\t4MTE6NjQ1OTo1MDczOjExODY5OjA6ODc5Mjo5OjE2NTA6NDU4NTkzMiIsImIiOiJ\tkdnAtMTk5LTE1LTIxNS0xNjQiLCJyIjoiamFtZXMuY2xlcmttYXh3ZWxsQHNzc2N\tsaWVudC5jb20iLCJnIjoiYmctc2pyLTA2In0="
},
{
"header_name": "Received",
"header_value": "from [10.0.15.43] ([10.0.15.43:38376] helo=sjmas03.marketo.org)\tby sjmta02.marketo.org (envelope-from <thinkwithgoogle-noreply@google.com>)\t(ecelerity 4.2.38.62370 r(:)) with ESMTP\tid 08\/94-14989-A567EED5; Mon, 09 Dec 2019 10:29:14 -0600"
},
{
"header_name": "Date",
"header_value": "Mon, 9 Dec 2019 10:29:14 -0600"
},
{
"header_name": "From",
"header_value": "Think with Google <thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "Reply-To",
"header_value": "<thinkwithgoogle-noreply@google.com>"
},
{
"header_name": "To",
"header_value": "<james.clerkmaxwell@sssclient.com>"
},
{
"header_name": "Message-ID",
"header_value": "<1802064205.-995280616.1575908954104.JavaMail.mktmail@sjmas03.marketo.org>"
},
{
"header_name": "Subject",
"header_value": "[Marketing Medium] Just Published - The 2019 Data Review"
},
{
"header_name": "MIME-Version",
"header_value": "1.0"
},
{
"header_name": "Content-Type",
"header_value": "multipart\/alternative"
},
{
"header_name": "X-PVIQ",
"header_value": "mkto-172GOP811-000001-000000-001650"
},
{
"header_name": "X-Binding",
"header_value": "bg-sjr-06"
},
{
"header_name": "X-MarketoID",
"header_value": "172-GOP-811:6459:5073:11869:0:8792:9:1650:4585932"
},
{
"header_name": "X-MktArchive",
"header_value": "false"
},
{
"header_name": "List-Unsubscribe",
"header_value": "<mailto:NRKWE5JUOVAXQUJUKZKVA4LQOF2DMLLDGRIT2PI.1650.8792.9@unsub-sj.mktomail.com>"
},
{
"header_name": "X-Mailfrom",
"header_value": "172-GOP-811.0.1650.0.0.8792.9.4585932@google.com"
},
{
"header_name": "X-MSYS-API",
"header_value": "{\"options\":{\"open_tracking\":false,\"click_tracking\":false}}"
},
{
"header_name": "X-OriginalArrivalTime",
"header_value": "09 Dec 2019 16:30:09.0637 (UTC) FILETIME=[EC6BF150:01D5AEAD]"
}
],
"smtp_conversation": null
}]
Schema Description
Field | Type | Length | Description |
direction | string | 32 | The direction of the email message. Possible values: |
delivery_state | string | 32 | The delivery status for the message. Possible values: |
email_guid | string | 64 | A unique identifier for the email message e.g. 1cd4f6fc-2fce-4a36-a1c7-a8fb3037e95e |
email_size | float | 24 | The size of the email message in bytes. |
originating_ip | string | 64 | The IPv4 address of the sender. |
from_address | text | The email address of the sender. | |
to_address | text | A JSON array of objects in the format {"to_address": "email@domain.com"} | |
cc_address | text | A JSON array of objects in the format {"cc_address": "email@domain.com"} | |
bcc_addresses | text | A JSON array of objects in the format {"bcc_address": "email@domain.com"} | |
received_date | timestamp | The timestamp with timezone that the server received the email message for processing, e.g. 2021-06-17 18:58:45+01 | |
subject | text | The subject of the email message. | |
attachments | text | See Attachments Array below. | |
verdict | string | 32 | Deprecated. Possible values: |
verdict_details | text | Deprecated. | |
urls | text | A JSON array of objects in the format {"url": "https://domain.com/path?query"} | |
rules | text | See Rules Array below. | |
email_headers | text | See Email Headers Array below. | |
smtp_conversation | text | See SMTP Conversation Array below. |
Attachments Array
Field | Type | Length | Description |
attachment_name | string | The filename of the attachment e.g. document1.docx | |
attachment_digest | string | The SHA256 hash of the file attachment. | |
attachment_mimetype | string | The MIME type for the attachment e.g. image/png |
Rules Array
Field | Type | Length | Description |
rule_data | string | The meta data captured by the rule, e.g. 5.0.1 user unknown | |
recipients | array | An array of email address strings. | |
rule_action | string | The rule action that was triggered e.g. Permanent Reject Error | |
final_action | int | 32 | Indicates whether the rule was a final action. Possible values: |
rule_data_name | string | Internal name for the rule_data. | |
rule_description | string | The name of the rule. See Connection Rules and Message Rules. |
Email Headers Array
Field | Type | Length | Description |
header_name | string | The header name e.g. Date, Received, Content-Type | |
header_value | string | The value of the header. |
SMTP Conversation Array
Field | Type | Length | Description |
date_time | string | Date and Time of the server message in the format | |
servery_reply | string | The server message text, e.g. RECEIVE:ehlo sonic313-20.consmr.mail.ir2.yahoo.com | |
destination_server | string | The IPv4 address of the delivery email server. |
Admin Audit - Event
[
{
"utc_timestamp": "2019-11-19 14:35:45",
"username": "tom.moreton@emaildomain.com",
"ip_address": "10.0.0.207",
"model": "KeywordCategory",
"url": "https:\/\/api2.clouduss.com\/keyword_categories",
"method": "PUT",
"payload": {
"original": {
"id": 65321,
"name": "tomtest2",
"description": "",
"match_all_patterns": 0,
"match_url_content": 1,
"match_appdata_content": 1,
"match_dlp_content": 1,
"dlp_enabled": 0
},
"changed": {
"id": 65321,
"name": "jankyqa",
"description": "",
"match_all_patterns": 0,
"match_url_content": 1,
"match_appdata_content": 1,
"match_dlp_content": 1,
"dlp_enabled": 0
}
}
}
]
Schema Description
Field | Type | Length | Description |
utc_timestamp | timestamp | Date and time the audit event was received, in UTC. | |
username | string | 256 | The authenticated username that performed the audited action. |
ip_address | string | 128 | The IPv4 address of the user that performed the audited action. |
model | string | 128 | A string representing a name for the related API request. |
url | string | 512 | The URL of the API request that relates to the audited action. |
method | string | 16 | The HTTP method of the related API request. Possible values: |
payload | object | An object containing two further objects: |