{"id":4575,"date":"2025-03-17T15:47:05","date_gmt":"2025-03-17T14:47:05","guid":{"rendered":"https:\/\/andreas-klingler.de\/infosec\/?p=4575"},"modified":"2025-03-21T13:34:39","modified_gmt":"2025-03-21T12:34:39","slug":"logging-best-practices","status":"publish","type":"post","link":"https:\/\/infosec.andreas-klingler.de\/?p=4575","title":{"rendered":"Logging and data collection best practices"},"content":{"rendered":"\n<ul class=\"wp-block-list\">\n<li>Logs should be pro\u00adtect\u00aded like backups.&nbsp;<ul class=\"wp-block-list\">\n<li>Remote<\/li>\n\n\n\n<li>Mul\u00adti\u00adple versions<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Logs should be read-only<\/li>\n\n\n\n<li>Logs should be aggra\u00adgat\u00aded dynam\u00adi\u00adcal\u00adly when necessary.<\/li>\n\n\n\n<li>Logs should be avail\u00adable also after an attack.<\/li>\n\n\n\n<li>Logs should con\u00adtain use\u00adful&nbsp;data&nbsp;<ul class=\"wp-block-list\">\n<li>A web serv\u00ader logs which has only the request IPs of the fire\u00adwall before it, is not so useful.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>The log for\u00admat should be ver\u00adsa\u00adtile and robust (e.g. <span class=\"caps\">JSONL<\/span> ist bet\u00adter than <span class=\"caps\">CSV<\/span>&nbsp;files)<\/li>\n\n\n\n<li>The time\u00adstamps should be in the same for\u00admat and the same time\u00adzone or con\u00adtain also the timezone.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Ques\u00adtions, who should be answered easily:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Find every\u00adthing what A&nbsp;did.<\/li>\n\n\n\n<li>Find every\u00adthing releat\u00aded to&nbsp;X.<\/li>\n\n\n\n<li>Find events which are not like the others.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Tips for analyzing log&nbsp;files<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Search a log and count the&nbsp;hosts:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cat log | jq -c '.host' | sort | uniq -c | sort -n<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">To see more about the host \u201cApache\u201d<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cat log | grep \"Apache\" | jq<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Search in gzipped logs:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">zgrep log.gz<br>zcat log.gz | grep \"Apache\"<\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Con\u00advert time\u00adstamps on the&nbsp;fly:<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">cat log | jq -c '.ts|=strftime(\"%Y-%m-%d %H:%M:%S %z\")' | jq<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>Ques\u00adtions, who should be answered eas\u00adi\u00adly: Tips for ana\u00adlyz\u00ading log&nbsp;files Search a log and count the&nbsp;hosts: cat log | jq \u2011c \u2018.host\u2019 | sort | uniq \u2011c | sort \u2011n To see more about the host \u201cApache\u201d cat log | grep \u201cApache\u201d | jq Search in gzipped logs: zgrep log.gzzcat log.gz | grep \u201cApache\u201d Convert&nbsp;[\u2026]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"wp_typography_post_enhancements_disabled":false,"footnotes":""},"categories":[476],"tags":[196,197],"class_list":["post-4575","post","type-post","status-publish","format-standard","hentry","category-defence-tool","tag-logging","tag-logs"],"_links":{"self":[{"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/posts\/4575","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4575"}],"version-history":[{"count":9,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/posts\/4575\/revisions"}],"predecessor-version":[{"id":4822,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=\/wp\/v2\/posts\/4575\/revisions\/4822"}],"wp:attachment":[{"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4575"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4575"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/infosec.andreas-klingler.de\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4575"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}