When I visit organizations, I’m often asked what should be logged, how long the data should be retained, and how it should be reviewed. These are interesting questions, as logging is very important for security and process improvement reasons.
A good starting point is the SP800-14 NIST special publication on securing computer systems. It covers audit trails from a high-level perspective. The intent of this article is to highlight and explore some of the audit log concepts laid out in the publication.
Before we begin, there is a very important caveat to bear in mind – log data doesn’t help if it isn’t reviewed! As today’s systems are often generating thousands of lines of data per day, if not per hour, manual review isn’t very realistic. Log centralization and analysis tools should be used to automatically alert on certain conditions as well as help facilitate meaningful log review. Let’s focus now on the details.
Inevitably, someone asks why event data should be logged on a given system. Essentially there are four categories of reasons:
Essentially, for each system monitored and likely event condition there must be enough data logged for determinations to be made. At a minimum, you need to be able to answer the standard who, what and when questions.
The data logged must be retained long enough to answer questions, but not indefinitely. Storage space costs money and at a certain point, depending on the data, the cost of storage is greater than the probable value of the log data.
The same can be said for costs associated with performance degradation that the log analysis tools suffer if the data sets are simply allowed to grow indefinitely.
For the log data to be useful, it must be secured from unauthorized access and integrity problems. This means there should be proper segregation of duties between those who administer system/network accounts and those who can access the log data.
The idea is to not have someone who can do both or else the risk, real or perceived, is that an account can be created for malicious purposes, activity performed, the account deleted and then the logs altered to not show what happened. Bottom-line, access to the logs must be restricted to ensure their integrity. This necessitates access controls as well as the use of hardened systems.
Consideration must be given to the location of the logs as well – moving logs to a central spot or at least off the sample platform can give added security in the event that a given platform fails or is compromised. In other words, if system X has catastrophic failure and the log data is on X, then the most recent log data may be lost. However, if X’s data is stored on Y, then if X fails, the log data isn’t lost and can be immediately available for analysis. This can apply to hosts within a data center as well as across data centers when geographic redundancy is viewed as important.
The trick is to understand what will be logged for each system. Log review is a control put in place to mitigate risks to an acceptable level. The intent is to only log what is necessary and to be able to ensure that management agrees, which means talking to each system’s stakeholders. Be sure to involve IT operations, security, end-user support, the business and the legal department.
Work with the stakeholders and populate a matrix wherein each system is listed and then details are spelled out in terms of: what data must be logged for security and operational considerations, how long it will be retained, how it will be destroyed, who should have access, who will be responsible to review it, how often it will be reviewed and how the review will be evidenced. The latter is from a compliance perspective – if log reviews are a required control, how can they be evidenced to auditors?
Finally, be sure to get senior management to formally approve the matrix, associated policies and procedures. The idea is to be able to attest both that reviews are happening and that senior management agrees with the activity being performed.
Audit logs are beneficial to have for a number of reasons. To be effective, IT must understand log requirements for each system, then document what will be logged for each system and get management’s approval. This will reduce ambiguity over the details of logging and facilitate proper management.
Note: A sample log matrix template will be placed on the author’s website at http://www.spaffordconsulting.com/articles.html next to the link to this article.
Huawei’s AI Update: Things Are Moving Faster Than We Think
FEATURE | By Rob Enderle,
December 04, 2020
Keeping Machine Learning Algorithms Honest in the ‘Ethics-First’ Era
ARTIFICIAL INTELLIGENCE | By Guest Author,
November 18, 2020
Key Trends in Chatbots and RPA
FEATURE | By Guest Author,
November 10, 2020
FEATURE | By Samuel Greengard,
November 05, 2020
ARTIFICIAL INTELLIGENCE | By Guest Author,
November 02, 2020
How Intel’s Work With Autonomous Cars Could Redefine General Purpose AI
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
October 29, 2020
Dell Technologies World: Weaving Together Human And Machine Interaction For AI And Robotics
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
October 23, 2020
The Super Moderator, or How IBM Project Debater Could Save Social Media
FEATURE | By Rob Enderle,
October 16, 2020
FEATURE | By Cynthia Harvey,
October 07, 2020
ARTIFICIAL INTELLIGENCE | By Guest Author,
October 05, 2020
CIOs Discuss the Promise of AI and Data Science
FEATURE | By Guest Author,
September 25, 2020
Microsoft Is Building An AI Product That Could Predict The Future
FEATURE | By Rob Enderle,
September 25, 2020
Top 10 Machine Learning Companies 2020
FEATURE | By Cynthia Harvey,
September 22, 2020
NVIDIA and ARM: Massively Changing The AI Landscape
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
September 18, 2020
Continuous Intelligence: Expert Discussion [Video and Podcast]
ARTIFICIAL INTELLIGENCE | By James Maguire,
September 14, 2020
Artificial Intelligence: Governance and Ethics [Video]
ARTIFICIAL INTELLIGENCE | By James Maguire,
September 13, 2020
IBM Watson At The US Open: Showcasing The Power Of A Mature Enterprise-Class AI
FEATURE | By Rob Enderle,
September 11, 2020
Artificial Intelligence: Perception vs. Reality
FEATURE | By James Maguire,
September 09, 2020
Anticipating The Coming Wave Of AI Enhanced PCs
FEATURE | By Rob Enderle,
September 05, 2020
The Critical Nature Of IBM’s NLP (Natural Language Processing) Effort
ARTIFICIAL INTELLIGENCE | By Rob Enderle,
August 14, 2020
Datamation is the leading industry resource for B2B data professionals and technology buyers. Datamation's focus is on providing insight into the latest trends and innovation in AI, data security, big data, and more, along with in-depth product recommendations and comparisons. More than 1.7M users gain insight and guidance from Datamation every year.
Advertise with TechnologyAdvice on Datamation and our other data and technology-focused platforms.
Advertise with Us
Property of TechnologyAdvice.
© 2025 TechnologyAdvice. All Rights Reserved
Advertiser Disclosure: Some of the products that appear on this
site are from companies from which TechnologyAdvice receives
compensation. This compensation may impact how and where products
appear on this site including, for example, the order in which
they appear. TechnologyAdvice does not include all companies
or all types of products available in the marketplace.