There are hundreds of possible log sources around your environment, and choosing which bubble to the top of your IT consciousness can be difficult. In a job where everything seems to be a top priority, understanding all the log types and sources available for selection can be daunting. In your environment, some logs may be more valuable than others, but having general guidance about logging. What types of records may be available to monitor can help make you a better technologist.
There’s no way we could ever think to cover every possible source of logs, but let’s start with some of the classics and go from there.
What Is A Log File?
Enterprise organisations are increasingly choosing to deploy new applications and migrate existing ones to private and public cloud computing environments. Cloud computing, especially in the public cloud, provides significant benefits that include cost savings through economies of scale, streamlined processes and simplified management with fewer administrative tasks.
As organisations depend on the cloud for more critical applications and services, there is a growing need to maintain network transparency and visibility, also called observability. Observability in the context of cloud computing depends on two factors. the presence of data outputs that accurately reflect activities and behaviours on the network and the ability to aggregate and analyse that data.
Log files are the primary data source for network observability. A log file is a computer-generated data file that contains information about usage patterns, activities, and operations within an operating system, application, server or another device. IT organisations can implement security event monitoring (SEM), security information management (SIM), security information and event management (SIEM), or another analytics tool to aggregate and analyse log files from throughout a cloud computing environment.
Log File Categories For Common Operating Systems
Log files are automatically computer-generated whenever an event with a specific classification takes place on the network. The reason log files exist is that software and hardware developers find it easier to troubleshoot and debug their creations when they access a textual record of the events that the system is producing. Each of the leading operating systems is uniquely configured to generate and categorise event logs in response to specific types of events.
Windows Event Logs
The windows operating system can generate an event log in response to activity on any hardware or software components. Network security and operations analysts can use specialised software tools to aggregate and analyze these logs, detect patterns and trends, and respond to incidents or potential user issues. Windows is pre-configured to classify events in six categories:
- Application Logs - an application log is created when an event takes place inside an application. These logs help code developers understand and measure how applications are behaving during development and before release.
- Directory Service Logs - a computer configured to respond to security authentication requests within a Windows Server domain (known as a domain controller) may generate directory service logs. These logs record user privilege changes, authentication operations, and requests and other operations that take place in Windows Active Directory.
- DNS Server Logs - a Domain Name System (DNS) server contains the databases that match hostnames of websites on the internet with their appropriate IP addresses. Each time you navigate to a new web page, DNS servers are involved in processing the request and helping your browser get to the right page. DNS server logs are a particular type of log file for recording activity on a DNS server.
- File Replication Service Log - another type of log file that is only available for domain controllers. They record information about file replications that take place on the computer.
- Security Log - security logs are created in response to security events that take place on the computer. These can include various events such as failed log-ins, password changes, failed authentication requests, file deletion and more. Network administrators can configure which types of events are application events and should be entered into the security log.
- System Log - system logs record events that occur within the operating system itself, such as driver errors during start-up, sign-in and sign-out events and other activity.
Linux Event Logs
The Linux operating system is uniquely configured to generate and store log files. Linux creates a continuous timeline of events on the system, including every event related to the server, kernel, and running applications. Linux places events in four distinct categories:
- Application logs
- Event logs
- Service logs
- System logs
These categories are analogous to those used by Windows O/S.
Ios Event Logs
iOS takes a unique approach to event log generation when compared to other operating systems. iOS does not log every event that happens in the system, but it does generate documentation for application crashes. Later versions of iOS (10.0 and beyond) offer an API that can be used to log application events that take place on the system. The iOS logging API allows network administrators to access log file data from:
- App security
- Apple pay
- Data encryption
- Device controls
- Internet services
- Network security
- Privacy controls
- User password management
Why Do It Organisations Monitor Log Files?
Large IT organisations depend on an extensive network of IT infrastructure and applications to power key business services. Log File monitoring and analysis increase the observability of this network, creating transparency and allowing visibility into the cloud computing environment. Observability should not be treated as an ultimate goal. However - it should always be seen as a mechanism for achieving fundamental business objectives, such as improving the reliability of systems, meeting security and compliance objectives and driving revenue growth.
Log File monitoring and analysis can help IT organisations improve the reliability of their systems for the end-user. Log files include information about system performance that can determine when additional capacity is needed to optimise the user experience. Log files can help analysts identify slow queries, errors causing transactions to take too long or bugs that impact website or application performance.
IT organisations can use log file monitoring to maintain the security posture of cloud computing environments and prevent data breaches. Log files capture things like unsuccessful log-in attempts, failed user authentication, or unexpected server overloads, all of which can signal an analyst that a cyber attack might be in progress. The best security monitoring tools can send alerts and automate responses as soon as these events are detected on the network.
IT organisations can also use log file monitoring to improve their business decision-making. Log files capture the behaviour of users within an application, giving rise to an area of inquiry known as user behaviour analytics. By analysing the actions of users within an application, developers can optimize the application to get users to their goals more quickly, improving customer satisfaction and driving revenue in the process.
Top Log Sources You Should Monitor
These are those devices that are the “information superhighway” of your infrastructure. Switches, routers, wireless controllers, and access points can be teased to provide logging information about the health and state of your environment. The logs can provide insights ranging from wireless AP hopping to hardware failures. Probably most impactful to your domain are notifications of configuration changes. They knew who changed what and when can help you diagnose and recover from any misconfigurations.
As organisations push towards a cloud-first methodology, the edge devices in your environment can become even more vital to your business. Your firewalls and other security devices are handling more and more traffic as loads are shifted to cloud infrastructures. The logs on these security devices can provide a plethora of interesting information—not least is blocked traffic, health of the VPN, intrusion detection and prevention systems, and unusual user activity. These Security Information and Event Management (SIEM) logs may be your first defence in understanding an attack or isolating an anomaly in your user experience. Tools like SolarWinds Security Event Manager are designed to monitor event logs for any suspicious activity, allowing you to respond in real-time to potential threats.
It may go without saying, but I’m going to tell you anyway: server logs can offer much information about the state of your environment. Windows and Linux servers are constantly pumping out logs that give you an understanding of how and why systems are behaving the way they are. There are hundreds of thousands of events that can trigger within an operating system and its associated applications. Knowing which log events are frivolous and which require immediate action is a skill honed on the battlefield. Regardless, it would be best if you didn’t overlook server logs as a viable source of information.
Yes, I’m aware that capturing web server logs can be construed as a tedious process, but it is one of the best ways, if not the best way, to understand how end-users interact with your web properties. IIS, Apache, Tomcat, Web Sphere, NGINX, and every other web engine out there can provide some measure of web server logging. Depending on your needs, sometimes just understanding when people are going to your site and from where can prove invaluable to understanding the needs of your customers. Unfortunately, a web server log is a typical log type that can sometimes be overlooked when organisations develop their logging strategy.
Whether you use Active Directory, an implementation of OpenLDAP, or another alternative, knowing who and what is poking around your infrastructure can be vital to maintaining a good security posture. Each of your authentication servers will provide some logging measure, but what’s key for you is understanding what to look for. Most commonly, it would be best to look for token requests, authorisation revocation, and authentication failures. These types of logs can aid in determining failing logins due to account expiration, isolate the source of a potential attack, and pinpoint problem areas that need to be addressed.
Hypervisors can let us IT professionals do our jobs better by balancing workloads and utilising resources more efficiently. Clusters can now run hundreds, if not thousands, of simultaneous workloads. However, much of the work associated with hypervisors is behind the curtain, and you never get to see. Your hypervisors are juggling all the time—allocate resources from this virtual machine to this one, move the storage from this cluster node to this other one, shift this entire virtual machine to another node—and it’s a precarious balance. Capturing and monitoring hypervisor logs can be one of the best ways to understand what your hypervisors are doing when you aren’t watching.
Although relatively new compared to most other log types on this list, containers are becoming more and more business-critical. Extrapolating to a higher level would be container management services. These services are like hypervisors in many ways, but just different enough to warrant a separate category. Understanding why the host felt it was necessary to drop back your scaled-out deployment from eight endpoints to only four would help diagnose and tune. Most of this information is located only in the container logs, so make sure you get them.
This may seem an odd addition to this list of the best log types to monitor because of the IT trend to move towards a more hyper-converged infrastructure or moving everything to the cloud, but it’s something that’s frequently overlooked. If your fibre switch loses connectivity to a server-side transceiver, then that data is no longer available to that server. In today’s world, there are generally redundant pathways so that connectivity is not truly lost, but the scenario still applies in a multi-path environment. Say you have four connections from your server to your SAN infrastructure, but after a series of unfortunate events over several months, three of them have failed. This means that you have restricted data movement by 75%. You’ve not encountered a failure in the traditional sense because the connectivity still exists, and data is moving, but with performance hampered this badly, is it any wonder end users are complaining? In my opinion, this is one of the top overlooked log sources.
This applies to pretty much any application log. Although some software applications will leverage the operating system’s existing logging functionality for log management, these are becoming fewer and fewer. Most critical records for applications are stored in flat files on your disks somewhere. Often, these logs are used by your application support people for troubleshooting, but what about multi-tier applications? If you have a front-end, middleware, and back-end deployment, each may collect logs slightly differently. Make sure you aren’t sleeping on managing and monitoring these logs—from each tier—and getting them into a system so that you can compare transactions by lining up the timestamps.
Yes, really. In IT, a common trope is to blame the end-user, but sometimes it’s not their fault. Sometimes it’s the fault of the endpoint itself. I’m not saying that every log on every machine needs to be collected all the time; I’m saying that you should probably not do that, but selective log collection from endpoints can be critical in gaining a more significant grasp of the scope of the problem. This is perhaps the most overlooked log type needed for actively troubleshooting issues.
Types Of Log Data
There are six different types of logs monitored by SIEM solutions:
- Perimeter device logs
- Windows event logs
- Endpoint logs
- Application logs
- Proxy logs
- IoT logs
Perimeter Device Logs
Perimeter devices monitor and regulate traffic to and from the network. Firewalls, virtual private networks (VPNs), intrusion detection systems (IDSs), and intrusion prevention systems (IPSs) are some of the perimeter devices. These devices generate logs containing a large amount of data, and perimeter device logs are vital for understanding the security events occurring in the network. Log data in the Syslog format helps IT admins perform security audits, troubleshoot operational issues, and better understand the traffic passing through and from the corporate network.
Windows Event Logs
Windows event logs are a record of everything that happens on a Windows system. This log data is further classified into.
Endpoints are devices that are connected across the network and communicate with other devices across servers. Some examples include desktops, laptops, smartphones, and printers. With organisations increasingly adopting remote work, endpoints create points of entry to the network that could be exploited by malicious actors.
Businesses run on various databases, web server applications, and other in-house apps to perform specific functions. These applications are often vital for the effective functioning of the business. All of these applications generate log data that provide insights into what is happening within the applications.
Proxy servers play an essential role in an organisation's network by providing privacy, regulating access. Since all web requests and responses pass through the proxy server, proxy logs can reveal valuable information about usage statistics and the browsing behaviour of endpoint users.
Internet of Things (IoT) refers to a network of physical devices that exchange data with other devices on the internet. These devices are embedded with sensors, processors, and software to enable data collection, processing, and transmission. Like endpoints, devices that make up an IoT system generate logs. Log data from IoT devices provides insights into the functioning of hardware components, such as microcontrollers, the firmware update requirements of the device, and the flow of data in and out of the device. A crucial part of logging data from IoT systems is the storage location of log data. These devices do not possess sufficient memory to store the logs. So, the records must be forwarded to a centralized log management solution to be stored for extended periods. The SIEM solution then analyses the logs to troubleshoot errors and detect security threats.
There are additional log sources that I’ve neglected, like proxy servers, load balancers, and cloud management systems, to name a few, but this isn’t meant to be an exhaustive list. Hopefully, after reviewing these ten log types, you gain a little perspective into what would be relevant for your situation. It’s also something to keep in mind as new hardware and software enters your infrastructure.
Whether you choose one, all, or none of these as potential log sources to monitor is dependent on your exact needs. Simply thinking about what types of watching or log analysis tools you need moving forward could help you choose those relevant to your situation. Every bit of information can help you gain a deeper understanding of your infrastructure and how to handle its care and feeding best. Remember, it’s not if something will go sideways. It’s when. Having the best log types to back up your decision-making can be a welcome tool in your IT arsenal.