No other container orchestration tool is as popular as Kubernetes, so it’s become the de facto standard in automation, management, monitoring and logging of containers. Kubernetes manages containers, but you also need to configure it for monitoring and logging. Several native logging options exist in Kubernetes so that you can track the health of your container environment. Native logging solutions can have too much overhead, so some administrators choose to work with third-party solutions. If you decide to work with Kubernetes logging, we discuss the basic functionality and commands to set up monitoring and view log information from the command line.
When you think of logging, you think of simple error information sent to a file or database so that you can review it later, but logs contain valuable data used in analysis and monitoring of an environment. These logs can also be used with analytics and visualization tools as a way to monitor the health of the environment. With containers, logging is more of a challenge than basic application logging due to the many moving parts and resources. When containers are deleted or crash, the system cleans up the container, creates a new one, and logs are removed along with the deleted container. Any log data captured for anomalies before deletion are removed.
If you’re familiar with virtual server monitoring, the challenges of logging Kubernetes events are similar. Virtual servers spin up and down as needed, and once the server is destroyed any events contained within them are also deleted. Containers also have their own events, and when administrators or Kubernetes deletes container logs, any data contained within them that log anomalies and reasons why the container failed are also destroyed.
Regardless of the challenges, it’s important for organizations to set up monitoring to ensure stability of the environment. There are a few options for logging Kubernetes events:
Of the above three solutions, the most efficient, scalable, and flexible solution is using a log monitoring tool that works in the cloud. Since it’s likely that your containerized environment will run in the cloud, a cloud-hosted logging solution is quicker to provision, configure, and start monitoring events compared to the others. A third-party solution is also vendor agnostic. Using a vendor-specific option is also quick to provision, but it’s locked to the vendor’s platform.
Not being tied to a specific vendor is important in enterprise environments. In an enterprise environment, it’s not uncommon for administrators to use a secondary provider as a disaster recovery hot site where data and applications are replicated to the secondary provider should the main provider suffer from catastrophic failure. In this case, a secondary vendor-specific logging solution would be necessary. With a vendor \-agnostic solution, you can use the logging tool in both environments.
Servers write application and environment events to logs in a configured text file. For example, Kubernetes logs are stored in Linux to files such as /var/log/containers/app.log. This works well for a server environment, but containers work differently than a simple application running on an operating system.
With Kubernetes, pods dynamically spin up or down, and they can move across nodes in a cluster. Most environments run several containers across clusters, and these containers store their own logs. This means that you should have a solution that collects and aggregates logs into one location.
Kubernetes has a logging framework that can be used to capture standard output (stdout) and standard errors (stderr) from each container and send it to a central log. Logs are managed by the Kubernetes service, and administrators can view each container log by typing “kubectl logs <pod> -c <name of container>" at the command line.
Another option is to configure Kubernetes to use application-level logging. This configuration relies on each container to have its own logging component configured individually, but it’s much less convenient than using the previous solution. Should any containers change configurations, it would require changes to the logging solution.
Containerized environments have several layers unlike traditional environments where an application simply runs on an operating system installed on a virtual machine or physical server. In Kubernetes, the three main layers are the containers themselves, the nodes that run the containers, and the clusters that host multiple nodes. For a full overview of all errors, logging all three are necessary and should be aggregated where a visualization tool can be used to analyze the output. To understand Kubernetes logging, here is a brief overview of each logging type.
Every pod captures errors and logs generated by the application. As soon as you configure and deploy a pod, you can see its logs in Kubernetes from the command line. Suppose that you have a pod with the metadata name set as “myapplication.” Type the following in the command line after you’ve deployed it:
The output from the above command would show logs generated by the application container. For each container, you have a log file linked to it.
Containers running within nodes write their information to stdout or stderr (depending on the type of event) streams. These streams get picked up by the Kubernetes kubelet service running on the node. The kubelet service outbound information is handled by a configured logging driver, which sends information to the log file. The file is normally located in the /var/log/containers directory.
The kubelet service runs at the operating system level, so it logs the health of the underlying application environment. It also handles logging rotation so that log files do not fill up the available storage space. To collect kernel logs, you need systemd installed on your node. System-level events can be found in these logs to identify issues with the kernel, which could lead to more serious downtime.
Node logs can be viewed using the following command:
Any changes to the system will be seen in these logs. For example, if administrators make changes to environment variables, it will show in node logs.
To log entire clusters, you need a third-party log aggregator. Kubernetes has no native support for cluster-level logging and aggregation. This is where LogDNA can benefit administrators responsible for Kubernetes monitoring and logging. A few other alternatives administrators can use include:
The three logging types can be used if you choose to manually configure each one, but this requires considerable overhead to maintain all three. Should any configurations change, it could affect the way your monitoring and logging solution runs. To overcome the many challenges with node and container logging, and to get a centralized cluster logging solution, a third-party tool that aggregates information across your entire environment will simplify Kubernetes monitoring.
Several advantages of a centralized cloud logging solution for Kubernetes include: