Servers and cloud services are chatty little critters. They like to keep a note of everything they do, and those notes are known as log files. They’re usually written whether you like it or not, but if you take an interest in them, you might learn a lot about your workloads. Logfiles And Logging In Your Cloud.
Who Creates Logfiles?
It’s not really a who, it’s a what. And the what is Services. A service is just what it sounds like – it’s something which provides a service to another entity. It could be a service which is consumed by another service running on a computer, and so on. Or it could be a service which is used by a human user.
Let’s think about a cash machine. When you put your card in and punch in your PIN, there are services which authenticate that you are who you say you are. Next there is an authorisation step, which is where the bank computers decide what information you’re allowed to see, now that we know you are who you say you are. Those services happen in the background, but then there must be another service which writes the information to the screen you’re looking at, and interoperates your input.
All those services will keep a log of everything they do, in which order, and at exactly what time. That means if you report your card as stolen etc., the police and the bank can check the logs, and piece together the events that happened.
What Is Logged?
This blog is all about helping none-technical folks understand more about the cloud, so there’s a good chance you’ve never seen a logfile before. So what does a logfile look like? Well they’re not pretty, let me tell you that!
But it mostly depends on what information is important for that service, and for that environment. For example, in a development environment, where things are changing constantly, and you’re trying out new things, you want to know a lot about what’s happening behind the scenes. Development environments are frequently broken – that’s just the nature of development, and technicians need to understand exactly what went wrong.
In our bank scenario as above, we might record things like;
- Card reader started pulling the card in
- Card reader completed the card read
- Completed card reader stage successfully
- “Enter PIN” message printed to screen
- Keypad listener started
- Number 1 key signal received from keypad module
- Number 2 key signal received from keypad module
And so on.
So as you can see, there is usually a lot of information recorded in a logfile under normal circumstances.
What Should You Do With Logfiles?
Let’s say your bank card is stolen, and money is taken out of your account from a cash machine down the road. We’re going to need the logfile from that cash machine aren’t we. So does a bank employee have to head down to the particular cash machine with a cable and laptop to grab the log? Nah, of course not. The logfiles are securely sent over the bank’s network to a special set of servers which collect the logfiles from all the cash machines, and all the point of sale terminals in all the shops in the country. The logging server’s job is to put all these logs in order, and to make sure all that information is kept safe and secure, just in case it is ever needed.
So you might be thinking, my business isn’t as big as a bank; surely I don’t need a logging server. Well, you’d be right. But it might be useful to have a logging service. After all, we’re talking about the cloud – we don’t concern ourselves with running pesky servers just because we might want to use a service it could provide. No, we use serverless technology to help.
Logfile Services
We love a bit of AWS over here – that’s no secret. And AWS brilliantly has a logfile service which is super easy to use. It’s called CloudWatch Logs.
CloudWatch Logs is a place to send the logfiles from any number of services running in your cloud. Any serverless functions you might be running in Lambda, or API calls coming throu API Gateway. But we can also install a lightweight agent service on any virtual servers you might be using, and tell it to send a given logfile over to CloudWatch Logs.
That means that if you’re running an autoscaling group, where servers are created when times are busy and destroyed when they’re not needed, you don’t loose any logs created by that server which may be deleted. It’s all fine – just check the logs in CloudWatch Logs, and don’t worry about which server they were generated from.
What Can We Do With Logs?
It’s great having all your logfiles in the same place, and sequenced properly so you can move forwards and backwards in time to read them, no matter which server created that log. But, that’s only half of the story. With CloudWatch Logs, you can set up filters and alerts based on certain things appearing in the logfile.
A popular example might be a 404 error generated by a web server. A 404 error means Page Not Found. In English, that means someone asked for a page on your website that doesn’t exist. Something like beatyconsultancy.co.uk/bog for example. That’s probably a typo, and I meant /blog. But what if I posted that web link to social media, or worse, ran an advertising campaign sending people to that link? That would generate loads of 404 – Page Not Found errors in the logfile. I probably want to know about that!
In CloudWatch logs, we could apply a filter to look out for “404” in the web server’s logfile, and count the number of times it’s found in each 5 minute period. If the count of 404 errors is greater than, let’s say 3, send me a message. Now that’s really useful, and it’s something we couldn’t achieve without a logging server.
Do you fancy getting more from your logs and logfiles? Get in touch with us today!
