Fog Creek

Mini and Tagged Logging, Open Source Go Packages – Tech Talk

In this Tech Talk, Stephen, a Developer here at Fog Creek talks about two Go packages, Mini and Logging, that we recently Open-Sourced. He describes their functionality and provides a few examples of their use.

Find them both on GitHub: Mini and Logging.

About Fog Creek Tech Talks

At Fog Creek, we have weekly Tech Talks from our own staff and invited guests. These are short, informal presentations on something of interest to those involved in software development. We try to share these with you whenever we can.

Content and Timings

  • Introduction (0:00)
  • Mini (0:39)
  • Logging (3:30)

Transcript

Introduction

All I wanted to talk about today was a couple of libraries that we open sourced. These are two Go libraries that we wrote. The first package that we released is a little library that reads INI files and the second one is a logging package. The INI files, there really wasn’t a good alternative when we started working on the two projects. The logging I wrote for two reasons. One, the Go logging is pretty limited. The second is that I wanted to provide this feature called Tagged Logging.

Mini – INI file reader package

Let me start with this package called MINI, which basically is a small library to read INI files. For people who don’t know, an INI file is a really simple format. You can have comments, you can have key-value pairs, you can have sections. In the example that’s on the slides now, there’s a comment at the top with a semicolon. You can also do comments with hashtags. The bracketed owner has a section called owner. Name=John Doe is a key-value pair. As you can see online eight, you can quote the values. What you’re not seeing here, you can also do arrays of values. You can have 10 things and the syntax for that is that you just put empty brackets after the names. So name empty brackets would mean that there’s an array of names. That format is documented online. I put a link to the Wikipedia.

The way it works in the library, you can load this configuration in from an INI file or from what’s called a reader which, just like a stream if you’re a Java person or whatever. So you can create the reader any way you want. You get back whether or not there’s an error reading or an object that represents the configuration. I’m skipping the error handling code. If you look at this, this code won’t compile because I skipped the error handling. Then you can just go ask. You can ask for, give me a string from a section, and in the case of the file we’re looking at, all the data is in sections. I have to get it from a section. If I go back for a second. If you had put data online one that was a key-value pair, it’d be global, and you wouldn’t have to refer to it with a section.

The library also supports defaults, so if you look at this string from section, you can put in a default. That’s pretty much the whole API, which is pretty simple. There’s one other thing I’m going to talk about in a second. I also wanted to add, that the way people use this, Go has a package, they have a really simple way to get packages. They don’t have a really good way to manage versions of packages. We’ve posted this up on GitHub and people can use it just by typing go get, and then the URL there, the path to the package. It’s really easy to use the package, but, if you want to update it, if you want to deal with managing these or having multiple versions, it’s a little bit more of a pain.

The other thing we support is, if you have a structure, you can actually read the data from a section into the structure. That uses reflection to fill in the values and the structure to match whatever the values are in the data file, and that supports a raise, and ints and floats, and booleans and strings as well. That’s a pretty simple little package. The logging packs is more interesting.

Logging

The logging package is actually a pretty complete logging package. It has named loggers, it has the concept of a default logger. One of the things that I wanted to do was make it so that you could log really easily. In 99% of the cases, you just want to log to the same place. You don’t really need all of this concept of named logging.

There is this need to have different levels of logging, because, you maybe don’t want to spit out every message all the time. It can get excessive. We have log levels. There are five log levels. Four of them are pretty standard, ERROR, WARN, INFO, and DEBUG. One of them is less standard which is, VERBOSE, and I’ll talk about that a little bit. You can different formats for your log, you can have appenders. It hooks in with the Go log package. It has a couple of things that are unique to this. One is, logging is all going to happen in the background. We’re using Go routines to do logging, the concurrency feature. When you do a log message, all the processing of the log message happens in a separate Go routine which could be in a separate thread or not. It’s not happening within your run and kind of your line of running code on the processor.

Which means that, if an error happens appending to a file, we can’t give it right back to you. What we do is, we actually give that back to you through a channel which you can ignore or read or whatever. The second thing that’s interesting, is, because we have log levels, you might have the situation where you say, “Okay, I’m debugging everything as info, because the debug level is very verbose.” I don’t want to see all the log messages all the time because then I don’t want to see them all the time. If something bad happens, I may lose the ones that were important, because they are the ones that would have printed when something bad happens, but they didn’t print because I didn’t have the log level at the right place. What I implemented is this idea of a buffer. We actually saved the last n log messages.

If an error happens, if you feel like you have the memory, you can set that buffer to a very large number. We default it a couple of thousand, but you could set it very high if you wanted. If you catch the error soon enough, and change the log level, which we can do in some of the services. You would actually, it will replay the messages that are in the buffer, which means, that you could go back and see what happened. That could get a little crazy, because if you go back and say, “Well, now I want to see all the debug messages,” well there could have been a million of those. The last feature that we have is this tagged logging. What tagged logging is, is this idea that every time you call log, you can pass in tags that somehow decorate the log message. The library let’s you associate levels with tags.

As well as associating a level with the whole logging process itself, you can say something like, ‘if any message is logged with a tag HDP, I want it to be allowed at this level’, like say debug, but if its logged with the tag rabbit mq, either I just want to use the default or I want to log it at info, or something like that. Now you can filter, not just by the logger itself and maybe the named logger, but you can actually filter across, kind of perpendicular or orthogonal, based on what the message is related to, not just where the message is in the code. What’s even cooler about this is, you can use tags like account ID. If a request comes in, say to the search service that we were doing, it automatically sets up a tag for the account ID. We could potentially turn on debug logging for that account, but not for all other accounts.

Which means, if a certain account is having a problem, we could debug that account without debugging everybody else. Moreover, if you add that with the buffer, that means you can go back, and potentially if an account had a problem, go back and look at the errors that happen, or the log messages that happen for that account without having to see everything that happened for everybody. I think the tagged logging is a really cool feature. Here’s a simple example. We have this idea, when you import the package, it’s going to be called logging, you could name it something else if you want. You could set a default log level. When you do that, it only logs things at that level and above. You can set a tag level, and when you do that, the tag level can override the default level.

Then it just has normal methods to log. We have formatted methods and non-formatted methods. We’re hooked into the Go logging, so if you use the standard Go package, you can say, “I want all the standard Go things to log with these tags at this level. Then we have this buffer logging I mentioned, where you can set how big you want the buffer to be by default at zero and nothing gets buffered. Then, if things are logging and you later reset the log level to something lower, all the messages in the buffer are checked and printed out. Both the MINI package and the logging package are out on GitHub. We’ve already got some comments, we’ve already fixed a couple of bugs and added some new tests.