Log4mongo for Java

The log4mongo-java project includes several Log4J appenders for storing log messages in a MongoDB collection.

MongoDbAppender - stores the LogEvent in a BSONified form
ExtendedMongoDbAppender - Extends MongoDbAppender by allowing you to add top level elements
MongoDbPatternLayoutAppender - supports logging data in a custom format

QuickStart

Download and install MongoDB
Start mongod running on localhost on the default port
Download the Mongo Java driver jarfile
Download Log4J jarfile
Download log4mongo-java jarfile
Place these three jar files in your classpath
Create a file called log4j.properties in your classpath
Add the following lines to the log4j.properties file
a. log4j.rootLogger=INFO, MongoDB
b. log4j.appender.MongoDB=org.log4mongo.MongoDbAppender
c. log4j.appender.MongoDB.databaseName=appname
d. log4j.appender.MongoDB.collectionName=log
Create a Java class like:

import org.apache.log4j.Logger;
public class LogTest {
    public static void main(String[] args) {
        Logger logger = Logger.getLogger(LogTest.class);
        logger.error("Don't panic");
    }
}

Compile and run the class. Then, start a mongo shell and confirm that an event was logged.

$ mongo
MongoDB shell version: 2.0.3
connecting to: testconnecting to: test
> use appname
switched to db appname
> db.log.findOne()
{{
"_id" : ObjectId("4c60e21be54744706960af09"),
"timestamp" : "Thu Mar 29 2012 22:22:35 GMT-0700 (PDT)",
"level" : "ERROR",
"thread" : "main",
"message" : "Don't panic",
"fileName" : "LogTest.java",
"method" : "main",
"lineNumber" : "8",
"class" : {
"fullyQualifiedClassName" : "LogTest",
"package" : [
""
],
"className" : "LogTest"
}
"loggerName" : {
"fullyQualifiedClassName" : "LogTest",
"package" : [
""
],
"className" : "LogTest"
}
}

Appenders

An abstract BsonAppender class (introduced in 0.5) provides basic functionality for serializing a Log4J LogEvent to a BSON DBObject. The MongoDB specific appenders extend this class. This abstract class is useful if you want to store the same information outside of MongoDB.

MongoDbAppender

MongoDbAppender stores the Log4J LogEvent object as a document in a MongoDB collection. The document is serialized in a format that is very similar to the structure of the LogEvent class. Where possible, the data is stored as appropriate BSON objects, rather than just strings.

A future enhancement will optionally store the Map in the MDC (Mapped Diagnostic Context) as part of the document.

MongoDbPatternLayoutAppender

MongoDbPatternLayoutAppender stores a log message as a document in a MongoDB collection based on a user-specified conversion pattern using the standard Log4J pattern layout, parser and converter functionality. While requiring some additional configuration, this appender allows significant flexibility over the format of the stored document. The MongoDbPatternLayoutAppender also allows custom information to be stored with the document. The custom values are determined at the time an event is logged.

Log4J Properties

The root directory of the project distribution contains a sample properties file named log4j.properties.sample. All of the properties used by log4mongo-java are documented in the sample file.

Replica Sets

The log4mongo-java 0.6 release added full support for replica sets. While earlier releases would work with replica sets, you could specify only one host in the hostname property. If the appender happened to be initialized when that host was not available, the appender would not be able to connect to other members of the set.

With the 0.6 release, you should specify the names of all the hosts that can become master. The Mongo Java driver will automatically find the current master. For example:

og4j.appender.MongoDB.hostname=mongo1.example.com mongo2.example.com

If MongoDB is listening on the same port on each host, you can specify just that port. If it's the default port of 27017, you can omit the property.

log4j.appender.MongoDB.port=27000

If the same port is not used by each host, then you must specify the port for all hosts. For example, if the hosts mongo1 and mongo2 are listening on 27000 and 28000, respectively:

log4j.appender.MongoDB.port=27000 28000

When a master becomes available, the next attempt to log will fail and the insert will be lost. The Mongo driver will then check for the new master and the next insert should go to the new master. Log inserts between the time of the original master's failure and the election of the new master will be lost.

MongoDB Collection vs. Capped Collection

Although events can be logged to any MongoDB collection, it's common to use a capped collection. Capped collections provide very high performance and automatic removal of old information. One drawback, though, is that the collection is capped by size in bytes or in objects, rather than by a time period. A common desire is to keep logs for a certain period of time. Capped collections cannot be used for this purpose, though. If you are confident that logging data will rarely exceed a certain size over the desired period of time, though, capped collections can be a big win.

Another advantage of capped collections is that they allow you to control the amount of disk space used for logging. History is full of sad stories of apps grinding to a halt because the log files filled up a disk partition to which the app needed to write other data.

Assuming you are using a database named appname and a collection named log, you can use the following commands in the mongo shell to create a capped collection of size 100,000,000 bytes.

> use appname
> db.createCollection('log', {capped:true, size:100000000})

Authentication

MongoDB authentication is optional. If the userName property is specified, the appender will attempt to authenticate against the specified MongoDB database using the values for the userName and password properties. (TODO: Add description of behavior and a test case for when userName is specified, but password isn't).

For example, if an appender is defined as:

log4j.appender.MongoDB=com.google.code.log4mongo.MongoDbAppender

and the host, database, user name and password are specified as:

log4j.appender.MongoDB.hostname=localhost
log4j.appender.MongoDB.databaseName=appname
log4j.appender.MongoDB.userName=open
log4j.appender.MongoDB.password=sesame

then when the appender is initialized, it will attempt to authenticate as user open with password sesame against the appname database in a MongoDB instance listening on the default port on localhost.

Write Concern

The 0.7.3 adds support for setting the write concern when logging. The default is UNACKNOWLEDGED. You can override the write concern in your log4j properties file. For example, if you named your appender MongoDB in the properties file and you wanted the driver to wait for log event data to be written to disk on at least the primary:

log4j.appender.MongoDB.writeConcern=FSYNCED

You should definitely do performance testing, though, since I found FSYNCED to be 1 to 2 orders of magnitude slower than UNACKNOWLEDGED when writing 1000 consecutive log events.

Troubleshooting

When logging an event, this exception is thrown - java.lang.NoSuchMethodError: com.mongodb.DBCollection.insert(Lcom/mongodb/DBObject\;)

If you use log4mongo-java 0.3.2 or earlier with MongoDB Java driver 2.1 or later, you will probably get this error.

A source compatible, but binary incompatible, change was made in the 2.1 driver to the DBCollection.insert() method used by log4mongo-java. That method now takes a varargs argument instead of a single DBCollection object. In client code, the Java compiler converts the varargs argument to an array whether you pass one DBCollection or more. However, log4mongo-java 0.3.2 was compiled against the 2.0 Java driver, so it passes a standalone DBCollection, instead of the array that is expected by the compiled 2.1 driver code.

The workaround is to either upgrade to log4mongo-java 0.4.0\+ or to compile your own log4mongo-java jarfile against MongoDB Java driver 2.1+.

When logging an event, this exception is thrown - java.lang.NoSuchMethodError: java.util.Arrays.copyOf(\[Ljava/lang/Object;I)\[Ljava/lang/Object;

Versions of log4mongo-java up to and including 0.6.0 contain Java 1.6 specific code. You may see the above exception if you are using Java 1.5 or earlier. This has been fixed on GitHub and will be included in the next release of log4mongo-java, likely to be 0.6.1.

NullPointerException when app exits and appender is closed when using MongoDB Java Driver 2.2

The MongoDB Java Driver 2.2 has a bug, JAVA-180 causes a NullPointerException if you call Mongo.close() and are not using MongoDB in a replica set. The bug is fixed in Robert's fork of the 2.2 driver, as well as in the trunk for the 2.3 release. The bug was not present in the 2.1 driver.

Can't Serialize Proxied Object

Libraries such as Hibernate may use libraries like CGLib for generating class proxies. In your code it will look like you are passing a regular object to one of the Logger methods for logging a message, but at runtime a proxy will be passed. The BSONEncoder can't serialize this proxy.

Here's an example stack trace:

java.lang.IllegalArgumentException: can't serialize class com.example.MyClass$$EnhancerByCGLIB$$8a4801c9
at org.bson.BSONEncoder._putObjectField(BSONEncoder.java:205)
at org.bson.BSONEncoder.putObject(BSONEncoder.java:121)

The simplest workaround is usually to call .toString() on the object before passing it to one of the Logger methods.

Future Enhancements

Performance improvements