Archive

Posts Tagged ‘MapReduce’

Using a Spring Context within Hadoop Mappers and Reducers with environment specific property files

Bootstrapping a Spring Context from the classpath is quite simple, even from Mappers/Reducers in Hadoop. When you want to use enviroment specific property files it gets a bit harder. In our Mappers and Reducers we rely heavily on Spring Beans who perform the magic. Besides that we load different properties per environment (ie a MongoDB). After several iterations we came up with a nice solution. You can also leave out the Spring part and use this solution to broadcast Properties to all your Mappers/Reducers.

Read more…

Overriding Hadoop jars with MapReduce v2 – How to fix the classpath

We ran into a nasty problem with Hadoop when we wanted to use a ‘new’ version of Jackson. Hadoop decided to throw a NoSuchMethodError. It appeared Hadoop uses an ancient version of Jackson. Some terrible memories of classloading and JBoss EAP came into my head. After I was calmed down by some colleagues I ultimately found a solution.

Read more…

Categories: English, java, work Tags: , , ,