Using a Spring Context within Hadoop Mappers and Reducers with environment specific property files

Bootstrapping a Spring Context from the classpath is quite simple, even from Mappers/Reducers in Hadoop. When you want to use enviroment specific property files it gets a bit harder. In our Mappers and Reducers we rely heavily on Spring Beans who perform the magic. Besides that we load different properties per environment (ie a MongoDB). After several iterations we came up with a nice solution. You can also leave out the Spring part and use this solution to broadcast Properties to all your Mappers/Reducers.

Lees verder…

Overriding Hadoop jars with MapReduce v2 – How to fix the classpath

7 september 2014 1 reactie

We ran into a nasty problem with Hadoop when we wanted to use a ‘new’ version of Jackson. Hadoop decided to throw a NoSuchMethodError. It appeared Hadoop uses an ancient version of Jackson. Some terrible memories of classloading and JBoss EAP came into my head. After I was calmed down by some colleagues I ultimately found a solution.

Lees verder…

Categorieën:English, java, work Tags:, , ,

How to solve a “does not contain shard key for pattern” at a Mongo WriteConcernException

A few days ago I got stuck a bit longer than normally because of a WriteConcernException in MongoDB. Inside the exception I got the message : “does not contain shard key for pattern { _id: 1.0 }”. The id was in the query, but apparently not in the right place

Lees verder…

Categorieën:English, work Tags:

Creativity

This post is a collection of interesting things I found about creativity on the internet

Lees verder…

Categorieën:creativity, English

Food

This post is a collection of interesting things I found about food on the internet

 

Lees verder…

Categorieën:English, food

How to control your Android phone with a broken screen from adb

A few days ago I dropped my Nexus 4 on a parking lot. A few cracks in the screen and unfortunately I only was able to control the lower left quarter of the screen. Fortunately USB debugging was enabled (when it’s not, stop reading, this article won’t help you) and I was able to control my phone. All I needed was a factory reset so I’m able to sell my phone to someone who’s willing to pay for a screen repair.

Lees verder…

Categorieën:android, English, work Tags:,

The Cargo Daemon – For ‘intense’ redeployments with Maven

When you’re doing continuous integration with frequent redeployments you might run into out of memory errors or some things that won’t unload properly. The Cargo daemon will solve this for you. Before reading any further you have to promise that this is no excuse to skip testing for memory leaks!
In this article I will explain how to deploy multiple applications in a Tomcat container including some jdbc-drivers. I will use the cargo-maven2-plugin since this is the easiest option when using Jenkins (or any other ci-environment).

I won’t go too deep into all the configuration options since the manual is pretty good. I just want to get you up and running a bit faster.

Lees verder…

Categorieën:English, java, work Tags:, ,
Volg

Ontvang elk nieuw bericht direct in je inbox.