How to fix the classpath in Spark – Rebuilding Spark to get rid of old jars

After fixing an earlier classpath problem a new and more difficult problem showed up. Again the evil-doer is and ancient version of Jackson.
Unfortunately all classes are loaded from one big container with Spark/Hadoop/Yarn, this causes a lot of problems. The spark.files.userClassPathFirst option is still ‘experimental’ which, in our case, meant it just didn’t work. But again, we found a solution. Our system engineers also wanted a reproducible solution so in the end it’s just a small recipe.

Lees verder…

Categorieën:English, java, work Tags:, , , ,

Using a Spring Context within Hadoop Mappers and Reducers with environment specific property files

Bootstrapping a Spring Context from the classpath is quite simple, even from Mappers/Reducers in Hadoop. When you want to use enviroment specific property files it gets a bit harder. In our Mappers and Reducers we rely heavily on Spring Beans who perform the magic. Besides that we load different properties per environment (ie a MongoDB). After several iterations we came up with a nice solution. You can also leave out the Spring part and use this solution to broadcast Properties to all your Mappers/Reducers.

Lees verder…

Overriding Hadoop jars with MapReduce v2 – How to fix the classpath

7 september 2014 1 reactie

We ran into a nasty problem with Hadoop when we wanted to use a ‘new’ version of Jackson. Hadoop decided to throw a NoSuchMethodError. It appeared Hadoop uses an ancient version of Jackson. Some terrible memories of classloading and JBoss EAP came into my head. After I was calmed down by some colleagues I ultimately found a solution.

Lees verder…

Categorieën:English, java, work Tags:, , ,

How to solve a “does not contain shard key for pattern” at a Mongo WriteConcernException

A few days ago I got stuck a bit longer than normally because of a WriteConcernException in MongoDB. Inside the exception I got the message : “does not contain shard key for pattern { _id: 1.0 }”. The id was in the query, but apparently not in the right place

Lees verder…

Categorieën:English, work Tags:

Creativity

This post is a collection of interesting things I found about creativity on the internet

Lees verder…

Categorieën:creativity, English

Food

This post is a collection of interesting things I found about food on the internet

 

Lees verder…

Categorieën:English, food

How to control your Android phone with a broken screen from adb

A few days ago I dropped my Nexus 4 on a parking lot. A few cracks in the screen and unfortunately I only was able to control the lower left quarter of the screen. Fortunately USB debugging was enabled (when it’s not, stop reading, this article won’t help you) and I was able to control my phone. All I needed was a factory reset so I’m able to sell my phone to someone who’s willing to pay for a screen repair.

Lees verder…

Categorieën:android, English, work Tags:,
Volg

Ontvang elk nieuw bericht direct in je inbox.