I currently find myself working on a project where Grails and MongoDB are the technology stack... in fact that in part is some of the reason for being interested in the project. I thought I would share 2 aspects of my learning so far:

Grails + Mongo

We spent at least 2 weeks trying to understand which approach we were going to take for our persistent tier. The general concensus was to use the Mongo GORM. We started down that path and ran into a bug and then a limitation on embedding. We got amazing support on the bug followed by the next release of the GORM option which fixed our initial document embedding concerns. We ran into more problems.

We switched to the morphia driver... once again... challenges caused lots of frustration. The challenges included getting other plugins to work with this driver, the fact that the morphia "domain" objects couldn't be in the standard grails domain folder, which further complicated mocking. Frustrations with getting full capabilities of getting spring security to work. It turns out that with spring security there are a number of relational assumptions (along with assumptions of objects being domain classes.. meaning they live in the grails domain package space) which cause problems when you are using NoSQL.

The morphia had another limitation we couldn't live with... we wanted the ability to have different types of objects mapped to the same collection. It appears to me that the Mongo efforts in the Grails space are bolt ons to a rational model. You can get it to work when thinking with a RDBMS mindset. They seem to break down with a NoSQL mindset... this certainly could be attributed to user error (meaning me). I believe we gave it a good try... however the project is under the gun and we didn't have time to "play" with it or wait for responses.

Alas we created our own solution... one of our significant constraints was to be prepared for extremely large scale. This had us duplicating a lot of documents not common in RDBMS solutions... along with the fact we needed to control the number of queries. Ideally we want 1 query for a page... certainly for the main page which is high laden with dynamic content.

Mongo Copying Documents - Collection to Collection

I'm really getting to like Mongo... a lot. One of the downsides is there is limited documentation... including limited forum comments on it. Because of this, I've decided to post interesting solutions to problems as I discover them... time permitting. so for todays tip. I recently needed to duplicate a "record" in mongo speak a document in a collection. There is so much information out there on how not to duplicate records that a google search at this point usually points you in the wrong direction. My situation was such that we have a CMS component to our application, which sets a publish day on content. What I needed was the ability to duplicate a record (so the BSON id would need to change and set a new publish day.

The first thing to recognize for those new to Mongo is that the Mongo console (the command line client to Mongo) works with javascript... so get your javascript fu ready... no.. not your jQuery fu... your plain jane generic javascript fu:)

Here is the solution I came up with:

db.NewsItem.find({newsType: "Blah"}).forEach( function(x){ x._id = new ObjectId(); x.publishDay=283;  db.NewsItem.insert(x)} );

So it turns out that you can pass a lambda expression to the forEach method of the collection. The trick is to assign a new ObjectId() to the _id, make any other assignments and then insert into your collection of choice... in this case back into the same collection. This comes in handy as well when you want to backup a collection real quick. (yea, I know you could use the mongodump and mongorestore, for a quick temp solution this just seems more convenient)

db.NewsItem.find({publishDay : 285}).forEach( function(x){ db.NewsItemBackup.insert(x)} );

Happy Coding!

Presenting Rocking the Gradle at JavaOne tomorrow 12:30pm at the Parc 55... see you there!

Wow… has it really be 11 months since I’ve blogged. I had many ideas for blogs half written or half thought out… but last year was extremely busy. It’s a new year… and I’m back :) Let’s not focus on the past, let’s get right into it.

I had a client recently asking how to improve throughput. It took me a few minutes of questioning to understand what the subject really was. Thoughtput? Are you having scaling issues? It turns out, the interest is closer to developer throughput, although I don’t think they used that term and it may have been broader than that. The interest is in the acceleration of moving an idea into a product. Immediately coming to mind is the book “The Goal”, whose focus it is to increase the throughput of a manufacturing plant. Although the creation of software is nothing like an assembly line (although the analogies seem to be common place), it is however interesting to consider some of the similarities involved with the theory of constraints. The challenge is, in my opinion, that the constraints for software development are more virtual and harder to measure than assembly line constraints.

Consider what slows down the development of software. Of all my years of software development, the issue has never been an issue of programmer to keyboard. In other words, it isn’t how you type or how fast you type, although it certainly needs to be reasonable. It often times isn’t how long a developer has been developing… although again there is a required threshold. Where have there been issues? Let’s see if you list is similar to my list:

• Poorly written requirements. The user seems to know what they want, but was unable to communicate it effectively or the BA was unable to capture it appropriately.
• Lack of direction, mission and purpose of the entire team… either we have too much time, too much money or a total lack of leadership. Let’s chalk this up to priorities.
• Lack of knowing what to do next… BA or programmer… or PM
• Lack of knowing how what a programmer is working on fits into the whole
• Lack of team coordination
• Large team sizes

I’m sure there are plenty more… but what I would say is the #1 issue for teams in trying to achieve high velocity is clarity of thought. I mean the ability to see a problem as the user sees the problem. Being able to see a solution and understand the problem in abstract terms and abstract ways; For a team to have the same thought or to hold within their minds the same mental model / picture. This is a tough task as differing priorities, differing locations, differing experiences and differing thoughts among the team members results in a number of differing ideas to the solution. It brings to mind to me of the scene in the last samurai, where Koyamada tells Tom Cruise “Too Many Mind”… “No Mind”

As Dr. Covey says in 7 Habits of Highly Effective People (which is what we are talking about… right?), you must start with the end in mind. Clarity positions people to be as effective as they can be. In the real world, a user really may not have the clarity themselves or may not understand the problem fully to articulate the desired clarity. This is why an agile development process is so effective. It provides an opportunity for the user to discover the clarity through regular feedback with the developers. Or in other words to detect deltas in the forming solution from the ideal or user desired solution. It also places the developer close to the user, so that through the creation of software when the output drifts from the user mental model it is caught quickly and corrected. Also developers are pairing with, and communicating with other developers regularly so that differences in the mental model are detected early.
What we are seeking for efficiencies sake is the fastest mechanism to detect that something is just not right.

So at the time of development what are the common practices that occur in a corporate development which cause inefficiencies or put another way what are the practices that cause the error detection to be delayed, either the detection itself is delayed or the communication of the detection is delayed.

• Developers are several degrees removed from the user. So there are either 2 human “hops” for error recognition and for communication.
• Developers are too junior. Junior developers are just less capable of thinking and communicating in abstract software terms and are less able to detect deficiencies in the solution model.
• Separation of the team members. Numerous studies show the increase cost of communication of people in just 2 different rooms, its more expensive for 2 different floors and way expensive when there is an ocean between members of the team. There is an additional cost that isn’t well documented in the fact the difference in the mental model or solution set are very difficult off-shore.
• Communication is all facilitated through paper. This is the worse of all the communication modalities.

So what can you do to increase throughput? What can you do to increase team velocity?

1. Hire highly skilled developers, developers skilled in their language who also have a proven record of quickly translating problem sets into solution sets.
2. Provide them with quick, and easy access to the user
3. Put the developers in the same work space
4. Reduce the team size, which results in less possibilities of mental model differences.

You are looking for a team with a high attention to detail and ability to detect differences in the mental model during the development process. Then you are looking for well-lubricated spontaneous communication.

This isn’t fool proof but it does solve one of the largest most common problems I see in corporate software development.

I was off for the holidays which gave me some time to clean out the storage area. I ran across some notes from a conference I attended in 1998 and 3 core principles stood out that I thought I would share as we start this new year.

Core Principles (as I wrote them many years ago):
1. Smaller is better than larger
2. Understood is better than unknown
3. Progress is better than promises

Smaller is better than larger
There was nothing else written and I left nothing by way of context... regardless of what it meant then, it is clear that this is a great principle when it comes to code. Less code that does the same amount of work is better. The paradox is that it may take a little more time to develop a smaller code solution, but it pays off. The skill is in not being so abstract and so small as to not be readable or maintainable. It is this balance that makes a true software craftsman.

Understood is better than unknown
If there is one word that we would use to describe the issues of software development... it is the word "unknown". Estimating the known is easy. Estimating the unknown, is unknowable. The skill in software development is to separate the unknowns from the knows. Estimate the knowns and provide a SWAG estimate for the unknowns... these are estimates that you need to keep in check. It is best to delay the estimates and work of the unknowns until ( what Kevlin Henney brilliantly describes as) the last responsible moment. Of course this only works for the known unknowns... what will get you is the unknown unknowns:)

Progress is better than promises
This is why I have been a practitioner and trainer for XP and agile practices for many years. It is all about developing business value on a regular and iterative basis. Promises are meaningless... progress is all that matters.

Happy New Year!!

As part of the Lambda Lounge, established by Alex Miller (Thanks!!), we have started a group to study the SICP. We just started and are going virtual... if you would like to participate, email or tweet me (@kensipe). In the process of studying SICP and LISP, I plan on focusing on Clojure. Clojure was previously on my machine as I was reading Stu's book Programming Clojure, however increased usage would require some maturing of my tools. This led to the discovery of Mark Reid's blog on Setting up Clojure for Mac OS X, however I assume based on updates and changes over the last year, it requires some updates.

Read Mark's blog first... getting clojure, the clojure-contrib and jline. Here are the differences:

• Create an environment variable CLOJURE_HOME assigned to the directory for clojure
• Create the /bin directory for the clj script included below
• Add $CLOJURE_HOME/bin in the $PATH
• Clojure is distributed with clojure.jar in the CLOJURE_HOME... so I maintain that. It is expected that contrib and jline are in the same directory.
• Mark's clj script has been updated below... The previous script invokes classes which are depreciated.
  

Here is the updated script:

#!/bin/bash
# Runs Clojure using the classpath specified in the `.clojure` file of the
# current directory.
#
# Ken Sipe   2010-02-20
# Inspired by Mark Reid 
# original: http://github.com/mreid/clojure-framework/blob/e1c80cc650f448713243be8272dba1fa3c1a7cea/clj
#
# This scripts expects $JAVA_HOME and $CLOJURE_HOME to be defined for the system it is running on
# The clojure.jar is standardly in the clojure_home dir (and not in the lib directory), so
# the expectation is that clojure-contrib.jar and the jline.jar are placed in the same dir

CLJ_DIR=$CLOJURE_HOME
CLOJURE=$CLJ_DIR/clojure.jar
CONTRIB=$CLJ_DIR/clojure-contrib.jar
JLINE=$CLJ_DIR/jline-0_9_5.jar
CP=$PWD:$CLOJURE:$JLINE:$CONTRIB

# Add extra jars as specified by `.clojure` file
if [ -f .clojure ]
then
   CP=$CP:`cat .clojure`
fi

if [ -z "$1" ]; then
   java -server -cp $CP \
        jline.ConsoleRunner clojure.lang.Repl
else
   java -server -cp $CP clojure.main -i $*
fi

Love the TextMate support and the history support in JLine!