Why you should pair program for six hours each day - half a year later

A while a go, I created this Medium post on pairing and how many hours of pairing is best for you and for clients.  You can read it in full on Medium. 

Some observations, half a year later:

- People find this confusing.  For far too long, we have been drumming the "we pair all the time" beat and it feels weird to roll this back now.  But then again, is good agile process not all about continuous improvement and reacting to changing circumstances?  When pairing started, most of us were not aware of the drawbacks (and those that were, voted with their feet by leaving).   To quote the Big Lebowski: "New stuff has come to light" (slightly edited for content).

- My management is not sure about this:  A company-wide rollout would remove barriers to using this, but they (understandably) worry about the economic consequences.  On the other hand, almost all other consultants never pair and they still bill for all of their hours, so there must be a way to resolve this. 

- Our clients get it.  There has been near-zero push-back from the client stakeholders on projects where we tried this. 

- Under-represented minorities are disproportionally affected by the lack of flexibility that is inherent in pairing for 40 hours a week. I don't have any public, hard data to prove this yet, but if you think about combining 9 to 6 with, for example, being a parent you can get a good idea of how hard this is.  

  

Marking Jenkins builds with Badges

We can adjust the status of a Jenkins job to Unstable, Failed or Successful, and we can add handy badges to the builds to signal warnings or errors.  All from the Groovy PostBuild Plugin. 

Jenkins is like your older uncle who helps you get through college: Dependable but a little clunky, stuffy but also necessary.  Turns out uncle is wearing a brightly patterned sports coat at times, in the shape of build badges:

What is also nice is that a Jenkins build can have final status other than failed (red ball) or passed (blue ball).  You may have seen NOT_BUILD (grey ball) and broken your head over UNSTABLE (yellow ball).

Aside: What is unstable supposed to mean, you ask?  In Jenkins' slightly warped view of the world, if the build step is successful (ie. the compiler did its job) but the tests failed, a job is unstable.  In normal agile terms, that is just a plain fail.  To add to the confusion, only certain post-build actions are able to move the status to unstable, so you may rarely encounter this.  But it is a great status to (ab)use if you want to show alerts on your job.

Setting build status and badges

So how can we access all this goodness?  Groovy is the answer here.  Note that I am applying groovy to any style Jenkins jobs, not only pipeline jobs that already groovy-fied.

You want to add the Groovy postbuild plugin to your Jenkins and then create a Groovy post-build action.  Here is some sample code that unconditionally set the build to Unstable.   Run it and you should see a yellow ball and an unstable outcome.
   import hudson.model.*
   def build = manager.build
   build.@result = hudson.model.Result.UNSTABLE
   manager.addErrorBadge("That really wasn't very stable")
   return


There are lots of other ways to do the same, and other things you can do, see the documentation for this plugin.  For example, you can also use 'manager.buildUnstable()' instead of that last line.

The Jenkins interface has an nice 'run in sandbox' checkbox, which you really want to check because it limits the abilities of your Groovy scripts.  However, it didn't work for me:  I basically could not run any Groovy code with this set.  The docs say that all the methods we used above are whitelisted, but apparently things have changed (or my Jenkins version is out of date).

Reading and checking build files

For my problem, I needed to get slightly more fancy and tests a few files in the current job.  Note that it is much easier to test for log lines in the current build, the method 'manager.logContains(REGEXP)' can help you with that.

To test for certain files in the current build directory, you can use:
def build = manager.build
def workspace = manager.getEnvVariable('WORKSPACE')

if (new File(workspace + '/mySpecialFile').exists()) {
   manager.buildUnstable()
   manager.addErrorBadge("Unit tests were not run")
   manager.listener.logger.println("Marked unstable")
} else {
   manager.listener.logger.println("All well") }

return

This checks for the presence of 'mySpecialFile' under the project root.  If you build has produced such a file, it will be marked unstable and a log line will be emitted.

Getting a list of all variables available to you

How did I find out about the workspace variable?  I ran this piece of code, modified from one of the comments in the Groovy PostBuild Plugin page:

def build = Thread.currentThread().executable
def buildMap = build.getBuildVariables()
buildMap.keySet().each { var -> 
       manager.listener.logger.println( var + "= " + buildMap.get(var)) }
def envVarsMap = build.parent?.lastBuild.properties.get("envVars")
envVarsMap.keySet().each { var -> 
       manager.listener.logger.println( var + ": " + envVarsMap.get(var)) }

This will show the build variables (parameters to the build) and the environment variables.  My result (slightly redacted; no build variables):

BUILD_DISPLAY_NAME: #29
BUILD_ID: 29
BUILD_NUMBER: 29
BUILD_TAG: jenkins-test-29
BUILD_URL: http://localhost:8080/job/test/29/
CLASSPATH: 
DERBY_HOME: /usr/lib/jvm/java-8-oracle/db
EXECUTOR_NUMBER: 1
HOME: /var/lib/jenkins
HUDSON_HOME: /var/lib/jenkins
HUDSON_SERVER_COOKIE: 7......d
HUDSON_URL: http://localhost:8080/
J2REDIR: /usr/lib/jvm/java-8-oracle/jre
J2SDKDIR: /usr/lib/jvm/java-8-oracle
JAVA_HOME: /usr/lib/jvm/java-8-oracle
JENKINS_HOME: /var/lib/jenkins
JENKINS_SERVER_COOKIE: 7.....d
JENKINS_URL: http://localhost:8080/
JOB_BASE_NAME: test
JOB_DISPLAY_URL: http://localhost:8080/job/test/display/redirect
JOB_NAME: test
JOB_URL: http://localhost:8080/job/test/
LANG: en_US.UTF-8
LOGNAME: jenkins
MAIL: /var/mail/jenkins
MANPATH: /opt/OpenPrinting-Gutenprint/man:/opt/OpenPrinting-Gutenprint/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/lib/jvm/java-8-oracle/man
NLSPATH: /usr/dt/lib/nls/msg/%L/%N.cat
NODE_LABELS: master
NODE_NAME: master
NODE_PATH: /usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
PATH: /opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
PATH+CHROMEDRIVER: /var/lib/jenkins/tools/chromedriver
PWD: /var/lib/jenkins
QT_QPA_PLATFORMTHEME: appmenu-qt5
RUN_CHANGES_DISPLAY_URL: http://localhost:8080/job/test/29/display/redirect?page=changes
RUN_DISPLAY_URL: http://localhost:8080/job/test/29/display/redirect
SHELL: /bin/bash
SHLVL: 1
USER: jenkins
WORKSPACE: /var/lib/jenkins/workspace/test
XDG_RUNTIME_DIR: /run/user/125
XDG_SESSION_COOKIE: f...........0
XDG_SESSION_ID: c20
XFILESEARCHPATH: /usr/dt/app-defaults/%L/Dt

You can check this output against http://localhost:8080/env-vars.html.



 Hope you can use this in your work with Jenkins!

A pragmatic Git strategy for fast moving teams

Git is a wonderful tool that is so flexible you can use it in many ways.  With that has come the discussion of how to use it best.  Of course, the answer to depends on your priorities.

Here are the priorities I see often in agile teams delivering software fast:

1. Members of the team need to work in temporary isolation (read: branches)
2. Git and git history is a way to communicate your code changes to the rest of your team
3. You have tests that give you confidence in your final code
4. Recent Git history is valuable if something goes wrong
5. When you build something, you ship it asap

Let's apply these principles to our thinking about various parts of a Git strategy:

Feature branches:  A pair (or a soloing dev) should work on a branch (isolation), and branches are best organized by topic.  Feature work should be short lived so it does not diverge.  Merge master back into the feature if in doubt.  

git checkout feature

git merge master

Feature branches can all start with a common prefix (ft_) but they do not have to.   

Release branches:  If we really ship all we have, every time, there should usually be no need for release branches.  Just release master, optionally from a couple of commits ago.   If you are just in the middle of a big change, either do not merge that feature yet or make a once-in-a-lifetime release branch. Tag your release:

git tag release-1.1 

git push origin release-1.1

Merging features:  A feature branch will have detailed commit history on it, which is great if you need to go back to it next week.  But to keep master easy to follow, you squash your work into one commit when it goes to master.  Keep the feature branch around for 4 weeks, then delete it.

You can accomplish this in many ways, this one is easy to understand:

git checkout feature # Use the name of your branch here

git checkout -b temp  

# Solve any conflicts between feature and master:

git rebase master 
## if conflicts occur, fix, git add and git rebase --continue

git checkout master
git pull

git merge --squash temp

git commit 

git branch -D temp

At this point, you feature branch will not have changed and all the work of merging it with master is hidden in one commit on master.  Write a recurring calendar event to delete old branches on the first of each month.

During the 'git rebase master' step, you may have to merge files multiple times.  This is tedious but it makes the individual merges easier.  If you are impatient, skip this step and deal with the conflicts during the merge step. 

If your feature branch warrants two or more commits: It happens, you did two things in one feature.  Proceed as follows:

git checkout feature

git checkout -b temp  

# Solve any conflicts between feature and master:

git rebase master 

# Reduce to a few commits:

git rebase -i master 

## An editor will open up, change the 'pick' to 'squash' on all 

## lines except the first one and other final commit.  

# Now update the commit message if you haven't already done that:

git commit --amend  

# Merge and clean up

git checkout master

git merge temp

git branch -D temp

Happy working with Git!

Finding memory leaks in Java

Here is what we found out about finding memory leaks today - we have not fixed our problem yet so there may be another installment in the works :-)

Choose your tool

There are a number of good tools to observe and inspect memory use in a Java app.  We were alerted to our problem via AppDynamics, which is suite for monitoring live apps.  Back on the dev station, you can use VisualVM (free) or YourKit (with a 14 day free trial) to visualize Memory usage, CPU usage, Garbage Collection, etc.

Integration with IntelliJ

We found that YourKit integrates very well: Install the YourKit Java Profiler Integration plugin.  You get a new run option that starts your app and connects it to YourKit.  The alternative is to start the app normally and connect to it, like you would with any remote debugger.

Monitoring Local Server

This is the memory view before we start hitting the server:  Notice how the VM starts up and creates lots of heap objects (left), on the right you see the class meta data (metaspace, in light green) grow.  If you are unsure about Java memory spaces, check out YourKit help page,  and this page about the transition from PermGen to MetaSpace.

Monitoring Remote Server

Testing on a dev station is all nice and well but the real challenge was connecting to a Java app running on a CloudFoundry sandbox.  It turns out that the default Java buildpack has support for a large number of options, one of which is YourKit.

How it works:
The java buildpack has a your-kit-profiler.yml.  This Yaml file disables the profiler by default.  So the trick is to either deploy the app with a custom buildpack, or somehow override the contents of this yaml file.

It turns out that the buildpack has a interesting mechanism for (partially?) overriding the contents of yaml files:  You include a special environment variable in the env section of your manifest.  The name of the variable is 'JBP_CONFIG_' + the name of the yaml file but without '.yml'.  The value of the variable will be the new contents of the yaml file, overriding what was there before.

The only limitation is that you have to squeeze all your values on one line and be careful with quoting rules.  For us, the result looked like this:

applications:- host: HostAway
  memory: 1024M
  env:    SPRING_PROFILES_ACTIVE: sb
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "sandboxdocservice"}'

The one line of yaml contents does not come out so well, it contains:
applications:
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "HostAwayWithYourKit"}'

Bring up your app by pushing it to CF as normal.  To reach it, you will have to create a secure tunnel to your workstation:

cf ssh -N -T -L 10101:localhost:10101 HostAway

Back in YourKit, go to the Welcome screen and choose 'connect to remote application'. Enter 'localhost:10101' and you should be good to go.

Thanks!!

Tons of thanks to Tiffany Chang and Ajay Pillai for working with me on this.

Building a Frontend Toolchain

• front end tools
• Using Spring for assets
• NPM
• Bower
  • Useful bower commands:
  • Dependencies in Bower
• NPM vs Bower
• Using Gulp
  • Moving the resources
  • Main-bower-files
  • ProcessHtml
  • gulp watch
  • More gulp
• Back to maven
• Making this work in IntelliJ
• Final workflow
• Conclusion
  • Some further links

We have so far focussed on getting our application working and have not bothered much with setting things up right.  This post is about setting up a good workflow for our frontend files and assets, that is, our html, css and javacript files, including the angular framework.  

Using Spring for assets

As explained in this 2013 blog post, Spring will copy static resources that are located in src/main/resources/static for us at compile time. There are actually a few more directories where Spring will copy files from (such as src/main/resources/public) and it will put these files intarget/classes/static/ (or public, etc).
However, putting our javascript into this directory is getting tedious and we don't have any way to track versions and pull in updates to our javascript. On top of that, we would like to combine all our javascript in one file and minify that file. For these tasks, we need no fewer than three tools: Npm, Bower and Gulp.

NPM

NPM apparently does not mean Node Package Manager, but it sure behaves a lot like that would be a good name for it. It requires node installed on your dev machine, which is no big deal.
Basic characteristics of NPM are:

• NPM is based on its own npm repo, which has 1000s of packages in it. You can publish your own js pacakages to that.
• NPM is the first tool on the scene for most pipelines, it is used to pull other tools in via a npm install bower etc.

How to use it:

• Basic command npm install .... This will get you the most recent version and put it in node_modules/
• Variant for global installation (ie not in current directory, but in some /usr/lib folder: npm install -g ...
• Command to search: npm search ..., you can also look at https://www.npmjs.com/
• More structured approach is to have a package.json file, see below for details. If that file is present, you can just run npm install to download and set up all dependencies the project needs.

This last option is great: It means that somebody can clone the repo for this project and run npm install; bower update to get all requirements in.
Using package.json:

• First create a file package.json with {} as the contents
• Then run npm install ... --save from the command line and each install will update the package.json for you!
• Normally, the dependency will go into the dependencies section, if you run npm install .. --save-dev the name will go into the devDependencies section of the package.json. This is what we do, see below.
• You may want to edit your package.json to specify exact version numbers, ie from bower: ^1.7.7 (which means that version and up) remove the caret.

Setup for NPM for this project:

$ echo "{}" > package.json
$ npm install bower --save
$ npm install jshint --save
$ npm install gulp --save

Bower

Bower will record the files you install in bower.json, the bower init will create a first version of that file for you.
Setup of Bower for this project:

$ bower init

Just answer all questions, leave 'main', 'moduletype' empty for now and accept defaults for 'ignore' and 'private'.

$ bower install angular#1.4.9 --save
$ bower install angular-animate#1.4.9 --save
$ bower install angular-uuid --save
  # told bower to choose the 1.4.9 version and record it 
$ bower install angular-cookies#1.4.9 --save

Useful bower commands:

• Install a new package with bower install angular --save, similar to npm, this will download the package into bower_components and update the bower.json file for you.
• In my case, this downloaded Angular 1.5, while I wanted to stay on 1.4, so…
• To uninstall a package, run bower uninstall angular --save, here the save flag removes the reference from the bower.json file.
• You can use hash syntax to request a specific version, bower install angular#1.4.9 --save.
• If you forget the --save flag you can simply run the command again with the flag.
• To see which packages you have installed, and which one can be updated, bower list
• To update a package, run bower update ..

Dependencies in Bower

Bower keeps track of dependencies for you and will also record version numbers in the bower.json file, as we saw above. However, bower is not smart about dependencies, you will have to do that for it.
For example, I installed angular version 1.4.9. When I tried to install angular-animate however, the latest version of that package is 1.5.3 and it requires angular 1.5.3. So I will end up with a mixed situation, which is undesirable (to be fair, Bower does warn you about this). Bower does not work like apt-get(the Debian/Ubuntu package installer) and others, which will ask you to upgrade angular or fail the installation of angular-animate.

NPM vs Bower

Separation of concerns between NPM and Bower

• use package.json and npm install to keep your tools up to date
• use bower.json and bower .. to keep your front end dependencies up to date
• This keeps your tools in node_modules and your dependencies in bower_components, which is convenient for further processing of the dependencies.

Using Gulp

Once, at the dawn of computer ages, there was make, which 'made' your computer program for your from sources. Make was the first of many task runners, programs that help you to compile sources, compress javascript, move assets around and package things up.
We have earlier talked about maven and gradle, the two main task runners in java land. For various reasons, javascript has its own task runners, with grunt and gulp as the two main contenders at the time of writing. We chose gulp here because it is generally easier to set up and it provides the watch functionality out of the box.
We will first need a fair few gulp components. I added them with the code below, but you can just do npm install.

npm install main-bower-files --save
npm install gulp-concat --save
npm install gulp-uglify --save
npm install gulp-print --save
npm install gulp-rename --save
npm install gulp-add-src --save

Finally, I created a symbolic link from my top level directory to the gulp binary, as follows

ln -s node_modules/gulp/bin/gulp.js gulp
git add gulp

This works in all unixes (Linux and OS-X).
Gulp's settings and tasks are in gulpfile.js, which is pretty human readable once you get used to the format: First, all dependencies are required and the results are assigned to variables. These variables are used as functions further down the line.
Gulp has a streaming concept, similar to functional programming or flow-based programming. Basically, you start a stream of files with a gulp.src command, then operate on that list of files in various steps. Each step is wrapped in a .pipe() function, similar to the use of pipes in bash.

Moving the resources

Until now, our frontend resources lived in src/main/resources/static and as we saw above, Spring will copy them to the target directory for us (Spring will copy everything under src/main/resources).
We will have to move the resources now, Gulp can take care of copying things and if we leave them, they will be copied twice. I have chosen src/main/frontend/ in this project. After you create the directory, make sure to right click it in IntelliJ and choose Mark Directory As ..: Sources Root. The folder will turn blue (in my color scheme) and IntelliJ wil index the files in this directory.
Without explaining Gulp in full yet, copying files is easy: Below is the source for a css task, which will copy all .css files from source to dest. It starts by declaring the task. Then we have a gulp.src call which will typically produce a list of files/filenames (Gulp bundles a filename and the file contents together in something it calls a Vinyl. Also note that technically, this is not a list but a stream).
Then we do several operation on these files/filenames, each wrapped in a .pipe() call as mentioned above. Our first step is print, which prints out the filename to the terminal. Our second and last step is gulp.dest, which will write the file contents out to the directory specified.
We use two small magic tricks: In the gulp.src, we use a javascript variable src instead of a full path; and we use /**/*.js to denote any js files in src or subdirectories of src (recursive copy).

var print = require('gulp-print');
var src = 'src/main/frontend/';
var destprod = 'target/classes/static/';

// copy css files
gulp.task('css', function() {
    gulp.src(src + '/**/*.css')
 .pipe(print())
 .pipe(gulp.dest(destprod));
});

Read this excellent, pretty and short post on the principles of Gulp. This introduction helped me a ton.

Main-bower-files

Above, we introduced a separation between tools (managed with NPM) and front end libraries (managed with Bower). Now, we would like to combine and minify all this JS code into one file. There are standard Gulp plugins for these actions, but how do we get the names of all the frontend libraries we need?
The main-bower-files plugin can help us with that: It will essentially do a bower list command for us and create a list of front-end libraries to ship to the user, in the right order (if file B depends on file A, it should come after A in the combined js). Of course, we also have our own js file(s) and these should come after the library files. gulp-add-src plugin can take care of that.

var bowerfiles = require('main-bower-files');
var addsrc = require('gulp-add-src');

var destdev = 'target/classes/static/devresources/'

// copy all js files for dev targets
gulp.task('js-dev', function() {
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(gulp.dest(destdev + 'js/'));
});

You may have noted this version does not do any minification etc, and is called js-dev. It also copies to destdev instead of destprod. We have a matching js-prod, which concatenates all files together under the name 'steamvoat.js', then renames that to 'steamvote.min.js' (we could have combined and renamed in one step), then runs the 'uglify' minifier, and finally copies the files out to the production target directory. To avoid confusion, the prod tasks starts out by removing all the full-length dev files under the destdev directory.

var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');
var del = require('del');

// minify and package all js files for production targets
gulp.task('js-prod', function() {
    del(destdev);
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(concat('steamvoat.js'))
 .pipe(rename({suffix: '.min'}))
 .pipe(uglify())
 .pipe(gulp.dest(destprod));
});

The benefit of this approach is that production is fast, using minified files, while we can easily inspect and debug the full files in our development environment.

ProcessHtml

There is one small detail to deal with: We have replaced five js files with one combined and minified file in production, but the <script src commmands on our main page do not reflect that. We could simply source all full files as browsers throw a harmless error when files are not found, but there is a better way. The ProcessHtml node module, which allows you to replace html, use templates and generally modify html to your needs.
You can see it in action at the tail end of our index.html. It starts with a html comment with build: as the first word. The 'build:js= subcommand will replace many lines of js scripts into one, the second argument specifies the name of the combined js file.

<!-- build:js  steamvoat.min.js -->
<script src="/devresources/js/angular.js"></script>
<script src="/devresources/js/angular-cookies.js"></script>
<script src="/devresources/js/angular-uuid4.js"></script>
<script src="/devresources/js/angular-animate.js"></script>
<script src="/devresources/js/index.js"></script>
<!-- /build -->
</body>
</html>

When run through ProcessHtml, this will show up as

<script src="steamvoat.min.js"></script> </body></html>

A good overview of what you can do with ProcessHtml is given in this blog post. For full information, you have to know that there is an underlying node version of ProcessHtml which does all the work, with specific grunt and gulp versions built on top of that. The best docs are over at the grunt version.
The build commands should have modifier to make them work only on certain Grunt targets, but this does not seem to work for Gulp. This may be a good thing, now all the logic resides in the gulpfile.
In our gulpfile, there are therefore two rules for copying html files: A dev version which plain copies, and a production version which runs processHtml on the html.

gulp watch

Half the reason for using Gulp is the excellent support for 'watching' files. Imagine that every time we change a html file, it automatically is copied over to the target directory so that our test server will use it if we refresh the page. This can be done by running gulp watch

// Watch for changes in files
gulp.task('watch', ['dev'], function() {
    // Watch .js files
    gulp.watch(src + '**/*.js', ['js-dev']);
    // Watch html files
    gulp.watch(src + '/**/*.html', ['html-dev']);
    // Watch css files
    gulp.watch(src + '/**/*.css', ['css']);
    // Watch image files
    gulp.watch(src + '/img/*', ['images']);
});

Now if you run ./gulp watch, gulp will build the dev version and then your terminal will seem to hang but whenever you make a change to any of your resources, it will run the appropriate target to copy the modified files over for you.

More gulp

You can create endless complexity with gulp and it is reasonably manageable. We can for examples combine our css into one file, and preprocess Sass or Less files first.

Back to maven

To tie it all together, it would be great to have a unified interface to npm, bower and gulp. Also, we have various maven commands that we already need to build java, create a package for CloudFoundry etc. These take care of the java side, but our app will not work if the Gulp dev task has not run.
So how do we tie Gulp to Maven, our backend build tool? There is a great plugin "Front End Maven Plugin" for maven that takes care of all these tasks for us. I did not use it here because I did not want my gulp tasks to be run all the time and I don't want to install a local copy of node either. For larger projects with Continuous Integration, the plugin would be great. For this small size project, setting it up by hand was easy enough.
I use a small plugin to Maven that allows you to run any shell command. This does make the setup dependent on gulp and the exact location of gulp, but I can live with that.
First, we declare a plugin exec-maven-plugin. This plugin is run in the process-resources phase and it will run the exec goal. The plugin will run the executable mentioned in the configuration section, here gulp production:

<build>
    <plugins>
     //   ... stuff deleted ...
 <plugin>
     <artifactId>exec-maven-plugin</artifactId>
     <groupId>org.codehaus.mojo</groupId>
     <executions>
  <execution><!-- Run our version calculation script -->
      <id>Run Gulp</id>
      <phase>process-resources</phase>
      <goals>
   <goal>exec</goal>
      </goals>
  </execution>
     </executions>
     <configuration>
  <executable>node_modules/gulp/bin/gulp.js</executable>
  <arguments><argument>production</argument></arguments>
     </configuration>
 </plugin>
    </plugins>
</build>

Making this work in IntelliJ

We want the gulp task 'production' to be run when we start the application from IntelliJ. There are two ways to do this.
The simplest way is to not use the IntelliJ configuration target (our main application file), but use the maven task spring-boot:run as the configuration to run. This has several drawbacks, mainly that you cannot configure how you run the target so well.
The nicer way is to set up an IntelliJ configuration to run the gulp task (see also IntelliJ help) and then make our main IntelliJ configuration run this gulp configuration by default. Here is how that goes:

• Go to the configurations dropdown of your IntelliJ. For me, this sits in the top right corner. Choose 'Edit Configurations'. Alternatively, go to Run: Edit Configurations.
• Create a new configuration with the plus button (top left)
• From the dropdown, choose Gulp.js
• An 'unnamed' gulp task is created, call it 'production' (or whatever)
• Make sure that the gulpfile textbox points to your gulpfile.js
• Under task, select 'production'
• All other settings should be fine and can be left open if they are blank.

Now try your new gulp configuration by choosing it from the drop down, then pressing the Run icon (green triangle). Alternatively, use Run: Run.... A 'gulp' tab should open on your "Run" view (bottom of IntelliJ screen) and you should see the production task copy files and conclude that with a satisfying "Process finished with exit code 0".
Finally, we need to tell our main configuration (which runs our app) to run the gulp task first.

• Go to Edit Configurations again
• Under 'Spring Boot', you should see a configuration that runs your app. Click on the configuration.
• On the bottom of the window, there is a box called "Before Launch: .." with one task in it, called "Make" (if you do not see that box, stop all running processes and run this configuration once).
• Click the plus button, then click "Run another configuration" and select your gulp 'Production' task.
• Use the triangle icons to have Make be the last item.

Final workflow

When a new clone of the project is made, you will have to run a few commands to get the dependencies:

npm install
bower update

You will also have to run these commands every now and then to pick up new versions of the packages you are using. I think this is a great workflow, as it means you will not be surprised by new versions. You have to remember to run the commands on occasion.
To build and run the project, you can use maven or IntelliJ, as outlined above.
When making changes to the frontend files, you can view the changes with a simple browser reload if you run a watch before you start editing:

gulp watch

And that is it!

Conclusion

This seems a lot of work but it is definitely worth it if you want to maintain your app easily and if you want to automate the build and deploy process.

Some further links

• http://www.radcortez.com/javascript-package-management-npm-bower-grunt/
• http://andy-carter.com/blog/a-beginners-guide-to-package-manager-bower-and-using-gulp-to-manage-components
• http://andy-carter.com/blog/a-beginners-guide-to-the-task-runner-gulp

Making your own Google Forms

Summary

For a short assignment, we wanted to replicate the basics of Google Forms without using the actual product. The assignment was to set up a form on our own server but feed the results into a Google Sheets (the name for their spreadsheet product). Another requirement was to set this up as fast a possible, so we had to take some shortcuts. We navigated a lot of new terrain and in this blog post, I will discuss our findings and the solution we arrived at.
If you want you can jump straight to An outline of the code or to the repository

Why we chose this approach

Using Google Forms

We decided to not use the standard Google Forms interface:

• Forms is the standard interface to sheets, which does exactly what we want: It allows you to set up form with fields and the results will automatically be posted to a google spreadsheet. The users may have to log in to get access to the form (at your option), but the data are always saved in one central form that is owned by the creator of the form.
• In other words, for writing to sheets, form creator credentials are used, not user credentials.
• Forms does not support full html formatting out of the box, and does not easily allow you to run javascript functions to automatically update the page based on the user's entries.
• By default, if you Submit a form, this leads to one row in the spreadsheet. We needed a form that would lead to 1..6 lines in the spreadsheet.

In retrospect, this particular assignment could probably have been solved by dropping a few of the requirements and using Google Forms after all. Little did we know… However, the solution discussed here provides you much more flexibility than a pure Google Forms one.

Google sheets API

Google Sheets has its own API, which we did not use either:

• Sheets has its own legacy API, but using GAS (below) is recommended
• Sheets own legacy API is not as powerful, available for Java and .Net only, requires you to install Google jars, and requires you to set up Oauth. These were all reasons to reject this for a quick solution.

GAS: Google Apps Scripts

• GAS stands for Google Apps Scripts, which is a javascript environment that allows you to access all of the Google Apps (mail, docs, sheets, etc.) from the cloud.
• GAS is great, but it is really its own thing with a lot of domain specific knowledge built into it.
• GAS is geared towards having scripts operate on the users own documents, the case of multiple users all writing to the same sheet is supported but less well documented
• GAS is really good for creating new formula operators in Sheets, or to set up fancy dashboards on top of sheets.
• GAS scripts have access to a wealth of built-in functionality which you can just use, without any includes or injection. Critical to our project were PropertiesService (to store information between invocations of the script), `LockService (to prevent concurrent access to the spreadsheet), `ContentService` (to construct and format http replies).
• GAS scripts can render html forms, but the support for it is rather involved (you have to write a doGet method that returns the html, as if you are a writing a web server in javascript). It is unclear to us whether you can use CSS, standard libraries and other tools for modern fast-paced development.

GAS as an API

We did not use GAS directly, for reasons outlined above. Instead we will write an application which interacts with a GAS script:

• There is no set API for GAS, instead you write a javascript script that has a doGet and/or a doPost method. You publish this script and a URL is generated that functions as the external call point for your script. You can then add parameters to the GET or add a body to the POST to pass information into the GAS script.
• It seems that GAS wants its POST data in the old fashioned form-data format, which requires you to do some work on the Spring side (see code below).
• This may be a misunderstanding on our end, the documentation on the parameters to a external call make it sound like other post body types are possible too (csv in the example on GAS). However, this Stack Overflow posting makes it clear that x-www-form-urlencoded (aka old-fashioned forms) is the best choice and that it is the default.
• GAS restricts input parameters to a few primitive javascript types, more than enough to work with but it is good to be aware of the limitations.
• GAS scripts are not easy to debug, esp. when it comes to the interaction with the outside world. There is a logger but we had mixed success with it; you may also try the other options under the View menu.
• GAS as an API has a few idiosyncrasies: It seems necessary to give your script a new version number every time you publish it, there is a `/dev` endpoint that you cannot use from the external world, and so on
• There are subtle differences between publish as web app and 'publish as API' which are not clear to me. We ended up using the 'publish as web app' for our API, even though we only use our app as an API.
• GAS does not provide any headers that say it is ok to use this API from another website (ie. it does not disable CORS with Access-Control-Allow-Origin: *). CORS is Cross Origin Resource Sharing, a browser technology that prevents scripts on host A to touch endpoints on host B, unless host B specifically allows this. Because CORS is not specifically allowed on GAS, you cannot call your GAS API from javascript or anywhere in the browser. We thought we would build a front-end only app, but we ended up building a simplistic backend just to avoid CORS.
• GAS calls are very slow, they easily take up to five seconds on our state of the art machines on a fast network.

Related links

• Using Google sheets as a database, by Martin Hawksey. This blog post was very informative and put us on the path to our final solution. There is a live demo on the site that does not seem to work.
• Martin's gist updated by Corey Phillips.

An outline of the code

You can of course just peruse the repository, but here is an overview and some details that took us longer than they should have.
In a nutshell:

• We build a front end app with angular, bootstrap and ui-router to have a three page web form, much like in this blog post on scotch.io.
• This front end was served by a standard Spring Boot application, with exactly one endpoint for posting data to our GAS script.
• The back end was pushed out to Pivotal Web Services, our in-house CloudFoundry instance.
• We used a slight variation on the GAS script provided by Martin Hawksey. The GAS script is a bit slow to run so we would like to submit all rows of data simultaneously, but for now we make multiple calls to the GAS script.

Spring project, pom.xml

Via start.spring.io, we created a standard project with test and web options.

Manifest for cloudfoundry

The repo has a manifest.yml.template file which you should rename to manifest.yml after filling in the blanks, which define where your app will run on pws and where your GAS script lives (Google will display this URL when you publish your GAS script).

---
applications:
  - name: <YOUR APP NAME HERE>
    memory: 512M
    path: target/vacationform-0.0.1-SNAPSHOT.jar
  env:
    VACATIONFORM_URL: <YOUR GAS SCRIPT URL HERE>

With this in hand, you should be able to `cf login` to your PWS account and `cf push` the code.
We did not specify a buildpack as java was automatically recognized.

Packaged files

We included the necessary files to run bootstrap, jquery, angular and angular ui-router. Style sheets for the bootstrap-theme. These files are `src/main/resources/static/`.

Main page: index.html

This is a ui-view and nothing more (well, add headers and script includes)

<body ng-app="vacation-form">
<div class="container">
    <div ui-view></div>
</div>

The steps

There are three steps, named step1.html etc with associated controllers also named step1 etc. In step 1, we create a factory that provides us with a shared data structure vacationData:

app.factory("vacationData", function() { // to share information
    var vacationData = {};    
    vacationData.name = "Michael Dirk";    
    vacationData.weeks = 0;
    //   [[ etc ]]
    return vacationData;});

Each of the steps starts by calling the factory and putting the vacationData object on the scope, as we will need it there:

app.controller("step1", function($scope, vacationData) {
    $scope.vacationData = vacationData;});

Step 1 and 2 contain forms that are linked to fields of our vacationData object. We also have a few computed fields, which are resolved in the init of the step 3.

<form>    
  <input type="text" ng-model="vacationData.name" > &nbsp; Your name

(Apologies to the bootstrap designers for the nbsp there)
You can move to the next step through a button with a symbolic state target:

<a ui-sref="step2">    
   <button class="btn btn-primary">Next Step</button>
</a>

Step 3 deals with sending the data to GAS, asking the user to wait, and displaying the result of the GAS call.

<p class="row">    
We have submitted your information for you.  
This request returned status:  {{returnStatus}}
</p>

javascript, index.js

For maximum hackiness, we put all our javascript in one file. It is a total of 70-some lines. It includes a dog standard router:

app = angular.module("vacation-form", ['ui.router']);

app.config(function($stateProvider, $urlRouterProvider) {
    $stateProvider
 .state('step1', {
     url: '/step1',            
     templateUrl: 'step1.html'        })
    // [[ stuff deleted ]]

In step 3, we have all the data and we make a call to our Spring backend:

$scope.returnStatus = "pending";
var url = "/api/setVacationPlans";
$http.post(url, vacationData).then(
    function (success) {
 $scope.returnStatus = success.data; },    
    function (failure) {
 $scope.returnStatus = failure.data; });

The endpoint on the backend

The POST endpoint takes a vacationRequest in its body:

@RequestMapping(path = "/api/setVacationPlans", 
  method = RequestMethod.POST, 
  consumes = {"application/json"})
public ResponseEntity vacationPlans(@RequestBody VacationRequest vacationRequest) {

Note the consumes argument to RequestMapping. The actual input to this endpoint is the vacationData over on the JSON side.

VacationData data structure

I have not detailed it above, but vacationData is a JSON object (aka dictionary or hashmap) with a few user details at the top level, such as the name. It has an embedded list of vacationPlan objects, each of which have dates in them. We want to generate a row in the google spreadsheet for each vacation plan that is submitted.
We rely on Jackson JSON to parse this JSON object for us. To enable this, we created a VacationRequest object, which contains a list of VacationRequest objects.

Posting to GAS from the endpoint

Continuing on the end point, we make one call to GAS for each vacation plan. First, we need to set up a RestTemplate to make the call for us. Setting this up feels unnecessarily complicated to me, but these calls are apparently necessary to make the RestTemplate ship the data we put into it with the form-field convention (which begs the question why there is no such thing as a FormFieldTemplate):

RestTemplate restTemplate = new RestTemplate();
HttpMessageConverter formHttpMessageConverter = new FormHttpMessageConverter();
HttpMessageConverter stringHttpMessageConverter = new StringHttpMessageConverter();
restTemplate.getMessageConverters().add(formHttpMessageConverter);
restTemplate.getMessageConverters().add(stringHttpMessageConverter);

We then make the call to GAS by making a map of arguments and passing it into the restTemplate. There are probably neater ways to do this, but this worked. The only bit of sophistication is that we declare a class variable URL which is filled in from a environment variable (or application.properties, if you feel so inclined).

@Value("${VACATIONFORM_URL}")
private String url = "https://example.com/exec";

 // stuff deleted

 @RequestMapping(path = "/api/setVacationPlans", method = RequestMethod.POST, consumes = {"application/json"})
 public ResponseEntity vacationPlans3(@RequestBody VacationRequest vacationRequest) {

 // stuff deleted

MultiValueMap<String, String> map = new LinkedMultiValueMap<String, String>();    
map.add("name", vacationRequest.getName());    
map.add("fromDate", plan.getFromDate());
 // stuff deleted
String response = restTemplate.postForObject(url, map, String.class);

The GAS script

This is a universal "add to spreadsheet" script, that matches the column headers in the spreadsheet to the incoming data. An new row in the spreadsheet is created and for each column that has a matching input parameter, the value is copied to the spreadsheet. Columns that are not in the input parameters are set to 'undefined' and input parameters that do not correspond to any column header are ignored (note that the match is case sensitive, and mind trailing spaces).
We used a very slight variation on the script by Martin Hawksey, we are using the updated version in this gist. Line numbers below refer to that version:

• The script will receive a request on doPost (line 39). Google calls the request an event. It is stuffed into the 'e' parameter, documented here.
• Before doPut can run, the script requires you to run setup once (from the GAS dashboard) so that the PropertyService has the spreadsheet's ID value (setup function on line 85).
• After a lock is required, the script opens the spreadsheet specified by the ID (line 49-54).
• A row array is created and populated by looping over the names in the header row (which is normally row 1) (line 57-66)
• In the successful case, the number of the new row is returned
• A reasonable error case is created, and the lock is released.

Unsolved problems

This code works but it still has a few issues:

• The dates that come out of the bootstrap date picker are not in the format that Sheets expects.
• There is no check on the user, we wanted to require google authentication with pivotal.io, but the current version does not do that. You can add that requirement to the GAS script when you publish it, but that will break things as you end up on our SSO login page.
• If a user does not list any specific plans in step 2, their submission is effectively ignored
• The code has zero tests. Not great but in our defense, most of it is devoid of any logic and the interesting parts (submission to google Sheets) will be hard to test mechanically.

There is a larger authentication issue which we found hard to solve: The user is authenticated with Google and Pivotal on the front end, but it is the backend that is contacting the GAS script. The back end does not have access to the cookies and authentication tokens that exist on the front end. This is a standard browser security precaution: Our web page lives on a site that is not part of Google or Pivotal, so it cannot read cookies or authentication tokens from those sites. We can therefore not ship the authentication to the backend and the backend has to contact GAS as an anonymous user.

In conclusion

It is a lot of explanation but not that much code. In the end, I think using the GAS script was the best way forward: I dislike adding a third layer to the project (frontend, backend and GAS) but it is a lot easier than going the Sheets API way, which requires you to set up Oauth. Because the GAS script is basically a universal add-to-sheet routine, it can be used as a standard component.

Thanks

To Mike Oleske who went on a three-day deep dive in Apps land with me. All the good code is his, all the mistakes are mine.

Updates

• 2016-03-24 Reformatted to look like my other posts

Coffee with Lady Gaggia

Lady Gaggia is our Gaggia Academia coffee maker.   I gave a lightning talk about how to use this machine.  There are three videos and a link to the quick start guide.  Enjoy a nice coffee on me!

Step one: Making a Latte or Cappuccino

Step 2: Steaming your own milk

Step 3: Cleaning the drip tray

You won't need all three steps to make a good coffee, step 1 should do it most of the time.

If you want to know more about the details, follow this link to the Gaggia Academia QuickStart Guide over at Gaggia USA.