Tuesday, November 27, 2018

Why you should pair program for six hours each day - half a year later






A while a go, I created this Medium post on pairing and how many hours of pairing is best for you and for clients.  You can read it in full on Medium

Some observations, half a year later:
- People find this confusing.  For far too long, we have been drumming the "we pair all the time" beat and it feels weird to roll this back now.  But then again, is good agile process not all about continuous improvement and reacting to changing circumstances?  When pairing started, most of us were not aware of the drawbacks (and those that were, voted with their feet by leaving).   To quote the Big Lebowski: "New stuff has come to light" (slightly edited for content).
- My management is not sure about this:  A company-wide rollout would remove barriers to using this, but they (understandably) worry about the economic consequences.  On the other hand, almost all other consultants never pair and they still bill for all of their hours, so there must be a way to resolve this. 
- Our clients get it.  There has been near-zero push-back from the client stakeholders on projects where we tried this. 
- Under-represented minorities are disproportionally affected by the lack of flexibility that is inherent in pairing for 40 hours a week. I don't have any public, hard data to prove this yet, but if you think about combining 9 to 6 with, for example, being a parent you can get a good idea of how hard this is.  
  

Wednesday, December 13, 2017

Marking Jenkins builds with Badges


We can adjust the status of a Jenkins job to Unstable, Failed or Successful, and we can add handy badges to the builds to signal warnings or errors.  All from the Groovy PostBuild Plugin. 

Jenkins is like your older uncle who helps you get through college: Dependable but a little clunky, stuffy but also necessary.  Turns out uncle is wearing a brightly patterned sports coat at times, in the shape of build badges:



What is also nice is that a Jenkins build can have final status other than failed (red ball) or passed (blue ball).  You may have seen NOT_BUILD (grey ball) and broken your head over UNSTABLE (yellow ball).

Aside: What is unstable supposed to mean, you ask?  In Jenkins' slightly warped view of the world, if the build step is successful (ie. the compiler did its job) but the tests failed, a job is unstable.  In normal agile terms, that is just a plain fail.  To add to the confusion, only certain post-build actions are able to move the status to unstable, so you may rarely encounter this.  But it is a great status to (ab)use if you want to show alerts on your job.

Setting build status and badges

So how can we access all this goodness?  Groovy is the answer here.  Note that I am applying groovy to any style Jenkins jobs, not only pipeline jobs that already groovy-fied.

You want to add the Groovy postbuild plugin to your Jenkins and then create a Groovy post-build action.  Here is some sample code that unconditionally set the build to Unstable.   Run it and you should see a yellow ball and an unstable outcome.
   import hudson.model.*
   def build = manager.build
   build.@result = hudson.model.Result.UNSTABLE
   manager.addErrorBadge("That really wasn't very stable")
   return


There are lots of other ways to do the same, and other things you can do, see the documentation for this plugin.  For example, you can also use 'manager.buildUnstable()' instead of that last line.

The Jenkins interface has an nice 'run in sandbox' checkbox, which you really want to check because it limits the abilities of your Groovy scripts.  However, it didn't work for me:  I basically could not run any Groovy code with this set.  The docs say that all the methods we used above are whitelisted, but apparently things have changed (or my Jenkins version is out of date).

Reading and checking build files


For my problem, I needed to get slightly more fancy and tests a few files in the current job.  Note that it is much easier to test for log lines in the current build, the method 'manager.logContains(REGEXP)' can help you with that.

To test for certain files in the current build directory, you can use:
def build = manager.build
def workspace = manager.getEnvVariable('WORKSPACE')

if (new File(workspace + '/mySpecialFile').exists()) {
   manager.buildUnstable()
   manager.addErrorBadge("Unit tests were not run")
   manager.listener.logger.println("Marked unstable")
else {
   manager.listener.logger.println("All well") }

return

This checks for the presence of 'mySpecialFile' under the project root.  If you build has produced such a file, it will be marked unstable and a log line will be emitted.


Getting a list of all variables available to you

How did I find out about the workspace variable?  I ran this piece of code, modified from one of the comments in the Groovy PostBuild Plugin page:

def build = Thread.currentThread().executable
def buildMap = build.getBuildVariables()
buildMap.keySet().each { var -> 
       manager.listener.logger.println( var + "= " + buildMap.get(var)) }
def envVarsMap = build.parent?.lastBuild.properties.get("envVars")
envVarsMap.keySet().each { var -> 
       manager.listener.logger.println( var + ": " + envVarsMap.get(var)) }

This will show the build variables (parameters to the build) and the environment variables.  My result (slightly redacted; no build variables):

BUILD_DISPLAY_NAME: #29
BUILD_ID: 29
BUILD_NUMBER: 29
BUILD_TAG: jenkins-test-29
BUILD_URL: http://localhost:8080/job/test/29/
CLASSPATH: 
DERBY_HOME: /usr/lib/jvm/java-8-oracle/db
EXECUTOR_NUMBER: 1
HOME: /var/lib/jenkins
HUDSON_HOME: /var/lib/jenkins
HUDSON_SERVER_COOKIE: 7......d
HUDSON_URL: http://localhost:8080/
J2REDIR: /usr/lib/jvm/java-8-oracle/jre
J2SDKDIR: /usr/lib/jvm/java-8-oracle
JAVA_HOME: /usr/lib/jvm/java-8-oracle
JENKINS_HOME: /var/lib/jenkins
JENKINS_SERVER_COOKIE: 7.....d
JENKINS_URL: http://localhost:8080/
JOB_BASE_NAME: test
JOB_DISPLAY_URL: http://localhost:8080/job/test/display/redirect
JOB_NAME: test
JOB_URL: http://localhost:8080/job/test/
LANG: en_US.UTF-8
LOGNAME: jenkins
MAIL: /var/mail/jenkins
MANPATH: /opt/OpenPrinting-Gutenprint/man:/opt/OpenPrinting-Gutenprint/man:/usr/local/man:/usr/local/share/man:/usr/share/man:/usr/lib/jvm/java-8-oracle/man
NLSPATH: /usr/dt/lib/nls/msg/%L/%N.cat
NODE_LABELS: master
NODE_NAME: master
NODE_PATH: /usr/lib/nodejs:/usr/lib/node_modules:/usr/share/javascript
PATH: /opt/OpenPrinting-Gutenprint/sbin:/opt/OpenPrinting-Gutenprint/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/snap/bin:/usr/lib/jvm/java-8-oracle/bin:/usr/lib/jvm/java-8-oracle/db/bin:/usr/lib/jvm/java-8-oracle/jre/bin
PATH+CHROMEDRIVER: /var/lib/jenkins/tools/chromedriver
PWD: /var/lib/jenkins
QT_QPA_PLATFORMTHEME: appmenu-qt5
RUN_CHANGES_DISPLAY_URL: http://localhost:8080/job/test/29/display/redirect?page=changes
RUN_DISPLAY_URL: http://localhost:8080/job/test/29/display/redirect
SHELL: /bin/bash
SHLVL: 1
USER: jenkins
WORKSPACE: /var/lib/jenkins/workspace/test
XDG_RUNTIME_DIR: /run/user/125
XDG_SESSION_COOKIE: f...........0
XDG_SESSION_ID: c20
XFILESEARCHPATH: /usr/dt/app-defaults/%L/Dt

You can check this output against http://localhost:8080/env-vars.html.



 Hope you can use this in your work with Jenkins!

Thursday, August 31, 2017

A pragmatic Git strategy for fast moving teams

Git is a wonderful tool that is so flexible you can use it in many ways.  With that has come the discussion of how to use it best.  Of course, the answer to depends on your priorities.

Here are the priorities I see often in agile teams delivering software fast:

  1. Members of the team need to work in temporary isolation (read: branches)
  2. Git and git history is a way to communicate your code changes to the rest of your team
  3. You have tests that give you confidence in your final code
  4. Recent Git history is valuable if something goes wrong
  5. When you build something, you ship it asap
Let's apply these principles to our thinking about various parts of a Git strategy:

Feature branches:  A pair (or a soloing dev) should work on a branch (isolation), and branches are best organized by topic.  Feature work should be short lived so it does not diverge.  Merge master back into the feature if in doubt.  
git checkout feature
git merge master

Feature branches can all start with a common prefix (ft_) but they do not have to.   

Release branches:  If we really ship all we have, every time, there should usually be no need for release branches.  Just release master, optionally from a couple of commits ago.   If you are just in the middle of a big change, either do not merge that feature yet or make a once-in-a-lifetime release branch. Tag your release:
git tag release-1.1 
git push origin release-1.1

Merging features:  A feature branch will have detailed commit history on it, which is great if you need to go back to it next week.  But to keep master easy to follow, you squash your work into one commit when it goes to master.  Keep the feature branch around for 4 weeks, then delete it.

You can accomplish this in many ways, this one is easy to understand:
git checkout feature # Use the name of your branch here
git checkout -b temp  
# Solve any conflicts between feature and master:
git rebase master 
## if conflicts occur, fix, git add and git rebase --continue

git checkout master
git pull
git merge --squash temp
git commit 
git branch -D temp

At this point, you feature branch will not have changed and all the work of merging it with master is hidden in one commit on master.  Write a recurring calendar event to delete old branches on the first of each month.

During the 'git rebase master' step, you may have to merge files multiple times.  This is tedious but it makes the individual merges easier.  If you are impatient, skip this step and deal with the conflicts during the merge step. 


If your feature branch warrants two or more commits: It happens, you did two things in one feature.  Proceed as follows:
git checkout feature
git checkout -b temp  
# Solve any conflicts between feature and master:
git rebase master 
# Reduce to a few commits:
git rebase -i master 
## An editor will open up, change the 'pick' to 'squash' on all 
## lines except the first one and other final commit.  

# Now update the commit message if you haven't already done that:
git commit --amend  
# Merge and clean up
git checkout master
git merge temp
git branch -D temp

Happy working with Git!

Thursday, November 17, 2016

Finding memory leaks in Java

Here is what we found out about finding memory leaks today - we have not fixed our problem yet so there may be another installment in the works :-)

Choose your tool

There are a number of good tools to observe and inspect memory use in a Java app.  We were alerted to our problem via AppDynamics, which is suite for monitoring live apps.  Back on the dev station, you can use VisualVM (free) or YourKit (with a 14 day free trial) to visualize Memory usage, CPU usage, Garbage Collection, etc.

Integration with IntelliJ

We found that YourKit integrates very well: Install the YourKit Java Profiler Integration plugin.  You get a new run option that starts your app and connects it to YourKit.  The alternative is to start the app normally and connect to it, like you would with any remote debugger.

Monitoring Local Server


This is the memory view before we start hitting the server:  Notice how the VM starts up and creates lots of heap objects (left), on the right you see the class meta data (metaspace, in light green) grow.  If you are unsure about Java memory spaces, check out YourKit help page,  and this page about the transition from PermGen to MetaSpace.

Monitoring Remote Server

Testing on a dev station is all nice and well but the real challenge was connecting to a Java app running on a CloudFoundry sandbox.  It turns out that the default Java buildpack has support for a large number of options, one of which is YourKit.

How it works:
The java buildpack has a your-kit-profiler.yml.  This Yaml file disables the profiler by default.  So the trick is to either deploy the app with a custom buildpack, or somehow override the contents of this yaml file.

It turns out that the buildpack has a interesting mechanism for (partially?) overriding the contents of yaml files:  You include a special environment variable in the env section of your manifest.  The name of the variable is 'JBP_CONFIG_' + the name of the yaml file but without '.yml'.  The value of the variable will be the new contents of the yaml file, overriding what was there before.

The only limitation is that you have to squeeze all your values on one line and be careful with quoting rules.  For us, the result looked like this:

applications:- host: HostAway
  memory: 1024M
  env:    SPRING_PROFILES_ACTIVE: sb
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "sandboxdocservice"}'


The one line of yaml contents does not come out so well, it contains:
applications:
    JBP_CONFIG_YOUR_KIT_PROFILER: '{version: "2015.15086.0", repository_root: "{default.repository.root}/your-kit/{platform}/{architecture}", enabled: true, port: "10101", default_session_name: "HostAwayWithYourKit"}'

Bring up your app by pushing it to CF as normal.  To reach it, you will have to create a secure tunnel to your workstation:

cf ssh -N -T -L 10101:localhost:10101 HostAway

Back in YourKit, go to the Welcome screen and choose 'connect to remote application'. Enter 'localhost:10101' and you should be good to go.

Thanks!!


Tons of thanks to Tiffany Chang and Ajay Pillai for working with me on this.


Friday, March 25, 2016

Building a Frontend Toolchain


We have so far focussed on getting our application working and have not bothered much with setting things up right.  This post is about setting up a good workflow for our frontend files and assets, that is, our html, css and javacript files, including the angular framework.  

Using Spring for assets

As explained in this 2013 blog post, Spring will copy static resources that are located in src/main/resources/static for us at compile time. There are actually a few more directories where Spring will copy files from (such as src/main/resources/public) and it will put these files intarget/classes/static/ (or public, etc).
However, putting our javascript into this directory is getting tedious and we don't have any way to track versions and pull in updates to our javascript. On top of that, we would like to combine all our javascript in one file and minify that file. For these tasks, we need no fewer than three tools: Npm, Bower and Gulp.

NPM

NPM apparently does not mean Node Package Manager, but it sure behaves a lot like that would be a good name for it. It requires node installed on your dev machine, which is no big deal.
Basic characteristics of NPM are:
  • NPM is based on its own npm repo, which has 1000s of packages in it. You can publish your own js pacakages to that.
  • NPM is the first tool on the scene for most pipelines, it is used to pull other tools in via a npm install bower etc.
How to use it:
  • Basic command npm install .... This will get you the most recent version and put it in node_modules/
  • Variant for global installation (ie not in current directory, but in some /usr/lib folder: npm install -g ...
  • Command to search: npm search ..., you can also look at https://www.npmjs.com/
  • More structured approach is to have a package.json file, see below for details. If that file is present, you can just run npm install to download and set up all dependencies the project needs.
This last option is great: It means that somebody can clone the repo for this project and run npm install; bower update to get all requirements in.
Using package.json:
  • First create a file package.json with {} as the contents
  • Then run npm install ... --save from the command line and each install will update the package.json for you!
  • Normally, the dependency will go into the dependencies section, if you run npm install .. --save-dev the name will go into the devDependencies section of the package.json. This is what we do, see below.
  • You may want to edit your package.json to specify exact version numbers, ie from bower: ^1.7.7 (which means that version and up) remove the caret.
Setup for NPM for this project:
$ echo "{}" > package.json
$ npm install bower --save
$ npm install jshint --save
$ npm install gulp --save

Bower

Bower will record the files you install in bower.json, the bower init will create a first version of that file for you.
Setup of Bower for this project:
$ bower init
Just answer all questions, leave 'main', 'moduletype' empty for now and accept defaults for 'ignore' and 'private'.
$ bower install angular#1.4.9 --save
$ bower install angular-animate#1.4.9 --save
$ bower install angular-uuid --save
  # told bower to choose the 1.4.9 version and record it 
$ bower install angular-cookies#1.4.9 --save

Useful bower commands:

  • Install a new package with bower install angular --save, similar to npm, this will download the package into bower_components and update the bower.json file for you.
  • In my case, this downloaded Angular 1.5, while I wanted to stay on 1.4, so…
  • To uninstall a package, run bower uninstall angular --save, here the save flag removes the reference from the bower.json file.
  • You can use hash syntax to request a specific version, bower install angular#1.4.9 --save.
  • If you forget the --save flag you can simply run the command again with the flag.
  • To see which packages you have installed, and which one can be updated, bower list
  • To update a package, run bower update ..

Dependencies in Bower

Bower keeps track of dependencies for you and will also record version numbers in the bower.json file, as we saw above. However, bower is not smart about dependencies, you will have to do that for it.
For example, I installed angular version 1.4.9. When I tried to install angular-animate however, the latest version of that package is 1.5.3 and it requires angular 1.5.3. So I will end up with a mixed situation, which is undesirable (to be fair, Bower does warn you about this). Bower does not work like apt-get(the Debian/Ubuntu package installer) and others, which will ask you to upgrade angular or fail the installation of angular-animate.

NPM vs Bower

Separation of concerns between NPM and Bower
  • use package.json and npm install to keep your tools up to date
  • use bower.json and bower .. to keep your front end dependencies up to date
  • This keeps your tools in node_modules and your dependencies in bower_components, which is convenient for further processing of the dependencies.

Using Gulp

Once, at the dawn of computer ages, there was make, which 'made' your computer program for your from sources. Make was the first of many task runners, programs that help you to compile sources, compress javascript, move assets around and package things up.
We have earlier talked about maven and gradle, the two main task runners in java land. For various reasons, javascript has its own task runners, with grunt and gulp as the two main contenders at the time of writing. We chose gulp here because it is generally easier to set up and it provides the watch functionality out of the box.
We will first need a fair few gulp components. I added them with the code below, but you can just do npm install.
npm install main-bower-files --save
npm install gulp-concat --save
npm install gulp-uglify --save
npm install gulp-print --save
npm install gulp-rename --save
npm install gulp-add-src --save
Finally, I created a symbolic link from my top level directory to the gulp binary, as follows
ln -s node_modules/gulp/bin/gulp.js gulp
git add gulp
This works in all unixes (Linux and OS-X).
Gulp's settings and tasks are in gulpfile.js, which is pretty human readable once you get used to the format: First, all dependencies are required and the results are assigned to variables. These variables are used as functions further down the line.
Gulp has a streaming concept, similar to functional programming or flow-based programming. Basically, you start a stream of files with a gulp.src command, then operate on that list of files in various steps. Each step is wrapped in a .pipe() function, similar to the use of pipes in bash.

Moving the resources

Until now, our frontend resources lived in src/main/resources/static and as we saw above, Spring will copy them to the target directory for us (Spring will copy everything under src/main/resources).
We will have to move the resources now, Gulp can take care of copying things and if we leave them, they will be copied twice. I have chosen src/main/frontend/ in this project. After you create the directory, make sure to right click it in IntelliJ and choose Mark Directory As ..: Sources Root. The folder will turn blue (in my color scheme) and IntelliJ wil index the files in this directory.
Without explaining Gulp in full yet, copying files is easy: Below is the source for a css task, which will copy all .css files from source to dest. It starts by declaring the task. Then we have a gulp.src call which will typically produce a list of files/filenames (Gulp bundles a filename and the file contents together in something it calls a Vinyl. Also note that technically, this is not a list but a stream).
Then we do several operation on these files/filenames, each wrapped in a .pipe() call as mentioned above. Our first step is print, which prints out the filename to the terminal. Our second and last step is gulp.dest, which will write the file contents out to the directory specified.
We use two small magic tricks: In the gulp.src, we use a javascript variable src instead of a full path; and we use /**/*.js to denote any js files in src or subdirectories of src (recursive copy).
var print = require('gulp-print');
var src = 'src/main/frontend/';
var destprod = 'target/classes/static/';

// copy css files
gulp.task('css', function() {
    gulp.src(src + '/**/*.css')
 .pipe(print())
 .pipe(gulp.dest(destprod));
});
Read this excellent, pretty and short post on the principles of Gulp. This introduction helped me a ton.

Main-bower-files

Above, we introduced a separation between tools (managed with NPM) and front end libraries (managed with Bower). Now, we would like to combine and minify all this JS code into one file. There are standard Gulp plugins for these actions, but how do we get the names of all the frontend libraries we need?
The main-bower-files plugin can help us with that: It will essentially do a bower list command for us and create a list of front-end libraries to ship to the user, in the right order (if file B depends on file A, it should come after A in the combined js). Of course, we also have our own js file(s) and these should come after the library files. gulp-add-src plugin can take care of that.
var bowerfiles = require('main-bower-files');
var addsrc = require('gulp-add-src');

var destdev = 'target/classes/static/devresources/'

// copy all js files for dev targets
gulp.task('js-dev', function() {
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(gulp.dest(destdev + 'js/'));
});
You may have noted this version does not do any minification etc, and is called js-dev. It also copies to destdev instead of destprod. We have a matching js-prod, which concatenates all files together under the name 'steamvoat.js', then renames that to 'steamvote.min.js' (we could have combined and renamed in one step), then runs the 'uglify' minifier, and finally copies the files out to the production target directory. To avoid confusion, the prod tasks starts out by removing all the full-length dev files under the destdev directory.
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');
var del = require('del');

// minify and package all js files for production targets
gulp.task('js-prod', function() {
    del(destdev);
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(concat('steamvoat.js'))
 .pipe(rename({suffix: '.min'}))
 .pipe(uglify())
 .pipe(gulp.dest(destprod));
});
The benefit of this approach is that production is fast, using minified files, while we can easily inspect and debug the full files in our development environment.

ProcessHtml

There is one small detail to deal with: We have replaced five js files with one combined and minified file in production, but the <script src commmands on our main page do not reflect that. We could simply source all full files as browsers throw a harmless error when files are not found, but there is a better way. The ProcessHtml node module, which allows you to replace html, use templates and generally modify html to your needs.
You can see it in action at the tail end of our index.html. It starts with a html comment with build: as the first word. The 'build:js= subcommand will replace many lines of js scripts into one, the second argument specifies the name of the combined js file.
<!-- build:js  steamvoat.min.js -->
<script src="/devresources/js/angular.js"></script>
<script src="/devresources/js/angular-cookies.js"></script>
<script src="/devresources/js/angular-uuid4.js"></script>
<script src="/devresources/js/angular-animate.js"></script>
<script src="/devresources/js/index.js"></script>
<!-- /build -->
</body>
</html>
When run through ProcessHtml, this will show up as
<script src="steamvoat.min.js"></script> </body></html>
A good overview of what you can do with ProcessHtml is given in this blog post. For full information, you have to know that there is an underlying node version of ProcessHtml which does all the work, with specific grunt and gulp versions built on top of that. The best docs are over at the grunt version.
The build commands should have modifier to make them work only on certain Grunt targets, but this does not seem to work for Gulp. This may be a good thing, now all the logic resides in the gulpfile.
In our gulpfile, there are therefore two rules for copying html files: A dev version which plain copies, and a production version which runs processHtml on the html.

gulp watch

Half the reason for using Gulp is the excellent support for 'watching' files. Imagine that every time we change a html file, it automatically is copied over to the target directory so that our test server will use it if we refresh the page. This can be done by running gulp watch
// Watch for changes in files
gulp.task('watch', ['dev'], function() {
    // Watch .js files
    gulp.watch(src + '**/*.js', ['js-dev']);
    // Watch html files
    gulp.watch(src + '/**/*.html', ['html-dev']);
    // Watch css files
    gulp.watch(src + '/**/*.css', ['css']);
    // Watch image files
    gulp.watch(src + '/img/*', ['images']);
});
Now if you run ./gulp watch, gulp will build the dev version and then your terminal will seem to hang but whenever you make a change to any of your resources, it will run the appropriate target to copy the modified files over for you.

More gulp

You can create endless complexity with gulp and it is reasonably manageable. We can for examples combine our css into one file, and preprocess Sass or Less files first.

Back to maven

To tie it all together, it would be great to have a unified interface to npm, bower and gulp. Also, we have various maven commands that we already need to build java, create a package for CloudFoundry etc. These take care of the java side, but our app will not work if the Gulp dev task has not run.
So how do we tie Gulp to Maven, our backend build tool? There is a great plugin "Front End Maven Plugin" for maven that takes care of all these tasks for us. I did not use it here because I did not want my gulp tasks to be run all the time and I don't want to install a local copy of node either. For larger projects with Continuous Integration, the plugin would be great. For this small size project, setting it up by hand was easy enough.
I use a small plugin to Maven that allows you to run any shell command. This does make the setup dependent on gulp and the exact location of gulp, but I can live with that.
First, we declare a plugin exec-maven-plugin. This plugin is run in the process-resources phase and it will run the exec goal. The plugin will run the executable mentioned in the configuration section, here gulp production:
<build>
    <plugins>
     //   ... stuff deleted ...
 <plugin>
     <artifactId>exec-maven-plugin</artifactId>
     <groupId>org.codehaus.mojo</groupId>
     <executions>
  <execution><!-- Run our version calculation script -->
      <id>Run Gulp</id>
      <phase>process-resources</phase>
      <goals>
   <goal>exec</goal>
      </goals>
  </execution>
     </executions>
     <configuration>
  <executable>node_modules/gulp/bin/gulp.js</executable>
  <arguments><argument>production</argument></arguments>
     </configuration>
 </plugin>
    </plugins>
</build>

Making this work in IntelliJ

We want the gulp task 'production' to be run when we start the application from IntelliJ. There are two ways to do this.
The simplest way is to not use the IntelliJ configuration target (our main application file), but use the maven task spring-boot:run as the configuration to run. This has several drawbacks, mainly that you cannot configure how you run the target so well.
The nicer way is to set up an IntelliJ configuration to run the gulp task (see also IntelliJ help) and then make our main IntelliJ configuration run this gulp configuration by default. Here is how that goes:
  • Go to the configurations dropdown of your IntelliJ. For me, this sits in the top right corner. Choose 'Edit Configurations'. Alternatively, go to Run: Edit Configurations.
  • Create a new configuration with the plus button (top left)
  • From the dropdown, choose Gulp.js
  • An 'unnamed' gulp task is created, call it 'production' (or whatever)
  • Make sure that the gulpfile textbox points to your gulpfile.js
  • Under task, select 'production'
  • All other settings should be fine and can be left open if they are blank.
Now try your new gulp configuration by choosing it from the drop down, then pressing the Run icon (green triangle). Alternatively, use Run: Run.... A 'gulp' tab should open on your "Run" view (bottom of IntelliJ screen) and you should see the production task copy files and conclude that with a satisfying "Process finished with exit code 0".
Finally, we need to tell our main configuration (which runs our app) to run the gulp task first.
  • Go to Edit Configurations again
  • Under 'Spring Boot', you should see a configuration that runs your app. Click on the configuration.
  • On the bottom of the window, there is a box called "Before Launch: .." with one task in it, called "Make" (if you do not see that box, stop all running processes and run this configuration once).
  • Click the plus button, then click "Run another configuration" and select your gulp 'Production' task.
  • Use the triangle icons to have Make be the last item.

Final workflow

When a new clone of the project is made, you will have to run a few commands to get the dependencies:
npm install
bower update
You will also have to run these commands every now and then to pick up new versions of the packages you are using. I think this is a great workflow, as it means you will not be surprised by new versions. You have to remember to run the commands on occasion.
To build and run the project, you can use maven or IntelliJ, as outlined above.
When making changes to the frontend files, you can view the changes with a simple browser reload if you run a watch before you start editing:
gulp watch
And that is it!

Conclusion

This seems a lot of work but it is definitely worth it if you want to maintain your app easily and if you want to automate the build and deploy process.

Thursday, March 17, 2016

Making your own Google Forms



Summary

For a short assignment, we wanted to replicate the basics of Google Forms without using the actual product. The assignment was to set up a form on our own server but feed the results into a Google Sheets (the name for their spreadsheet product). Another requirement was to set this up as fast a possible, so we had to take some shortcuts. We navigated a lot of new terrain and in this blog post, I will discuss our findings and the solution we arrived at.
If you want you can jump straight to An outline of the code or to the repository

Why we chose this approach

Using Google Forms

We decided to not use the standard Google Forms interface:
  • Forms is the standard interface to sheets, which does exactly what we want: It allows you to set up form with fields and the results will automatically be posted to a google spreadsheet. The users may have to log in to get access to the form (at your option), but the data are always saved in one central form that is owned by the creator of the form.
  • In other words, for writing to sheets, form creator credentials are used, not user credentials.
  • Forms does not support full html formatting out of the box, and does not easily allow you to run javascript functions to automatically update the page based on the user's entries.
  • By default, if you Submit a form, this leads to one row in the spreadsheet. We needed a form that would lead to 1..6 lines in the spreadsheet.
In retrospect, this particular assignment could probably have been solved by dropping a few of the requirements and using Google Forms after all. Little did we know… However, the solution discussed here provides you much more flexibility than a pure Google Forms one.

Google sheets API

Google Sheets has its own API, which we did not use either:
  • Sheets has its own legacy API, but using GAS (below) is recommended
  • Sheets own legacy API is not as powerful, available for Java and .Net only, requires you to install Google jars, and requires you to set up Oauth. These were all reasons to reject this for a quick solution.

GAS: Google Apps Scripts

  • GAS stands for Google Apps Scripts, which is a javascript environment that allows you to access all of the Google Apps (mail, docs, sheets, etc.) from the cloud.
  • GAS is great, but it is really its own thing with a lot of domain specific knowledge built into it.
  • GAS is geared towards having scripts operate on the users own documents, the case of multiple users all writing to the same sheet is supported but less well documented
  • GAS is really good for creating new formula operators in Sheets, or to set up fancy dashboards on top of sheets.
  • GAS scripts have access to a wealth of built-in functionality which you can just use, without any includes or injection. Critical to our project were PropertiesService (to store information between invocations of the script), `LockService (to prevent concurrent access to the spreadsheet), `ContentService` (to construct and format http replies).
  • GAS scripts can render html forms, but the support for it is rather involved (you have to write a doGet method that returns the html, as if you are a writing a web server in javascript). It is unclear to us whether you can use CSS, standard libraries and other tools for modern fast-paced development.

GAS as an API

We did not use GAS directly, for reasons outlined above. Instead we will write an application which interacts with a GAS script:
  • There is no set API for GAS, instead you write a javascript script that has a doGet and/or a doPost method. You publish this script and a URL is generated that functions as the external call point for your script. You can then add parameters to the GET or add a body to the POST to pass information into the GAS script.
  • It seems that GAS wants its POST data in the old fashioned form-data format, which requires you to do some work on the Spring side (see code below).
  • This may be a misunderstanding on our end, the documentation on the parameters to a external call make it sound like other post body types are possible too (csv in the example on GAS). However, this Stack Overflow posting makes it clear that x-www-form-urlencoded (aka old-fashioned forms) is the best choice and that it is the default.
  • GAS restricts input parameters to a few primitive javascript types, more than enough to work with but it is good to be aware of the limitations.
  • GAS scripts are not easy to debug, esp. when it comes to the interaction with the outside world. There is a logger but we had mixed success with it; you may also try the other options under the View menu.
  • GAS as an API has a few idiosyncrasies: It seems necessary to give your script a new version number every time you publish it, there is a `/dev` endpoint that you cannot use from the external world, and so on
  • There are subtle differences between publish as web app and 'publish as API' which are not clear to me. We ended up using the 'publish as web app' for our API, even though we only use our app as an API.
  • GAS does not provide any headers that say it is ok to use this API from another website (ie. it does not disable CORS with Access-Control-Allow-Origin: *). CORS is Cross Origin Resource Sharing, a browser technology that prevents scripts on host A to touch endpoints on host B, unless host B specifically allows this. Because CORS is not specifically allowed on GAS, you cannot call your GAS API from javascript or anywhere in the browser. We thought we would build a front-end only app, but we ended up building a simplistic backend just to avoid CORS.
  • GAS calls are very slow, they easily take up to five seconds on our state of the art machines on a fast network.

Related links

  • Using Google sheets as a database, by Martin Hawksey. This blog post was very informative and put us on the path to our final solution. There is a live demo on the site that does not seem to work.
  • Martin's gist updated by Corey Phillips.

An outline of the code

You can of course just peruse the repository, but here is an overview and some details that took us longer than they should have.
In a nutshell:
  • We build a front end app with angular, bootstrap and ui-router to have a three page web form, much like in this blog post on scotch.io.
  • This front end was served by a standard Spring Boot application, with exactly one endpoint for posting data to our GAS script.
  • The back end was pushed out to Pivotal Web Services, our in-house CloudFoundry instance.
  • We used a slight variation on the GAS script provided by Martin Hawksey. The GAS script is a bit slow to run so we would like to submit all rows of data simultaneously, but for now we make multiple calls to the GAS script.

Spring project, pom.xml

Via start.spring.io, we created a standard project with test and web options.

Manifest for cloudfoundry

The repo has a manifest.yml.template file which you should rename to manifest.yml after filling in the blanks, which define where your app will run on pws and where your GAS script lives (Google will display this URL when you publish your GAS script).
---
applications:
  - name: <YOUR APP NAME HERE>
    memory: 512M
    path: target/vacationform-0.0.1-SNAPSHOT.jar
  env:
    VACATIONFORM_URL: <YOUR GAS SCRIPT URL HERE>
With this in hand, you should be able to `cf login` to your PWS account and `cf push` the code.
We did not specify a buildpack as java was automatically recognized.

Packaged files

We included the necessary files to run bootstrap, jquery, angular and angular ui-router. Style sheets for the bootstrap-theme. These files are `src/main/resources/static/`.

Main page: index.html

This is a ui-view and nothing more (well, add headers and script includes)
<body ng-app="vacation-form">
<div class="container">
    <div ui-view></div>
</div>

The steps

There are three steps, named step1.html etc with associated controllers also named step1 etc. In step 1, we create a factory that provides us with a shared data structure vacationData:
app.factory("vacationData", function() { // to share information
    var vacationData = {};    
    vacationData.name = "Michael Dirk";    
    vacationData.weeks = 0;
    //   [[ etc ]]
    return vacationData;});
Each of the steps starts by calling the factory and putting the vacationData object on the scope, as we will need it there:
app.controller("step1", function($scope, vacationData) {
    $scope.vacationData = vacationData;});
Step 1 and 2 contain forms that are linked to fields of our vacationData object. We also have a few computed fields, which are resolved in the init of the step 3.
<form>    
  <input type="text" ng-model="vacationData.name" > &nbsp; Your name
(Apologies to the bootstrap designers for the nbsp there)
You can move to the next step through a button with a symbolic state target:
<a ui-sref="step2">    
   <button class="btn btn-primary">Next Step</button>
</a>
Step 3 deals with sending the data to GAS, asking the user to wait, and displaying the result of the GAS call.
<p class="row">    
We have submitted your information for you.  
This request returned status:  {{returnStatus}}
</p>

javascript, index.js

For maximum hackiness, we put all our javascript in one file. It is a total of 70-some lines. It includes a dog standard router:
app = angular.module("vacation-form", ['ui.router']);

app.config(function($stateProvider, $urlRouterProvider) {
    $stateProvider
 .state('step1', {
     url: '/step1',            
     templateUrl: 'step1.html'        })
    // [[ stuff deleted ]]
In step 3, we have all the data and we make a call to our Spring backend:
$scope.returnStatus = "pending";
var url = "/api/setVacationPlans";
$http.post(url, vacationData).then(
    function (success) {
 $scope.returnStatus = success.data; },    
    function (failure) {
 $scope.returnStatus = failure.data; });

The endpoint on the backend

The POST endpoint takes a vacationRequest in its body:
@RequestMapping(path = "/api/setVacationPlans", 
  method = RequestMethod.POST, 
  consumes = {"application/json"})
public ResponseEntity vacationPlans(@RequestBody VacationRequest vacationRequest) {
Note the consumes argument to RequestMapping. The actual input to this endpoint is the vacationData over on the JSON side.

VacationData data structure

I have not detailed it above, but vacationData is a JSON object (aka dictionary or hashmap) with a few user details at the top level, such as the name. It has an embedded list of vacationPlan objects, each of which have dates in them. We want to generate a row in the google spreadsheet for each vacation plan that is submitted.
We rely on Jackson JSON to parse this JSON object for us. To enable this, we created a VacationRequest object, which contains a list of VacationRequest objects.

Posting to GAS from the endpoint

Continuing on the end point, we make one call to GAS for each vacation plan. First, we need to set up a RestTemplate to make the call for us. Setting this up feels unnecessarily complicated to me, but these calls are apparently necessary to make the RestTemplate ship the data we put into it with the form-field convention (which begs the question why there is no such thing as a FormFieldTemplate):
RestTemplate restTemplate = new RestTemplate();
HttpMessageConverter formHttpMessageConverter = new FormHttpMessageConverter();
HttpMessageConverter stringHttpMessageConverter = new StringHttpMessageConverter();
restTemplate.getMessageConverters().add(formHttpMessageConverter);
restTemplate.getMessageConverters().add(stringHttpMessageConverter);
We then make the call to GAS by making a map of arguments and passing it into the restTemplate. There are probably neater ways to do this, but this worked. The only bit of sophistication is that we declare a class variable URL which is filled in from a environment variable (or application.properties, if you feel so inclined).
@Value("${VACATIONFORM_URL}")
private String url = "https://example.com/exec";

 // stuff deleted

 @RequestMapping(path = "/api/setVacationPlans", method = RequestMethod.POST, consumes = {"application/json"})
 public ResponseEntity vacationPlans3(@RequestBody VacationRequest vacationRequest) {

 // stuff deleted

MultiValueMap<String, String> map = new LinkedMultiValueMap<String, String>();    
map.add("name", vacationRequest.getName());    
map.add("fromDate", plan.getFromDate());
 // stuff deleted
String response = restTemplate.postForObject(url, map, String.class);

The GAS script

This is a universal "add to spreadsheet" script, that matches the column headers in the spreadsheet to the incoming data. An new row in the spreadsheet is created and for each column that has a matching input parameter, the value is copied to the spreadsheet. Columns that are not in the input parameters are set to 'undefined' and input parameters that do not correspond to any column header are ignored (note that the match is case sensitive, and mind trailing spaces).
We used a very slight variation on the script by Martin Hawksey, we are using the updated version in this gist. Line numbers below refer to that version:
  • The script will receive a request on doPost (line 39). Google calls the request an event. It is stuffed into the 'e' parameter, documented here.
  • Before doPut can run, the script requires you to run setup once (from the GAS dashboard) so that the PropertyService has the spreadsheet's ID value (setup function on line 85).
  • After a lock is required, the script opens the spreadsheet specified by the ID (line 49-54).
  • A row array is created and populated by looping over the names in the header row (which is normally row 1) (line 57-66)
  • In the successful case, the number of the new row is returned
  • A reasonable error case is created, and the lock is released.

Unsolved problems

This code works but it still has a few issues:
  • The dates that come out of the bootstrap date picker are not in the format that Sheets expects.
  • There is no check on the user, we wanted to require google authentication with pivotal.io, but the current version does not do that. You can add that requirement to the GAS script when you publish it, but that will break things as you end up on our SSO login page.
  • If a user does not list any specific plans in step 2, their submission is effectively ignored
  • The code has zero tests. Not great but in our defense, most of it is devoid of any logic and the interesting parts (submission to google Sheets) will be hard to test mechanically.
There is a larger authentication issue which we found hard to solve: The user is authenticated with Google and Pivotal on the front end, but it is the backend that is contacting the GAS script. The back end does not have access to the cookies and authentication tokens that exist on the front end. This is a standard browser security precaution: Our web page lives on a site that is not part of Google or Pivotal, so it cannot read cookies or authentication tokens from those sites. We can therefore not ship the authentication to the backend and the backend has to contact GAS as an anonymous user.

In conclusion

It is a lot of explanation but not that much code. In the end, I think using the GAS script was the best way forward: I dislike adding a third layer to the project (frontend, backend and GAS) but it is a lot easier than going the Sheets API way, which requires you to set up Oauth. Because the GAS script is basically a universal add-to-sheet routine, it can be used as a standard component.

Thanks

To Mike Oleske who went on a three-day deep dive in Apps land with me. All the good code is his, all the mistakes are mine.

Updates

  • 2016-03-24 Reformatted to look like my other posts

Tuesday, January 26, 2016

Coffee with Lady Gaggia


Lady Gaggia is our Gaggia Academia coffee maker.   I gave a lightning talk about how to use this machine.  There are three videos and a link to the quick start guide.  Enjoy a nice coffee on me!

Step one: Making a Latte or Cappuccino



Step 2: Steaming your own milk



Step 3: Cleaning the drip tray


You won't need all three steps to make a good coffee, step 1 should do it most of the time.

If you want to know more about the details, follow this link to the Gaggia Academia QuickStart Guide over at Gaggia USA.

Tuesday, December 22, 2015

Working with users and security (springdo-6)

Working with different users

So far, the User Interface of our application has behaved as if there is only one user. All todo items share one space and there is no login necessary. In reality, we do want our fantastic application to have more than one user, in fact we want 1000s of them! And our users probably do not want to share their todo items for all to see. So let's go to the other extreme and make the notes private to a user. We will later want to introduce note sharing between users, but that will be another episode.
In essence the user story here is there be users. It is useful to split that up in smaller parts: First we will secure our site, so only authenticated users can log in. They will still see the one large list of todo items because items are not linked to users yet.
User Story 6a: A user can log in with a password to see the otherwise secured site.
After we have users and a login method, we can link items to users:
User Story 6b: A user should only see the items that belong to them.
and of course
User Story 6c: When a user creates a new item, it belongs to them.

Spring Components

The nice part of Spring is that there are many, many components to it. If you have a problem that is more or less commonplace, like authentication, there probably is a solution for it somewhere in the larger Spring framework. The Spring sitehttps://spring.io/projects is dedicated to the projects and you can see which projects are available and what they do (in its current form (Fall 2015), the site fails to live up on its promises a bit: Subprojects are not directly visible and it is low on actual information what the projects do, but this will surely be improved upon soon).
Luckily, Spring Security is there and a few clicks later, you will find the usual '5 minute' introduction to Spring Authorization and you can find more info on Spring Security in general. Note that this introduction is not necessary for this tutorial and it also was build with normal Spring in mind, not Spring Boot, but it makes for good reading regardless. Much of what follows below is based on it.

Adding Spring Boot Security

To add a new part of Spring to our project we have to first tell our build tool (Maven) about the new dependency and then have it download the necessary files. To do so, double click the pom.xml file on the top level of the project.
In IntelliJ, you can right click anywhere in the file and choose 'Generate', then 'Managed Dependency'. IntelliJ will show a scrollable list of all available dependencies, from which you should choose 'spring-boot-starter-security'. This should generate a new dependency in your pom.xml, like so:
<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-security</artifactId>
</dependency>
Aside: There is also a 'Generate: Dependency' option and the difference does not seem to be covered the in the IntelliJ documentation. According to this older question, the position in the pom.xml differs between managed and normal dependencies (for the curious, managed dependencies explained by Maven). For us, the interface of 'Generate: Dependency' is easier but some dependencies cannot be included that way, so I advise to use 'Generate: Managed Dependency'.
If you were to run the project now, you would likely see the following error:
Screen+Shot+2015-10-16+at+1.34.45+PM.png
After you have made any changes to the pom.xml, you need to run the Maven targets (harder) or tell IntelliJ to act upon the new information (easier). Often, IntelliJ will pop up a message saying that the Maven project files need to be reread. Say yes to this. Alternatively, enable automatic reimporting under the settings menu, "Import Maven projects automatically":
Screen+Shot+2015-10-16+at+12.32.26+PM.png
A third way to have IntelliJ sync to the maven settings is to open the Maven tab (View menu, then Tool windows, Maven project) and use the shortcuts presented there. To alert IntelliJ to your maven changes, click the first button (two arrows in a circle). To download files, click the second button (folder icon with two arrows). Always make sure the 7th item, offline mode (representing by a disconnected cable icon) is in its deselected state, as Maven will not download any required dependencies in offline mode.
Screen+Shot+2015-10-16+at+1.35.39+PM.png
If all else fails, you can run the project from Maven directly, using the 'spring-boot:run' action. On the command line, you would do this by typing: `mvn spring-boot:run`. Before running the project, Maven will want to compile it and to do that, it will first pull in (download) all the dependencies.
TLDR We use Maven here but we like Gradle better / Working with Maven is not always very intuitive. The original version of this project was set up with Gradle instead of Maven. These two build systems are both very good, exhaustively documented and more than a little challenging when you first use them. Luckily IntelliJ (and any other IDE) makes things easier for you by giving you attractive buttons to click on. Gradle has the distinct advantage that its configuration file build.gradle is precisely 100 times more readable than Maven's pom.xml and there are a few other advantages too. However, almost all of the Spring (Boot) docs are made with Maven in mind, so we stuck to that convention here.

What is inside spring-boot-starter-security?

We used the dependency 'spring-boot-starter-security'. The 'starter' packages are convenient high-level dependencies supplied by Spring Boot: Instead of having to include a number of things, you just include the starter which will pull in the packages you would normally need for this task. In IntelliJ, you can inspect the file springdo.iml to see how IntelliJ has interpreted this. Here, I went to the 'version control' tab to do a diff on that file, showing that next to 'spring-boot-starter-security', nine other packages were pulled in:
Screen+Shot+2015-10-16+at+1.33.57+PM.png
Because the way Spring Boot works, reasonable defaults are assumed whenever no configuration is given. For the starter-security this means that without any further confirmation, your site will be locked down. When you go to any page, a browser-native login screen will be shown like this one:
Screen+Shot+2015-10-19+at+11.18.59+AM.png
This is not very helpful, but it is a reasonable default. The security suite can also create a default login form and other goodies, but we will not explore those here as they depend on Thymeleaf, a template engine which can help you serve pages from the server directly. Because we use Angular for the front end, our back end only outputs 'information', in the form of JSON packets, and never actual pages.
This reflects a change in style that has happened over the last few years: It used to be the case that frontend systems were minimal or non-existent, the server would simply produce a page which the browser would display as-is. Servers used technology like JSPor Thymeleaf to take a page template and populate it with user specific information ("Welcome User4829! You have 0 new messages"). Every click or input action made by the user would be sent back to the server and would produce a new page.
This system is still useful for static pages, but front end systems like Angular (and it predecessors BackboneFoundation and others) have now taken over the role of interacting with the user. The obvious benefit is speed of interaction: A click can be handled by javascript much faster by cutting out two network trips and replacing a busy server with a dedicated and lean javascript engine in the user's browser. This is even more true on mobile devices using a cellular data connection: When I turn my phone sideways, I want an instantaneous animation from portrait to landscape which would be impossible to do from the server side over a relatively slow link.

Making login work

To make login work, we need to give the users a place to login. A page 'credentials.html' was created (specifically named like that because Spring security will automatically provide a 'login' page and we wanted to avoid confusion). I have deleted some source (marked // ..) to provide focus on the essential parts of credentials.html:
<html ng-app="credentials">
  // ...
<div class="container">
    <div ng-controller="credentials" ng-cloak class="ng-cloak">
 <h1>Please log into SpringDo</h1>

 <form name="credentialform" ng-submit="gosubmit()" role="form">
     <div><label> User Name : <input type="text" ng-model="username" autofocus="true"/> </label></div>
     <div><label> Password: <input type="password" ng-model="password" /> </label></div>
     <div><input type="submit" value="Sign In"/></div>
 </form>
 <div></div>
   // ...
    </div>
</div>
We set up a simple form which is bound to the javascript variables username and password. The 'Submit' button is tied to the gosubmit function, defined in the matching javascript, credentials.js:
 1: angular.module('credentials', ['ngRoute'])
 2:     .controller('credentials', function($scope, $http, $window) {
 3:  $scope.username = "navya";
 4:  $scope.password = "secret";
 5:  $scope.reminder = false;
 6: 
 7:  $scope.gosubmit = function() {
 8:      var request = {
 9:   method: 'POST',
10:   url: '/credentials.html',
11:   data: $.param({username: $scope.username, password: $scope.password}),
12:   headers: {
13:       'Content-Type': 'application/x-www-form-urlencoded', }, };
14:      $http(request).then(
15:   function (success) {
16:       console.log("success", success);
17:       $window.location.href = "/index.html";  },  // rather heavy handed
18:   function (failure) {
19:       console.log("failure", failure); });
20:      };
21:     });
By setting the javascript variables $scope.username and .password, we populate the form elements, which makes testing much faster. Obviously we would remove this for a production version of the app. A nicer approach would only populate these fields if we are on localhost.
The work is done in the gosubmit function, which has to check the supplied username and password with the backend. It does that through a POST back to the same endpoint, with the username and password in the body of the POST. The $param function will format a javascript map into a http body for us. We need to set the content-type correctly otherwise Spring will not accept the POST (this is a gotcha that took me at least one full hour to fix).
If the authorization is successful the success function is called and we forward the user to the main page with the $window.location.href function. This causes a rather ugly refresh, but we will leave that for now. If you are interested, there is a matching$location directive in Angular that is more lightweight and will create a smoother transition. It is a little more work to make it work in this context, so we left it out here.
Aside on persistent console logs: It is hard to follow the console messages that are displayed during login, as the page changes. To make this easier, have the log persist over page changes: In the Chrome developer tools, go to the settings menu (three vertical dots) and check 'Preserve log upon navigation'. The same setting can be found in Firebug.
If the authorization fails, a counter-intuitive thing happens: The post action is still succesful but the user is not authorized. Because the post is successful, the 'success' branch is run and the user's browser is forwarded to index.html. However, because authorization has not succeeded, the Spring security framework will throw a 302 error and redirect the user back to the credentials page. This does cause a fair amount of flashing and on the whole is not much of a great user experience, but we will leave it in place for now.
The failure branch is only reached if the server has gone offline. In general, it is hard to handle that situation correctly: The server was up when the page we were on was served but has gone down since. The best we can do is tell the user to try again and not showing any reaction will (implicitly) give the same message. Only in large javascript apps that maintain extensive local state (like Gmail), does it make sense to try to retry the server and send the new information after all. Keep in mind that at that point, you will also have to disambiguate between two possible causes of this error: Either the data never reached the server; or the data did reach the serve and the server state was adjusted but the server reply never reached us.
Aside on cookies and browser state: When working with authentication, you will sometimes run into weird errors. My favorite one is an AngularJS message in the console showing that an AngularJS assertion (areq) had failed. If this happens to you, often the solution is as simple as opening a fresh anonymous browser window to make sure you do not have any tokens and cookies floating around. A hard refresh (shift-control-R or shift-command-R) will not do the same thing.

Backend changes

This login function does not work without a large number of changes in the backend.
First of all, we need to create a database of users. To this end, we create a User class, which is also a JPA Entity. To later interface with the Spring Security framework, we inherit from UserDetails, a Spring Security Core interface which requires a number of methods like getPasswordgetUsername, and isEnabled (see the Spring docs for full details).
The essence of the User class is only a few lines of code:
@Entity
public class User implements org.springframework.security.core.userdetails.UserDetails {

    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    public long id;

    @OneToMany(mappedBy = "user")  // what this is called on the 'many' side
    private List<Item> items = new ArrayList<>();

    @JsonView(JsonViews.ItemList.class)
    private String username;  // login credential

    @JsonView(JsonViews.UserDetails.class)
    private String password;

    @JsonView(JsonViews.UserDetails.class)
    private String email;  // for password reminders etc

    @JsonView(JsonViews.UserDetails.class)
    private Boolean isAdmin = false;  // has access to admin pages?
We declare the data fields we need, they will be given getters further down in the code. There are no setters currently, as we are not modifying users yet, only creating them. The items field warrants some explanation: Each todo item is owned by one user, and each user has many items. Hence we have a many-to-one relationship between items and user. On the user side, we express this with a @OneToMany, which takes as an argument the name of the variable on the Item side. On the Item.java side, we have a matching line:
@ManyToOne
public User user;
Together, these two code fragments allow us to use a natural interface on both entities: Items have a user variable which holds the User class that owns this item, from the outside we can do anItem.getUser to access it. Similarly, we a User object has a list of entities accessible via items, or aUser.getItems, neither of which are currently used but may come in handy soon.
Note that there is no actual User variable on an Item, and there is no List<Item> on the User. Instead, these are magically supplied by the JPA from a hidden join table which has two columns, one for the UserId and one for the ItemId. AccessinganItem.getUser() is doing a query on that table for that ItemId, similar to SELECT userid from JOINTABLE where itemid = ?1. The returned UserId is converted into a User object. Similarly, accessing aUser.getItems() is equivalent to doing a query for all items by that user, SELECT itemid from JOINTABLE where userid = ?1, followed by a map to convert the itemIds into their matching Item objects.
A few more details need to be taken care of: Each object of the Item class now has a user field as we just saw. And when the ListOfItems endpoint is hit, we return the JSON representations of a number (zero or more) items to the front end. However, we do not want this user field to contain all the information we have on the user: Specifically, we would like to omit the password and the email. However, on a (future) admin view, we would like to see those fields?
How do we return two different types of JSON from the same object, depending on the context it was called in? Jackson JSON, the JSON generator that is used by Spring, has created JsonViews for that. Views are first declared as interfaces on a small class,JsonViews.java in our project:
public class JsonViews {
    interface ItemList {}
    interface UserDetails extends ItemList {}   }
Now we have two views with a hierarchical relationship, such that UserDetails also includes all fields that that ItemList declares. Next, we annotate each of the data fields to be in one of these two views, as shown above: Username is part of the ItemList view, and the password and email are part of the UserDetails view. We wil see below how you request these views on a controller.

The admin page

To edit, add and remove users, we will need an admin page. We will not be adding the functionality to edit users just yet, but this is also a good place to display the list of all items by all users. This is basically the functionality that ListOfItems used to provided, before it changed to only return the items of this user.
Three endpoints will be implemented in AdminController.java: First, /resource/admin/list/: list all items, regardless of user. Second, /resource/userlist/: returns a list of all usernames (strings) on the system. Third, /resource/user/{username}/: shows full details of the user, including password. Finally, we need to see who is currently logged in and we do this via the /who/ endpoint, which shows name of the currently logged in user.
For full details, please consult the AdminController.java file. Below is the code for one endpoint, the others are very similar:
@RequestMapping(value = "/resource/admin/userlist/", method = RequestMethod.GET)
List<String> userList(Principal principal) {
    User user = userRepository.findByUsername(principal.getName());
    if (user != null && user.isAdmin()) {
 Iterable <User> iterable = userRepository.findAll();
 List<String> result = new ArrayList<>();
 iterable.iterator().forEachRemaining((User u) -> {result.add(u.getUsername()); });
 return result;
    } else {
 System.err.println("Unauthorized request to /resource/admin/userlist/ by " + principal.getName());
 return new ArrayList<String>();
    }
}
We connect this function to an endpoint with RequestMapping as usual. This function will return a List of Strings (usernames), and it takes a Principal as an argument. When Spring Security is active, any function can have Principal as an argument and the value, a Spring object describing the currently logged in user, will be filled in automatically.
To work with principal, we usually just do a getName(). In this code, we proceed to look up that user in our own UserRepository, using an automatically created JPA function findUserByUsername (the interface for this function is defined in the UserRepository; the implementation is provided under the hood). Only admin users should be able to see the full list of users, so we check whether our user lookup was successful (non-null answer) and whether that user has isAdmin set.
In this first pass, we have put some code in SpringdoApplication.java to create two users, Navya and Dirk, both with password "secret". Navya is an admin user, and Dirk is not.
If the authorization is successful, we can retrieve all users on the system and return a list of their names. This is done via a findAll on the UserRepository. findAll is a built-in JPA function which does what it says on the can. FindAll returns an iterable, from which we derive an iterator. A lambda function is applied to each item with forEachRemaining, using a simple Java 8 lambda which takes a user u as an input and runs the add method on our result array. This is a little convoluted, partially because of Java design choices regarding iterators and iterables, and partially because we can only return a list of real items to the front end so we have to populate an arraylist with string values instead of passing the iterator or iterable.
If the authorization is not successful, we print an error on the console and return an empty list. Ideally, the front end would check for this with an ng-if and display some placeholder text ("No users found, are you admin?") in case the list is empty. This has not been implemented in the current version, but it should be less than 10 minutes work for you.
Lesson Learned If you look at the AdminController.java file, you see that each method is wrapped in an if, checking whether the user is authorized to call this routine. This is getting tedious and we will explore a Spring method to do this more efficiently later.
Back on the admin page, we show the list of users on the system. Each name has a details button next to it, and clicking that will show the user details after retrieving them from the backend. This separation between a user list and user details is good style: If the user list is long, only returning a minimal amount of information speeds up the page load. Secondly, sensitive information is only sent over the wire when explicitly requested.
The bottom half of the admin page shows all the todo items, their item IDs and the user who owns them. I have not implemented the ability to click on an item here to expand it inline, but you should be able to create this swiftly using my code for the user list as an example.
The admin html page uses two ng-repeat statements to loop over users and items. It also uses two partials to render the users and items inside the loops, consistent with the pattern we used on the main page. On the javascript side, admin.js does all the expected contacting of the backend and storing the information in variables on the scope.
The only interesting code is the userQuery array shown below: This is initialized to {} and filled with user details when requested via the button press. The buttons are wired to userDetails(username) calls (with the username filled in) and the routine will contact the backend and populate the userQuery map. On the html side (admin.html), an ng-if"userQuery[user]"= hides details that have not been retrieved yet: If there are no data yet, userQuery[user] will return undefined, which is falsy and will surpress the html fragment.
angular.module('admin', []).controller('ad1', function($scope, $http) {
 // .. 
    $scope.userQuery = {};
    $scope.userDetails = function (username) {
 $http.get('/resource/user/' + username + '/').then(
     function (success) {
  $scope.userQuery[success.data['username']] = success.data;
     }
 );
    };
We have not defined a 'failure' branch in the javascript above. This is consistent with my stance on this: I do not know how to substantially help the user if the request failed. Clearly, the username must exist on the system because our usernames have been retrieved from the server earlier. A failing call to /resource/user/<id> means that the server is down, the connection is very slow, or the user was since removed by another admin. So there are three reasons, which I cannot distinguish between and none of them can be easily addressed by my user.
What does the User Experience look like in case of failure? If no result is returned, the user details will stay hidden (thanks to the ng-if in the admin.html). The user will probably click on the username again, thinking that somehow their first click did not register. If the request is successful the second time, they will not think much of it; if it isn't, they will think the button is somehow broken. In either case, I don't think the user experience will be enhanced by flashing a message like "Sorry, but I cannot connect to the server right now, or maybe the user has been deleted? If you are on a mobile device, move to an area with better reception. If you are on a desktop, check your wifi connection or reinsert your IP cable. " This is adding unnecessary information to an already confusing situation, not a better User Experience. It reminds me of this (infamous) Windows 8 screen:
win8-stuck-download-dialog.png

Security settings

We have wrapped the endpoints in the AdminController file in authentication testing logic. However, an unauthenticated user should not even be able to reach these methods. The same holds for the endpoints in ListOfItemsController: Without a valid login, we should not be able to access these methods. Spring Security provides a convenient method to enforce this: We basically block access to all of the site (with some exceptions for static content) unless the users is authenticated. If an unauthenticated request is made, Spring Security will redirect to the login page for a smooth user experience.
To set this up, we need to create a class that extends WebSecurityConfigurerAdapter and that is annotated with @EnableWebMvcSecurity (for Spring Boot 1.2.5 and earlier only, see below). That class should contains some 'fluent' declarations that do precisely what we outlined above:
@Configuration
@EnableWebMvcSecurity
// only exactly one class should have this annotation and it should override 'configure'

public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
                .authorizeRequests()
                    .antMatchers("/js/**").permitAll()
                    .antMatchers("/css/**").permitAll()
                    .antMatchers("/fonts/**").permitAll()
                    .antMatchers("/who").permitAll()
                    .anyRequest().authenticated()
                    .and()
                .formLogin()
                    .loginPage("/credentials.html")
                    .permitAll()
                    .and()
                .logout()
                    .permitAll()
                    .and()
                .csrf().disable()  // not great
        ;
    }

    @Override
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth.userDetailsService(inMemoryUserDetailsManager());
    }

    @Bean
    public InMemoryUserDetailsManager inMemoryUserDetailsManager () {
        Properties users = new Properties();
        return new InMemoryUserDetailsManager(users);
    }

}
The first configure method allows access to pages as outlined, allows everybody to access the login and logout pages, and tell Spring that our login can be found at credentials.html. Finally, it disables Cross Site Request Forgery (Wikipedia), which is not a good thing to disable but makes our life much easier for now.
The 2nd and 3rd methods set up an in-memory database of users to be used by Spring. We will discuss this database in a minute, but in a nutshell: The InMemoryUserDetailsManager is a fast and easy way to set up authentication for first versions of your app and for testing-specific setups.

Further changes

Overview of all changes, please browse through the commit to read all the changes:
  • index.html: Minor changes to show the currently logged in user and to log out.
  • index.js: Matching changes to angular.
  • credentials.html and .js: Login page as discussed above.
  • admin.html: Page to view all items (regardless of user). In the future, this page will also allow us to add and remove users.
  • admin.js: Matching angular code.
  • AdminController.java: A new backend for the admin pages.
  • AdminControlerTest.java: A test for the backend.
  • Item.java: Added a user to each todo list item. Also changed the items so that the user name is included in the JSON for the normal view (itemlist view). Finally, added a factory method, Item.empty(User user), which will create a new empty item that belongs to the given user.
  • User.java: A new entity and class that extends Spring's UserDetails class. It basically just stores the name, password and email of each user, with some added methods to satisfy the UserDetail interface.
  • JsonViews.java: Creates a new class with two hierarchical interface: ItemList (the default view) and UserDetails (a more detailed view showing confidential user details).
  • ListOfItemsController.java: The main endpoint (/resource/list/) will now return a list of items owned by the current user. Nex to that, a few GET requests changed to PUT requests to be more conformant with REST. Spring Security will work better when creating and deleting items is done via PUT.
  • SpringDoApplication.java: Small changes to include the UserReposity (autowire), to set up the authentication framework and to add two users and five todo items to the database on startup. The five todo items were there previously, but now they are tied to users.
  • MvcConfig.java: Configuration to show login pages with Thymeleaf, no longer used.
  • WebSecurityConfig.java: Configuration settings as per above.
A short note about the Empty factory function in Item.java: This class already has several constructors (a zero-argument constructor is required by the JPA and we wrote two further constructors in the course of these tutorials). Why a factory function instead of another constructor? The main advantage here is that we can give it a good name instead of having to remember what the constructor with 1,2,3 or 4 arguments does again. We use a similar factor method to create an AdminUser in User.java, check it out.
At this point, we have a working login system: The user can log in and we have two users (hardcoded into the application for now). Stories belong to a user internally and only the stories owned by a user are visible to that user.

Upgrading Spring Boot to run tests

Building tests to work with this secured site is not as easy as it should be: The latest Spring security version (4.0.3 at the time of writing) has support for mocking logged in users, but current spring-boot pulls in a much older version (3.2.8). You can check this by opening the Maven project tab in IntelliJ and clicking on the spring-security. It is also shown in the first screenshot above, on line 66, you can see security core being included at version 3.2.8. I tried overriding Spring Boot's defaults and manually pulled in the latest Spring Security version, but this lead to version inconsistency errors.
Luckily, a new version of Spring Boot (1.3.0) was released while I was writing this, and upgrading is as simple as changing the version number in the pom.xml file and refreshing Maven (see directions above):
<parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>1.3.0.RELEASE</version>
If you look at the springdo.iml file, you can see that Spring uses different version numbering schemes for each of its components: Spring Boot 1.3.0 pulls in the Spring Framework 4.2.3, Spring Data JPA 1.9.1 and Spring Security 4.0.3, among others.
Lesson Learned: Keep this in mind when looking at the documentation for Spring components: Consult the iml file or otherwise determine the exact version number used before you hit the internet. I wasted a good amount of time trying to make Spring Security 4.0 features work in the Spring Security 3.2 setting, because I didn't realize Spring Security has a different versioning system from Spring Framework (which was at 4.x already).
The migration does not come for free: We use a @EnableWebMcvSecurity annotation in our WebSecurityConfig.java class, but this annotation is now deprecated and replaced by @EnableWebMcvSecurity:
@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
There are many other migration concerns (see resources section below), but none of those affect us directly. Our previous commit had the security section excluded from the autoconfiguration, because this was necessary for that version. Now this line of code has to be removed or Spring will not start up (this is not mentioned in any of the migration documents, as far as I can tell). In SpringdoApplication.java, remove the line that reads:
@EnableAutoConfiguration(exclude = SecurityAutoConfiguration.class)

Setting up and writing tests

In a real world application, we should write tests before we write the implementation but this would make this tutorial harder to read. Also note that we are still not testing any frontend functionality: We will do all of that in a later episode of this tutorial. Because we have no tool to test the front end, we cannot do end-to-end testing either, which is really starting to become a bit of an issue. But again, more on that later.
As it is, we do not have any tests for the new admin endpoints, so these need to be written. We do not have any logic in our other java files (which contain repositories, entities, and such) so there is nothing to test there. But before we write the new tests, let's run the old ones: Our one functional test is ListOfItemsControllerTest, and it currently has one failing test: 'whenCreateIsHitANewItemIsCreatedAndReturned' fails with the following (compressed) logs:
Unauthorized request to /resource/create/ by null
...
java.lang.IllegalArgumentException: root can not be null
The problem here is that the routine to create a new todo item (code shown below) requires a user. In the production version, Spring will pass in a user (called a Principal in Spring security parlance) automatically and it will be picked up by our method argument principal. So if you test this by hand on the server, the code works. However, in the test, we are not logged in as a user and we therefore get an error on the terminal and a null object is passed into principal, leading to the subsequent NullPointer exception because we cannot do a getName on null:
@RequestMapping(value="/resource/create/", method=RequestMethod.POST)
Item postCreate(Principal principal) {
    User user = userRepository.findByUsername(principal.getName());
The reason we upgraded to the new version is that Spring Security has a new annotation in version 4.x: @WithMockUser. This should give us an easy way to mock a standard user (username 'user', password 'password') for this situation. We can optionally supply a username, which is what we do here. However, adding this to the test code still does not work, the principal argument does not get populated in the test. We will see below how to fix this the right way, here let's first step over a quick fix.
Here is the first part of the test code:
@WithMockUser("Navya")
@Test
public void whenCreateIsHitANewItemIsCreatedAndReturned() throws Exception {
    mvc.perform(post("/resource/create/"))
     .andExpect(status().isOk())
This test currently fails again with a NullPointer exception. I debugged the test by putting a breakpoint on the first line of the end point handler postCreate and we again have a null value for principal. To validate that this is a valid login problem, I temporarily allowed access to the /resource/** path in the WebSecurityConfig.java and then accessed the create endpoint (/resource/create/) with the excellent Postman tool, without logging into the site first: The result is the same NullPointer exception, but this time it is expected/valid as this endpoint should not be accessed without being authenticated. (You can have an interesting Friday afternoon discussion on whether the app should handle this error, or a NullPointer is good enough for something that is not supposed to happen. I suggest doing this over cheese and wine so it is enjoyable, because IMHO there is no right answer to this question.)
As the docs point out, another way to access the currently logged in user is to access the global SecurityContextHolder with a call like:
principal = SecurityContextHolder.getContext().getAuthentication();
This is not particularly pretty and it is, as we just saw, it is only necessary for tests. I 'solved' it by creating a helper function and calling it on the first line of all Test functions that involve principal:
public Principal fixPrincipalForTest(Principal principal) {
    if (principal == null) {
 principal = SecurityContextHolder.getContext().getAuthentication();
 if (principal == null) {
     System.err.println("Unauthorized request, principal is null");
 }
    }
    return principal;
}
If you debug this code, you can see that the principal object has inner principal data, which actually holds the username and password. Apparently this is how Spring has chosen to organize its principal data structure, but it is at least slightly confusing: Running principal.getPrincipal() will not return a Principal object, but a UserDetails.User object. Keep this in mind if IntelliJ tells you that you are dealing with incompatible types. We will encounter that 'User' datatype again below.
Using fixPrincipalForTest we can make things work and I actually have left it in the code of ListOfItemsController, even though it creates a (minimal) amount of overhead in the runtime. For AdminController, it is worth exploring a better way as there are four endpoints that all need this treatment. More importantly, in AdminControllerTest, all tests require authentication of some type.
On the Spring-Security channel of Slack, Rob Winch pointed me to the very obvious solution for this problem with principal: We need to tell the MockMvc that we use for testing to use the security framework. In AdminControllerTest, you will find the following code:
@Before
public void setUp() throws Exception {
    mvc = MockMvcBuilders
     .webAppContextSetup(context)
     .apply(springSecurity())
     .build();
}
The apply(..) line make the MockMvc use Spring Security. The @WithMockUser annotations will work as advertised and the principal argument will be set in the tests. A drawback of kinds is that all interactions with the Mvc now use security, which means that we cannot access any endpoints under /resource/ without logging in. For this reason, I stuck with the unsecured MockMvc and the fixPrincipalForTest hack in the ListOfItemsController.
Once security is working in the MockMvc, Spring provides some goodies for smooth testing. For example, we can easily test whether an unauthorized request indeed returns the correct response from the webserver. You can see this in action in this test, which confirms that user details cannot be viewed when you have not logged in:
@Test
public void AdminShowUser_failsWithoutAuthentication() throws Exception {
    mvc.perform(get("/resource/user/Navya/"))
     .andExpect(unauthenticated());
}

AdminControllerTest

With that we can finally write the code for the new Admin endpoints. We can use the handy IntelliJ Navigate: Test Subject method for navigating to the matching test file: If the test file does not exist, it will prompt you to generate the test file. A popup will show you the four functions in this file ask for which ones to create tests: I checked all of them. Next, I copied the test setup from ListOfItemsControllerTest and started modifying the provided empty tests. Instead of one test per function, we prefer more 'behavior based' tests that tell a user story and its outcome.
Here is a the first two, which test the outcome of the admin/list endpoint: Recall that this endpoint returns all items in the database for the admin users, but only the users own items for non-admin.
@WithMockUser("Navya")
@Test
public void AdminList_returnsAListOfAllItemsForAdmin() throws Exception {
    mvc.perform(get("/resource/admin/list/"))
     .andExpect(status().isOk())
     .andExpect(jsonPath("$", hasSize(5)))
     .andExpect(jsonPath("$[0].title", containsString("swim")))  // owned by Navya
     .andExpect(jsonPath("$[4].title", containsString("sleep")));  // owned by Dirk
}

@WithMockUser("Dirk")
@Test
public void AdminList_returnsAListOfOwnItemsForUser() throws Exception {
    mvc.perform(get("/resource/admin/list/"))
     .andExpect(status().isOk())
     .andDo(print())
     .andExpect(jsonPath("$", hasSize(2)))
     .andExpect(jsonPath("$[1].title", containsString("sleep")));
}
Note again that these tests will not work unless we use the fixPrincipalForTest function above (as we did for ListOfItemsController) or set up our MockMvc mvc to be aware of Spring Security (what we do here). The reason for these two different approaches are historical and educational, but setting up the MockMvc with Spring Security also means that each test in AdminControllerTest has to be wired up with MockUser etc. Because ListOfItemsController contains secured and unsecured content, it is easier to use the fixPrincipalForTest helper there. (You could argue that it is not optimal and confusing to have secured and unsecured functions in ListOfItemsController; on the other hand it is good to keep related functionality together).
#+COMMENT TODO update the ListOfItemsController file at a later stage.
The tests assume the setup created in 'SpringdoApplication.java': 2 users with 5 items between them. We could make more robust tests here by actually asserting that items belong to more than one user etc, but this is the simplest thing that works. For the user test, I quickly researched whether we could write a simple jsonPath to assert no item is owned by Navya, but because our json is an array with maps inside it, jsonPath's quite expressive syntax does not work here.

Lesson Learned: Although an array as the outer element is perfectly valid JSON, many tools expect a map as the outer element with named inner elements, or a 'data' key to hold the array. In other words:
valid:      [ {"map": 1}, {"map": 2} ]
expected:   { "data": [ {"map": 1}, {"map": 2} ] }

While testing the /who/ endpoint (which is not under the protected /resource/ tree, ie. accessible by anybody), I discovered that although the intention was that /who/ would show who, if anybody was logged in, it did not really deal with the case of not being logged in. Nice catch, and another good reason to write the tests first next time.

Improving the user database

There is one issue about this setup that is particularly not satisfactory to me: We have a UserRepository, but the Spring Security framework is not using it to authenticate the users. Instead, Spring has its own in-memory database of users and at startup, we copy all users from our repository into the Spring's database (this is done in SpringDoApplication.java, relevant code shown below). This is not only duplication of data, but it will also become quite hard to manage once we add the ability to edit, add or remove users.
@Override
public void run(String... strings) throws Exception {
    User defaultUser = userRepository.save(User.AdminUser("Navya", "secret", "n@example.com"));
    User secondUser = userRepository.save(new User("Dirk", "secret", "t@example.com"));
    //  ..
    userRepository.findAll().forEach(user -> inMemoryUserDetailsManager.createUser(user));
The underlying issue here is that Spring's concept of users more or less hardcoded into the framework. Spring provides an interface UserDetails which has specifies all the things you would expect: Name, password, enabled and a bunch of other reasons why a user may be considered inactive, and finally a list of GrantedAuthority to signify which roles the user has in the system. Our User class extends the UserDetails class, but Spring Security wants to talk to a UserDetailsManager class.
There are two major ways forward: Create an extra layer of indirection to the system, which will wrap around our UserRepository and supply the methods Spring expects from a UserDetailsManager. Alternatively, we can have a two-layer User representation, with an inner User that is a org.springframework.security.core.userdetails.UserDetails type, an contains all the administrative fields like isCredentialsNonExpired. This option is well discussed in this blog. We stay on our course and follow the first option, as our User class already has the necessary security 'cruft'. Personally, I would argue that it is good style to keep all the user data in one place, even if that means that we need to store some variables that we are pretty unlikely to ever use (and have ugly names to boot). Regrettably for us, Spring Security requires these and, to me, the easiest way forward it to implement them.
First, we create a new class the UserManager, which interacts with the UserRepository:
@Component
public class UserManager implements UserDetailsManager {
    @Autowired
    UserRepository userRepository;

    public void createUser(UserDetails user){
 userRepository.save((io.pivotal.User) user);
    }
    // ... code deleted
Because our user class implements UserDetails, we can pass our users into these functions, which are required to have UserDetail parameters by the interface. However, the repository does not know how to handle UserDetails so we have to cast them back to the User class they are. I have used the full path on User in this code to disambiguate the many versions of the User class here. Below is the code for creating a user, the other methods are similar:
public void createUser(UserDetails user) {
    userRepository.save((io.pivotal.User) user);
}
In the setup of the security component (WebSecurityConfig.java), we need to initialize our new UserManager. This setup is not particularly pretty, but apparently it is what Spring requires and it gives us the flexibility to specify the UserManager at runtime. We can use this later on to only start up with our 2 users and 5 stories for tests, and use an empty database for production use.
@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
    auth.userDetailsService(myUserManager());
}

@Bean
public UserManager myUserManager () {
    return new UserManager();
}
One final change is on the frontend, of all places: We have so far pre-filled the username field with "navya" (lowercase), which was apparently enough to satisfy the inMemoryUserDetailsManager, but our new login system does not convert case so we have to change it to "Navya". Similarly, use "Dirk" to login as the second user.
With that, we have a very long episode finished: Our three stories have all been addressed and can be accepted. Creating a secure app is not even that hard, it just requires a lot of background reading and understanding of how Spring Security works.
We have not implemented some straightforward things like adding, editing and removing users, but this is quite similar to the same operations on todo items. It does make for some great practice to implement this.
A related issue is that anybody can delete items at the moment if they are willing to create a custom url: All they have to do is to log in and then hit the delete endpoint with the todo item id. A better implementation would only allow the owner of the item to delete it.

Resources

Thanks

A big thanks to everyone who read and commented on this tutorial (Ehren Murdick; Michael Oleske), to Pivotal Labs for letting me write these tutorials, and to Navyasri Canumalla, who sat down with me to learn Spring and wrote half the code you see here.