Building a Frontend Toolchain

• front end tools
• Using Spring for assets
• NPM
• Bower
  • Useful bower commands:
  • Dependencies in Bower
• NPM vs Bower
• Using Gulp
  • Moving the resources
  • Main-bower-files
  • ProcessHtml
  • gulp watch
  • More gulp
• Back to maven
• Making this work in IntelliJ
• Final workflow
• Conclusion
  • Some further links

We have so far focussed on getting our application working and have not bothered much with setting things up right.  This post is about setting up a good workflow for our frontend files and assets, that is, our html, css and javacript files, including the angular framework.  

Using Spring for assets

As explained in this 2013 blog post, Spring will copy static resources that are located in src/main/resources/static for us at compile time. There are actually a few more directories where Spring will copy files from (such as src/main/resources/public) and it will put these files intarget/classes/static/ (or public, etc).
However, putting our javascript into this directory is getting tedious and we don't have any way to track versions and pull in updates to our javascript. On top of that, we would like to combine all our javascript in one file and minify that file. For these tasks, we need no fewer than three tools: Npm, Bower and Gulp.

NPM

NPM apparently does not mean Node Package Manager, but it sure behaves a lot like that would be a good name for it. It requires node installed on your dev machine, which is no big deal.
Basic characteristics of NPM are:

• NPM is based on its own npm repo, which has 1000s of packages in it. You can publish your own js pacakages to that.
• NPM is the first tool on the scene for most pipelines, it is used to pull other tools in via a npm install bower etc.

How to use it:

• Basic command npm install .... This will get you the most recent version and put it in node_modules/
• Variant for global installation (ie not in current directory, but in some /usr/lib folder: npm install -g ...
• Command to search: npm search ..., you can also look at https://www.npmjs.com/
• More structured approach is to have a package.json file, see below for details. If that file is present, you can just run npm install to download and set up all dependencies the project needs.

This last option is great: It means that somebody can clone the repo for this project and run npm install; bower update to get all requirements in.
Using package.json:

• First create a file package.json with {} as the contents
• Then run npm install ... --save from the command line and each install will update the package.json for you!
• Normally, the dependency will go into the dependencies section, if you run npm install .. --save-dev the name will go into the devDependencies section of the package.json. This is what we do, see below.
• You may want to edit your package.json to specify exact version numbers, ie from bower: ^1.7.7 (which means that version and up) remove the caret.

Setup for NPM for this project:

$ echo "{}" > package.json
$ npm install bower --save
$ npm install jshint --save
$ npm install gulp --save

Bower

Bower will record the files you install in bower.json, the bower init will create a first version of that file for you.
Setup of Bower for this project:

$ bower init

Just answer all questions, leave 'main', 'moduletype' empty for now and accept defaults for 'ignore' and 'private'.

$ bower install angular#1.4.9 --save
$ bower install angular-animate#1.4.9 --save
$ bower install angular-uuid --save
  # told bower to choose the 1.4.9 version and record it 
$ bower install angular-cookies#1.4.9 --save

Useful bower commands:

• Install a new package with bower install angular --save, similar to npm, this will download the package into bower_components and update the bower.json file for you.
• In my case, this downloaded Angular 1.5, while I wanted to stay on 1.4, so…
• To uninstall a package, run bower uninstall angular --save, here the save flag removes the reference from the bower.json file.
• You can use hash syntax to request a specific version, bower install angular#1.4.9 --save.
• If you forget the --save flag you can simply run the command again with the flag.
• To see which packages you have installed, and which one can be updated, bower list
• To update a package, run bower update ..

Dependencies in Bower

Bower keeps track of dependencies for you and will also record version numbers in the bower.json file, as we saw above. However, bower is not smart about dependencies, you will have to do that for it.
For example, I installed angular version 1.4.9. When I tried to install angular-animate however, the latest version of that package is 1.5.3 and it requires angular 1.5.3. So I will end up with a mixed situation, which is undesirable (to be fair, Bower does warn you about this). Bower does not work like apt-get(the Debian/Ubuntu package installer) and others, which will ask you to upgrade angular or fail the installation of angular-animate.

NPM vs Bower

Separation of concerns between NPM and Bower

• use package.json and npm install to keep your tools up to date
• use bower.json and bower .. to keep your front end dependencies up to date
• This keeps your tools in node_modules and your dependencies in bower_components, which is convenient for further processing of the dependencies.

Using Gulp

Once, at the dawn of computer ages, there was make, which 'made' your computer program for your from sources. Make was the first of many task runners, programs that help you to compile sources, compress javascript, move assets around and package things up.
We have earlier talked about maven and gradle, the two main task runners in java land. For various reasons, javascript has its own task runners, with grunt and gulp as the two main contenders at the time of writing. We chose gulp here because it is generally easier to set up and it provides the watch functionality out of the box.
We will first need a fair few gulp components. I added them with the code below, but you can just do npm install.

npm install main-bower-files --save
npm install gulp-concat --save
npm install gulp-uglify --save
npm install gulp-print --save
npm install gulp-rename --save
npm install gulp-add-src --save

Finally, I created a symbolic link from my top level directory to the gulp binary, as follows

ln -s node_modules/gulp/bin/gulp.js gulp
git add gulp

This works in all unixes (Linux and OS-X).
Gulp's settings and tasks are in gulpfile.js, which is pretty human readable once you get used to the format: First, all dependencies are required and the results are assigned to variables. These variables are used as functions further down the line.
Gulp has a streaming concept, similar to functional programming or flow-based programming. Basically, you start a stream of files with a gulp.src command, then operate on that list of files in various steps. Each step is wrapped in a .pipe() function, similar to the use of pipes in bash.

Moving the resources

Until now, our frontend resources lived in src/main/resources/static and as we saw above, Spring will copy them to the target directory for us (Spring will copy everything under src/main/resources).
We will have to move the resources now, Gulp can take care of copying things and if we leave them, they will be copied twice. I have chosen src/main/frontend/ in this project. After you create the directory, make sure to right click it in IntelliJ and choose Mark Directory As ..: Sources Root. The folder will turn blue (in my color scheme) and IntelliJ wil index the files in this directory.
Without explaining Gulp in full yet, copying files is easy: Below is the source for a css task, which will copy all .css files from source to dest. It starts by declaring the task. Then we have a gulp.src call which will typically produce a list of files/filenames (Gulp bundles a filename and the file contents together in something it calls a Vinyl. Also note that technically, this is not a list but a stream).
Then we do several operation on these files/filenames, each wrapped in a .pipe() call as mentioned above. Our first step is print, which prints out the filename to the terminal. Our second and last step is gulp.dest, which will write the file contents out to the directory specified.
We use two small magic tricks: In the gulp.src, we use a javascript variable src instead of a full path; and we use /**/*.js to denote any js files in src or subdirectories of src (recursive copy).

var print = require('gulp-print');
var src = 'src/main/frontend/';
var destprod = 'target/classes/static/';

// copy css files
gulp.task('css', function() {
    gulp.src(src + '/**/*.css')
 .pipe(print())
 .pipe(gulp.dest(destprod));
});

Read this excellent, pretty and short post on the principles of Gulp. This introduction helped me a ton.

Main-bower-files

Above, we introduced a separation between tools (managed with NPM) and front end libraries (managed with Bower). Now, we would like to combine and minify all this JS code into one file. There are standard Gulp plugins for these actions, but how do we get the names of all the frontend libraries we need?
The main-bower-files plugin can help us with that: It will essentially do a bower list command for us and create a list of front-end libraries to ship to the user, in the right order (if file B depends on file A, it should come after A in the combined js). Of course, we also have our own js file(s) and these should come after the library files. gulp-add-src plugin can take care of that.

var bowerfiles = require('main-bower-files');
var addsrc = require('gulp-add-src');

var destdev = 'target/classes/static/devresources/'

// copy all js files for dev targets
gulp.task('js-dev', function() {
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(gulp.dest(destdev + 'js/'));
});

You may have noted this version does not do any minification etc, and is called js-dev. It also copies to destdev instead of destprod. We have a matching js-prod, which concatenates all files together under the name 'steamvoat.js', then renames that to 'steamvote.min.js' (we could have combined and renamed in one step), then runs the 'uglify' minifier, and finally copies the files out to the production target directory. To avoid confusion, the prod tasks starts out by removing all the full-length dev files under the destdev directory.

var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var rename = require('gulp-rename');
var del = require('del');

// minify and package all js files for production targets
gulp.task('js-prod', function() {
    del(destdev);
    gulp.src(bowerfiles())
 .pipe(addsrc.append(src + '**/*.js'))
 .pipe(print())
 .pipe(concat('steamvoat.js'))
 .pipe(rename({suffix: '.min'}))
 .pipe(uglify())
 .pipe(gulp.dest(destprod));
});

The benefit of this approach is that production is fast, using minified files, while we can easily inspect and debug the full files in our development environment.

ProcessHtml

There is one small detail to deal with: We have replaced five js files with one combined and minified file in production, but the <script src commmands on our main page do not reflect that. We could simply source all full files as browsers throw a harmless error when files are not found, but there is a better way. The ProcessHtml node module, which allows you to replace html, use templates and generally modify html to your needs.
You can see it in action at the tail end of our index.html. It starts with a html comment with build: as the first word. The 'build:js= subcommand will replace many lines of js scripts into one, the second argument specifies the name of the combined js file.

<!-- build:js  steamvoat.min.js -->
<script src="/devresources/js/angular.js"></script>
<script src="/devresources/js/angular-cookies.js"></script>
<script src="/devresources/js/angular-uuid4.js"></script>
<script src="/devresources/js/angular-animate.js"></script>
<script src="/devresources/js/index.js"></script>
<!-- /build -->
</body>
</html>

When run through ProcessHtml, this will show up as

<script src="steamvoat.min.js"></script> </body></html>

A good overview of what you can do with ProcessHtml is given in this blog post. For full information, you have to know that there is an underlying node version of ProcessHtml which does all the work, with specific grunt and gulp versions built on top of that. The best docs are over at the grunt version.
The build commands should have modifier to make them work only on certain Grunt targets, but this does not seem to work for Gulp. This may be a good thing, now all the logic resides in the gulpfile.
In our gulpfile, there are therefore two rules for copying html files: A dev version which plain copies, and a production version which runs processHtml on the html.

gulp watch

Half the reason for using Gulp is the excellent support for 'watching' files. Imagine that every time we change a html file, it automatically is copied over to the target directory so that our test server will use it if we refresh the page. This can be done by running gulp watch

// Watch for changes in files
gulp.task('watch', ['dev'], function() {
    // Watch .js files
    gulp.watch(src + '**/*.js', ['js-dev']);
    // Watch html files
    gulp.watch(src + '/**/*.html', ['html-dev']);
    // Watch css files
    gulp.watch(src + '/**/*.css', ['css']);
    // Watch image files
    gulp.watch(src + '/img/*', ['images']);
});

Now if you run ./gulp watch, gulp will build the dev version and then your terminal will seem to hang but whenever you make a change to any of your resources, it will run the appropriate target to copy the modified files over for you.

More gulp

You can create endless complexity with gulp and it is reasonably manageable. We can for examples combine our css into one file, and preprocess Sass or Less files first.

Back to maven

To tie it all together, it would be great to have a unified interface to npm, bower and gulp. Also, we have various maven commands that we already need to build java, create a package for CloudFoundry etc. These take care of the java side, but our app will not work if the Gulp dev task has not run.
So how do we tie Gulp to Maven, our backend build tool? There is a great plugin "Front End Maven Plugin" for maven that takes care of all these tasks for us. I did not use it here because I did not want my gulp tasks to be run all the time and I don't want to install a local copy of node either. For larger projects with Continuous Integration, the plugin would be great. For this small size project, setting it up by hand was easy enough.
I use a small plugin to Maven that allows you to run any shell command. This does make the setup dependent on gulp and the exact location of gulp, but I can live with that.
First, we declare a plugin exec-maven-plugin. This plugin is run in the process-resources phase and it will run the exec goal. The plugin will run the executable mentioned in the configuration section, here gulp production:

<build>
    <plugins>
     //   ... stuff deleted ...
 <plugin>
     <artifactId>exec-maven-plugin</artifactId>
     <groupId>org.codehaus.mojo</groupId>
     <executions>
  <execution><!-- Run our version calculation script -->
      <id>Run Gulp</id>
      <phase>process-resources</phase>
      <goals>
   <goal>exec</goal>
      </goals>
  </execution>
     </executions>
     <configuration>
  <executable>node_modules/gulp/bin/gulp.js</executable>
  <arguments><argument>production</argument></arguments>
     </configuration>
 </plugin>
    </plugins>
</build>

Making this work in IntelliJ

We want the gulp task 'production' to be run when we start the application from IntelliJ. There are two ways to do this.
The simplest way is to not use the IntelliJ configuration target (our main application file), but use the maven task spring-boot:run as the configuration to run. This has several drawbacks, mainly that you cannot configure how you run the target so well.
The nicer way is to set up an IntelliJ configuration to run the gulp task (see also IntelliJ help) and then make our main IntelliJ configuration run this gulp configuration by default. Here is how that goes:

• Go to the configurations dropdown of your IntelliJ. For me, this sits in the top right corner. Choose 'Edit Configurations'. Alternatively, go to Run: Edit Configurations.
• Create a new configuration with the plus button (top left)
• From the dropdown, choose Gulp.js
• An 'unnamed' gulp task is created, call it 'production' (or whatever)
• Make sure that the gulpfile textbox points to your gulpfile.js
• Under task, select 'production'
• All other settings should be fine and can be left open if they are blank.

Now try your new gulp configuration by choosing it from the drop down, then pressing the Run icon (green triangle). Alternatively, use Run: Run.... A 'gulp' tab should open on your "Run" view (bottom of IntelliJ screen) and you should see the production task copy files and conclude that with a satisfying "Process finished with exit code 0".
Finally, we need to tell our main configuration (which runs our app) to run the gulp task first.

• Go to Edit Configurations again
• Under 'Spring Boot', you should see a configuration that runs your app. Click on the configuration.
• On the bottom of the window, there is a box called "Before Launch: .." with one task in it, called "Make" (if you do not see that box, stop all running processes and run this configuration once).
• Click the plus button, then click "Run another configuration" and select your gulp 'Production' task.
• Use the triangle icons to have Make be the last item.

Final workflow

When a new clone of the project is made, you will have to run a few commands to get the dependencies:

npm install
bower update

You will also have to run these commands every now and then to pick up new versions of the packages you are using. I think this is a great workflow, as it means you will not be surprised by new versions. You have to remember to run the commands on occasion.
To build and run the project, you can use maven or IntelliJ, as outlined above.
When making changes to the frontend files, you can view the changes with a simple browser reload if you run a watch before you start editing:

gulp watch

And that is it!

Conclusion

This seems a lot of work but it is definitely worth it if you want to maintain your app easily and if you want to automate the build and deploy process.

Some further links

• http://www.radcortez.com/javascript-package-management-npm-bower-grunt/
• http://andy-carter.com/blog/a-beginners-guide-to-package-manager-bower-and-using-gulp-to-manage-components
• http://andy-carter.com/blog/a-beginners-guide-to-the-task-runner-gulp