Working with users and security (springdo-6)

Table of Contents

• Working with different users
• Spring Components
• Adding Spring Boot Security
• What is inside spring-boot-starter-security?
• Making login work
  • Backend changes
  • The admin page
  • Security settings
  • Further changes
  • Upgrading Spring Boot to run tests
  • Setting up and writing tests
  • AdminControllerTest
  • Improving the user database
• Resources
• Thanks

Working with different users

So far, the User Interface of our application has behaved as if there is only one user. All todo items share one space and there is no login necessary. In reality, we do want our fantastic application to have more than one user, in fact we want 1000s of them! And our users probably do not want to share their todo items for all to see. So let's go to the other extreme and make the notes private to a user. We will later want to introduce note sharing between users, but that will be another episode.
In essence the user story here is there be users. It is useful to split that up in smaller parts: First we will secure our site, so only authenticated users can log in. They will still see the one large list of todo items because items are not linked to users yet.

User Story 6a: A user can log in with a password to see the otherwise secured site.

After we have users and a login method, we can link items to users:

User Story 6b: A user should only see the items that belong to them.

and of course

User Story 6c: When a user creates a new item, it belongs to them.

Spring Components

The nice part of Spring is that there are many, many components to it. If you have a problem that is more or less commonplace, like authentication, there probably is a solution for it somewhere in the larger Spring framework. The Spring sitehttps://spring.io/projects is dedicated to the projects and you can see which projects are available and what they do (in its current form (Fall 2015), the site fails to live up on its promises a bit: Subprojects are not directly visible and it is low on actual information what the projects do, but this will surely be improved upon soon).
Luckily, Spring Security is there and a few clicks later, you will find the usual '5 minute' introduction to Spring Authorization and you can find more info on Spring Security in general. Note that this introduction is not necessary for this tutorial and it also was build with normal Spring in mind, not Spring Boot, but it makes for good reading regardless. Much of what follows below is based on it.

Adding Spring Boot Security

To add a new part of Spring to our project we have to first tell our build tool (Maven) about the new dependency and then have it download the necessary files. To do so, double click the pom.xml file on the top level of the project.
In IntelliJ, you can right click anywhere in the file and choose 'Generate', then 'Managed Dependency'. IntelliJ will show a scrollable list of all available dependencies, from which you should choose 'spring-boot-starter-security'. This should generate a new dependency in your pom.xml, like so:

<dependency>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Aside: There is also a 'Generate: Dependency' option and the difference does not seem to be covered the in the IntelliJ documentation. According to this older question, the position in the pom.xml differs between managed and normal dependencies (for the curious, managed dependencies explained by Maven). For us, the interface of 'Generate: Dependency' is easier but some dependencies cannot be included that way, so I advise to use 'Generate: Managed Dependency'.
If you were to run the project now, you would likely see the following error:

After you have made any changes to the pom.xml, you need to run the Maven targets (harder) or tell IntelliJ to act upon the new information (easier). Often, IntelliJ will pop up a message saying that the Maven project files need to be reread. Say yes to this. Alternatively, enable automatic reimporting under the settings menu, "Import Maven projects automatically":

A third way to have IntelliJ sync to the maven settings is to open the Maven tab (View menu, then Tool windows, Maven project) and use the shortcuts presented there. To alert IntelliJ to your maven changes, click the first button (two arrows in a circle). To download files, click the second button (folder icon with two arrows). Always make sure the 7th item, offline mode (representing by a disconnected cable icon) is in its deselected state, as Maven will not download any required dependencies in offline mode.

If all else fails, you can run the project from Maven directly, using the 'spring-boot:run' action. On the command line, you would do this by typing: `mvn spring-boot:run`. Before running the project, Maven will want to compile it and to do that, it will first pull in (download) all the dependencies.
TLDR We use Maven here but we like Gradle better / Working with Maven is not always very intuitive. The original version of this project was set up with Gradle instead of Maven. These two build systems are both very good, exhaustively documented and more than a little challenging when you first use them. Luckily IntelliJ (and any other IDE) makes things easier for you by giving you attractive buttons to click on. Gradle has the distinct advantage that its configuration file build.gradle is precisely 100 times more readable than Maven's pom.xml and there are a few other advantages too. However, almost all of the Spring (Boot) docs are made with Maven in mind, so we stuck to that convention here.

What is inside spring-boot-starter-security?

We used the dependency 'spring-boot-starter-security'. The 'starter' packages are convenient high-level dependencies supplied by Spring Boot: Instead of having to include a number of things, you just include the starter which will pull in the packages you would normally need for this task. In IntelliJ, you can inspect the file springdo.iml to see how IntelliJ has interpreted this. Here, I went to the 'version control' tab to do a diff on that file, showing that next to 'spring-boot-starter-security', nine other packages were pulled in:

Because the way Spring Boot works, reasonable defaults are assumed whenever no configuration is given. For the starter-security this means that without any further confirmation, your site will be locked down. When you go to any page, a browser-native login screen will be shown like this one:

This is not very helpful, but it is a reasonable default. The security suite can also create a default login form and other goodies, but we will not explore those here as they depend on Thymeleaf, a template engine which can help you serve pages from the server directly. Because we use Angular for the front end, our back end only outputs 'information', in the form of JSON packets, and never actual pages.
This reflects a change in style that has happened over the last few years: It used to be the case that frontend systems were minimal or non-existent, the server would simply produce a page which the browser would display as-is. Servers used technology like JSPor Thymeleaf to take a page template and populate it with user specific information ("Welcome User4829! You have 0 new messages"). Every click or input action made by the user would be sent back to the server and would produce a new page.
This system is still useful for static pages, but front end systems like Angular (and it predecessors Backbone, Foundation and others) have now taken over the role of interacting with the user. The obvious benefit is speed of interaction: A click can be handled by javascript much faster by cutting out two network trips and replacing a busy server with a dedicated and lean javascript engine in the user's browser. This is even more true on mobile devices using a cellular data connection: When I turn my phone sideways, I want an instantaneous animation from portrait to landscape which would be impossible to do from the server side over a relatively slow link.

Making login work

To make login work, we need to give the users a place to login. A page 'credentials.html' was created (specifically named like that because Spring security will automatically provide a 'login' page and we wanted to avoid confusion). I have deleted some source (marked // ..) to provide focus on the essential parts of credentials.html:

<html ng-app="credentials">
  // ...
<div class="container">
    <div ng-controller="credentials" ng-cloak class="ng-cloak">
 <h1>Please log into SpringDo</h1>

 <form name="credentialform" ng-submit="gosubmit()" role="form">
     <div><label> User Name : <input type="text" ng-model="username" autofocus="true"/> </label></div>
     <div><label> Password: <input type="password" ng-model="password" /> </label></div>
     <div><input type="submit" value="Sign In"/></div>
 </form>
 <div></div>
   // ...
    </div>
</div>

We set up a simple form which is bound to the javascript variables username and password. The 'Submit' button is tied to the gosubmit function, defined in the matching javascript, credentials.js:

 1: angular.module('credentials', ['ngRoute'])
 2:     .controller('credentials', function($scope, $http, $window) {
 3:  $scope.username = "navya";
 4:  $scope.password = "secret";
 5:  $scope.reminder = false;
 6: 
 7:  $scope.gosubmit = function() {
 8:      var request = {
 9:   method: 'POST',
10:   url: '/credentials.html',
11:   data: $.param({username: $scope.username, password: $scope.password}),
12:   headers: {
13:       'Content-Type': 'application/x-www-form-urlencoded', }, };
14:      $http(request).then(
15:   function (success) {
16:       console.log("success", success);
17:       $window.location.href = "/index.html";  },  // rather heavy handed
18:   function (failure) {
19:       console.log("failure", failure); });
20:      };
21:     });

By setting the javascript variables $scope.username and .password, we populate the form elements, which makes testing much faster. Obviously we would remove this for a production version of the app. A nicer approach would only populate these fields if we are on localhost.
The work is done in the gosubmit function, which has to check the supplied username and password with the backend. It does that through a POST back to the same endpoint, with the username and password in the body of the POST. The $param function will format a javascript map into a http body for us. We need to set the content-type correctly otherwise Spring will not accept the POST (this is a gotcha that took me at least one full hour to fix).
If the authorization is successful the success function is called and we forward the user to the main page with the $window.location.href function. This causes a rather ugly refresh, but we will leave that for now. If you are interested, there is a matching$location directive in Angular that is more lightweight and will create a smoother transition. It is a little more work to make it work in this context, so we left it out here.
Aside on persistent console logs: It is hard to follow the console messages that are displayed during login, as the page changes. To make this easier, have the log persist over page changes: In the Chrome developer tools, go to the settings menu (three vertical dots) and check 'Preserve log upon navigation'. The same setting can be found in Firebug.
If the authorization fails, a counter-intuitive thing happens: The post action is still succesful but the user is not authorized. Because the post is successful, the 'success' branch is run and the user's browser is forwarded to index.html. However, because authorization has not succeeded, the Spring security framework will throw a 302 error and redirect the user back to the credentials page. This does cause a fair amount of flashing and on the whole is not much of a great user experience, but we will leave it in place for now.
The failure branch is only reached if the server has gone offline. In general, it is hard to handle that situation correctly: The server was up when the page we were on was served but has gone down since. The best we can do is tell the user to try again and not showing any reaction will (implicitly) give the same message. Only in large javascript apps that maintain extensive local state (like Gmail), does it make sense to try to retry the server and send the new information after all. Keep in mind that at that point, you will also have to disambiguate between two possible causes of this error: Either the data never reached the server; or the data did reach the serve and the server state was adjusted but the server reply never reached us.
Aside on cookies and browser state: When working with authentication, you will sometimes run into weird errors. My favorite one is an AngularJS message in the console showing that an AngularJS assertion (areq) had failed. If this happens to you, often the solution is as simple as opening a fresh anonymous browser window to make sure you do not have any tokens and cookies floating around. A hard refresh (shift-control-R or shift-command-R) will not do the same thing.

Backend changes

This login function does not work without a large number of changes in the backend.
First of all, we need to create a database of users. To this end, we create a User class, which is also a JPA Entity. To later interface with the Spring Security framework, we inherit from UserDetails, a Spring Security Core interface which requires a number of methods like getPassword, getUsername, and isEnabled (see the Spring docs for full details).
The essence of the User class is only a few lines of code:

@Entity
public class User implements org.springframework.security.core.userdetails.UserDetails {

    @Id
    @GeneratedValue(strategy= GenerationType.AUTO)
    public long id;

    @OneToMany(mappedBy = "user")  // what this is called on the 'many' side
    private List<Item> items = new ArrayList<>();

    @JsonView(JsonViews.ItemList.class)
    private String username;  // login credential

    @JsonView(JsonViews.UserDetails.class)
    private String password;

    @JsonView(JsonViews.UserDetails.class)
    private String email;  // for password reminders etc

    @JsonView(JsonViews.UserDetails.class)
    private Boolean isAdmin = false;  // has access to admin pages?

We declare the data fields we need, they will be given getters further down in the code. There are no setters currently, as we are not modifying users yet, only creating them. The items field warrants some explanation: Each todo item is owned by one user, and each user has many items. Hence we have a many-to-one relationship between items and user. On the user side, we express this with a @OneToMany, which takes as an argument the name of the variable on the Item side. On the Item.java side, we have a matching line:

@ManyToOne
public User user;

Together, these two code fragments allow us to use a natural interface on both entities: Items have a user variable which holds the User class that owns this item, from the outside we can do anItem.getUser to access it. Similarly, we a User object has a list of entities accessible via items, or aUser.getItems, neither of which are currently used but may come in handy soon.
Note that there is no actual User variable on an Item, and there is no List<Item> on the User. Instead, these are magically supplied by the JPA from a hidden join table which has two columns, one for the UserId and one for the ItemId. AccessinganItem.getUser() is doing a query on that table for that ItemId, similar to SELECT userid from JOINTABLE where itemid = ?1. The returned UserId is converted into a User object. Similarly, accessing aUser.getItems() is equivalent to doing a query for all items by that user, SELECT itemid from JOINTABLE where userid = ?1, followed by a map to convert the itemIds into their matching Item objects.
A few more details need to be taken care of: Each object of the Item class now has a user field as we just saw. And when the ListOfItems endpoint is hit, we return the JSON representations of a number (zero or more) items to the front end. However, we do not want this user field to contain all the information we have on the user: Specifically, we would like to omit the password and the email. However, on a (future) admin view, we would like to see those fields?
How do we return two different types of JSON from the same object, depending on the context it was called in? Jackson JSON, the JSON generator that is used by Spring, has created JsonViews for that. Views are first declared as interfaces on a small class,JsonViews.java in our project:

public class JsonViews {
    interface ItemList {}
    interface UserDetails extends ItemList {}   }

Now we have two views with a hierarchical relationship, such that UserDetails also includes all fields that that ItemList declares. Next, we annotate each of the data fields to be in one of these two views, as shown above: Username is part of the ItemList view, and the password and email are part of the UserDetails view. We wil see below how you request these views on a controller.

The admin page

To edit, add and remove users, we will need an admin page. We will not be adding the functionality to edit users just yet, but this is also a good place to display the list of all items by all users. This is basically the functionality that ListOfItems used to provided, before it changed to only return the items of this user.
Three endpoints will be implemented in AdminController.java: First, /resource/admin/list/: list all items, regardless of user. Second, /resource/userlist/: returns a list of all usernames (strings) on the system. Third, /resource/user/{username}/: shows full details of the user, including password. Finally, we need to see who is currently logged in and we do this via the /who/ endpoint, which shows name of the currently logged in user.
For full details, please consult the AdminController.java file. Below is the code for one endpoint, the others are very similar:

@RequestMapping(value = "/resource/admin/userlist/", method = RequestMethod.GET)
List<String> userList(Principal principal) {
    User user = userRepository.findByUsername(principal.getName());
    if (user != null && user.isAdmin()) {
 Iterable <User> iterable = userRepository.findAll();
 List<String> result = new ArrayList<>();
 iterable.iterator().forEachRemaining((User u) -> {result.add(u.getUsername()); });
 return result;
    } else {
 System.err.println("Unauthorized request to /resource/admin/userlist/ by " + principal.getName());
 return new ArrayList<String>();
    }
}

We connect this function to an endpoint with RequestMapping as usual. This function will return a List of Strings (usernames), and it takes a Principal as an argument. When Spring Security is active, any function can have Principal as an argument and the value, a Spring object describing the currently logged in user, will be filled in automatically.
To work with principal, we usually just do a getName(). In this code, we proceed to look up that user in our own UserRepository, using an automatically created JPA function findUserByUsername (the interface for this function is defined in the UserRepository; the implementation is provided under the hood). Only admin users should be able to see the full list of users, so we check whether our user lookup was successful (non-null answer) and whether that user has isAdmin set.
In this first pass, we have put some code in SpringdoApplication.java to create two users, Navya and Dirk, both with password "secret". Navya is an admin user, and Dirk is not.
If the authorization is successful, we can retrieve all users on the system and return a list of their names. This is done via a findAll on the UserRepository. findAll is a built-in JPA function which does what it says on the can. FindAll returns an iterable, from which we derive an iterator. A lambda function is applied to each item with forEachRemaining, using a simple Java 8 lambda which takes a user u as an input and runs the add method on our result array. This is a little convoluted, partially because of Java design choices regarding iterators and iterables, and partially because we can only return a list of real items to the front end so we have to populate an arraylist with string values instead of passing the iterator or iterable.
If the authorization is not successful, we print an error on the console and return an empty list. Ideally, the front end would check for this with an ng-if and display some placeholder text ("No users found, are you admin?") in case the list is empty. This has not been implemented in the current version, but it should be less than 10 minutes work for you.
Lesson Learned If you look at the AdminController.java file, you see that each method is wrapped in an if, checking whether the user is authorized to call this routine. This is getting tedious and we will explore a Spring method to do this more efficiently later.
Back on the admin page, we show the list of users on the system. Each name has a details button next to it, and clicking that will show the user details after retrieving them from the backend. This separation between a user list and user details is good style: If the user list is long, only returning a minimal amount of information speeds up the page load. Secondly, sensitive information is only sent over the wire when explicitly requested.
The bottom half of the admin page shows all the todo items, their item IDs and the user who owns them. I have not implemented the ability to click on an item here to expand it inline, but you should be able to create this swiftly using my code for the user list as an example.
The admin html page uses two ng-repeat statements to loop over users and items. It also uses two partials to render the users and items inside the loops, consistent with the pattern we used on the main page. On the javascript side, admin.js does all the expected contacting of the backend and storing the information in variables on the scope.
The only interesting code is the userQuery array shown below: This is initialized to {} and filled with user details when requested via the button press. The buttons are wired to userDetails(username) calls (with the username filled in) and the routine will contact the backend and populate the userQuery map. On the html side (admin.html), an ng-if"userQuery[user]"= hides details that have not been retrieved yet: If there are no data yet, userQuery[user] will return undefined, which is falsy and will surpress the html fragment.

angular.module('admin', []).controller('ad1', function($scope, $http) {
 // .. 
    $scope.userQuery = {};
    $scope.userDetails = function (username) {
 $http.get('/resource/user/' + username + '/').then(
     function (success) {
  $scope.userQuery[success.data['username']] = success.data;
     }
 );
    };

We have not defined a 'failure' branch in the javascript above. This is consistent with my stance on this: I do not know how to substantially help the user if the request failed. Clearly, the username must exist on the system because our usernames have been retrieved from the server earlier. A failing call to /resource/user/<id> means that the server is down, the connection is very slow, or the user was since removed by another admin. So there are three reasons, which I cannot distinguish between and none of them can be easily addressed by my user.
What does the User Experience look like in case of failure? If no result is returned, the user details will stay hidden (thanks to the ng-if in the admin.html). The user will probably click on the username again, thinking that somehow their first click did not register. If the request is successful the second time, they will not think much of it; if it isn't, they will think the button is somehow broken. In either case, I don't think the user experience will be enhanced by flashing a message like "Sorry, but I cannot connect to the server right now, or maybe the user has been deleted? If you are on a mobile device, move to an area with better reception. If you are on a desktop, check your wifi connection or reinsert your IP cable. " This is adding unnecessary information to an already confusing situation, not a better User Experience. It reminds me of this (infamous) Windows 8 screen:

Security settings

We have wrapped the endpoints in the AdminController file in authentication testing logic. However, an unauthenticated user should not even be able to reach these methods. The same holds for the endpoints in ListOfItemsController: Without a valid login, we should not be able to access these methods. Spring Security provides a convenient method to enforce this: We basically block access to all of the site (with some exceptions for static content) unless the users is authenticated. If an unauthenticated request is made, Spring Security will redirect to the login page for a smooth user experience.
To set this up, we need to create a class that extends WebSecurityConfigurerAdapter and that is annotated with @EnableWebMvcSecurity (for Spring Boot 1.2.5 and earlier only, see below). That class should contains some 'fluent' declarations that do precisely what we outlined above:

@Configuration
@EnableWebMvcSecurity
// only exactly one class should have this annotation and it should override 'configure'

public class WebSecurityConfig extends WebSecurityConfigurerAdapter {
    @Override
    protected void configure(HttpSecurity http) throws Exception {
        http
                .authorizeRequests()
                    .antMatchers("/js/**").permitAll()
                    .antMatchers("/css/**").permitAll()
                    .antMatchers("/fonts/**").permitAll()
                    .antMatchers("/who").permitAll()
                    .anyRequest().authenticated()
                    .and()
                .formLogin()
                    .loginPage("/credentials.html")
                    .permitAll()
                    .and()
                .logout()
                    .permitAll()
                    .and()
                .csrf().disable()  // not great
        ;
    }

    @Override
    protected void configure(AuthenticationManagerBuilder auth) throws Exception {
        auth.userDetailsService(inMemoryUserDetailsManager());
    }

    @Bean
    public InMemoryUserDetailsManager inMemoryUserDetailsManager () {
        Properties users = new Properties();
        return new InMemoryUserDetailsManager(users);
    }

}

The first configure method allows access to pages as outlined, allows everybody to access the login and logout pages, and tell Spring that our login can be found at credentials.html. Finally, it disables Cross Site Request Forgery (Wikipedia), which is not a good thing to disable but makes our life much easier for now.
The 2nd and 3rd methods set up an in-memory database of users to be used by Spring. We will discuss this database in a minute, but in a nutshell: The InMemoryUserDetailsManager is a fast and easy way to set up authentication for first versions of your app and for testing-specific setups.

Further changes

Overview of all changes, please browse through the commit to read all the changes:

• index.html: Minor changes to show the currently logged in user and to log out.
• index.js: Matching changes to angular.
• credentials.html and .js: Login page as discussed above.
• admin.html: Page to view all items (regardless of user). In the future, this page will also allow us to add and remove users.
• admin.js: Matching angular code.
• AdminController.java: A new backend for the admin pages.
• AdminControlerTest.java: A test for the backend.
• Item.java: Added a user to each todo list item. Also changed the items so that the user name is included in the JSON for the normal view (itemlist view). Finally, added a factory method, Item.empty(User user), which will create a new empty item that belongs to the given user.
• User.java: A new entity and class that extends Spring's UserDetails class. It basically just stores the name, password and email of each user, with some added methods to satisfy the UserDetail interface.
• JsonViews.java: Creates a new class with two hierarchical interface: ItemList (the default view) and UserDetails (a more detailed view showing confidential user details).
• ListOfItemsController.java: The main endpoint (/resource/list/) will now return a list of items owned by the current user. Nex to that, a few GET requests changed to PUT requests to be more conformant with REST. Spring Security will work better when creating and deleting items is done via PUT.
• SpringDoApplication.java: Small changes to include the UserReposity (autowire), to set up the authentication framework and to add two users and five todo items to the database on startup. The five todo items were there previously, but now they are tied to users.
• MvcConfig.java: Configuration to show login pages with Thymeleaf, no longer used.
• WebSecurityConfig.java: Configuration settings as per above.

A short note about the Empty factory function in Item.java: This class already has several constructors (a zero-argument constructor is required by the JPA and we wrote two further constructors in the course of these tutorials). Why a factory function instead of another constructor? The main advantage here is that we can give it a good name instead of having to remember what the constructor with 1,2,3 or 4 arguments does again. We use a similar factor method to create an AdminUser in User.java, check it out.
At this point, we have a working login system: The user can log in and we have two users (hardcoded into the application for now). Stories belong to a user internally and only the stories owned by a user are visible to that user.

https://github.com/dirkjot/springdo/commit/37f70a12bd7e67e608dbb3572d86ebf5be2ca07f

Upgrading Spring Boot to run tests

Building tests to work with this secured site is not as easy as it should be: The latest Spring security version (4.0.3 at the time of writing) has support for mocking logged in users, but current spring-boot pulls in a much older version (3.2.8). You can check this by opening the Maven project tab in IntelliJ and clicking on the spring-security. It is also shown in the first screenshot above, on line 66, you can see security core being included at version 3.2.8. I tried overriding Spring Boot's defaults and manually pulled in the latest Spring Security version, but this lead to version inconsistency errors.
Luckily, a new version of Spring Boot (1.3.0) was released while I was writing this, and upgrading is as simple as changing the version number in the pom.xml file and refreshing Maven (see directions above):

<parent>
 <groupId>org.springframework.boot</groupId>
 <artifactId>spring-boot-starter-parent</artifactId>
 <version>1.3.0.RELEASE</version>

If you look at the springdo.iml file, you can see that Spring uses different version numbering schemes for each of its components: Spring Boot 1.3.0 pulls in the Spring Framework 4.2.3, Spring Data JPA 1.9.1 and Spring Security 4.0.3, among others.
Lesson Learned: Keep this in mind when looking at the documentation for Spring components: Consult the iml file or otherwise determine the exact version number used before you hit the internet. I wasted a good amount of time trying to make Spring Security 4.0 features work in the Spring Security 3.2 setting, because I didn't realize Spring Security has a different versioning system from Spring Framework (which was at 4.x already).
The migration does not come for free: We use a @EnableWebMcvSecurity annotation in our WebSecurityConfig.java class, but this annotation is now deprecated and replaced by @EnableWebMcvSecurity:

@Configuration
@EnableWebSecurity
public class WebSecurityConfig extends WebSecurityConfigurerAdapter {

There are many other migration concerns (see resources section below), but none of those affect us directly. Our previous commit had the security section excluded from the autoconfiguration, because this was necessary for that version. Now this line of code has to be removed or Spring will not start up (this is not mentioned in any of the migration documents, as far as I can tell). In SpringdoApplication.java, remove the line that reads:

@EnableAutoConfiguration(exclude = SecurityAutoConfiguration.class)

Setting up and writing tests

In a real world application, we should write tests before we write the implementation but this would make this tutorial harder to read. Also note that we are still not testing any frontend functionality: We will do all of that in a later episode of this tutorial. Because we have no tool to test the front end, we cannot do end-to-end testing either, which is really starting to become a bit of an issue. But again, more on that later.
As it is, we do not have any tests for the new admin endpoints, so these need to be written. We do not have any logic in our other java files (which contain repositories, entities, and such) so there is nothing to test there. But before we write the new tests, let's run the old ones: Our one functional test is ListOfItemsControllerTest, and it currently has one failing test: 'whenCreateIsHitANewItemIsCreatedAndReturned' fails with the following (compressed) logs:

Unauthorized request to /resource/create/ by null
...
java.lang.IllegalArgumentException: root can not be null

The problem here is that the routine to create a new todo item (code shown below) requires a user. In the production version, Spring will pass in a user (called a Principal in Spring security parlance) automatically and it will be picked up by our method argument principal. So if you test this by hand on the server, the code works. However, in the test, we are not logged in as a user and we therefore get an error on the terminal and a null object is passed into principal, leading to the subsequent NullPointer exception because we cannot do a getName on null:

@RequestMapping(value="/resource/create/", method=RequestMethod.POST)
Item postCreate(Principal principal) {
    User user = userRepository.findByUsername(principal.getName());

The reason we upgraded to the new version is that Spring Security has a new annotation in version 4.x: @WithMockUser. This should give us an easy way to mock a standard user (username 'user', password 'password') for this situation. We can optionally supply a username, which is what we do here. However, adding this to the test code still does not work, the principal argument does not get populated in the test. We will see below how to fix this the right way, here let's first step over a quick fix.
Here is the first part of the test code:

@WithMockUser("Navya")
@Test
public void whenCreateIsHitANewItemIsCreatedAndReturned() throws Exception {
    mvc.perform(post("/resource/create/"))
     .andExpect(status().isOk())

This test currently fails again with a NullPointer exception. I debugged the test by putting a breakpoint on the first line of the end point handler postCreate and we again have a null value for principal. To validate that this is a valid login problem, I temporarily allowed access to the /resource/** path in the WebSecurityConfig.java and then accessed the create endpoint (/resource/create/) with the excellent Postman tool, without logging into the site first: The result is the same NullPointer exception, but this time it is expected/valid as this endpoint should not be accessed without being authenticated. (You can have an interesting Friday afternoon discussion on whether the app should handle this error, or a NullPointer is good enough for something that is not supposed to happen. I suggest doing this over cheese and wine so it is enjoyable, because IMHO there is no right answer to this question.)
As the docs point out, another way to access the currently logged in user is to access the global SecurityContextHolder with a call like:

principal = SecurityContextHolder.getContext().getAuthentication();

This is not particularly pretty and it is, as we just saw, it is only necessary for tests. I 'solved' it by creating a helper function and calling it on the first line of all Test functions that involve principal:

public Principal fixPrincipalForTest(Principal principal) {
    if (principal == null) {
 principal = SecurityContextHolder.getContext().getAuthentication();
 if (principal == null) {
     System.err.println("Unauthorized request, principal is null");
 }
    }
    return principal;
}

If you debug this code, you can see that the principal object has inner principal data, which actually holds the username and password. Apparently this is how Spring has chosen to organize its principal data structure, but it is at least slightly confusing: Running principal.getPrincipal() will not return a Principal object, but a UserDetails.User object. Keep this in mind if IntelliJ tells you that you are dealing with incompatible types. We will encounter that 'User' datatype again below.
Using fixPrincipalForTest we can make things work and I actually have left it in the code of ListOfItemsController, even though it creates a (minimal) amount of overhead in the runtime. For AdminController, it is worth exploring a better way as there are four endpoints that all need this treatment. More importantly, in AdminControllerTest, all tests require authentication of some type.
On the Spring-Security channel of Slack, Rob Winch pointed me to the very obvious solution for this problem with principal: We need to tell the MockMvc that we use for testing to use the security framework. In AdminControllerTest, you will find the following code:

@Before
public void setUp() throws Exception {
    mvc = MockMvcBuilders
     .webAppContextSetup(context)
     .apply(springSecurity())
     .build();
}

The apply(..) line make the MockMvc use Spring Security. The @WithMockUser annotations will work as advertised and the principal argument will be set in the tests. A drawback of kinds is that all interactions with the Mvc now use security, which means that we cannot access any endpoints under /resource/ without logging in. For this reason, I stuck with the unsecured MockMvc and the fixPrincipalForTest hack in the ListOfItemsController.
Once security is working in the MockMvc, Spring provides some goodies for smooth testing. For example, we can easily test whether an unauthorized request indeed returns the correct response from the webserver. You can see this in action in this test, which confirms that user details cannot be viewed when you have not logged in:

@Test
public void AdminShowUser_failsWithoutAuthentication() throws Exception {
    mvc.perform(get("/resource/user/Navya/"))
     .andExpect(unauthenticated());
}

AdminControllerTest

With that we can finally write the code for the new Admin endpoints. We can use the handy IntelliJ Navigate: Test Subject method for navigating to the matching test file: If the test file does not exist, it will prompt you to generate the test file. A popup will show you the four functions in this file ask for which ones to create tests: I checked all of them. Next, I copied the test setup from ListOfItemsControllerTest and started modifying the provided empty tests. Instead of one test per function, we prefer more 'behavior based' tests that tell a user story and its outcome.
Here is a the first two, which test the outcome of the admin/list endpoint: Recall that this endpoint returns all items in the database for the admin users, but only the users own items for non-admin.

@WithMockUser("Navya")
@Test
public void AdminList_returnsAListOfAllItemsForAdmin() throws Exception {
    mvc.perform(get("/resource/admin/list/"))
     .andExpect(status().isOk())
     .andExpect(jsonPath("$", hasSize(5)))
     .andExpect(jsonPath("$[0].title", containsString("swim")))  // owned by Navya
     .andExpect(jsonPath("$[4].title", containsString("sleep")));  // owned by Dirk
}

@WithMockUser("Dirk")
@Test
public void AdminList_returnsAListOfOwnItemsForUser() throws Exception {
    mvc.perform(get("/resource/admin/list/"))
     .andExpect(status().isOk())
     .andDo(print())
     .andExpect(jsonPath("$", hasSize(2)))
     .andExpect(jsonPath("$[1].title", containsString("sleep")));
}

Note again that these tests will not work unless we use the fixPrincipalForTest function above (as we did for ListOfItemsController) or set up our MockMvc mvc to be aware of Spring Security (what we do here). The reason for these two different approaches are historical and educational, but setting up the MockMvc with Spring Security also means that each test in AdminControllerTest has to be wired up with MockUser etc. Because ListOfItemsController contains secured and unsecured content, it is easier to use the fixPrincipalForTest helper there. (You could argue that it is not optimal and confusing to have secured and unsecured functions in ListOfItemsController; on the other hand it is good to keep related functionality together).
#+COMMENT TODO update the ListOfItemsController file at a later stage.
The tests assume the setup created in 'SpringdoApplication.java': 2 users with 5 items between them. We could make more robust tests here by actually asserting that items belong to more than one user etc, but this is the simplest thing that works. For the user test, I quickly researched whether we could write a simple jsonPath to assert no item is owned by Navya, but because our json is an array with maps inside it, jsonPath's quite expressive syntax does not work here.

Lesson Learned: Although an array as the outer element is perfectly valid JSON, many tools expect a map as the outer element with named inner elements, or a 'data' key to hold the array. In other words:

valid:      [ {"map": 1}, {"map": 2} ]
expected:   { "data": [ {"map": 1}, {"map": 2} ] }

While testing the /who/ endpoint (which is not under the protected /resource/ tree, ie. accessible by anybody), I discovered that although the intention was that /who/ would show who, if anybody was logged in, it did not really deal with the case of not being logged in. Nice catch, and another good reason to write the tests first next time.

https://github.com/dirkjot/springdo/commit/840ec05dd68b8968d898c35e9ff5fd13ecf8c2fb

Improving the user database

There is one issue about this setup that is particularly not satisfactory to me: We have a UserRepository, but the Spring Security framework is not using it to authenticate the users. Instead, Spring has its own in-memory database of users and at startup, we copy all users from our repository into the Spring's database (this is done in SpringDoApplication.java, relevant code shown below). This is not only duplication of data, but it will also become quite hard to manage once we add the ability to edit, add or remove users.

@Override
public void run(String... strings) throws Exception {
    User defaultUser = userRepository.save(User.AdminUser("Navya", "secret", "n@example.com"));
    User secondUser = userRepository.save(new User("Dirk", "secret", "t@example.com"));
    //  ..
    userRepository.findAll().forEach(user -> inMemoryUserDetailsManager.createUser(user));

The underlying issue here is that Spring's concept of users more or less hardcoded into the framework. Spring provides an interface UserDetails which has specifies all the things you would expect: Name, password, enabled and a bunch of other reasons why a user may be considered inactive, and finally a list of GrantedAuthority to signify which roles the user has in the system. Our User class extends the UserDetails class, but Spring Security wants to talk to a UserDetailsManager class.
There are two major ways forward: Create an extra layer of indirection to the system, which will wrap around our UserRepository and supply the methods Spring expects from a UserDetailsManager. Alternatively, we can have a two-layer User representation, with an inner User that is a org.springframework.security.core.userdetails.UserDetails type, an contains all the administrative fields like isCredentialsNonExpired. This option is well discussed in this blog. We stay on our course and follow the first option, as our User class already has the necessary security 'cruft'. Personally, I would argue that it is good style to keep all the user data in one place, even if that means that we need to store some variables that we are pretty unlikely to ever use (and have ugly names to boot). Regrettably for us, Spring Security requires these and, to me, the easiest way forward it to implement them.
First, we create a new class the UserManager, which interacts with the UserRepository:

@Component
public class UserManager implements UserDetailsManager {
    @Autowired
    UserRepository userRepository;

    public void createUser(UserDetails user){
 userRepository.save((io.pivotal.User) user);
    }
    // ... code deleted

Because our user class implements UserDetails, we can pass our users into these functions, which are required to have UserDetail parameters by the interface. However, the repository does not know how to handle UserDetails so we have to cast them back to the User class they are. I have used the full path on User in this code to disambiguate the many versions of the User class here. Below is the code for creating a user, the other methods are similar:

public void createUser(UserDetails user) {
    userRepository.save((io.pivotal.User) user);
}

In the setup of the security component (WebSecurityConfig.java), we need to initialize our new UserManager. This setup is not particularly pretty, but apparently it is what Spring requires and it gives us the flexibility to specify the UserManager at runtime. We can use this later on to only start up with our 2 users and 5 stories for tests, and use an empty database for production use.

@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
    auth.userDetailsService(myUserManager());
}

@Bean
public UserManager myUserManager () {
    return new UserManager();
}

One final change is on the frontend, of all places: We have so far pre-filled the username field with "navya" (lowercase), which was apparently enough to satisfy the inMemoryUserDetailsManager, but our new login system does not convert case so we have to change it to "Navya". Similarly, use "Dirk" to login as the second user.
With that, we have a very long episode finished: Our three stories have all been addressed and can be accepted. Creating a secure app is not even that hard, it just requires a lot of background reading and understanding of how Spring Security works.
We have not implemented some straightforward things like adding, editing and removing users, but this is quite similar to the same operations on todo items. It does make for some great practice to implement this.
A related issue is that anybody can delete items at the moment if they are willing to create a custom url: All they have to do is to log in and then hit the delete endpoint with the todo item id. A better implementation would only allow the owner of the item to delete it.

https://github.com/dirkjot/springdo/commit/9a6ad767ba6b8cef920b25ff0a531857ac13eb0f

Resources

• A well-written blog post with a similar demonstration by Bartosz Kielczewski, this includes CSRF protection.
• Testing with CSRF protection on requires some extra work, see http://docs.spring.io/spring-security/site/docs/4.0.2.RELEASE/reference/htmlsingle/#test-mockmvc-csrf
• Blog entry on how to format security props: https://spring.io/blog/2013/07/11/spring-security-java-config-preview-readability/
• How to do a form post from angular is discussed on this StackOverflow post.
• There are differences between Spring Boot 1.2 and 1.3. These are apparently not included in the normal documentation but listed separately on a Github wiki page.
• Spring Security 4.x is a big upgrade over the 3.x series. However, this means the updates and migrations are not trivial either. This migration guide is very long but contains some good information. They provide a sample project which gets upgraded from 3.x to 4.x, here is the diff between them (several hundred lines).
• In an excellent answer to this SO question, Sam Brannen discusses various strategies for testing under Spring and Spring Boot and how you can disable security while testing.
• I can recommend this very long post (actually seven posts combined into one tutorial; reading the Github Readme.adoc files may be easier) that deals with all kinds of aspects of Spring, AngularJS and security. Part 2 (subdirectory 'Single' in the github repo) helped me write the code for this episode. Dave Syer starts out with the basics, then moves on to such topics as having security and content on different servers and using OAuth. It is highly recommended reading if you want to dive much deeper into that area. https://spring.io/guides/tutorials/spring-security-and-angular-js/ and https://github.com/dsyer/spring-security-angular/blob/master/overview.adoc.
• Full docs, syntax and online test pages for the excellent JsonPath library (JayWay version).

Thanks

A big thanks to everyone who read and commented on this tutorial (Ehren Murdick; Michael Oleske), to Pivotal Labs for letting me write these tutorials, and to Navyasri Canumalla, who sat down with me to learn Spring and wrote half the code you see here.