Seamless User Profile Migration from MySQL to OpenDJ

Following on from my previous post on OpenDJ password schemes, a common requirement is often to migrate users into the OpenDJ profile store from an existing database. There are numerous ways to do this, such as LDIF file imports or using OpenIDM reconciliation and livesync. However, both methods only really do a like for like comparison – no data cleansing takes place - unless you start to configure some logic processing in there.

This might be fine, but if your existing repositories contain millions of entries, some of which you don't know are live, a quick way to migrate across only active users, is to use OpenAM, with it's Dynamic Profile creation feature.

The above describes the process at a high level. Basically there are 3 authentication modules in a chain, using the flexibility of sufficient and optional modules. 

In this flow, there are basically 3 states.

	User in MySQL	User in OpenDJ	Authentication Works Against	Password Captured
1st Run	Yes	No	MySQL	No
2nd Run	Yes	Yes	MySQL	Yes
3rd Run	Yes	Yes	OpenDJ/MySQL	No

On the first run through, authentication fails against OpenDJ, as the user only exists in MySQL.  The chain then flows down to the JDBC module to authenticate the user.  The scripted module doesn't have an impact yet, as the user is only created in OpenDJ once the authentication chain has completed.  

With regards to the JDBC module, depending on how the password has been stored in the SQL database, it's quite likely you will need to write a password syntax transformation class, to alter the submitted clear text password, into an algorithm that the database is using to store the password.  This is pretty simple and documented process, with an example I wrote for SHA1 hashing available here.

On the second run through, the same thing happens, except this time the scripted module has something to update in the DJ repository - the user was created at the end of the 1st run through remember. The script simply does an idRepository.setAttribute against the newly created DJ user, to update the userPassword attribute with the password value from the sharedState.password.  The script I used is available here.

If all things are working as expected... the 3rd run through is somewhat different.  Not only does the user now exist in the DJ store, but that store also contains the existing user password from MySQL. 

So, whilst the user logs in using the same credentials as if nothing has happened, the authentication chain will authenticate successfully against OpenDJ and then exit the chain.

The main benefit of this approach, is that the end user has not been impacted - they log in with the same credentials as they did when using the MySQL repository.  No impact to their journey and no dreaded password reset use case.  Secondly, only the users that have successfully logged in are created in the new DJ store.  The bi-product of this process, is that a data cleansing aspect as has taken place.  Any users captured in the MySQL database that no longer use the service will no be migrated.

Another benefit of the migration is following my blog on password storage in OpenDJ, you can also seamlessly upgrade the hashing algorithm too.

NB - To allow the flow down of the shared state username and password down between the initial LDAP module and the secondary JDBC module, edit the module options setting within the authentication chain to conain iplanet-am-auth-shared-state-enabled=true.

Password Storage in OpenDJ

A common use case, is the migration of user profile data to OpenDJ.  Especially in large scale consumer facing identity projects, most clients already have repo's that contain user profile data.

Sometimes these stores also contain authentication data - that is the user name and password of the individuals.  Migrating data is relatively simple in this day and age regardless of whether that is identity data or not, but a common issue regarding login credentials, is how to migrate without impacting the login process.  For example, you don't necessarily want to get every user to reset their password for example, when they migrate to the new system.

Within OpenDJ this fortunately isn't a big deal.  A reason users might have to reset their password, is often to do with how the password has been stored on the source system.  When it comes to passwords there are generally two main approaches - symmetric encryption and hashing.  Symmetric encryption (meaning the password can be decrypted using the same encryption key) is seen a less secure method than something like hashing.  The argument for symmetric encryption was often around usability and speed and perhaps for password recovery style use cases - as opposed to password reset use cases if the password could not be recovered.

Password hashing is where a password is converted into a one-way set of opaque characters that visually have no relation to the clear text password - meaning hackers have a harder way of trying to get the original password.  The hash can also generally not be reversed - think of hashing like smashing a glass mirror - once smashed it's nearly impossible to get the mirror glued back together to look the same.  It's also nearly impossible to smash two identical mirrors in such a way that the broken pieces look the same.  So... hashing is seen as more secure and seen as irreversible.

But if it's irreversible...how do users login?  When the clear text password is entered, the password has the specific hashing algorithm applied to it and then compared to the existing hash that is stored. So we're performing a hash comparison not a clear text comparison.  I digress.

Back to OpenDJ.  OpenDJ provides a range of these different hashing algorithms out of the box. Take a look at the password storage schemes via the dsconfig interactive CLI (in ../bin/ of the main OpenDJ root folder).  Option 28 of the main menu takes you into the Password Storage Scheme area...

Most modern deployments will want to use a one way hash, generally with a salt, so something like Salted SHA512 is a nice bet.  Now the issue comes, when for example, the source data feed of users, has a hash of a lower security level than what you want in the modern world with OpenDJ.  So whilst OpenDJ supports things like SHA1 out of the box (and you can code new plugins for algorithms not supported...) you might want to migrate all users to a new more secure algorithm going forward.

Haha - the password reset scenario I mentioned above! Well not quite...OpenDJ has a neat feature that allows migration to new algorithms without getting users to reset their password.

Firstly you can set the appropriate default-password-storage-scheme to include the existing hashing algorithm (for example SSHA) when you migrate your users across.  This is done via the Password Policy option via the dsconfig main menu.  So we now have users in DJ with their password stored using the existing algorithm. A neat way to check this is the case, is to view the user via the ../bin/control-panel tool, switching to LDIF view.  Check for the userPassword attribute...and you will see the base64 encoded password.

Taking the encoded value and using something like the base64 utility that comes with most BASH distributions, you can decode the value to see the hashed value underneath.

Note the value is prefixed with the algorithm used, so it's easy to see what is happening. Next thing we can do, is to alter the default-password-storage-scheme to include our new algorithm...namely SSHA512.  Again, do this via editing the appropriate password policy.  At the same time, also alter deprecated-password-storage-scheme property include our initial algorithm - namely SSHA.

This on it's own doesn't alter the algorithm.  The change occurs, the next time the user authenticates. So logging into OpenAM with my existing user and their existing password...not only logs me successfully...it also updates the password in the background to be stored using the new algorithm.

This time checking the userPassword value in the LDIF view, I can instantly see the base64 value is much longer.

Doing a base64 decode, reveals the reason: we're now storing using the SSHA512 algorithm.

A quick and simple way to upgrade algorithms without impacting the user journey.

Of course, getting the data into DJ in the first place, would be a good use case for OpenIDM through basic reconciliation using the connector framework.  It is also simple to configure OpenIDM to leverage pass through authentication to leverage the password storage schemes just configured in DJ.

For more information on password storage schemes see here.

Stateless Tokens within OpenAM 13.0

The unstable OpenAM nightly build of 13.0, contains a great new feature: the ability to create stateless or client side tokens.  This brings a range of new use cases to the access management table, including increased scale (less server side storage, CTS replication and in memory storage) and the potential for "offline" token introspection for authorization.  Stateless does of course lack a lot of the key features of the stateful architecture.

What is a stateless token?

The stateless token is basically a JWT token, that is stored in the existing iPlanetDirectoryPro cookie (if accessing via a browser) or within the tokenId response if authenticating over REST.  The JWT contains all of the content that would stored on the server side in a stateful session - so things like uid, expiryTime and any other profile or session attributes you want to define.

To quote my colleague Ashley Stevenson "Stateful is a phone call and Stateless is a text message".

The token can also be signed and/or encrypted using standard algorithms such as HS256 (which uses a shared secret) or RS256 (which uses a public / private key combo) so adding a bit of security.

Config can be done at the realm level too, which makes for a flexible approach to which realms, users and applications should use it.

Offline Authentication

An interesting bi-product of using stateless tokens, is that introspection can be done on the token, without going back to the originating source - ie OpenAM.  Once OpenAM issues the token (this would need to be at least cryptographically signed and ideally encrypted if it contained sensitive PII required for authorization), verification and decoding of the token can be done by a 3rd party application.  This is pretty straight forward to do as OpenAM leverages open standards such as JSON Web Tokens (JWT) with standard signing and encryption algorithms.

I created a quick sample node.js application that does just that.  It does the following simply using a few lines of JavaScript and can be run from a command line for testing.

1. Authenticates to the pre-configured stateless realm in OpenAM over REST
2. Receives the JSON response with the tokenId value and strips out the JWT component
3. Verifies the tail signature using HS256 and a shared secret configured by OpenAM to prove the token hasn't been tampered with
4. Decodes the token from base64 and introspects the JSON contents

The code is available here.

The introspection aspect in step 4, could be easily expanded to perform additional queries of the contents, such as looking for certain claims or profile attributes that could be used by an application, in order to perform an authorization decision.

See the following draft documentation for further details on configuration of stateless tokens and the implications of the approach over stateful - http://openam.forgerock.org/doc/bootstrap/admin-guide/index.html#chap-session-state