Liquibase Core 1.5.0 is now available for download from http://www.liquibase.org/download

1.5.0 includes a major refactoring which should not affect most users except for the following items:

BREAKING CHANGES

• Servlet Migrator: The web.xml parameter names have changed. See http://www.liquibase.org/manual/servlet_listener for more information.

• If you used the “database.migrator.should.run”, it is must now be changed to “liquibase.should.run”

• If you have extended or embedded Liquibase classes or calls directly in your code, changes will be necessary.

BACKWARDS-COMPATIBLE CHANGES

• Servlet Migrator: The classes to reference in web.xml have changed to liquibase.servlet.LiquibaseStatusServlet and liquibase.servlet.LiquibaseServletListener. The old classes are now simply subclasses of the new and are deprecated so they should work.

• Spring Migrator: The class to reference in your spring config has changed to liquibase.spring.SpringLiquibase. The old class still exists as a subclass of the new so existing configurations should continue to work.

• Command Line: The “migrate” command has been changed to “update”. “migrate” is now an alias for “update” so existing calls should continue to work

ENHANCEMENTS

IMPROVED SCHEMA SUPPORT

There is now a “defaultSchemaName” parameter available for setting default schema. This schema will be used for all ambiguous database objects as well as for storing the databasechangelog and databasechangeloglock tables.

NEW COMMANDS

Ant support has been greatly expanded and now covers most of the functionality available in the command line application. See http://www.liquibase.org/manual/ant for more information.

NEW COMMANDS

• changeLogSync
• updateCount
• updateCountSQL

NEW REFACTORINGS

• update change

• delete change

OTHER CHANGES

• Custom Database implementations can be specified with the databaseClassName parameter
• “replaceIfExists” attribute added to createView
• createTable can specify uniqueConstraintName
• Setting value/valueNumeric/valueBoolean/valueDate on addColumn will update all existing rows with the given value
• Database table comments saved to generated change log
• Changelog file comparisons are case-insensitive on windows
• Output warning of old schema version
• Added comments tag to generated SQL output
• Rollback commands can specify contexts
• XSDs are not pulled from network
• Handles Postgres datatypes better
• Bug fixes

Upgrading

Upgrading is simply a matter of replacing the liquibase.jar file. To take advantage of newer change log features, change your XSD declaration to:

<databasechangelog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.5"
    xsi="http://www.w3.org/2001/XMLSchema-instance"
    schemalocation="http://www.liquibase.org/xml/ns/dbchangelog/1.5
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.5.xsd">

Depending on feedback received from this release, the 1.5.0.0 releases of the various plug-ins (Maven, Grails, IntelliJ, Eclipse) should be released over the next few days.

As usual, be sure to let us know if you have any questions or issues.

Liquibase Core 1.5.1 has been released along with the initial 1.5.x series Grails and IntelliJ plugins.

Changes to the core library are minor bug fixes including:

• Fixed failOnError logic
• Improved serial column detection in Postgres
• Rollback connections before close for DB2
• Other minor bug fixes

The Liquibase core library can be downloaded, as usual, from http://www.liquibase.org/download. The Grails and IntelliJ plugins are installed via their respective plugin managers

The 1.5 Maven plugin should be released next week after further testing.

There was a great post the other day on Coding Horror titled “Get Your Database Under Version Control”. It references a good series of posts by K. Scott Allen on database version control, but also makes the point that “When it comes to version control, the database is often a second or even third-class citizen.”

I have noticed this in the past as well, and have wondered why?

Lack of tools?

Subversion, CVS, RCS and others have been around for years, but when I came the the realization that the database must be version controlled I could find no tools that fit my need and so created Liquibase. I think lack of tools is a symptom of the problem rather than a cause, however. Developers have never been a group to sit around waiting for a tool to solve a pressing need they see.

DBA Overlords?

In some organization changes to the database must go through a database change process managed by DBAs. The fact that the database changes are managed by an external group could create a “not our problem” situation where the developers depend on the DBAs to track changes. Again, I don’t see this as a reason because the majority of projects do not have such a process in place and so wouldn’t be depending on it.

Only Now A Big Problem?

The answer that makes most sense to me is that database versioning is a relatively new problem. Code changes need to be propagated to every developer on a team quickly and reliably and therefore automated tools like Subversion and CVS have been around for a long time and no one would ever consider a group project without them. Most developers would not even consider a solo project without them.

Databases, on the other hand, do not change as often and so manual and error-ridden processes have worked well enough in the past. As databases have become more and more central to projects of all types the old manual database update scripts are showing their limitations.

I think the final straw in the ad-hoc database management schemes has come from the growth of agile processes. Pre-agile, even smaller projects would often design the database up-front and changes to it could be managed in a piecemeal fashion throughout the project. As agile does away with as much of the up-front design as possible, the number of database changes introduced throughout a project increases dramatically and a way to quickly, reliably, and automatically apply changes becomes a necessity.

I see the existing database versioning tools like Liquibase, DBDeploy, and ActiveRecord:Migration as the RCS of database versioning: they are a great start, but there is a lot of ground left to cover. Recent version of Liquibase have added features such as database change rollback, database comparisons, DBDoc, and change contexts, but there is still a lot to do. That doesn’t mean, however, that you shouldn’t Get your databse under version control!

Liquibase 1.5.2 has been released.

Changes include:

• Fixed bug with runOnChange and MD5sum check
• Handles generating changelog for tables with composite primary keys
• Other minor bug fixes

The Liquibase core library can be downloaded via http://www.liquibase.org/download.

The Maven, Grails, and IntelliJ plug-ins have been updated and can be downloaded via their respective plug-in manager.

The Liquibase team is proud to announce the 1.6.0 Liquibase Core release.

This release focuses primarily on Hibernate integration which allows you to use the diffChangeLog functionality available with the command line and Ant to automatically generate Liquibase change sets based on new Hibernate mapping information.

This functionality is similar to what is offered with hbm2ddl, but with the added ability to control and modify the changes before they are applied. For more information, see http://www.liquibase.org/manual/hibernate

Additional functionality includes a new “tagDatabase” tag, a “primaryKeyName” attribute on the column tag, as well as many bug fixes.

NOTE: Due to a bug in the change set md5 checksum generation in pre-1.6 releases, existing md5sum information needs to be cleared out of the databasechagelog table before running 1.6 for the first time. This can be done by running “liquibase clearCheckSums” or by running “UPDATE DATABASECHANGELOG SET MD5SUM=NULL”

Upgrading is simply a matter of replacing the liquibase.jar file. To take advantage of newer change log features, change your XSD declaration to:

<databasechangelog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.6"
    xsi="http://www.w3.org/2001/XMLSchema-instance"
    schemalocation="http://www.liquibase.org/xml/ns/dbchangelog/1.6
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.6.xsd">

Liquibase 1.6 can be downloaded from http://www.liquibase.org/download and, as always, let us know if you have any questions or issues.

The Maven and Grails plug-ins will be updated over the next week.

Liquibase 1.6.1 has been released.

This is primarily a bug fix release, but also includes a release of the 1.6 version to the Maven (still propagating), Grails, and IntelliJ repositories.

Download Liquibase 1.6.1 from: http://www.liquibase.org/download

It is Liquibase 1.0’s first birthday, but you get the gift: Liquibase 1.7.0 is now available.

Major changes include:

• Added loadData change which allows importing of data from CSV files

• Added support for changelog parameters for more dynamic and flexible change logs

• Allow preconditions in changeSet tags to act as sanity checks during a database update

• Can export data with generateChangeLog

• Added markNextChangeSetRan command

• Added validCheckSum tag to changeSet

• Can use any refactoring tags or reference previous changeSets inside a rollback tag

• Allow custom preconditions
• RowMapper interface is now public
• Misc bug fixes

The core library can be downloaded from http://www.liquibase.org/download while the Maven, Grails, and IntelliJ plug-ins can be installed from their respective plug-in managers.

We have moved our blog to a new hosting provider and to WordPress from Blogger.

Please let us know if you have any issues or if any RSS feeds don’t transfer correctly.

Martin Fowler makes good points in his Incremental Migration Bliki post. Data migration is easy to push off or forget, and can often kill an otherwise great release–especially when valuable historical data gets lost due to a bad script.

I think reason many people put off data migration scripts is because they mistakenly think that they are for release day only. In reality, however, data migration is needed throughout the development process including updating developer databases and migrating integration test data.

Doing your data migrations incrementally and appending to your database change log on every refactoring will help you catch migration logic errors early and keep everyone’s databases in sync.

To keep up with your database migration scripts, you need to use the update your test/code/test cycle to:

1. Write Test
2. Run Test
3. Code Change
4. Append database changes to change log
5. Run Liquibase to update database
6. Run Test, See Green

Making no database changes outside of your change log script and dev cycle will ensure that no changes get forgotten and that there are no differences between what happened on the developer’s database and what happens on other developer’s and production databases.

To integrate Liquibase into your build process, there are Ant, and Maven front-ends available. This will allow you to migrate your integration tests data sets before executing them and allow you to find errors in your database update logic with each run.

Liquibase 1.8.0 has been released and can be downloaded from http://www.liquibase.org/download.

Major changes include: #### Improvements to preconditions

• onFail and onError controls
• Several new precondition checks
• Custom preconditions can be passed parameters

SQLite Support

• Context checking is now case insensitive
• Specifying a column as autoincrement for a non-autoincrement table does not cause an error
• End delimiter can be specified with sql changes
• Indexes can be created as unique
• Required attributes for all changes are checked before execution
• Command line migrator return codes are better
• Many more bug fixes

The Maven repository will be updated over the next couple days. The Grails and IntelliJ plug-ins are available immediately.

I would also like to thank everyone who has helped with this release by submitting bug reports, feature requests, and patches. They are all greatly appreciated, so please keep them coming.

Liquibase 1.8.1 is now available for download. It is a bug fix release that fixes several minor bugs found in 1.8.0.

As usual, if you have any questions or comments, please let us know.

The goal of a Liquibase change log file is to track the linear sequence of changes required to take a database from a starting point to the current state, and it is built up one change at a time throughtout development. By following the “always append changes” rule you will ensure that all databases are consistent since all will have gone through the exact same set of changes.

If you go back and modify an old changeSet to incorporate a new change you will run into troubles because that change may have been ran against other databases and will not be ran again. To alert you to this problem, Liquibase stored checksums (md5 hashes) of each change that was ran. If the checksum of a changeSet in the current changelog file is different than what was executed before, Liquibase does not attempt to update anything.

That being said, things never go according to plan, and so Liquibase has several options for dealing with changes to existing changeSets depending on your needs. Generally I see 3 main reasons for a change:

Original Change Set was Wrong or Buggy

There are times when a change that was created wrong, especially if you are using the <sql> tag and have a complicated statement. When you catch the problem, you are often not able to recover original data or structures, but want the change to be correct for new databases going forward. In this case, it is best to use add a <validCheckSum> tag to the changeSet specifying that the new check sum is correct. To know the check sum to specify, simply make your change and try to update a database. The resulting verification error will include the new check sum.

Multiple change sets should be consolidated into a single change set

There are times when several change sets should be consolidated into a single change for performance reasons. There has been talk on the mailing list about creating syntax support for this, but for now the best option is to simply remove the old change sets from the change log and create a new one. To keep the new change set from being run on databases that have already made the change the old way, use the <changeSetExecuted>precondition.

Example:

Change should never have been applied

Perhaps you should never have moved data, or dropped a table. Whatevever the reason, there are times you want to reverse a done change set if it was executed and otherwise just skip it. The best way to handle this is similar to the combining change sets above. You simply delete the old changeSet from your changelog file and create a new changeset that will only run if the old changeset was ran.

Example:

<changeSet id="163-undo" author="nvoxland">
  <preConditions onFail="MARK_RAN">
      <changeSetExecuted id="163" author="nvoxland" changeLogFile="com/example/db.changelog.xml"/>
  </preConditions>
  .....
</changeSet>

More Options

Beyond these options, there are several other tools available to use including the changeLogSync command, markNextChangeSetRan, preconditions based on the database state (<tableExists>, <sqlCheck> etc.), or even manually editing the databasechangelog table to add or remove tables

Liquibase 1.9.0 has been released!

Major features include:

• modifySql support
• includeAll support
• Sybase Adaptive SQL Anywhere support
• Paths in <include> can be relative to changelog using the “relativeToChangeLog” attribute
• stop support
• runInTransaction attribute for changeSet
• Stronger validation in .xsd
• Better Derby support (drop column, rename column)
• Bug fixes

If you are using the standard Liquibase library, upgrading is simply a matter of replacing the liquibase.jar file. If you are using Grails, Maven, or the IntelliJ plugin, an updated version should be available via standard plugin repository within in the next day.

*NOTE: DUE TO A BUG IN THE “ADD NOT NULL CONSTRAINT” AND “DROP NOT NULL CONSTRAINT” CODE, YOU WILL GET CHECKSUM VALIDATION ERRORS FOR EXISTING CHANGESETS THAT USE THOSE TAGS. *

To solve the problem, either:

1. Add the <validCheckSum> tag to the failing changesets specifying the old MD5 sum (the one listed in the validation failed message)
2. Or, run UPDATE DATABASECHANGELOG SET MD5SUM=NULL

To take advantage of newer change log features, change your XSD declaration to:

<databasechangelog xmlns="http://www.liquibase.org/xml/ns/dbchangelog/1.9"
    xsi="http://www.w3.org/2001/XMLSchema-instance"
    schemalocation="http://www.liquibase.org/xml/ns/dbchangelog/1.9
    http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd">

Download Liquibase 1.9 from: http://www.liquibase.org/download

As usual, make sure you let us know if you have any questions, problems, or ideas. Also, thank you to everyone who submitted bug reports and patches over the last couple months.

There are times when I have seen projects with a version 1.9 that means “the last 1.x release before 2.0”. In our case, “1.9” simply means “the release after 1.8”

After a bit of discussion on the liquibase-user mailing list, we decided that we didn’t want to go to a 2.0 release at this point because there has not been any major changes and we have kept backwords compatibility.

Therefore, we are going to continue our current 1.x numbering strategy and continue to provide evolutionary improvements for the foreseeable future. 1.10, 1.11 etc. do look a bit confusing and like 1.1.0, 1.1.1 etc, but I think most Liquibase users will be smart enough to figure it out :)

Liquibase 1.9.1 has been released which addresses several bugs. Most bugs were relatively minor, but it is recommended you upgrade.

Some of the bugs fixed include:

• H2 schema capitalization issues
• Improved “local database” detection
• Improved database snapshot generation for Oracle and MSSQL
• Bug fixes in Sybase AS Anywhere support
• Handle urls with an authority component when using includeAll
• Fixed bug with “runInTransaction” attribute
• Support for Grails 1.1-SNAPSHOT
• Escape hsql tables/columns that are keywords

As usual, visit http://www.liquibase.org/download to download the update.

We store an md5 checksum with each changeset entry in the database changelog table to detect differences between what is currently in the changelog and what was actually ran against the database.

We try to keep as many false-positives and false-negitives out of the process by taking only the database change related tags, normalizing the XML, and then creating the checksum.

That means that you can reformat your XML (for the most part) as well as add or change preconditions, contexts, and validCheckSum tags without effecting the checksum. The only time that the xml reformat can cause problems is if a text block such as an <sql> tag is reformatted.

So don’t fear the checksum. It is there to help you out and you can add information around already ran changesets without problems. And if you do need to change your original changes, there are ways to deal with the checksum problems.

Liquibase 1.9.2 has been released. It is a bugfix release, mainly around issues with the includeAll tag.

Like always, it can be downloaded from http://www.liquibase.org/download

While the main goal of Liquibase is database change tracking, I found a very different use for it lately: moving test datasets.

I have an application that has been traditionally developed on MySQL, but am in the process adding MS SQL support as well. The application uses Hibernate so it should work in theory, but does it really work on SqlServer?

Since I have unit tests already set up that actually execute the data access layer against the database, I can run them against MS SQL instead, but the tests require data (and often specific data).

I wasn’t looking forward to duplicating my test data management effort, and so I started looking for a way to move the data from MySQL to SqlServer. I tried mysqldump (even with the “mssql compatability mode”) but it was still too MySQL specific. There are MS SQL migration tools, but I wanted it to be automated and be flexible enough to move the app to Oracle, DB2, etc. in the future. In particular, I was also wondering if running the DAO tests against H2 would be faster, but that is another blog post…

What I ended up doing was using Liquibase’s generateChangeLog functionality to output the database schema and data. The steps I run are:

1. Call generateChangeLog telling liquibase to export data as well. I used the Liquibase API to integrate it into my script easier.
2. Load the changelog into a DOM and update any mysql specific or SqlServer unsupported functionality using xpath. For example, convert BIGINT(20) to BIGINT.
3. Run the changeLog against the target database. This may take a bit of trial and error to find all your particular database incompatabilities.
4. Delete all rows from the target database’s databasechangelog table
5. Select all rows from the base database’s databasechangelog table and insert the values into the new database’s databasechangelog table. This will make sure that the database history liquibase sees in future attempts at updating is correct.

Now, I can continue to manage my dataset in MySQL like I always have but still magically have my test database and dataset available for testing against using MS SQL.

While I would not recommend using this for a production backup/restore scenario (the Liquibase diff tool does not export everything), it has been working great for managing and migrating the test dataset.

My plan for 1.10 is to focus on extensibility. I wanted to make sure I have good goals for that in mind, and that what we are planning to do will match up with those goals. 2. Create 3rd party community.

a. Ability to override what SQL is sent to the database for standard databases and refactorings (Add engine innodb to all create table statements, LoadData on MSSQL should call “set identity insert on/off”, etc)

b. Ability to add custom Change classes and use them from the changeset

c. Ability to create a custom changelog parser as an alternative to the standard XML

d. Better documentation of how the liquibase API can be used within an application

e. Ensure that all liquibase APIs will remain stable for the foreseeable future

2. Create 3rd party community. There are many refactorings that we do not want to support in the core because they are database specific or limited in scope (vacuum database, set identity_insert on, etc).

a. Create a “contrib” area on the site that people can upload useful extensions that are not in the core

b. Developer documentation on extending liquibase

c. Better issue/feature tracker that supports extensions?

d. Better mailing list/newsgroup setup

e. Ensure that all documented extension points will remain stable for the foreseeable future

3. Make core codebase more approachable to submissions:

a. Better developer documentation

b. Document migration path from 3rd party extension to core functionality

c. Refactoring of codebase to make it easier to add new Database, Change, and Statement classes without touching as much other code

d. Better unit test coverage to stop regression errors

Is there any particular use cases related to extensibility you want to make sure are addressed? What am I missing?

I plan to follow up with requests for in-depth comments on each of these steps as we get to them and can say exactly what we are implementing.

I am starting the extensibility refactoring at the *Statement class level and moving up from there. I committed an initial version of what I am thinking to trunk, and am looking for feedback (either in comments, or directly to me at nathan@voxland.net).

Renamed liquibase.database.sql package to liquibase.database.statement.

We may want to support non-“sql” statements in the future, so the name change seemeded best

Separated out the creation of SQL from the *Statement class.

Before each Statement class had a “String getSqlStatement(Database)” method that would create a string containing the database specific sql required to run the statement. Now, there is a liquibase.database.statement.generator package that contains classes that take a Statement instance containing the information about a statement (table name, column names, etc) and converts them into SQL strings.

There is now a SqlGeneratorFactory singleton with a “SqlGenerator getBestGenerator(SqlStatement statement, Database database)” method. When we get to the point where we have Statement classes that need to be executed, liquibase will call this method to get the correct SqlGenerator instance for a given statement and database and use that instance to create the sql to execute. (this change has not been made yet) .

The SqlGeneratorFactory has a list of all SqlGenerators available. That list is built up via a reflection package search in the liquibase.database.statement.generator package and sub-packages, or via a SqlGeneratorFactory.register() method. There will be a standard package of liquibase.database.statement.generator.ext that can be used by 3rd parties to have their statements registered automatically without using the register ethod.

I have started breaking up the Statement classes (although I haven’t removed the old getSqlStatement methods yet). In the liquibase.database.statement.generator package there is an SqlGenerator interface which provides a getSpecializationLevel method, an isValid method, and a generateSql method. In the getBestGenerator() method, the SqlGeneratorFactory will loop through all known generators and call the isValid method to determine if a given SqlGenerator would work for that combination of database and statement. Once it has a list of good generators, it will call the getSpecialiazationLevel() method which returns an int corresponding to how “good” that generator is for that statement/database combo. There should usually be a default generator for each statement that will have a level of 1. There will often be database-specific versions of the generator that return a level of 5. 3rd party Statement generators can return any level they want to override the built in ones or other 3rd party generators. If no generators are found for a database/statement combo, tyere is a NotImplementedGenerator that is returned that simply throws an exception.

New generateSql method returns an array.

Before, there was a one-to-one correspondence between a Statement class, and the SQL generated. I think there are situations, however, where a Statement may generate multiple SQL commands and so we should allow the return of an array.

generateSql returns instances of “Sql”, not Strings

This is the part I go back and forth on. Since the goal of this refactoring is to give 3rd party developers an API to code against, I want to do my best to make sure that API is not going to change. There has been talk in the past about building more flexibility into the system by giving back an syntax tree that can be modified before being executed. The idea with the Sql interface, is that we would only have an “UnparsedSql” implementation at first, but may make smarter implementations in the future.

The question is: is this too much? We now have a mechanism where we can override what sql string is generated for a particular database and/or statement, is that going to cover all the low-level-sql-tweaking that we would need? I feel like the Sql interface may allow some sort of cross-cutting sql modification, or similar, but it is just a half-thought out idea.

Improving tests

I am going to repurpose the test framework around the Statement classes that actually run them against the database. This has not been done yet.