Liquibase 3.2.3 and 3.3.0 have been released. As usual, they can be downloaded from the Liquibase download page and are available in the Maven repository as org.liquibase/liquibase-core.

Both 3.2.3 and 3.3.0 should be drop-in replacements for 3.2.2. A new batch of Liquibase extensions will be released over the next few days.

New “label” attribute on changeSet

Labels are general purpose way to categorize changeSets like contexts, but working in the opposite way. Instead of defining a set of contexts at runtime and then a match expression in the changeSet, you define a set of labels in the context and a match expression at runtime.

The most common time you would use labels instead of contexts is when the person doing the liquibase update has the knowledge of the types of changeSets to run, not the person writing the changeSet.

Labels can also be applied to modifySql

New change log commands and attributes

• New “empty” tag for explicitly marking a changeSet as unused
• New “output” tag for outputting a message during Liquibase update.
• New relativeToChangeLogFile attribute for loadData and loadUpdateDate
• New fullDefinition=true|false attribute on createView to support defining an entire view definition (including “column” names)

Support for clustered/nonclustered indexes and primary keys

A new “clustered=’true|false’” attribute is now avaiable on createIndex and createPrimaryKey to control whether they should be created as clustered or not.

And More

• Saving of “remarks” in MySQL and MSSQL
• Improved data type handling
• Performance improvements
• Official RPM and DEB packages built with release
• Major refactoring and updating of Ant integration
• Full release notes below

3.2.3 Change Log

3.2.3 is a patch release with smaller bug fixes. Even if not explicitly listed in the changelogs below, anything in 3.2.3 will be in 3.3.0.

• CORE-1919 - SpringLiquibase fails when dropFirst is true
• CORE-1987 - “mvn liquibase:diff” does not find any differences between databases
• CORE-1988 - Reported size for Oracle NVARCHAR2 columns is wrong
• CORE-1989 - Cannot set objectQuotingStrategy on root databaseChangeLog node
• CORE-2002 - AbstractResourceAccessor generates path in a unpredictable way
• CORE-2003 - Could not find implementation of liquibase.logging.Logger
• CORE-2042 - If liquibase.jar is nested in another jar/war/ear, it fails to start with a “cannot find implementation of liquibase.logging.Logger” error
• CORE-2058 - Load/Update tags should use “is null” not “= null” for null comparisons
• CORE-2070 - dropAllForeignKeyConstraints does not work on Firebird databases
• CORE-2075 - generateChangelog generates bad definition for TIME type
• CORE-2080 - Liquibase “empty” change not present in XSD version 3.2
• CORE-2065 - Use DOUBLE PRECISION for DOUBLE with Firebird
• CORE-54 - Support System Properties in Maven Plugin

3.3.0 Change Log

• CORE-16 - Support for “nonclustered” primary keys in mssql
• CORE-54 - Support System Properties in Maven Plugin
• CORE-1528 - Installer for Liquibase
• CORE-1598 - support for rename sequence
• CORE-1914 - New Change function: output
• CORE-1942 - Support for changeSet “labels”
• CORE-549 - relativeToChangelogFile for loadData, loadUpdateData, sqlFile
• CORE-1438 - createView should support having the entire view definition in the change body
• CORE-1502 - CLONE - UpdateSQL needs to append a “/” to the end of createProcedure for Oracle
• CORE-1654 - logicalFilePath support in formatted sql
• CORE-1660 - “remarks” attribute is ignored in MSSQL
• CORE-1932 - support for encrypted passwords / custom properties
• CORE-1946 - Have a rpm package for liquibase (built with maven)
• CORE-1963 - Ability to define full CREATE VIEW statement in <createView> change.
• CORE-1990 - Preserve inline comments in view snapshots in mssql
• CORE-2060 - Support liquibase.properties files with unknown properties
• CORE-2061 - Improvements to Informix support
• CORE-2062 - Add onlyUpdate flag to loadUpdateData
• CORE-2064 - Use ignoreClassPathPrefix for rollback as well
• CORE-2065 - Use DOUBLE PRECISION for DOUBLE with Firebird
• CORE-2066 - Support for –outputFile in command line
• CORE-2067 - Refactor Ant Task codebase
• CORE-2068 - New liquibase.hostDescription property for additional details in the DATABASECHANGELOGLOCK table
• CORE-2069 - Use prepared statement in <update> change whenever a clob type is used
• CORE-2072 - Do not include Oracle internal tables in snapshot/diff
• CORE-870 - Postgres, in an ALTER TABLE ALTER COLUMN statement, sometimes needs USING clause
• CORE-945 - Oracle : Temporary tables are created as regular tables
• CORE-1463 - Views not generated correctly with generateChangelog
• CORE-1556 - remarks attribute ignored for mysql
• CORE-1723 - unable to update on DB2/400, version V6R1, on jt400-6.7.jar
• CORE-1745 - afterColumn not working in MySQL
• CORE-1774 - Autocommit not restored on close in SpringLiquibase
• CORE-1882 - NullPointerException when MySQL foreign key points to an invalid table
• CORE-1919 - SpringLiquibase fails when dropFirst is true
• CORE-1922 - Sequence is not a reserved object name in HSQLDB
• CORE-1925 - liquibase scripts can not represent clustered indexes
• CORE-1937 - Oracle Float and VARCHAR precisions in changelog generated by generateChangeLog are incorrect
• CORE-1952 - liquibase loadData does not properly load numeric field in boolean always as false
• CORE-1956 - Double and float converted to FLOAT8(, 17) and FLOAT4(, 8) in PostgreSQL
• CORE-1958 - Column type of “TIMESTAMP(6)” under MySql converted to TIMESTAMP dropping fractional seconds
• CORE-1974 - dbchangelog-3.1.xsd missing <empty>
• CORE-1977 - CreateSequence with cacheSize=0 failing on Oracle
• CORE-1979 - MSSQL should not include parameters in SYSNAME data types
• CORE-1981 - Parameters set in included file are no longer set in 3.2.0
• CORE-1982 - Snapshot outputs defautlValueDate as defaultValueComputed on MSSQL for dates not in ISO format with a T in the middle
• CORE-1986 - includeAll from changeLogs within a jar is not working
• CORE-1988 - Reported size for Oracle NVARCHAR2 columns is wrong
• CORE-1993 - Drop table with cascade is not supported by Sybase
• CORE-1996 - addNotNullConstraint on h2 database has unexpected side effects
• CORE-1997 - Bit changelog default value of 1 executed as 0
• CORE-2002 - AbstractResourceAccessor generates path in a unpredictable way
• CORE-2010 - Oracle data type SDO_GEOMETRY snapshotted as SDO_GEOMETRY(1)
• CORE-2014 - applyToRollback property ignored when rollback changes are specified
• CORE-2015 - DiffChangeLog writes to the wrong point in the file on windows if file uses \n not \r\n
• CORE-2020 - Oracle default value current_timestamp converted to systimestamp
• CORE-2021 - Column remarks not snapshotted in mssql
• CORE-2026 - Oracle columns of type ANYDATA are snapshotted with a size
• CORE-2028 - generateChangeLog on SQL Anywhere 11.0.1 throws DatabaseException Driver Not Capable
• CORE-2032 - Snapshot incorrectly including clob/blob sizes on diff
• CORE-2051 - Not quoting VIEW params with spaces when snapshotting
• CORE-2054 - Add new “computed” column attribute to differentiate between an actual column name and a function as a column
• CORE-2063 - Fix for H2 autoincrement “start with” and “increment by” syntax
• CORE-2070 - dropAllForeignKeyConstraints does not work on Firebird databases
• CORE-2075 - generateChangelog generates bad definition for TIME type
• CORE-2080 - Liquibase “empty” change not present in XSD version 3.2
• CORE-2081 - PrimaryKeyExists precondition without tableName is broken
• CORE-2082 - Column snapshot on PostgreSQL does not include precision information for numeric data type
• CORE-2087 - Executing against Oracle doesn’t respect liquibaseSchemaName or liquibaseCatalogName
• CORE-2088 - outputDefaultSchema and outputDefaultCatalog command line parameters not respected
• CORE-2093 - Error: Property ‘relativeToChangelogFile’ not found on object type liquibase.change.core.LoadDataChange
• CORE-2094 - Liquibase.dropAll() should reset the lock service
• CORE-2095 - Invalid generated changeset for mysql bit with defaultValue 0

A new feature with Liquibase 3.3 is “labels”. Labels are similar to contexts in that both allow you to chose a subset of changeSets to execute at runtime. Labels are also similar to contexts in that both are purposely vague terms because they are fairly generic features can enable many different use cases. Where they differ is in who has the power to specify complex logic: the changeSet author or the deployment manager.

Contexts

Contexts in Liquiase have been available for quite a while, and they started out primarily as a way of “tagging” changeSets so they can be chosen at runtime. One common use is to mark changeSets that insert test data as context=”test” so that in your development and QA environments you you can run liquibase with –contexts=test to get the test data and in production you run with –contexts=prod to not have test data. Contexts are also helpful for marking changeSets based on feature sets to include (context=”shoppingCart”) or bundle (context=”pro”) or even customer (context=”acme_inc”). For complex cases, multiple contexts can be applied to a changeSet such as context=”acme_inc, pro” and multiple contexs can be chosen at runtime such as –contexts=free,qa.

With Liquibase 3.2, support was added to for context expressions in changeSets. Now, when you are defining your changeSet you can specify complex logic such as context=”!test” or context=”qa or (acme_inc and dev)”. The context logic can only be specified in your changeSet definition, however. When running Liquibase, you can still specify multiple contexts, but you are just listing out all the contexts that apply to the current Liquibase run.

Labels

Labels were added in Liquibase 3.3 to work like contexts, but “backwards” in who can specify logical expressions. In your changeSet you can only specify a simple list of “labels” that apply to the changeSet but at runtime you can write a complex expression to chose the labels you want to execute. This allows you to specify a changeSet with labels=”qa, acme_inc” and then at runtime use expressions such as –labels=”!acme_inc” or –labels=”pro or (free and beta)”.

Which is right for you?

Whether you should use contexts or labels comes down to whether the changeSet writer or the Liquibase executor best understands and/or needs the most control over which changeSets to execute.

• If the changeSet author needs to be able to specify complex logic based on the kind of environment that Liquibase will run in, use contexts.
• If the person executing Liquibase needs to specify complex logic to chose changeSets to run, use labels.
• If you do not use complex expressions, there is no functional difference between them.

Remember: you can use both.

Example Use Cases

Contexts work best when you can simply enumerate/describe features of the runtime environment:

• Free vs Pro versions
• QA vs. Dev. vs. Prod environments
• Customer A vs. Customer B

Labels work best when you can simply enumerate/describe what a changeSet is for, but the deployment time environment is complex to describe. An example of when labels would work well is when you can describe changeSets as for a particular feature or version such as “1.0” and/or “shopping_cart” but the decision on which features and/or versions needs to run is complex and chosen at deployment time. Labels in this case would allow you to run with –labels=”1.0 or (1.1 and shopping_cart)” to deploy the 1.0 changeSets and only the 1.1. features related to the shopping cart to one database and –labels=”1.0 or (1.1 and !shopping_cart)” to another database.

When in doubt, I usually go with contexts because that will simplify deployment configuration (to minimize release-day problems) while giving changeSet authors the option to handle complex logic if needed.

Liquibase 3.3.2 is officially released. It is primarily a bugfix release, but has one major new feature: object diffChangeLog/generateChangeLog object filtering.

includeObjects/excludeObjects logic

You can now set an includeObjects or excludeObjects paramter on the command line or Ant. For maven, the parameteres are diffExcludeObjects and diffIncludeObjects. The format for these parameters are:

• An object name (actually a regexp) will match any object whose name matches the regexp.
• A type:name syntax that matches the regexp name for objects of the given type
• If you want multiple expressions, comma separate them
• The type:name logic will be applied to the tables containing columns, indexes, etc.

NOTE: name comparison is case sensitive. If you want insensitive logic, use the (?i) regexp flag.

Example Filters:

• “table_name” will match a table called “table_name” but not “other_table” or “TABLE_NAME”
• “(i?)table_name” will match a table called “table_name” and “TABLE_NAME”
• “table_name” will match all columns in the table table_name
• “table:table_name” will match a table called table_name but not a column named table_name
• “table:table_name, column:*._lock” will match a table called table_name and all columns that end with “_lock”

Full 3.3.2 Change Log:

• CORE-875 - Ignore tables for diffs and generateChangelog
• CORE-1877 - SQLOutput prints endDelimiter regexes
• CORE-2114 - AddAutoIncrement on Postgres does not work when changes are applied on a specific schema
• CORE-2141 - handling dependencies and WAR as classpath
• CORE-2166 - SpringLiquibase: includeAll within jar causes SetupException
• CORE-2172 - dropPrimaryKey without constraint name on sql server doesn’t honour schema information
• CORE-2174 - Bad exception handling in OracleDatabase.setConnection
• CORE-2180 - NPE with bad name
• CORE-2182 - ClassLoader leak due to shutdown hooks

Since the 3.3.0 announcement, 3.3.1 was also released in December as a bugfix release with the following changes:

• CORE-1920 - SpringLiqubase includeAll is not including files
• CORE-2009 - ClassCastException when executing a custom task change (AntClassLoader problem)
• CORE-2097 - “mvn liquibase:futureRollbackSQL” asks for tag, count or date
• CORE-2099 - SQLAnywhere support (Driver not capable)
• CORE-2103 - changelogSchemaName/changelogCatalogName configuration options will not work on Oracle DB
• CORE-2104 - ConcurrentModificationException iterating over System.getProperties().entrySet()
• CORE-2105 - Maven profile performing dropAll and update on Oracle failing with an error on populated database.
• CORE-2107 - LOWER() keyword fails on Postgres createIndex task
• CORE-2108 - dropAll command trying to drop column on table that has already been dropped
• CORE-2116 - Could not find implementation of liquibase.logging.Logger
• CORE-2118 - Change default diffChangeLog/generateChangeLog objectQuotingStrategy back to LEGACY
• CORE-2119 - Bad finally block in SpringLiquibase.afterPropertiesSet()
• CORE-2120 - LoadUpdateData with value=NUMERIC quoting values
• CORE-2121 - DB2: DiffChangeLog/GenerateChangeLog/DropAll sees alias column and tries to drop/add them
• CORE-2127 - updateSQL creates duplicate DATABASECHANGELOGLOCK tables
• CORE-2130 - setFetchSize to a negative value breaks Oracle JDBC Driver
• CORE-2134 - ExecuteCommand won’t run with no os attribute.
• CORE-2136 - Mysql must quote PARTITION as a keyword
• CORE-2137 - Special characters (
) copied during generateChangelog on DB2/400
• CORE-2139 - H2Database.supportsDropTableCascadeConstraints() returns false
• CORE-2142 - generateChangeLog not including all columns in a table
• CORE-2146 - snakeyaml is pulled in as transitive dependency for using projects
• CORE-2149 - Liquibase command line fails
• CORE-2150 - On the 3.3.0-SNAPSHOT, liquibase –version returns 3.2.0
• CORE-2153 - Liquibase 3.2.1 is no longer compatible with Oracle 9
• CORE-2155 - diffTypes=data fails with java.sql.SQLException: Attribute value not valid (dataOutputDirectory attribute causes build to fail)
• CORE-2156 - Resource loader can’t load changelog file
• CORE-2157 - SQLException if there are single quotes in ChangeSet
• CORE-2159 - Datetime2 no longer used for MSSQL
• CORE-2161 - includeAll relativeToChangelogFile=”true” doesn’t work
• CORE-2164 - SpringLiquibase: includeAll within jar causes NullPointerException
• CORE-2179 - Creating functional indexes
• CORE-2115 - Really slow when using fat jars
• CORE-2125 - Make DatabaseChangeLog#include(String, boolean, ResourceAccessor) public
• CORE-2148 - Build failure on jdk-1.8
• CORE-2152 - Change logs in json format not processed by liquibase - parsing errors

Updated Extensions

The following extensions have also been recently updated with bugfixes, new features and support for Liquibase 3.3.x

• Liquibase-Hibernate 3.5
• Liquibase-Oracle 3.2
• Liquibase-Teradata 3.1
• Liquibase-Redshift 1.0

Download

As always, Liquibase can be downloaded from the Liquibase download page and is available in the Maven repository as org.liquibase/liquibase-core. The extensions can be downloaded from their corresponding github repository “Release” pages.

Liquibase 3.3.3 is primarily a bugfix release

As always, Liquibase can be downloaded from the Liquibase download page and is available in the Maven repository as org.liquibase/liquibase-core.

Fixed Issues:

• CORE-1768 - Oracle dropAll fails on spatial tables and sequences
• CORE-1840 - Liquibase fails when run on a computer that can’t connect to the internet
• CORE-1857 - Wrong column size detection on varchar2 fields with char as datatype
• CORE-1866 - Filtering changelog list by includeAll tag is not working
• CORE-1943 - Handle Error: InetAddress.getLocalHost().getHostName() UnknownHostException results in NoClassDefFoundError
• CORE-1958 - Column type of “TIMESTAMP(6)” under MySql converted to TIMESTAMP dropping fractional seconds
• CORE-1967 - includeAll uses full file path for sql changelogs
• CORE-2023 - Problem using includeAll with SpringLiquibase
• CORE-2126 - Postgres 9.3 - Drop table With Cascade - Not Supported
• CORE-2156 - Resource loader can’t load changelog file
• CORE-2186 - AbstractResourceAccessor#convertToPath(String, String) fails for processing includeAll from Classpath
• CORE-2192 - NoSuchMethodException when generating offline Oracle migration script
• CORE-2199 - Liquibase adds a semicolon after a stored proc definition making the stored proc unusable
• CORE-2202 - liquibase.should.run inverted boolean
• CORE-2204 - valueNumeric not being set when using prepared statements
• CORE-2206 - diffChangeLog with JPA-annotated entities causes ConcurrentModificationException
• CORE-2208 - Typo in message
• CORE-2210 - java.lang.NullPointerException when file is empty
• CORE-2214 - When inserting string value starting and ending with apostrophes (quotes) the value is not quoted in the generated SQL
• CORE-2218 - Regression on modifyDataType : VARCHAR2 was supported on 3.2…and fails on 3.3
• CORE-2239 - Remarks attribute in renameColumn causes parse error
• CORE-2240 - setDropFirst(true) still broken on empty database
• CORE-2262 - 3.3.2 ant task dies on NPE in ChangeLogParameters
• CORE-2263 - Index Snapshot - doesn’t include upper cased name indexes when db is NOT case sensitive
• CORE-2274 - Ant Upade Task does not consider changeLogFile correctly if it is contained in a JAR
• CORE-2279 - Rollback fails in MS SQL 2008 using liquibase 3.3.2
• CORE-2284 - Creating a DatabaseChangeLog() results in NPE
• CORE-2290 - Liquibase gives different results from Ant and the command line
• CORE-2301 - Regression from 3.2.3 in mssql 2000 unsupported usage of varchar(max) and sys.extenden_properties
• CORE-2304 - Autoincrement on type INT4 fails
• CORE-2310 - IncludeAll Fails with Unknown Reason Error
• CORE-2315 - NPE in CommandlineResourceAccessor
• CORE-2325 - Liquibase - New versions break DB create
• CORE-2329 - Escaped reserved keywords in HSQL are stored in lower case instead of upper case.
• CORE-2330 - includeAll uses full file path with includeAll
• CORE-2261 - UpdateSQL needs to append a “/” to the end of createProcedure for Oracle
• CORE-2287 - Improve support for Groovy-based tests in Eclipse
• CORE-2296 - Upgrade Groovy and Spock to maintained versions
• CORE-2318 - Add support for converting BigDecimal objects to a SQL string via DataTypeFactory

For those of you who don’t know, I’ve been working for Datical for the last year and a half as “Benevolent Dictator for Life.” I often joke that I feel like I’m retired because after years of working on Liquibase as a hobby on nights and weekends I’m now spending my days working on my “hobby”.

The rest of the Datical team works on Datical DB which uses Liquibase. Datical DB wraps additional functionality around Liquibase such as:

• Database specific objects such as Oracle Packages & SQL Server Functions
• Forecast to simulate deployments
• HTML reporting
• Out-of-the-box integrations with Jenkins, IBM UrbanCode, CA Release Automation, and others
• Rules Engine (using Drools) to help you control specific objects in your deployments.

I’ve always considered the scope of Liquibase to be “git for your database”–flexible but with a single, vendor neutral purpose that is part of an overall application deployment process. Datical DB builds out the rest of toolset for those who need it.

To help highlight and differentiate Liquibase and Datical, I’ve made a few updates to liquibase.org including a feature comparison on the download page and a more descriptive “Enterprise Version” menu link. Liquibase continues to be a separate product from Datical and will always remain open source. The changes are simply to point people to Datical DB if they are interested without getting in the way of those who simply use Liquibase.

Bugfix release Liquibase 3.3.4 is now available from liquibase.org/download and through the Maven repositories.

The most notable fix was to make the Maven plugin “liquibase.should.run” flag default back to true like it should have vs. “false” in 3.3.3.

Full changelog:

• CORE-2360 - Maven - Skip is active by default
• CORE-2199 - Liquibase adds a semicolon after a stored proc definition making the stored proc unusable
• CORE-2344 - Unknown host exception on OS RHEL 6.5
• CORE-2346 - IncludeAll does not work when runing liquibase from inside a jar
• CORE-2357 - alterSequence does not work as expected when you need to change the cache size
• CORE-2366 - Derby Network server works with command line but not with maven “Liquibase skipped due to maven configuration”
• CORE-2368 - No SQL outputted for <update> change

Yes, I know 3.3.4 just came out yesterday, but there was a configuration issue that affected dependencies in Maven.

3.3.5 resolves this with no other changes.

For people who have used Liquibase for a long time, a common question they have is how to clear out a changelog file that has gotten unwieldy.

The standard process for using Liquibase is to append individual change sets to your changelog file for each database change you need to make. Over time those changes can build up to thousands of entries, many of which are now redundant (create a table and later drop it) or inefficient (create a table, then add columns individually vs. just creating the table with all the columns). What is the best way to simplify all that cruft that has built up?

My first response is always “Do you really need to simplify it?” You built up that changelog over a long period of time and you have ran it and tested it countless times. Once you start messing with the changelog file you are introducing risk which has a cost of its own. Does whatever performance or file size concerns you have really outweigh the risk of messing with a script that you know works?

If it is worth the risk, why is it work the risk? Sometimes the problem is that your changelog file has just gotten so large that your editor chokes on it, or you get too many merge conflicts. The best way to handle this is to simply break up your changelog file into multiple files. Instead of having a single changelog.xml file with everything in it, create a master.changelog.xml file which uses the <include> tag to reference other changelog files.

<databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog/3.3"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog/3.3        http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd"><includefile="com/example/news/news.changelog.xml"/><includefile="com/example/directory/directory.changelog.xml"/></databaseChangeLog>

When you run liquibase update against the master.changelog.xml file, changeSets in com/example/news/news.changelog.xml will run and then the changeSets in com/example/directory/directory.changelog.xml will run. You can break up changeSets in whatever manner works best for you. Some break them up by feature, some break them up by release. Find what works best for you.

Other times, the problem is that liquibase update is taking too long. Liquibase tries to be as efficient as possible when comparing the contents of the DATBASECHANGELOG table with the current changelog file and even if there are thousands of already ran changeSets, an “update” command should take just seconds to run. If you are finding that update is taking longer than it should, watch the Liquibase log to determine why. Perhaps there is an old runAlways=”true” changeSet that no longer needs to run or there are preconditions which are no longer needed. Running Liquibase with –logLevel=INFO or even –logLevel=DEBUG can give additional output which can help you determine which changeSets are slow. Once you know what is slowing down your update, try to alter just those changeSets rather than throwing out the whole changelog and starting from scratch. You will still want to retest your changelog in-depth, but it is a far less risky change.

For other people, they find that liquibase update works well for incremental updates, but creating a database from scratch takes far too long. Again I would ask “is that really a problem?” Are you re-creating databases often enough that the risk of a change to the creation script makes sense? If you are, your first step should be to look for problem changeSets as described above. Databases are fast, especially when they are empty. Even if you create a table only to drop it again that is usually just a few milliseconds of overhead and not worth optimizing. The biggest performance bottlenecks in creating a database are usually indexes, so start with them. If you are creating and updating indexes frequently in your creation process, you may be able to combine those changeSets into something more efficient.

When you need to surgically alter your existing changeSets, remember how Liquibase works: each changeSet has an “id”, an “author”, and a file path which together uniquely identifies it. If the DATABASECHANGELOG table has an entry for that changeSet it will not run it. If it has an entry, it throws an error if the checksum for the changeSet in the file doesn’t match what was stored on the last run.

How you modify your existing changeSets will also depend on your environment and where in the changelog the problem changeSets are. If you are modifying changeSets that have been applied to all of your environments and are now only used on fresh database builds you can treat them differently than if they have been applied to some databases but not yet to others.

To merge or modify existing changeSets you will be doing a combination of editing existing changeSets, removing old changeSets, and creating new ones.

Removing unneeded changeSets is easy because Liquibase doesn’t care about DATABASECHANGELOG rows with no corresponding changeSets. Just delete out of date changeSets and you are done. For example, if you have a changeSet that creates the table “cart” and then another that drops it, just remove both changeSets from the file. You must make sure, however, that there are no changeSets between the create and the delete that make use of that table or they will fail on a fresh database build. That is an example of how you are introducing risk when changing your changelog file. Suppose instead you have a “cart” table that is created in one changeSet, then a “promo_code” column is created in another and an “abandoned” flag is created in another.

<databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd"><changeSetauthor="nvoxland"id="1"><createTabletableName="cart"><columnname="id"type="int"/></createTable></changeSet><changeSetauthor="nvoxland"id="2"><addColumntableName="cart"><columnname="promo_code"type="varchar(10)"/></addColumn></changeSet><changeSetauthor="nvoxland"id="3"><addColumntableName="cart"><columnname="abandoned"type="boolean"/></addColumn></changeSet></databaseChangeLog>

One option would be to combine everything into a new changeSet using the existing id=”1” and delete the other changeSets.

<databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd"><changeSetauthor="nvoxland"id="1"><validCheckSum>7:f24b25ba0fea451728ffbade634f791d</validCheckSum><createTabletableName="cart"><columnname="id"type="int"/><columnname="promo_code"type="varchar(10)"/><columnname="abandoned"type="boolean"/></createTable></changeSet></databaseChangeLog>

This will work well if all existing databases have the cart table with the promo_code and abandoned columns already added. Running Liquibase against existing databases just sees that id=”1” already ran and doesn’t do anything new. Running Liquibase against a blank database will create the cart table with all the columns right away. Notice that we had to add the flag or existing databases will thow an error saying that id=”1” has changed since it was run. Just use the checksum in the error message in the validCheckSum tag to mark that you know it changed and the new value is OK.

If you have some databases where the promo_code and/or abandoned columns have not yet been added, update the original createTable as before, but use preconditions with onFail=”MARK_RAN” to handle cases where the old changeSet ran while still not adding the columns again if the new changeSet ran.

<databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd"><changeSetauthor="nvoxland"id="1"><validCheckSum>7:f24b25ba0fea451728ffbade634f791d</validCheckSum><createTabletableName="cart"><columnname="id"type="int"/><columnname="promo_code"type="varchar(10)"/><columnname="abandoned"type="boolean"/></createTable></changeSet><changeSetauthor="nvoxland"id="2"><preConditionsonFail="MARK_RAN"><not><columnExiststableName="cart"columnName="promo_code"/></not></preConditions><addColumntableName="cart"><columnname="promo_code"type="varchar(10)"/></addColumn></changeSet><changeSetauthor="nvoxland"id="3"><preConditionsonFail="MARK_RAN"><not><columnExiststableName="cart"columnName="abandoned"/></not></preConditions><addColumntableName="cart"><columnname="abandoned"type="boolean"/></addColumn></changeSet></databaseChangeLog>

Now, on existing databases that have all 3 changeSets already ran, Liquibase will just continue on as before. For existing databases that have the old cart definition, it will see that the columns don’t exist for id=”2” and id=”3” and execute then as usual. For blank databases, it will create the table with the promo_code and abandoned columns and then in id=”2” and id=”3” it will see that they are already there and mark that they have ran without re-adding the columns. A word of warning, however: using preconditions will add a performance overhead to your update executions and are ignored in updateSQL mode because Liquibase cannot know how applicable they are when changeSets have not actually executed. For that reason it is best to avoid them if possible, but definitely use them when needed. Preconditions also add complexity to your changelog which will require additional testing so keep that in mind when deciding whether to modify your changelog logic. Sometimes it is easiest and safest to wait until all your databases have the columns and then modify the changeSets to avoid the preconditions.

The cart/promo_code/abandoned example shows some basic patterns you can use when modifying existing changeSets. Similar patters can be used to optimize whatever your bottlenecks are. Just remember when you change one changeSet, it can affect other changeSets below which may need to be modified as well. This can easily spider out of control so be mindful of what you are doing.

If you end up finding that it will work best to completely restart your changelog, see http://www.liquibase.org/documentation/existing_project.html which describes how to add Liquibase to an existing project (even if that project was previously managed by Liquibase).

(Originally posted to DZone)

When I first released Liquibase as open source in 2006, Sourceforge was the obvious place to host the project. Over the years, however, I’ve been slowing shifting functionality off sourceforge to liquibase.jira.com, github.com/liquibase and this blog.

For the last few years, the only thing remaining on SourceForge was the download file hosting. Given the recent issues SourceForge has had with GIMP, Nmap, and others it is time to move the file downloads off as well. Users entrust Liquibase with their most valuable asset–their data. I want to ensure that trust is sound which is why I am moving the download source.

What impact does this have? Very little. The main Liquibase download page is still http://liquibase.org/download and there have been no changes to that page except where the links point. If you were one of the few people who went to the sourceforge project page to download the files, they are no longer available and you must now go to the liquibase.org/download page instead.

I’ve moved the blog hosting off a wordpress server and am now hosting it on GitHub along with the rest of www.liquibase.org.

All of the existing content on blog.liquibase.org has been moved over and reformatted. URLs to previous posts should be the same except for using www.liquibase.org as the host instead of blog.liquibase.org. Requests to blog.liquibase.org should automatically redirect to the new site.

For those subscribing to the RSS feed, the feed URL changed from blog.liquibase.org/feed to www.liquibase.org/rss.xml but I did set up a redirect which most readers will hopefully follow.

Finally, the new setup does not support blog comments. That is OK with me because I prefer people to use forum.liquibase.org, liquibase.jira.com or even @liquibase for discussion.

If you run into any problems with the conversion, let me know.

Liquibase 3.4.0 has been released and is working it’s way through the Maven mirrors.

Downloads are available at http://liquibase.org/download. Question or comments can be directed here

Lots of good things in 3.4.0, but some highlights include:

You can now save snapshots and later reuse them for diff, generateChangeLog, and more

We previously added the ability to use “offline” databases which allows you to specify a url of “offline:mysql” for updateSql. With 3.4.0, there is a new --snapshotFormat=json attribute you can pass to the snapshot command which will output the snapshot in machine-readable JSON.

liquibase --url=jdbc:mysql://localhost/lbcat snapshot --snapshotFormat=json > snapshot.json

You can then use that snapshot with offline mode and any snapshot operations will use what is in the snapshot file as the database state.

• liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diff will compare the stored snapshot with the current database state
• liquibase –url=offline:mysql?snapshot=path/to/snapshot.json diff –referenceUrl=offline:mysql?snapshot=path/to/older-snapshot.json diff will compare two snapshots
• liquibase –url=offline:mysql?snapshot=path/to/snapshot.json generateChangeLog will generate a changelog based on what is in the snapshot
• liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diffChangeLog will generate a changelog based on what is new in the real database compared to what is in the snapshot.

Changelog properties can be defined as local to the changelog, not global

Previously, changelog properties were “write once” and global. Meaning that once you define a property, it cannot be overridden and is applicable to all changelogs.

You can now use the global="false" flag in properties defined in a changeSet and the property will only apply to that changeLog.

<databaseChangeLog><propertyname="varchar_type"value="NVARCHAR"/><propertyname="boolean_type"value="BIT"global="false"/></databaseChangeLog>

The type varchar_type will be set to NVARCHAR across all changelog files, but the boolean_type value will only be BIT within this file and will be undefined elsewhere.

Index descending can be more easily defined

Previously, if you wanted to control if an index is ASC or DESC you needed to use modifySql or just list the column name as “name DESC”. There is now a “descending” boolean flag that can be specified in createTable and createIndex changes.

<createIndextableName="my_table"indexName="my_index"><columnname="registration_time"/><columnname="id"descending="true"/></createIndex>

Performance improvements

There was significant work done to improve snapshot time as well as loaddata. The metadata queries were improved and the amount of metadata that needs to be read for some operations was greatly decreased.

The larger your databases are, the more of an improvement you should see.

New tagExists command

Are you wondering if you have already used a given tag? Use liquibase tagExists tagName to find out.

Improved SQL parsing

Parsing SQL with regular expressions is almost as bad of an idea as parsing HTML with regex but it is easier and what we always did. It usually worked, but would fall apart with things like semicolons or hyphens in strings and splitStatement and stripComments logic wouldn’t work as expected.

We’ve now switched over to a simple grammer which should make the SQL parsing much more resilient.

Full Release Notes

Thanks to everyone who helped with all these issues

• [CORE-14] - Dropping default values with MS-SQL
• [CORE-822] - Add a tag to add/update table/column remarks
• [CORE-864] - loaddata performance enhancement
• [CORE-1411] - MariaDB support
• [CORE-2254] - Ability to save snapshot for later comparison
• [CORE-2257] - Ability to use a previously saved database snapshot in diff and generateChangeLog
• [CORE-2302] - Add ability to load nested object/collection properties and BigDecimal properties automatically
• [CORE-2306] - Support passing in a script for rollback to override rollback logic included in the changelog
• [CORE-2308] - Track changeSet contexts and labels in databasechangelog table
• [CORE-2345] - Add XML Type
• [CORE-419] - Allowing ASC and DESC in index definitions
• [CORE-562] - Allow naming of not null constraints
• [CORE-715] - indexExists does a full snapshot
• [CORE-1731] - Support autoincrement in oracle 12c
• [CORE-2124] - Ability to pass properties to a JDBC driver required
• [CORE-2132] - Error message for missing sqlFile reference should be more descriptive
• [CORE-2147] - HsqlDatabase should emit uppercase names when quoting reserved words to preserve case insensitivity
• [CORE-2171] - New ChangeExecListener.runFailed method
• [CORE-2177] - Support NOT(X) syntax for labels
• [CORE-2185] - Few fields needs to be changed as protected and need additional field in RanChangeSet
• [CORE-2217] - Add DataTypeFactory support for delimited data type names, improve resolution of MSSQL data types
• [CORE-2228] - New usingIndexName attribute on addPrimaryKey
• [CORE-2236] - Support .yml extension in YamlChangeLogSerializer
• [CORE-2244] - Handle generating SQL Server DDL where ANSI NULL Default is false
• [CORE-2249] - Index and UniqueConstraint equivalence check should take name into account
• [CORE-2288] - Do not check/updatedatabasechangelog table on status
• [CORE-2292] - New tagExists command for command line
• [CORE-2298] - Allow HSQL to use defaultValueComputed for certain allowed functions on datetime type columns
• [CORE-2299] - Add capability to ignore missing or empty folder with includeAll
• [CORE-2307] - ChangeLog table name option in command line tool
• [CORE-2309] - global/local properties on changesets
• [CORE-2320] - MinGW (Git Bash) support for shell
• [CORE-2334] - Disable CREATE TABLE DATABASECHANGELOG generation when running on OfflineConnection
• [CORE-2336] - Use a grammer for parsing SQL rather than regexps
• [CORE-2358] - Improve data types for Liquibase tables in MSSQL
• [CORE-2359] - Consistently read dataTypeId for all databases
• [CORE-2363] - Improve robustness of MSSQL database case-insensitivity check
• [CORE-2371] - Improve Oracle snapshot performance
• [CORE-2386] - Set the connection default catalog/schema if defaultCatalogName or defaultSchemaName is set
• [CORE-2397] - MSSQL View Snapshot should not use sp_helptext
• [CORE-2399] - DBDoc improvements
• [CORE-2404] - Ability to preserve TEXT type in mssql snapshot and update
• [CORE-842] - Tag database not taking orderexecuted into account
• [CORE-1296] - drop column on ms sql server
• [CORE-1424] - SQL Generation ignores DATETIME parameters
• [CORE-1542] - Sequence is dropped not until a second run of dropAllDatabaseObjects on PostgreSQL
• [CORE-1738] - loadData from csv fails for boolean column (if another column present)
• [CORE-1749] - Update change command does not respect whereParams
• [CORE-1803] - DropAll doesn’t delete sequences if they are used as default value in postgres
• [CORE-1904] - Slow indexExists performance in Oracle
• [CORE-1924] - SQLServer diff - DATETIME2 not being handled correctly
• [CORE-2005] - /usr/bin/liquibase: Syntax error: “else” unexpected
• [CORE-2018] - Quotes stripped from index filter_conditions on snapshot
• [CORE-2019] - Comments in empty rollback prevent execution
• [CORE-2041] - Escaping of reserved keywords in HSQLDB
• [CORE-2096] - DiffChangeLog with changed indexes generates drop/add in the wrong order
• [CORE-2109] - dropAll fails for statements the database requires to run non-transactionally
• [CORE-2113] - Informix text datatype fixes
• [CORE-2133] - Oracle: GenerateChangeLog of a table with DEFAULT VALUE NULL creates defaultValueComputed=”NULL”
• [CORE-2167] - Issues with generateChangeLog of unique constraints on DB2
• [CORE-2169] - offline mode seems non-deterministic
• [CORE-2178] - Fatal exception acquiring lock in SQL Server databases with case-sensitive collation
• [CORE-2196] - Ant: “Unable to update database.” without explanation
• [CORE-2209] - Oracle snapshot sometimes creates “GENERATED ALWAYS AS (null)”
• [CORE-2211] - Liquibase tries to execute commented lines in custom SQL file
• [CORE-2219] - DB2 for zOs - adding primary key always emits REORG but REORG does not exist on Db2 for zOs
• [CORE-2222] - TIMESTAMP parameters dropped for PostgreSQL
• [CORE-2224] - Index uniqueness is not always recognized correctly
• [CORE-2227] - CLONE - UpdateSQL needs to append a “/” to the end of createProcedure for Oracle
• [CORE-2232] - Support schema other than public on PostgreSQL
• [CORE-2233] - Oracle Timestamp precision lost in generateChangeLog
• [CORE-2234] - columnExists precondition could be much faster (Oracle, mssql)
• [CORE-2237] - YamlChangeLogSerializer does not correctly serialize a changeset
• [CORE-2251] - Adding column with type DATETIME doesn’t work for PostgreSQL
• [CORE-2252] - XMLChangeLogSerializer writes array object for rollback tag content
• [CORE-2253] - Handle oracle varchar <-> clob conversions in diffChangeLog
• [CORE-2256] - Drop Sequences before Tables
• [CORE-2266] - DiffChangeLog: Invalid changelog when a primary key backing index is changed
• [CORE-2267] - Rollback fails for mixed-case objects created with QUOTE_ALL_OBJECTS
• [CORE-2270] - Doubledash inside quoted text causes parsing error
• [CORE-2271] - CLONE - DiffChangeLog: Invalid changelog when a unique constraint backing index is changed
• [CORE-2272] - DiffChangeLog must drop foreign keys before primary keys
• [CORE-2273] - Oracle char column snapshot not differentiating between a default value of 0 and ‘0’
• [CORE-2275] - YAML update fails with Unexpected error running Liquibase: java.util.LinkedHashMap cannot be cast to java.util.List
• [CORE-2281] - Oracle snapshot performance issue with many multiple views
• [CORE-2291] - ObjectQuotingStrategy not reset correctly after changeSet
• [CORE-2295] - includeAll tries to load all files instead only *.xml
• [CORE-2300] - Unsigned Int / Bigint cannot be created
• [CORE-2305] - Snapshot output too verbose
• [CORE-2316] - Data type registry occasionally returns wrong data type implementation
• [CORE-2321] - Liquibase tag command tags too much
• [CORE-2324] - diffChangeLog does not handle changes in sequence incrementBy, maxValue or ordered
• [CORE-2331] - Support for MSSQL collation in data type description broken
• [CORE-2340] - Add support for extensions to override the built-in change log, snapshot serializers
• [CORE-2355] - Improve updateSQL performance
• [CORE-2361] - preConditions, rollback, property, include, includeAll cannot be serialized
• [CORE-2373] - Local DTD files not found in subdirectory
• [CORE-2378] - Adding a new “CustomChange” triggers ClassNotFoundException
• [CORE-2380] - Support reading of gzip files
• [CORE-2381] - Fix unique constraint generator for informix
• [CORE-2383] - Change formatted SQL stripComments default from “true” to “false”
• [CORE-2385] - IncludeAll does not work when runing liquibase from inside a jar
• [CORE-2387] - dropPrimaryKey without constraint name on sql server doesn’t query schema correctly
• [CORE-2388] - Views not equal in different schemas
• [CORE-2390] - NullPointerException when generating changelog
• [CORE-2391] - column type doesn’t respect unsigned
• [CORE-2393] - changeSet contexts created with maven generateChangeLog are in parentheses
• [CORE-2401] - MSSQL handling timestamp according to sql standard, not sqlserver usage
• [CORE-2402] - Oracle NCLOB defaultValues not read correctly

There are many, many different processes and requirements companies have for managing their database schemas. Some allow the application to directly manage them on startup, some require SQL scripts be executed by hand. Some have schemas that can differ across customers, some have only one database to deal with.

For people who prefer to execute SQL themselves, Liquibase has always supported an “updateSQL” mode which does not update the database but instead outputs what would be run. This allows developers and DBAs to know exactly what will be ran and even make modifications as needed before actually executing the script.

Before version 3.2, however, Liquibase required an active database connection for updateSQL. It used that connection to determine the SQL dialect to use and to query the DATABASECHANGELOG table to learn what changeSets have already been executed.

Controlling updateSql SQL Syntax

With version 3.2, Liquibase added a new “offline” mode. Instead of specifying a jdbc url such as jdbc:mysql://localhost/lbcat you can use offline:mysql or offline:postgresql which lets Liquibase know what dialect to use. For finer dialect control, you can specify parameters like offline:mysql?version=3.4&caseSensitive=false

Available dialect parameters:

• version: Standard X.Y.Z version of the database
• productName: String description of the database, like the JDBC driver would return
• catalog: String containing the name of the default top-level container (‘database’ in some databases ‘schema’ in others)
• caseSensitive: Boolean value specifying if the database is case sensitive or not

Tracking History With CSV

These parameters let Liquibase know what SQL to generate for each changeSet, but without an active database connection you cannot rely on the DATABASECHANGELOG table to track what changeSets have already been ran. Instead, offline mode uses a CSV file which mimics the structure of the DATABASECHANGELOG table.

By default, Liquibase will use a file called “databasechangelog.csv” in the working directory, but it can be specified with a “changeLogFile” parameter such as offline:mssql?changeLogFile=path/to/file.csv

It is up to you to ensure that the contents of the csv file match what is in the database. Running updateSQL automatically appends to the CSV file under the assumption that you will apply the SQL to the database. Since the csv file matches a particular database, it isn’t something you normally would store or share under version control because every database can (and probably will) be in a different state. If you do store the files in a central location, you will probably want to at least have a separate file for each database.

By default, the SQL generated by updateSql in offline mode will still contain the standard DATABASECHANGELOG insert statements, so each database that you apply the SQL to will still have a correct DATABASECHANGELOG table. This means that you can switch between a direct-connection update and offline updateSQL as needed. It also means that you can also extract the current contents of the DATABASECHANGELOG table to a CSV file and use that as the file passed to the offline connection to ensure you have the right contents in the file.

If you do not want the DATABASECHANGELOG table SQL included in updateSQL output, there is an “outputLiquibaseSql” parameter which can be passed in your offline url.

Possible outputLiquibaseSql values:

• “none” will output no DATABASECHANGELOG statements
• “data_only” will output only INSERT INTO DATABASECHANGELOG statements
• “all” will output CREATE TABLE DATABASECHANGELOG if the csv file does not exist as well as INSERT statements (default value)

Offline Snapshots

The new 3.4.0 release of Liquibase expands The new 3.4.0 release of Liquibase expands offline support with a new “snapshot” parameter which can be passed to the offline url pointing to a saved database structure. Liquibase will use the snapshot anywhere it would have normally needed to read the current database state. This allows you to use preconditions and perform diff and diffChangeLog operations without an active connection and even between snapshots of the same database from different points in time.

To create a snapshot of your live databases, use the —snapshotFormat=json parameter on the “snapshot” command.

Command line example:

$ liquibase --url=jdbc:mysql://localhost/lbcat snapshot --snapshotFormat=json > snapshot.json

or

$ liquibase --url=jdbc:mysql://localhost/lbcat --outputFile=path/to/output.json snapshot --snapshotFormat=json

NOTE: currently only “json” is supported as a snapshotFormat.

You can then use that file with your offline url and any snapshot operations will use it as the database state.

• liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diff will compare the stored snapshot with the current database state
• liquibase –url=offline:mysql?snapshot=path/to/snapshot.json diff –referenceUrl=offline:mysql?snapshot=path/to/older-snapshot.json diff will compare two snapshots
• liquibase –url=offline:mysql?snapshot=path/to/snapshot.json generateChangeLog will generate a changelog based on what is in the snapshot
• liquibase –url=jdbc:mysql://localhost/lbcat –referenceUrl=offline:mysql?snapshot=path/to/snapshot.json diffChangeLog will generate a changelog based on what is new in the real database compared to what is in the snapshot.

Originally posted to java.dzone.com

Once you’ve created a database changelog file, what is the best way to run it? As always, it depends on what works best for you. There are three main ways to run Liquibase: “automatically on startup”, “manually as needed”, or “Just give me the SQL and I’ll do it myself”. All three work with any changelog file, so use the method (or combination of methods) which works best for your project.

Automatic Deployment

The easiest way to run Liquibase is to set it to run automatically on startup. Once set up, your database state always matches what your code expects and you have no manual steps to forget. This method works best in environments where you have less control over the deployment process or if you want a simpler deployment process.

I’ve seen this method used for web applications that use Continuous Delivery and have an automated release process from code check in through live production which gets executed multiple times per day. I’ve also seen this method used in packaged applications that are shipped to customers to make the database management portion completely transparent.

Don’t worry if you have multiple servers pointing to the same database. Liquibase uses a DATABASECHANGELOGLOCK table to ensure that only one instance of Liquibase runs at a time. Even if you have a cluster of servers all coming online at the same time and all automatically running Liquibase, the lock table will ensure that they will not all try to update the database concurrently and cause problems.

Liquibase ships with two hooks to automatically update your database on startup: a servlet listener and a Spring bean.

If neither of those hooks fit with your application, you can always call the Liquibase Java API directly. The most straightforward way of running Liquibase directly looks like this:

java.sql.Connectionconnection=openConnection();//your openConnection logic hereDatabasedatabase=DatabaseFactory.getInstance().findCorrectDatabaseImplementation(newJdbcConnection(connection));Liquibaseliquibase=newliquibase.Liquibase("path/to/changelog.xml",newClassLoaderResourceAccessor(),database);liquibase.update(newContexts(),newLabelExpression());

This code will create an instance of the liquibase.Liquibase façade and run the update() method which simply updates the database to match the passed changelog. There are many other methods on the Liquibase façade which can also be used if you are looking to automate Liquibase in different ways.

Manual Deployment

If automatic database updates don’t work well for you, you can execute Liquibase on demand. Liquibase ships with a command line application which supports both Windows and Linux. It also ships with an Ant task and a Maven goal for those who use those tools. These interfaces allow you to execute Liquibase commands whenever you need, without being tied to application startup.

One common use for the Ant and/or Maven interface is to integrate Liquibase into your build process. This allows you to catch errors in your changelog earlier, and also gives you a database which automated tests can be ran against. Developers can run the same tasks against their local environment for initial development and for fixing issues. Remember: contexts work well for embedding test data in your changelog and only deploying it to test environments.

You may also prefer to run Liquibase directly with more complex release processes that are designed to eliminate downtime. For example, if you make your database changes compatible with the existing codebase (no DROP commands), you can run Liquibase update while the old version of your site is still running. Once it has successfully updated, then begin a staged rollout of new code across your cluster.

Executing SQL

Manual updates with Liquibase allow you to control WHEN the database is updated, but WHAT is actually executed is still managed completely by Liquibase. For those who need to know exactly what is being done to their database, Liquibase supports an “updateSQL” mode in the command line, Ant, and Maven interfaces. When running updateSQL, Liquibase will simply output the SQL it would normally have ran. The output includes both the SQL to update your database and also the SQL to keep the DATABASECHANGELOG table up to date. Inspect the output as you need then execute it through whatever database tools you prefer. After running the SQL, your database will be in the correct state, and Liquibase will know what was ran and so future updateSQL calls will include only new changeSets.

Mix and Match

All the above methods can also be mixed and matched as needed to handle whatever schema management needs you may have. Some projects use automatic deployment for development and initial QA, then updateSQL for final QA and production. Some shipped products automatically run Liquibase on startup, but hotfixes are handled through a more manual process. Others automatically run Liquibase all the way through production, but the executed SQL is automatically saved and is watched by DBAs throughout the release progression to ensure nothing unexpected is happening.

No matter what your schema deployment needs are, you should be able to find a way to manage them with Liquibase.

Bugfix release Liquibase 3.4.1 is now available from liquibase.org/download and through the Maven repositories.

Fixes include:

• improvements to includeAll logic
• better handling of older-version DATABASECHANGELOG table structures with updateSQL and status
• improved data type handling
• extensions containing custom loggers should work again
• Correctly detect “Sybase IQ”

Full changelog:

• [CORE-998] - Changing index columns leads to wrong output order in the change log xml file.
• [CORE-2104] - ConcurrentModificationException iterating over System.getProperties().entrySet()
• [CORE-2385] - IncludeAll does not work when runing liquibase from inside a jar
• [CORE-2405] - Collation not preserved, depending on configuration
• [CORE-2406] - Escaped built-in data types should be lower case
• [CORE-2408] - Unknown column ‘LABELS’ in ‘field list’
• [CORE-2410] - Snapshot should not include paramaters for MSSQL geometry, geography or sql_variant types
• [CORE-2411] - BLOB string default values not quoted
• [CORE-2412] - Handle Oracle BFILE types
• [CORE-2414] - CLONE - generateChangeLog creates DOUBLE(22) instead of double in MySql
• [CORE-2415] - Custom Logger configuration does not work anymore
• [CORE-2416] - Diff drops and creates primary keys for all tables
• [CORE-2418] - Liquibase 3.4.0 tries to do INSERT instead of UPDATE-Statements with Postgres
• [CORE-2421] - MySQL column sizes are off by 1 in BIGINT and INT for diffChangeLog
• [CORE-2422] - Liquibase intialisation failed
• [CORE-2423] - Sybase IQ : strange procedure called
• [CORE-2426] - Default schema name missing quotes.
• [CORE-2427] - Better handle MSSQL stored procedures with a different defaultSchema and replaceOnExists=true
• [CORE-2428] - liquibase 2.0.3 to 3.3.3
• [CORE-2435] - includeSystemClasspath switch actually includes SystemClassLoader if false
• [CORE-2436] - Logging in ClassLoaderResourceAccessor prevents installation of custom Logger
• [CORE-2437] - Index.toString() contains “unique” if and only if index is NOT unique
• [CORE-2438] - DeleteGenerator does not handle parameter names and values with $ or \ properly
• [CORE-2440] - Not possible to override DefaultLogger using a Logger in a non-liquibase package.
• [CORE-2441] - Creation of foreign key fails in MySQL if database name contains dashes
• [CORE-2442] - Creating MD5 checksum fails if changeSet id contains the character “?”
• [CORE-2443] - Liquibase 3.4.0 ignores third party loggers in certain situations
• [CORE-2446] - endDelimiter splitting does not work in plain SQL files (regression)
• [CORE-2452] - Index names should be quoted on SQL Server
• [CORE-2458] - loadUpdateData will not update
• [CORE-2460] - Postgres index names cannot include schema name
• [CORE-2433] - quoting error in table creation
• [CORE-2359] - Consistently read dataTypeId for all databases
• [CORE-2419] - Support fluent/builder-style change properties
• [CORE-2449] - Correctly detect “Sybase IQ”
• [CORE-2450] - Non-bash /bin/sh gives “[[ not found” error
• [CORE-2451] - SQL scripts should have “USE database” in the header on SQL Server
• [CORE-2453] - Informix: Return null for connection schema name
• [CORE-2459] - Un-change Formatted SQL stripComments default back to true
• [CORE-2461] - Don’t do DATABASECHANGELOG ALTER statements if column types are different

This is the first in a planned monthly post designed to give a better window into Liquibase development and the Liquibase community. I’d appreciate any feedback you have on how helpful (or not) it is and/or any suggestions you have.

My work on Liquibase was split last month between a 3.4.1 release and continuing work on Liquibase 4.0. The full scope (and timeline) of 4.0 is still being determined, but the high-level goals are to:

• Simplify and standardize the codebase
• Improve the testability
• Make larger internal API changes to support new functionality

So far, the 4.0 work has focused on simplifying the Statement/Change/Generator/Snapshot logic and improving the testing of the interaction between Liquibase and the database. For more information and to help review the code so far, see http://forum.liquibase.org/#Topic/49382000001343003

The goal of 4.0 is to have no end-user breaking changes, but there will be API changes that will affect extension writers. Documentation on the upgrade process will be part of the release.

Beyond codebase changes, I switched the blog hosting from a separate server to being a part of the liquibase.org github pages sites. Anyone using an RSS reader should have been redirected, and for everyone else the main difference should be posts now showing in the left navigation of liquibase.org and a more consistent design. For me, I now have one less server to maintain which is always good.

I also recently became the proud owner of the @liquibase, twitter account so feel free to follow and/or contact me there.

Finally, I want to use each monthly update as an opportunity to recognize a different contributor. As an open source project, Liquibase is a group effort with usually very little recognition.

This month I’d like to thank Mark Chesney. Over the last few months he has sent nearly 50 pull requests and has not only done a good job of managing the corresponding liquibase.jira.com issues, but has also helped answer questions, find work-arounds, and triage issues on other issues as well. Most of his work has centered around improving MS SqlServer support, and I appreciate all he has done.

One problem with running an open source project is that we don’t really have “customers” to contact for feedback and product suggestions. I can always just add features that I think make sense, but one thing I’ve learned is that people use Liquibase in amazing ways I would never have imagined.

That is where the Liquibase survey comes in: it is your chance to provide feedback on how you use Liquibase and what is important to you so that I can better prioritize efforts over the next year.

Your participation and time is greatly appreciated

—–> Liquibase Survey 2015 Form<—–

If you know others that use Liquibase, please forward them the survey as well. Even better, if you know someone who is NOT using Liquibase, send them the survey as well.

The more information I can gather the better picture I will have.

Survey runs until September 18th, 2015

If you missed last month, I’ve started a monthly status post to give a better window into Liquibase development and the Liquibase community. I’d appreciate any feedback you have on how helpful (or not) it is and/or any suggestions you have.

New this month is the Liquibase 2015 Survey. Since I don’t have direct contact with the majority of Liquibase users, it is my best opportunity to understand how Liquibase is actually used in the wild to help prioritize features. If you’ve not filled out the survey yet, you can find it at http://goo.gl/forms/TNXBl5GsPv. The survey ends September 18th, 2015.

No new releases in the last month, but I’m planning on a 3.4.2 bugfix release in September.

Some interesting statistics from the liquibase.org traffic

• We get nearly 14,000 sessions per week
• Bing and DuckDuckGo are neck and neck for referrals, but Google still accounts for 98.6% of search engine traffic
• Nearly 15% of people are visiting from a Linux machine and more people use Android than iOS
• Over 68% use Chrome while less than 4% use IE

This month’s “contributor of the month” is Matt Bertolini. He saw that the Ant integration was getting fairly out of date and took it upon himself to refactor it. Not only was the pull request one of the best I’ve seen but he watched for related Jira issues and forum questions after his code was released and addressed any questions and problems that came up. While I always appreciate pull requests, the on-going maintenance of new features and helping other users with related problems is at least as valuable. As contributor of the month, Matt gets a 13,000 mAh USB battery graciously donated by datical.com in addition to my thanks.

Early on, Liquibase only supported XML changelogs with each changeSet listed in it. There are many advantages to using that setup, but there are now many different ways to store your database changes.

One way to manage your changes is to store them in individual SQL files. Some people prefer this setup because it is similar to other tools they have used, because they find SQL more natural to work with than XML, or they prefer to work with smaller files in a directory than changeSets stored in a large file. It is also helpful when migrating to Liquibase from an older system that manages changes as individual files.

Master Changelog

To bootstrap the process, you need a master changelog file that uses the includeAll command. I know the title said “Liquibase Without Changelogs,” but once you have created this file you never have to touch it again.

File “com/example/liquibase/changelog.xml”

<?xml version="1.0" encoding="UTF-8" standalone="no"?><databaseChangeLogxmlns="http://www.liquibase.org/xml/ns/dbchangelog"xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.4.xsd"><includeAllpath="com/example/liquibase"filter="sql"/></databaseChangeLog>

The includeAll command is telling Liquibase to run all the SQL files in the “com/example/liquibase” directory.

Now, when you run liquibase --changeLogFile=com/example/liquibase/changelog.xml update Liquibase will look for files with an SQL extension in com/example/liquibase and treat each as a changeSet. The files will be executed in alphabetical order.

Example Usage

Create a file “com/example/liquibase/0010-create-address.sql” containing

createtableaddress(idintprimarykey,line1varchar(20),line2varchar(20));

Now run liquibase update and you will see an “Update Successful” message and the new address table in your database.

If you run liquibase update update again, Liquibase knows that the SQL file already ran and does not try to re-create your table.

You can see what Liquibase is tracking by running select id, author, filename from databasechangelog and you will see a row with id=”raw”, author=”includeAll”, filename=”com/example/liquibase/0010-create-address.sql”. The SQL file is being managed like any other changeSet, using a generated author of “includeAll” and id of “raw”.

Now create another file “com/example/liquibase/0020-address-insert.sql” containing:

insertintoaddress(id,line1,line2)values(1,'121 Main Ave',null);insertintoaddress(id,line1,line2)values(2,'662 Broadway','Suite 3317');insertintoaddress(id,line1,line2)values(3,'412 Riverview',null);

and run liquibase update again.

This time data will be inserted into the table and if you select from databasechangelog there will now be a second row containing id=”raw”, author=”includeAll”, filename=”com/example/liquibase/0020-insert-address.sql”

Continue adding SQL files to build your database.

Things to consider: File Naming

Liquibase simply execute files in alphabetical order and will track what has ran based on the filename.

Therefore, choose a naming pattern that is easy for you to use but also leaves you open to sneak in new changeSets between existing ones down the road if you need. In the above example, I used a pattern of a zero-padded 4 digit number that increments by 10 followed by a simple description of what is in the file. The zero-padding will preserve the correct alphabetical file ordering while the incrementing by 10 allows me to add a “0011-increase-address-size.sql” file if I realize one needs to be added “before” the 0020 file. If I hit the limit of the 10 increment or the 1000 zero padded length, continue on with letters. 001a.sql will come after 0019 alphabetically.

Also make sure you use a consistent case for your filenames. Some systems will sort them case sensitively and others will not, so for best portability pick a case and stick to it.

Things to consider: checksums

Like all changeSets, Liquibase tracks the checksum of the changeSet when it was executed against each database. So, if you modify a SQL file after it has been executed somewhere, Liquibase will throw an error on the next update saying that the checksum has changed. Therefore, each new step in your database migration should be in its own file–don’t append to an existing file.

Things to consider: transactions

Like all changeSets, Liquibase will try to run each SQL file in its own transaction. Since most statements are auto-committing on most databases, it is best to have a single statement per SQL file. Otherwise, if the second statement fails but the first auto-commits, Liquibase will be in an indeterminate state due to the half-ran changeSet.

Liquibase Formatted SQL

Rather than using standard SQL in your files, consider using Liquibase-formatted SQL for some or all of your files. Liquibase-formatted SQL is designed to be backwards-compatible with standard SQL but allows you more flexibility and power within Liquibase, such as the ability to run preconditions, use changelog parameters, filter statements by database or context, and more. Liquibase-formatted SQL also allows you to specify multiple changeSet blocks within a single file.

To try a Liquibase-formatted file, create a file com/example/liquibase/0030-start-cart.sql

--liquibase formatted sql--changeset nvoxland:1createtablecart(idintprimarykey,created_datedate);--changeset nvoxland:2createtablecart_item(idintprimarykey,created_datedate,cart_idint,line_itemint,quantityint);

After running liquibase update, when you select from databasechangelog you will see a row with id=”1” author=”nvoxland” filename=”com/example/liquibase0030-start-cart.sql” and a row with id=”2” author=”nvoxland” filename=”com/example/liquibase0030-start-cart.sql”. Because of the formatting, Liquibase is seeing each chunk of the file as a separate changeSet and any new changesets you append to the file will be handled correctly as well.

(Originally posted to the Datical.com blog)

REMINDER: If you’ve not filled out the 2015 Liquibase survey yet, you can find it at http://goo.gl/forms/TNXBl5GsPv

The survey ends Friday, September 18th, 2015–MAKE SURE YOUR VOICE IS HEARD!

There were 136 responses to the 2015 Liquibase Survey. Since I’m relying on self-selection for people answering the survey, the first question is how representative is the data. The only two “demographic” survey questions I have alternate data points for are version used and country.

Liquibase Version

Maven downloads is as good of a proxy for active users as I have since it will be including re-downloads more often. Based on these results, the survey results are more skewed toward people who keep their Liquibase version more up to date as you’d probably expect since they are the ones visiting liquibase.org more often and subscribing to the blog where the survey announcements were.

Country

The survey is interestingly heavy on people from Germany and under-representative of India, but otherwise seems like a reasonable cross-section.

Usage Information

What I found most surprising was the length of time people have used Liquibase. For a project that was first released in 2006, I would have expected a larger contingent of people using it for longer than 4 years. It may be a result of newer users being most likely to visit liquibase.org and see the survey link, but it also a good sign for the project and the community that the user base is continuing to grow. Continued growth should be expected as well, since most people are using Liquibase daily or weekly as part of their normal development process, but many have so far only deployed to less than 1/4 of their projects.

Existing and Future Functionality

One goal of the survey is to get an idea of what functionality people use and what could be improved upon. The general “how well does it meet your needs” peaks at a 4 and averages to 4, whereas the documentation averages only 3.3. Documentation is something that I’ve not spent nearly enough time on, and the results definitely confirms that. The “meets needs” response is OK, but could definitely be better. The extension question had 75% of people not writing extensions, so most people are making do with what ships with Liquibase vs. building their own functionality.

The most helpful section of the survey was the free response questions since they gave people a chance to give specific details about what they liked and what they want improved.

Some common themes in the answers included:

What do you like most about Liquibase?

• Use of non-sql XML and other formats to describe changes while still supporting SQL when needed
• Database agnostic and independent
• Simple/easy to use
• Can set up automatic database updates that just work
• Flexibility
• Project reliability and community
• Multiple options for defining changelogs

What features do you wish Liquibase had?

• Improved error messages and feedback
• Improved groovy support
• Ability to verify schemas
• Support for more databases and database-specific functionality
• NoSQL support
• PostConditions
• Looping construct
• Consolidation of old changeSets
• Static analysis/linting of changelogs

What are your greatest pain points with Liquibase?

• Migration of current workflow / adaption
• Documentation
• File encoding issues
• Checksums
• The logging system
• Using Liquibase in OSGi
• Functionality that may not work consistently across all database types
• Lack of baselining support
• Performance
• Only Java-based
• Mixed-case handling
• Data type logic
• Sometimes slow bugfixes and community responses
• Required to use XML

There was a lot of other great answers as well, both for specific features to add and more general suggestions.

The areas for improvement generally lined up with what I was expecting, and are in the queue for upcoming releases which is always good. Issues with documentation appears not only as a called-out pain point, but also shows up as other issues (such as required use of XML) where existing functionality is not described well enough to be found.

Other demographics

I always find real-life database usage interesting. For all the hype some databases get, Oracle squeaks out ahead of Postgresql and Mysql with a big drop-off to the rest. MS SqlServer is probably under-represented compared to the industry as a whole since Liquibase is Java-based, but I was also surprised at H2’s popularity. I use H2 often while testing Liquibase because it is significantly faster for my tests than anything else, so it’s good to see it is getting wider use.

Not a big surprise that Java and Javascript dominated the technologies of Liquibase users, but I was surprised to see the other languages at nearly 10% usage each.

I’ve always seen Liquibase as more of a developer-friendly tool, but was surprised to see just how dominant the use by developers are vs. other roles was. Perhaps it’s all the XML that scares off everyone else…

I was also surprised to see the agility responses not farther to the right given that Liquibase is an agile/refactoring type tool.

Comparisons to 2013 Results

I purposefully kept the survey questions very close to the survey I did in 2013 to compare. Some highlights from comparing results:

• Daily use increased from 43% to 53% while weekly use remained consistent
• “How well does Liquibase meet your needs” increased from 3.6 to 4.0
• “How helpful is the documentation” increased from 3.2 to 3.3
• Database use was very consistent, other than an increase in DB2 use from 4% to 10%
• Non-Java technology use increased from about 5% to about 10%
• Agility increased from 3.4 to 3.5

Final Thoughts

Thanks again to everyone who filled out the survey this year.

While I get a good feel for how people are using Liquibase on an ongoing basis from questions asked, issues opened, and pull requests made, it is very helpful to have an alternate view of the project. There wasn’t anything completely unexpected to alter the roadmap I have in my head, but it is also good to see that the changes I’m planning on making will address many of the pain points people are having. Plus, there are were several “that is a great idea, why didn’t I think of that” suggestions which will be incorporated in future releases.

My plan was to continue making incremental improvements to the 3.x codeline while concurrently working on a more major 4.0 release, and based on the survey results I think that plan continues to make sense. It appears that for the most part people are satisfied with Liquibase and it does most of what is needed. Consistency and reliability are important to them, and I don’t want to throw major changes at them. However, while there are safe improvements such as performance and file encoding that can be made in 3.x, fully addressing things like checksums, mixed-case, and data types will require significant-enough changes that they are best wrapped up into a “you know there is going to require testing” 4.0 release.

Documentation is also re-highlighted as something that needs more love as well. Liquibase is very flexible and has a lots of options, and not only does the raw features need to be better documented but how the various pieces can fit together to solve use cases and pain points needs to be better highlighted as well.

If you have any question or suggestions please let me know or discuss it on the forum.