Upgrade your own package

 

Occasionally you want to write upgrade procedures for your own packages and, if you work on a product team or have write access e.g. to OpenACS CVS also for the product. This guide will let you know how you can make changes to any package and upgrade it correctly. I will not go through the steps for actually  running the upgrade, as this is already covered. This focuses on the description for the developer how to make sure that the upgrade for his package will run through without problems.

OpenACS version naming convention 

OpenACS version numbers help identify at a high-level what is in a particular release and what has changed since the last release. A "version number" is really just a string of the form:

major.minor.dot[ milestone ]

  • A major number change indicates a fundamental change in the architecture of the system, e.g. OpenACS 3 to ACS 4. A major change is required if core backwards compatibility is broken, if upgrade is non-trivial, or if the platform changes substantially.

  • A minor change represents the addition of new functionality or changed UI.

  • A dot holds only bug fixes and security patches. Dot releases are always recommended and safe.

  • A milestone marker indicates the state of the release:

    • d, for development, means the release is in active development and is not in its intended released form.

    • a, for alpha, means new development is complete and code checkins are frozen. Alpha builds should work well enough to be testable.

    • b, for beta, means most severe bugs are fixed and end users can start trying the release.

    • Final releases have no milestone marker. (Exception: In CVS, they are tagged with -final to differentiate them from branch tags.)

    Milestone markers are numbered: d1, d2, ..., a1, b1, etc.

A complete sequence of milestones between a couple of releases:

 5.2.1 => 5.2.2d1 => 5.2.2d2 => 5.2.2a1 => 5.2.2 => 5.3d1 => 5.3.a1 => 5.3b1 => 5.3 => 6.0d1 => 6.0

As you can see, we can skip the alpha and beta milestones altogether if we are not needing them. In a normal development cycle where you do not have a lot of development between feature freeze and release you will most likely only go from d (development) to release (as shown in the last upgrade step above) 

Upgrading

If you make a change to the parameters, provide new localizations, need DB upgrades or have a new dependency on a different package, you need to upgrade. Apart from the common scenarios described below an upgrade of a package always involves increasing the version number in the package.info file. This means you have to do that in three locations:

  • version_name
  • url
  • provides

In general you should only upgrade to a full release when NONE of the common scenarios are in place. A full release should only be used when your package is stable enough to be released. No upgrades should depend on you calling the new version a release version.

In addition to that information you can see that the .info files has information about callbacks that are executed when a certain event happens. acs-subsite has a good example of this:

        <callbacks>
            <callback type="after-install"  proc="subsite::package_install"/>
            <callback type="after-mount"  proc="subsite::after_mount"/>
            <callback type="after-upgrade"  proc="subsite::after_upgrade"/>
            <callback type="before-uninstantiate"  proc="subsite::before_uninstantiate"/>
            <callback type="before-upgrade"  proc="subsite::before_upgrade"/>
        </callbacks>

As we are only concerned about upgrades, before-upgrade and after-upgrade are the only conditions that concern us at this time. To understand their difference we need to look at the execution logic during an upgrade.

  • before-upgrade callback
  • database .sql updates (not encouraged anymore)
  • upgrading of package parameters
  • loading of catalog files
  • upgrading package version number
  • after-upgrade callback

As the before-upgrade callback is only useful in special situations (if you need to execute code before loading catalog files or parameter upgrades) we recommend only to use the after-upgrade callback.

 

Why not use before-upgrade and other upgrade woes 

Some users might ask why not use the before-upgrade callback as this allows you to keep the package not updated if anything fails. The reason for this is that if the upgrade fails, you still need to do manual actions. You sadly cannot upgrade your package though as part of the upgrade callbacks could have been already executed, leaving your package partially upgraded. Additionally the files are already of the higher version number, so technically you are already running the source of the latest package version, just your DB is not up to date.

Furthermore, the moment you do upgrades across multiple versions you are stuck if e.g. the upgrade from 3.1 to 3.2 introduced a parameter on which you depend on your upgrade from 3.5-3.6. Now you have an old site still on 3.1 which needs to be upgraded to 3.6. If you have the upgrades on the before callback callback, the parameter will not be there and the upgrade from 3.5 to 3.6 will fail. 

But if you thought, live is great I will only use after upgrade callbacks then (good choice!) you might still be getting screwed by the fact that you package relies on other packages at a given version. Sadly, you the upgrade from 3.2 - 3.3 depended on a procedure in acs-subsite 5.2 which became deprecated in 5.4, the package manager will upgrade acs-subsite to 5.4 first (because your package depends on it). Then it will run 3.2-3.3 upgrade script which will fail, because the procedure is not there anymore. Therefore you have to be aware that you need to doublecheck your upgrade scripts if they are still working after X major releases. My recommendation for OpenACS is to make sure for two major releases back (so you can savely upgrade from 5.2 to 5.4). How long you do that for your custom packages is up to you. Just be aware of this sideeffect when you change e.g. the API :-). 

 

Upgrade procedures 

After talking about all the risks involved in upgrading, lets take a look at the actual procedures you are going to call. The procedures for the callbacks are defined in /packages/yourpackage/tcl/apm-callback-procs.tcl. acs-subsite again gives us a fairly good example:

ad_proc -private subsite::after_upgrade {
    {-from_version_name:required}
    {-to_version_name:required}
} {
    After upgrade callback for acs-subsite.
} {
    apm_upgrade_logic \
        -from_version_name $from_version_name \
        -to_version_name $to_version_name \
        -spec {...}
}

As you can see the procedures have two parameters, "from_version_name" and "to_version_name". The package manager will on upgrade call this procedure with the from and the to_version_name parameters correctly set to the current version in the database (from) and the new version from the .info file (to).

How does a spec look like?

5.2.0d1 5.2.0d2 {
  set type_id [content::type::new -content_type "email_image" -pretty_name "Email_Image" \
                 -pretty_plural "Email_Images" -table_name "users_email_image" -id_column "email_image_id"]
                
  set folder_id [content::folder::new -name "Email_Images" -label "Email_Images"]
  content::folder::register_content_type -folder_id $folder_id -content_type "email_image"                
}
5.2.0a1 5.2.0a2 {
  set value [parameter::get -parameter "AsmForRegisterId" -package_id [subsite::main_site_id]]
  if {$value eq ""} {
    apm_parameter_register "AsmForRegisterId" "Assessment used on the registration process." "acs-subsite" "0" "number" "user-login"
  }
  apm_parameter_register "RegImplName" "Name of the implementation used in the registration process" "acs-subsite" "asm_url" "string" "user-login"
}

As you can see the spec for a single upgrade is made up of a list of three parameters. from_version, to_version and code. You can obviously define multiple upgrades  in the spec, each describing the upgrade from one version to the next. In the example above the upgrade from 5.2.0d1 to 5.2.0d2 will register a new content type. The second upgrade adds a new parameter and makes sure a different parameter exists.

As mentioned before, if you upgrade all specs which fall between your from and to version will be executed. If you upgrade e.g. from 5.1.5 to 5.2.0d6 then the 5.2.0d1-5.2.0d2 will be executed, but NOT the 5.2.0a1-5.2.0a2. Additionally they will be executed in order, so if you upgrade from 5.1.5 to 5.2 then first 5.2.0d1-d2 and after that 5.2.0a1-a2 is run. 

Common scenarios

These are the most common scenarios when you would like to do an upgrade of your package

 

Database upgrade 

You need to make a change to your database tables. This could be the need to create an index, change in the table structure or even inserting new data. Previously it was recommended that you do these upgrades in upgrade-sql files (and this is why you still see in each of the sql directories the "upgrades" directory). This is no longer the case.

To make a change to the database in the upgrade of your package, use the "db_*" statements within your apm-callback-procs.

 
    • Upgrade your table: "db_dml"  
      db_dml update_table "alter table malte add column handicap(integer)"
    • Insert data into your table: 
      db_dml insert_data "insert into malte (handicap) values (54)"
    • Execute PL/SQL: "db_exec_plsql" 
      db_exec_plsql delete_note {
  begin  note.del(:note_id);  end;
}

     

    Parameter changes

    If you need to work on parameters, here are some guidelines:

    • ADD: To add a new parameter you can just add this parameter to the parameters section of the .info file and you are fine.
    • Delete: To delete a parameter first remove it from the .info file. Then write a procedure to remove the parameter from apm_parameters and apm_parameter_values
    • Change: Changing the default value of a parameter is a three step process.
      • The easiest thing is to make sure the new default value is in the .info file so it is kept for future installs of the package.
      • To change the default value in the database, make a call to parameter::set_default and change the default value.
      • Last figure out the package_ids of the package in question using apm_package_ids_from_key
      • For each of the package_ids call parameter::set_value to change the value

     

    Language Keys

    If you added new language keys to your package you want to make sure that the language keys make it to the upgrade. There are different philosophies about how and when to add a language key in the first place, I will describe here my philosophy.

    When writing code, I will add the language key into the code. Once I test the page I will use the translation mode to add the language key into my installation. Before committing the change to the repository I will increase the version number of the package, save the keys into the catalog file and commit not only my change but also the .info file and the catalog files. This way I can be sure that any other developer picking up my code change has the message_key installed as well. Sadly over 50% of the OpenACS developers forget to export and commit the message keys, which is why one of the most common errors posted in OpenACS is "message key missing".

    A site doing the upgrade will in the upgrade process load the new message keys, and, dependent on the setting of the KeepLocalTranslations parameter, overwrite the existing locales if you made an upgrade in the catalog file or not. A conflict might also be raised, asking the administrator to resolve the conflict.

    The quick summary for the impatient on how to add a language key:

     

    • Add the language key into your code.
    • While testing, use translation mode to add the language key.
    • Export the packages messages to catalog files.
    • Increase the version number of the package.
    • Commit the whole package (including .info file and changed catalog files).

     

    You can execute any TCL code you want in the apm-callback-procs. This should make it even easier for you to achieve all the necessary things for the upgrade of your package. One example where this might be useful is if you want to move from lars-blogger for weblogs to xowiki::News.