The OME Blog Because metadata is worth a thousand pictures

OMERO 5.3 Status Update

2016 has been a busy year for OME. Many of you will have noticed the number of Bio-Formats releases and the fact that OME Files C++ is now available for implementing the OME Data Model and OME-TIFF support in C++ software (you can read our preprint here).

That doesn’t mean we haven’t been busy on the OMERO front though, we’ve pushed our deadline to make sure we can deliver several new features:

Data Management

As well as implementing the new Data Model, we have introduced the Folders feature discussed in a previous post. Folders will allow you to organize your images’ ROIs into a hierarchical structure so that, for example, you can sort cells into phenotypes or assign ontology terms to them, or use an analysis script to track entities across a set of images and then use a folder for each entity to store sightings of it in OMERO.insight.

ROI Folder screenshot

UI development

We’ve been working on our metadata capturing and display, drawing lessons from our IDR project. We’ve also revamped the display of Screen-Plate-Well data to allow browsing of all fields within a well and display of their positions within each well. Zooming of the plate and fields are both supported. It is also possible to add and view annotations on individual wells (and not just the Images they contain).

New Well UI screenshot

Permissions improvements

To aid the workflows of facilities managers, we are designing a new ‘Light Admin’ role to the OMERO permissions system. This will allow a manager to import data for any other user (i.e. it will belong to the other user), delete other users’ data (for clean up e.g. after someone has left the lab), and manage user groups (creating groups and adding existing users to them) without having the full rights and responsibilities of a full Administrator. If you are interested in all the technical details, there is a design issue on GitHub.

Reading data

OMERO 5.3.0 will bundle Bio-Formats 5.3.x. This will include, amongst other changes, support for JPEG-XR compressed CZI data. This work was funded by a partnership between Glencoe Software and ZEISS. See announcement

Additionally, OMERO 5.3.x will benefit from the improvements in Bio-Formats 5.3.x for tile-base image writing for TIFF and derived formats like OME-TIFF. For more information about the plan for Bio-Formats 5.3, see Bio-Formats Development Status

Image rendering

We’ve introduced support for Lookup Tables and reverse intensity and improved the rendering engine to allow full projection thumbnails.

LUT support screenshot

We’ve also added histograms of image pixel intensities in the OMERO.insight and OMERO.web clients.

Histograms screenshot

OMERO.web architecture

A big effort has gone into making OMERO.web deployable as a separate component from the server and the various web apps installable from Python Package Index (PyPI). This has involved reorganizing all the packages. OMERO.web now has an ‘Open With’ function for using custom viewers and our next generation web image viewer (OMERO.iviewer) is coming on too. There has also been a lot of work behind the scenes to improve the Web API.

In line with this, our future client development will be very much focused on web clients and OMERO.insight will enter maintenance mode once the updates for 5.3.0 are complete.

Developer previews

If you’re a developer with your own OMERO code, you can check out the work so far to see if it’ll affect you. There are further details in the milestone version history and as ever, the code is all on GitHub (our latest development milestone is here).

Heads up for sysadmins

We’re changing our recommended way to install OMERO.web so you can deploy it separately to the main OMERO.server, making it easier to get your your virtual environment set up with all the necessary prerequisites. We have also deprecated support for Apache and recommend you deploy using Nginx as we are likely to drop Apache completely during the 5.3.x development line.

We are moving to Ice 3.6 as the minimum supported version. Other updates to version requirements include Python 2.7 but we will continue to support Java 1.7 for OMERO 5.3.x, as we are aware that upgrading to 1.8 would be incompatible with current MATLAB distributions.

With the requirements changes mentioned above, we are also considering no longer deploying and building OMERO on CentOS 6 to reduce our testing matrix. It will still be possible to use OMERO on CentOS 6 with the minimum requirements. See OMERO.server installation on CentOS 6 with Python 2.7 and Ice 3.6

Note that we are in the process of updating the documentation so although a preview of the code changes is publicly available, you should not expect all the milestone documentation to reflect these changes before the final release.

Release schedule

As a consequence of all this work, OMERO 5.3.0 is still under development and is now not expected until the first quarter of 2017.

OME Files C++ and C++11

This blog is an update about the use of C++11 and C++14 (“modern C++”) within OME Files C++.

We have been evaluating the switch from C++98 to C++11 for over 18 months. Up until now, we have remained using C++98 in order to continue to support older systems with compilers which do not support C++11. However, circumstances have recently changed which will necessitate a switch at some point in the near future.

Platform support

Most current platforms support C++11 and C++14, with some exceptions (workarounds are noted)

Platform Compiler C++11 C++14
CentOS 6 GCC 4.6 Unsupported by default* Unsupported by default*
CentOS 7 GCC 4.8 Mostly supported Partially supported
FreeBSD 10 Clang/LLVM 3.4 Fully supported Fully supported
FreeBSD 11 Clang/LLVM 3.8 Fully supported Fully supported
MacOS X 10.10 Apple LLVM 7.3 Fully supported Fully supported
MacOS X 10.11 Apple LLVM 8.0 Fully supported Fully supported
MacOS X 10.12 Apple LLVM 8.0 Fully supported Fully supported
Ubuntu 14.04 GCC 4.8 Mostly supported Partially supported
Ubuntu 16.04 GCC 5.3 Fully supported Fully supported†
Visual Studio 2013 VC12‡ Partially supported Mostly unsupported
Visual Studio 2015 VC13‡ Supported Partially supported
Visual Studio 2017 VC15‡ Supported Supported

* The software collections devtoolset-4 (GCC 5.2) and devtoolset-3 (GCC 4.9) packages provide updated compilers which can build OME Files C++ using C++11

† Boost 1.58 provided by this release does not build OME Files with C++14, but does build with C++11; see trac ticket

‡ Microsoft Visual C++ lags significantly behind GCC and Clang/LLVM, but VS2013 contains a useful subset of features, which are improved upon significantly with VS2015 and later

The support classifications above are broad; please see the following references for full feature coverage:

Third-party dependencies

All of our third-party dependencies have supported building with a C++11 or C++14 compiler for some time. However, more recently some of our dependencies have begun to require it, or will require it in their next major release.

For many users, these libraries are provided by package repositories, including Linux distributions’ package managers, MacOS X homebrew and FreeBSD ports. As these package repositories begin to provide the updated C++11 versions of these libraries, OME Files C++ will no longer build with them. While some of these C++11 libraries are optional (Qt5), others such as ICU are required by Boost and Xerces-C++ for full Unicode support, which is needed by the OME-XML data model. As a result, C++11 support will become a necessity for continued use of these libraries as the versions requiring C++11 enter mainstream use.

The benefits of C++11

C++11 adds a number of useful features to the language, including:

  • auto type
  • decltype keyword
  • defaulted and deleted functions
  • delegating constructors
  • forward-declared enums
  • initialiser lists
  • lambdas
  • nullptr keyword
  • override and final qualifiers
  • range-based for loops
  • rvalue references
  • static_assert
  • strongly-typed enums
  • variadic templates

and library features, including:

  • std::array
  • std::shared_ptr
  • std::thread
  • std::tuple
  • std::unique_ptr

These features are all supported by the compilers on the above platforms. Some additional features (not included here) are not yet supported by the compilers on every platform, and so can’t yet be used.

These features benefit the project in several ways:

  • they reduce the code size dramatically by permitting much more concise code with reduced boilerplate (by around 50% in one test conversion of a single component)
  • they allow the use of standard library features in place of Boost equivalents, which aids interoperability (e.g. std::thread and std::shared_ptr in place of boost::thread and boost::shared_ptr)
  • they allow improved performance of standard library containers through the use of rvalue references, and also allow the use of new features such as std::unique_ptr to replace boost::shared_ptr in certain situations, again improving performance
  • they make the library easier to use, for example the use of initialiser lists can reduce the PixelBuffer nD array addressing from multiple lines to a single line of code
  • they make the library more understandable and more familiar for users already using these features with other libraries; in particular features such as std::shared_ptr have been available since VS2010 for Windows developers and their use is already widespread

Proposal

Given that:

  • all of our supported platforms can support a sizeable subset of C++11
  • our dependencies will increasingly require C++11
  • there are significant benefits to switching to C++11 as a result of the features and performance gains it brings

we plan to make a C++11 compiler a requirement in our upcoming 0.3.0 release of OME Files C++, and begin using a subset of the C++11 features available across our supported platforms. For most users, the transition should be entirely transparent and will require no immediate code changes as we will leave compatibility typedefs in place through a transitional period.

If this will present any problems, please get in touch through our mailing lists or forums so that we can discuss any concerns you may have. Likewise, if the change is something which you will find useful and beneficial, we would also like to hear from you.

Bio-Formats Development Status

This blog is an update about what we are working on in the OME Files and Bio-Formats codebase for the next few months.

OME Files/Bio-Formats status update

Following our Data Model post, we spent the first semester of 2016 working on a new version of the OME Data Model which was released in June 2016. This schema release was then followed by two releases in August 2016:

  • the release of our reference implementation for the OME file formats, OME Files C++ 0.2.0 with full support for all versions of the OME Data Model
  • the release of Bio-Formats 5.2.0 including upgrade to the 2016-06 model, API additions and bug fixes.

Since August, a series of patch releases have been delivered both for OME Files C++ 0.2.x and Bio-Formats 5.2.x. These releases extended the C++ reference implementation to support all modalities of OME-TIFF allowed by our format specification, and also provided a series of bug fixes for the community following a major release.

In addition, we also performed a minor release of the OME Data Model extending the AcquisitionMode enumeration to support new values.

Since the beginning of October 2016, our focus has switched to the development of Bio-Formats 5.3.0. This release includes two major parts: a re-architecture of the components and new backwards-compatible API additions.

Bio-Formats 5.3.0

As part of the development of OME Files, a large part of the C++ code constituting the reference implementation has been migrated out of the Bio-Formats repository into its own set of components.

In Bio-Formats 5.3.0, we will be applying a similar re-architecture effort for our Java codebase and the OME Data Model. While Bio-Formats has been historically a large multi-module project, the idea here is to facilitate development and shorten build time, provide a better separation of the functional components and allow faster releases of the low-level components. In particular the following components have been filtered out to their own repository and will be managed separately:

All decoupled components have been released and uploaded to the Maven Central repository under the org.openmicroscopy groupId.

Bio-Formats 5.3.0 will also include API additions. In order to minimize the impact of these changes, the codebase is migrating towards semantic versioning. This means that all the API added in 5.3.0 will be fully backwards-compatible and should require no changes for consumers using Bio-Formats 5.2.0. Two main parts of the API will be reviewed and extended in 5.3.0:

Timeline

We hope to release Bio-Formats version 5.3.0 at the beginning of December 2016. You can follow our progress on the Trello board.

Building open collaborations for the sustainable support of proprietary file formats

The challenges and cost associated with the development and maintenance of software for reading images stored in proprietary file formats (PFFs) have been discussed at length in previous blog posts 1, 2. One approach to help address these issues is the development of community collaborations that provide sustainable solutions to PFF support in Bio-Formats.

In this blog post, we want to describe two successful examples of partnerships established with well-recognized commercial entities, Carl Zeiss Microscopy GmbH and Intelligent Imaging Innovations (“3i”) and how this will result in more open, reusable code and better tools for the imaging community.

ZEISS: Open support for JPEG-XR compression

Some image acquisition systems built by Carl Zeiss Microscopy GmbH store binary image data using JPEG-XR compression. Open-source efforts to decode data stored using this technology have to date been unsuccessful. As we have noted previously, the complexity of providing a pure Java implementation of this compression scheme was simply too high to be fully assumed by a non-profit, grant-funded organization like OME.

In response, the user community has raised their concerns to ZEISS and a partnership has been established with Glencoe Software to add a Java-based JPEG-XR decoder to Bio-Formats. Thanks to extensive, fruitful discussions with ZEISS all outputs of this partnership are publicly available and all source code licensed identically to other OME projects. Some public examples of this ongoing work are:

In addition to being a powerful example of how commercial partners can work together, its philosophical implications should not be understated. In particular, the openness of the result demonstrates a cultural shift in the commercial instrumentation community towards the appreciation of community-based open-source projects.

3i: Maintaining a native reader

3i designs and manufactures systems for biological imaging, powered by their SlideBook software. Maintaining Bio-Formats support alongside the fast development of the SlideBook format and their multiple versions has also proven to be challenging. Over the last five years, developers in the 3i team have been building contact with the Bio-Formats development team to implement a native solution to read SlideBook image formats.

A first version of a native Slidebook reader for 32-bit Windows was integrated and released as part of Bio-Formats 5.1.2. Following discussions with members of the 3i team who attended the 2016 OME Users Meeting, this reader has been separated from the Bio-Formats source code and is completely maintained by the 3i team. As of Bio-Formats 5.2.0, the 3i reader became fully pluggable into Bio-Formats e.g. via ImageJ/Fiji. The 3i library is the first Bio-Formats reader where the development, quality control and release processes are fully managed by a third-party.

Support has also been added for Linux and Mac OS X platforms with 32 and 64 bit architectures. Integration into a platform like OMERO is an obvious next step.

What’s next

Collaboration and integration with different organisations undoubtedly comes at some cost: there are differences in objectives, priorities, process and culture to reconcile. However, our experience in these partnerships has been uniformly positive and the collaborations have been efficient and productive. We believe one under-appreciated aspect of open-source development is the ability to adapt to different scenarios and requirements, in order to achieve outcomes that benefit the whole community. We look forward to growing these kinds of collaborations for the community’s benefit.

OMEROImageChooser

This is the first of what will hopefully be a series of posts about tools developed by members of the community. Ian Munro of Imperial writes:

Over the course of adding OMERO capabilities to a number of pre-existing software tools, I felt that there was a need for a lightweight graphical tool that would make it possible to easily add the capability of allowing users to select images from OMERO, to desktop tools.

What I needed was an OMERO equivalent of JFileChooser or Matlab’s uigetfile or Qt’s QFilelDialog.

OMEROImageChooser is the result.

It is a graphical tool which allows a variety of OMERO objects (Images, Datasets, Plates, Attachments) to be selected with a look and feel modelled on OMERO.insight. It can easily be called from Java or Matlab code and is presented as a quick and relatively easy path to either developing a new OMERO desktop client or adding an OMERO interface to existing code.

The source is available from https://github.com/imperial-photonics/omeUiUtils.

A ready-to-use .jar file compatible with OMERO 5.2.x can be also downloaded from https://bintray.com/imperial-photonics/omeUiUtils/omeUiUtils/.

For instructions on how to integrate with Maven/Gradle, refer to the Bintray user documentation.

There are a number of options available, depending what feedback is needed from the user. The screenshots below illustrate some of these.

  1. Selecting one or more images: selecting images in the UI

  2. Selecting an image or attachment: selecting images or attachments in the UI

  3. Selecting a plate: selecting a plate in the UI

  4. Selecting a Dataset and file name for importing files to OMERO - e.g for use from acquisition tools: selecting a dataset in the UI