24 Oct 2017
The year so far…
2017 has been a very busy year for the OME team so far with:
- 2 major OMERO releases plus 6 security or patch releases - introducing
ROI Folders, a whole new UI layout for HCS data, a configurable restricted
admin user role and many other fixes and improvements
- 12 Bio-Formats releases to date (and another in the pipeline) -
featuring 2 new formats and improved support for many more
- 4 OME Files C++ releases - improving support for reading and writing
OME-TIFF in an open, liberally-licensed, native library
- 4 OMERO.figure releases - adding support for loading ROIs and choosing
look-up tables from OMERO, using markdown syntax for italics and bold
labels, setting background colors, plus other fixes and improvements
- various other web app releases - all now available from PyPI,
including OMERO.FPBioimage, a volumetric visualization tool
- the launch of our new OMERO.iviewer - a web browser-based interactive
multidimensional image viewing app that includes ROI drawing and measurement
functionality. OMERO.iviewer is approaching a full v1.0 release, a sign of
our commitment to maintaining and growing the functionality in this app
- AND a new website!
Plus of course, our usual Annual User Meeting bringing together members of the
community from across the globe (if you couldn’t make it, talks and workshops
are available from the event page).
The Image Data Resource (IDR) also continues
to go from strength to strength. If you missed it, our
Nature Methods paper (or the open
access PubMedCentral version)
discusses how the IDR can be used to obtain new biological insights from
existing datasets, plus an in-depth explanation of how the resource is set up.
It also includes information on how you can set up your own IDR.
With all that, hopefully you’ll forgive us for neglecting this blog!
What’s still to come…
We’re running two days of Bioinformatics training at Cambridge University
in December. In the run up to this, we’ll be developing new training materials
which will all be available via our website for those of you who can’t attend.
We’ll also be expanding our collection of Jupyter notebooks providing you with
examples of how to carry out image analysis via the OMERO API and likely
adding to our collection of how-to movies on
our YouTube channel.
The IDR will also be updated, publishing several new datasets (our first light
sheet fluorescence microscopy dataset is almost ready to go!) and improvements
to the Jupyter analysis tools.
In terms of releases, we expect to put out patch releases for OMERO,
Bio-Formats and OME Files. As always the content of these will be driven by
both the community and our own projects, for example the IDR continues to
challenge us in terms of format support and display, and the way our tools
connect and interact with analysis packages. You can always keep up with the
latest upcoming releases and more via our
Trello boards (you can
sign up here with just an email address).
Beyond this, we are continuing to push forward our functionality and the scale
at which our tools operate. As imaging technology takes huge leaps in scale,
this is challenging to say the least and we’ll need the support of the
community more than ever. If you have resources to contribute, we’d love you
to get involved - write to the mailing lists or forums,
check out our contributing developer docs,
read our blog posts on helping us support new file formats. Even if you
don’t have resources to spare, you can always help us secure grant funding by
citing our work in your
publications.
21 Dec 2016
2016 has been a busy year for OME. Many of you will have noticed the number of
Bio-Formats releases and the fact
that OME Files C++
is now available for implementing the
OME Data Model and
OME-TIFF
support in C++ software (you can read our preprint here).
That doesn’t mean we haven’t been busy on the OMERO front though, we’ve pushed
our deadline to make sure we can deliver several new features:
Data Management
As well as implementing the new Data Model,
we have introduced the Folders feature discussed in a
previous post.
Folders will allow you to organize your images’ ROIs into a hierarchical
structure so that, for example, you can sort cells into phenotypes or assign
ontology terms to them, or use an analysis script to track entities across a
set of images and then use a folder for each entity to store sightings of it
in OMERO.insight.
UI development
We’ve been working on our metadata capturing and display, drawing lessons from
our IDR project. We’ve also revamped the
display of Screen-Plate-Well data to allow browsing of all fields within a
well and display of their positions within each well. Zooming of the plate and
fields are both supported. It is also possible to add and view annotations on
individual wells (and not just the Images they contain).
Permissions improvements
To aid the workflows of facilities managers, we are designing a new ‘Light
Admin’ role to the OMERO permissions system. This will allow a manager to
import data for any other user (i.e. it will belong to the other user), delete
other users’ data (for clean up e.g. after someone has left the lab), and
manage user groups (creating groups and adding existing users to them) without
having the full rights and responsibilities of a full Administrator. If you
are interested in all the technical details, there is a
design issue on GitHub.
Reading data
OMERO 5.3.0 will bundle Bio-Formats 5.3.x. This will include,
amongst other changes, support for JPEG-XR compressed CZI data.
This work was funded by a partnership between Glencoe Software and ZEISS.
See announcement
Additionally, OMERO 5.3.x will benefit from the improvements in Bio-Formats 5.3.x
for tile-base image writing for TIFF and derived formats like OME-TIFF.
For more information about the plan for Bio-Formats 5.3, see Bio-Formats Development Status
Image rendering
We’ve introduced support for Lookup Tables and reverse intensity and improved
the rendering engine to allow full projection thumbnails.
We’ve also added histograms of image pixel intensities in the OMERO.insight
and OMERO.web clients.
OMERO.web architecture
A big effort has gone into making OMERO.web deployable as a separate component
from the server and the various web apps installable from Python Package Index
(PyPI). This has involved reorganizing all the packages. OMERO.web now has an
‘Open With’ function for using custom viewers and our next generation web
image viewer (OMERO.iviewer) is coming
on too. There has also been a lot of work behind the scenes to improve the Web
API.
In line with this, our future client development will be very much focused on
web clients and OMERO.insight will enter maintenance mode once the updates for
5.3.0 are complete.
Developer previews
If you’re a developer with your own OMERO code, you can check out the work so
far to see if it’ll affect you. There are further details in the
milestone version history
and as ever, the code is all on GitHub (our latest development milestone is here).
Heads up for sysadmins
We’re changing our recommended way to install OMERO.web so you can deploy it
separately to the main OMERO.server, making it easier to get your your virtual
environment set up with all the necessary prerequisites. We have also
deprecated support for Apache and recommend you deploy using Nginx as we are
likely to drop Apache completely during the 5.3.x development line.
We are moving to Ice 3.6 as the minimum supported version. Other updates to
version requirements include Python 2.7 but we will continue to support Java
1.7 for OMERO 5.3.x, as we are aware that upgrading to 1.8 would be
incompatible with current MATLAB distributions.
With the requirements changes mentioned above, we are also considering no longer
deploying and building OMERO on CentOS 6 to reduce our testing matrix.
It will still be possible to use OMERO on CentOS 6 with the minimum
requirements.
See OMERO.server installation on CentOS 6 with Python 2.7 and Ice 3.6
Note that we are in the process of updating the documentation so although a
preview of the code changes is publicly available, you should not expect all
the milestone documentation to reflect these changes before the final release.
Release schedule
As a consequence of all this work, OMERO 5.3.0 is still under development and
is now not expected until the first quarter of 2017.
05 Dec 2016
This blog is an update about the use of
C++11 and
C++14 (“modern C++”) within
OME Files C++.
We have been evaluating the switch from C++98 to C++11 for over 18
months. Up until now, we have remained using C++98 in order to
continue to support older systems with compilers which do not support
C++11. However, circumstances have recently changed which will
necessitate a switch at some point in the near future.
Most current platforms support C++11 and C++14, with some exceptions
(workarounds are noted)
| Platform |
Compiler |
C++11 |
C++14 |
| CentOS 6 |
GCC 4.6 |
Unsupported by default* |
Unsupported by default* |
| CentOS 7 |
GCC 4.8 |
Mostly supported |
Partially supported |
| FreeBSD 10 |
Clang/LLVM 3.4 |
Fully supported |
Fully supported |
| FreeBSD 11 |
Clang/LLVM 3.8 |
Fully supported |
Fully supported |
| MacOS X 10.10 |
Apple LLVM 7.3 |
Fully supported |
Fully supported |
| MacOS X 10.11 |
Apple LLVM 8.0 |
Fully supported |
Fully supported |
| MacOS X 10.12 |
Apple LLVM 8.0 |
Fully supported |
Fully supported |
| Ubuntu 14.04 |
GCC 4.8 |
Mostly supported |
Partially supported |
| Ubuntu 16.04 |
GCC 5.3 |
Fully supported |
Fully supported† |
| Visual Studio 2013 |
VC12‡ |
Partially supported |
Mostly unsupported |
| Visual Studio 2015 |
VC13‡ |
Supported |
Partially supported |
| Visual Studio 2017 |
VC15‡ |
Supported |
Supported |
* The software collections
devtoolset-4
(GCC 5.2) and
devtoolset-3
(GCC 4.9) packages provide updated compilers which can build OME
Files C++ using C++11
† Boost 1.58 provided by this release does not build OME Files with
C++14, but does build with C++11; see trac
ticket
‡ Microsoft Visual C++ lags significantly behind GCC and Clang/LLVM,
but VS2013 contains a useful subset of features, which are improved
upon significantly with VS2015 and later
The support classifications above are broad; please see the following
references for full feature coverage:
Third-party dependencies
All of our third-party dependencies have supported building with a
C++11 or C++14 compiler for some time. However, more recently some of
our dependencies have begun to require it, or will require it in
their next major release.
For many users, these libraries are provided by package repositories,
including Linux distributions’ package managers, MacOS X homebrew and
FreeBSD ports. As these package repositories begin to provide the
updated C++11 versions of these libraries, OME Files C++ will no
longer build with them. While some of these C++11 libraries are
optional (Qt5), others such as ICU are required by Boost and
Xerces-C++ for full Unicode support, which is needed by the OME-XML
data model. As a result, C++11 support will become a necessity for
continued use of these libraries as the versions requiring C++11 enter
mainstream use.
The benefits of C++11
C++11 adds a number of useful features to the language, including:
auto typedecltype keyworddefaulted and deleted functions- delegating constructors
- forward-declared
enums
- initialiser lists
- lambdas
nullptr keywordoverride and final qualifiers- range-based
for loops
- rvalue references
static_assert- strongly-typed
enums
- variadic templates
and library features, including:
std::arraystd::shared_ptrstd::threadstd::tuplestd::unique_ptr
These features are all supported by the compilers on the above
platforms. Some additional features (not included here) are not yet
supported by the compilers on every platform, and so can’t yet be
used.
These features benefit the project in several ways:
- they reduce the code size dramatically by permitting much more
concise code with reduced boilerplate (by around 50% in one test
conversion of a single component)
- they allow the use of standard library features in place of Boost
equivalents, which aids interoperability (e.g.
std::thread and
std::shared_ptr in place of boost::thread and
boost::shared_ptr)
- they allow improved performance of standard library containers
through the use of rvalue references, and also allow the use of new
features such as
std::unique_ptr to replace boost::shared_ptr in
certain situations, again improving performance
- they make the library easier to use, for example the use of
initialiser lists can reduce the
PixelBuffer nD array addressing
from multiple lines to a single line of code
- they make the library more understandable and more familiar for
users already using these features with other libraries; in
particular features such as
std::shared_ptr have been available
since VS2010 for Windows developers and their use is already
widespread
Proposal
Given that:
- all of our supported platforms can support a sizeable subset of
C++11
- our dependencies will increasingly require C++11
- there are significant benefits to switching to C++11 as a result of
the features and performance gains it brings
we plan to make a C++11 compiler a requirement in our upcoming 0.3.0
release of OME Files C++, and begin using a subset of the C++11
features available across our supported platforms. For most users,
the transition should be entirely transparent and will require no
immediate code changes as we will leave compatibility typedefs in
place through a transitional period.
If this will present any problems, please get in touch through our
mailing
lists or
forums so that we can
discuss any concerns you may have. Likewise, if the change is
something which you will find useful and beneficial, we would also
like to hear from you.
01 Nov 2016
This blog is an update about what we are working on in the OME Files and
Bio-Formats codebase for the next few months.
Following our Data Model post,
we spent the first semester of 2016 working on a new version of the OME Data
Model which was released in June 2016.
This schema release was then followed by two releases in August 2016:
- the release of our reference implementation for the OME file formats,
OME Files C++ 0.2.0
with full support for all versions of the OME Data Model
- the release of Bio-Formats 5.2.0
including upgrade to the 2016-06 model, API additions and bug fixes.
Since August, a series of patch releases have been delivered both for OME
Files C++ 0.2.x and Bio-Formats 5.2.x. These releases extended the
C++ reference implementation to support all modalities of OME-TIFF allowed by
our format specification, and also provided a series of bug fixes for the
community following a major release.
In addition, we also performed a minor release
of the OME Data Model extending the AcquisitionMode enumeration
to support new values.
Since the beginning of October 2016, our focus has switched to the development
of Bio-Formats 5.3.0. This release includes two major parts: a re-architecture
of the components and new backwards-compatible API additions.
As part of the development of OME Files, a large part of the C++ code
constituting the reference implementation has been migrated out of the
Bio-Formats repository into its own set of components.
In Bio-Formats 5.3.0, we will be applying a similar re-architecture effort for
our Java codebase and the OME Data Model. While Bio-Formats has been
historically a large multi-module project, the idea here is to facilitate
development and shorten build time, provide a better separation of the
functional components and allow faster releases of the low-level components.
In particular the following components have been filtered out to their own
repository and will be managed separately:
All decoupled components have been released and uploaded to the
Maven Central repository under the
org.openmicroscopy groupId.
Bio-Formats 5.3.0 will also include API additions. In order to minimize the
impact of these changes, the codebase is migrating towards
semantic versioning. This means that all the API added
in 5.3.0 will be fully backwards-compatible and should require no changes for
consumers using Bio-Formats 5.2.0. Two main parts of the API will be reviewed
and extended in 5.3.0:
Timeline
We hope to release Bio-Formats version 5.3.0 at the beginning of December 2016.
You can follow our progress on the Trello board.
31 Aug 2016
The challenges and cost associated with the development and maintenance of
software for reading images stored in proprietary file formats (PFFs) have
been discussed at length in previous blog posts
1,
2.
One approach to help address these issues is the development of community
collaborations that provide sustainable solutions to PFF support in
Bio-Formats.
In this blog post, we want to describe two successful examples of partnerships
established with well-recognized commercial entities,
Carl Zeiss Microscopy GmbH and
Intelligent Imaging Innovations (“3i”)
and how this will result in more open, reusable code and better tools for the
imaging community.
ZEISS: Open support for JPEG-XR compression
Some image acquisition systems built by Carl Zeiss Microscopy GmbH store
binary image data using JPEG-XR compression.
Open-source efforts to decode data stored using this technology have to date
been unsuccessful. As we have noted previously, the complexity of providing a
pure Java implementation of this compression scheme was simply too high to be
fully assumed by a non-profit, grant-funded organization like OME.
In response, the user community has raised their concerns to ZEISS and a
partnership has been established with
Glencoe Software
to add a Java-based JPEG-XR decoder to Bio-Formats. Thanks to extensive,
fruitful discussions with ZEISS all outputs of this partnership are publicly
available and all source code licensed identically to other OME projects. Some
public examples of this ongoing work are:
In addition to being a powerful example of how commercial partners can work
together, its philosophical implications should not be understated. In
particular, the openness of the result demonstrates a cultural shift in the
commercial instrumentation community towards the appreciation of
community-based open-source projects.
3i: Maintaining a native reader
3i designs and manufactures systems for biological imaging, powered by their
SlideBook software. Maintaining Bio-Formats support alongside the fast
development of the SlideBook format and their multiple versions has also
proven to be challenging. Over the last five years, developers in the 3i team
have been building contact with the Bio-Formats development team to implement
a native solution to read SlideBook image formats.
A first version of a native Slidebook reader for 32-bit Windows was integrated
and released as part of
Bio-Formats 5.1.2.
Following discussions with members of the 3i team who attended the 2016 OME
Users Meeting, this reader has been separated from the Bio-Formats source code
and is completely maintained by the 3i team. As of Bio-Formats 5.2.0,
the 3i reader became fully pluggable into Bio-Formats e.g. via ImageJ/Fiji.
The 3i library is the first Bio-Formats reader where the development, quality
control and release processes are fully managed by a third-party.
Support has also been added for Linux and Mac OS X platforms with 32 and 64 bit architectures. Integration into a platform like OMERO is an obvious next step.
What’s next
Collaboration and integration with different organisations undoubtedly comes
at some cost: there are differences in objectives, priorities, process and
culture to reconcile. However, our experience in these partnerships has been
uniformly positive and the collaborations have been efficient and productive.
We believe one under-appreciated aspect of open-source development is the
ability to adapt to different scenarios and requirements, in order to achieve
outcomes that benefit the whole community. We look forward to growing these
kinds of collaborations for the community’s benefit.
