05 Jul 2016
The recent EU referendum will undoubtedly have implications for the future of
UK science. It now seems equally clear that those implications will remain
unknown for some time to come. No matter what happens next, we want to
emphasize our ongoing commitment to the OME project and to our community.
It is unlikely that our current funding will be altered by the outcome of the
referendum or by the apparent upcoming changes to the political landscape. The
major awards that support OME’s work–from the Wellcome Trust and BBSRC–are
from UK-based institutions. We also participate in several EU projects–
Euro-BioImaging,
Global BioImaging,
MULTIMOT, and
CORBEL–that will
expire within the two-year window being discussed in the media as the minimum
time needed to negotiate a UK departure from the EU. Our understanding is that
our participation in these projects is not in question. (We also have newly
awarded funding to announce–once we have the final documentation!)
Furthermore, our recent Users Meeting in Dundee
once again confirmed strong support from our community and the critical need
to develop the tools that will enable the discoveries from next generation
imaging technologies that are now appearing in the community’s laboratories.
None of our immediate milestones will be modified, and we will keep delivering
Bio-Formats,
OMERO, and the
Image Data Resource just as our users
expect. We will continue to explore new avenues for funding, based on our
community’s need and validation of our software, just as we have done since
2002.
Since the project’s origins, our identity and values have been rooted in
openness, and international, cross-cultural collaboration. OME’s achievements
have always been made possible by the combined talents of an international
team, dedicated to open source software, open development practice, and open
engagement with our broad, diverse, and dedicated user community. Today, the
members of our development team represent more than ten different nations from
both within and outside the EU. Many of us are “immigrants”–we’ve actively
chosen to live in new countries with our families to deliver on the promises
of interoperability and scale in scientific image informatics. We are a
dedicated, multi-national, multi-lingual team that builds and delivers tools
that support the global bioimaging community. We think diversity is one of our
strengths, as it ensures a range of viewpoints and ideas. We therefore
reaffirm our support for inclusion and diversity, and strongly object to the
politics of fear and racism that have characterised some of the recent
political discussions, referenda and elections. We are an example of the best
aspects of the modern, interconnected world that unites dedicated, talented
individuals, regardless of race, religion, gender, sexual identity or national
origin, with a mission to drive scientific insight and discovery for the
benefit of all humankind.
20 Jun 2016
The OME Data Model defines a
Shape.Transform
element with an
AffineTransform
type that is used to transform a Shape’s position in the image in
various ways: scaling, rotations and more. These transformations are
performed within the two dimensions of the image’s x, y plane. In
order to properly locate a Region of Interest (ROI) on an image any
geometric transformations of its Shapes must be taken into account.
The OME Data Model’s AffineTransform defines the elements of a
transformation matrix,
where the attributes’ geometric meaning is,
- A00
- scaling of x
- A10
- shearing of y
- A01
- shearing of x
- A11
- scaling of y
- A02
- translation of x
- A12
- translation of y
The accompanying graph shows an example translation of in x and in y. The
above representation of affine transformations reflects a common
approach in computational geometry and works well.
In OMERO 5.2 the
Shape.transform
property uses a single string to represent transformations. Historically
a variety of formats have been used for this string, most from the World
Wide Web Consortium’s Scalable Vector
Graphics
(SVG), anything from "none" to "rotate (12.000 262.000 174.000)".
Unfortunately OMERO
clients
must thus bear the burden of parsing the possible string formats simply
to be able to determine a ROI’s position on the image. From the example
illustrated in the previous section, given a Point at (40, 85) with
transform "translate(35 -65)", that Point should be understood to be
located at (75, 20) in the image.
The database upgrade script from OMERO 5.2 to 5.3 will include a parser
written in
PL/pgSQL that
moves these string-based Shape.transform property values of existing
ROIs to a new OMERO model object, AffineTransform, whose properties
mirror the numeric attributes of the corresponding OME-XML element.
From OMERO 5.3 a Shape that does have an associated transform thus has
the named matrix elements A00, A10, A01, A11, A02, A12
conveniently available to OMERO clients as floating-point numbers. This
makes it far easier to write clients that properly take account of how a
ROI’s Shapes have been transformed.
Adapting existing code
The OME Data Model and Bio-Formats remain unchanged in regard to affine
transformations. However, the above constitutes a breaking change for
OMERO users. We do not make such changes lightly.
It is highly likely that existing client code that works with OMERO
5.2’s SVG-based Shape.transform property strings already translates them
to fit the properties of the new OMERO AffineTransform model object.
Such code may parse the "matrix(a00, a10, a01, a11, a02, a12)" SVG
format or use transformation matrices in its actual calculation or
display. For example, in adapting some of our Java code for OMERO 5.3 we
found that the AffineTransform object’s properties directly correspond
to properties of the
AWT’s
AffineTransform
Java class that we use already. We are therefore optimistic that, while
some ROI-related code does have to be adjusted, it will be a relatively
easy process that leaves the code clearer and simpler.
15 Jun 2016
For those of you who couldn’t make it to Dundee for OME 2016, or maybe just
didn’t get to all the sessions you wanted to, we have a range of content on
our downloads site–notes, slides and even movie versions of workshop
presentations–available to browse from
downloads.openmicroscopy.org/presentations/2016/Users-Meeting/ or
linked via the programme page if there is specific content you’re interested in.
One of the things the user meeting is really valuable for is hearing about
what the rest of the community is doing with our tools so the
lightning talks
have been a really valuable edition to the format over the past two years.
Together with the formal talks,
this year’s examples illustrate people doing everything from archiving
to data visualization to image processing, from the scale of individual
institutions to international research collaborations.
It’s always great to hear positive stories from the community rather than only
getting feedback when something needs fixing. To this end, we’d always invite
you to make use of the
mailing lists
and forums to discuss the
challenges you’re working on even if you don’t necessarily need help. We’d
also be keen to feature your stories on this blog if you’d like to submit
something longer e.g. with screenshots - you can open a PR at
github.com/ome/blog/ or just drop us
an email and we’ll get back to you for details. Did you know we maintain a
list of publications using our tools on
www.openmicroscopy.org/site/about/publications? We are always happy to hear of new citations
to add.
An even newer addition this year was the Unconference sessions on the third
day. Again, we’ve put some notes from the session up on our
downloads site.
The topics covered in the
Technical Developer discussions include data pipelines,
federation and scaling, while the
Admining OMERO section
is more about managing data and users, having grown out of a session aimed at
facility managers.
The File Formats discussion
provides a good example of how we carry issues raised forward. The
Unconference session on Reader integration and decentralisation prompted the
creation of a GitHub Design issue where the
interested members of the community can follow the options we’re considering
and make comments.
If you haven’t noticed our Design repo before, you can find
Issues relating to a whole
range of design scoping for the UI, OMERO.web viewer, Folders etc. You do need
a GitHub account to comment but signing up only requires a valid email
address. Similarly, we have a selection of public Trello boards at
trello.com/ome to help you track what’s
going on with the project, and receive notifications and give feedback if you
sign up for free.
We’ve had some great feedback from the meeting attendees and would love to
carry forward more positive engagement and to help empower our user community
to help each other. We’ll be looking at ways to try to promote the activities
of our community better going forward and in the meantime, please keep in
touch!
23 May 2016
For several months the OME team has been working on what will soon be
released as the new 2016 OME Data Model. The OME Data Model specifies
Regions of Interest (ROIs) in terms of a set of Shapes. As OMERO 5.3
will use the new Data Model, the upcoming changes include an initial
round of adjustments that improve consistency between Shape
representations in the Data Model and OMERO. They will each also include
a significant addition: Folders, a new top-level model object.
Using ROI Folders
Our initial focus is on supporting ROI-based Folder workflows.
OMERO.insight, OMERO.web and OMERO.cli will offer some support for users
to have their images’ ROIs organized into a hierarchy of folders. We
have several use cases in mind: for example, one may wish to sort cells
into phenotypes or assign ontology terms to them, or an analysis script
may track entities across a set of images and use a folder for each
entity to store sightings of it. We expect the community to think of
many more uses for folders.
Folders may seem rather like Datasets in their implementation: in our
current draft of the Data Model one may name Folders, add a description,
even annotate them, exactly as with datasets. The most obvious
difference from Datasets is what Folders may contain: ROIs, but also
other folders, even a mix of both. Folders may be nested arbitrarily
deeply but with a caveat: a Folder may have only one parent. The same
Dataset can be put into many projects but the Folder hierarchy is a
strict tree. This simplifies the user interface and speeds up
processing.
Graphical clients
An ROI may be both on an image and in some folders. When a scientist
views the ROIs of an image in OMERO.insight or OMERO.web they will be
able to see how those ROIs are organized into folders. Work is ongoing
in both clients to deliver at least some ability to work with ROIs and
folder hierarchies in OMERO 5.3.0. On
Twitter,
YouTube
and
elsewhere we have
already published some screenshots of how OMERO.insight can be used to
organize ROIs into folders and we are working on usability features such
as making images searchable by the name of a folder that contains any of
the image’s ROIs.
OMERO.server and scripts
Users of the Blitz
API
are well aware that it allows a wider range of actions than the
graphical clients support. Those exploring how Folders are represented
will see that a Folder has a parentFolder property for its parent, if
any, and childFolders and roiLinks properties for its contents, as
they might expect.
In our current draft of the new OME Data Model folders may directly
contain images regardless of ROIs. One may expect the graphical clients
to ignore this experimental feature, relying instead on folder-image
linkage via ROIs as above, but the additional imageLinks property of
Folder objects may be useful for grouping images in a different way
within a hierarchy.
One concern with writing scripts may be that processing an arbitrarily
deep Folder hierarchy could require many separate calls to the IQuery
query service. OMERO 5.3.0 will include two new Request subclasses for
queries, FindParents and FindChildren, that can be used to traverse
hierarchies arbitrarily far in one call: for instance, to ask which
images are contained in a set of Folders or in any of their subfolders.
The future of Folders
We have considered proposals for containing other objects, like plates,
wells and annotations, in Folders. One can imagine that someday Folders
may supersede everything from tag sets to datasets but, in starting out
by allowing only a couple of kinds of content, and then only in a strict
tree of Folders, we take a conservative approach that allows us to
adjust our plans based on how the community reacts to the introduction
of Folders in OMERO. We welcome your input on how you see Folders being
used in your workflows.
22 Mar 2016
The OME team has always been committed to building specifications and software
that are cross-platform solutions and that support as many different types of
users as possible, including those with limited IT support/budget. In keeping
with our open source ethos and the fact we rely on public and charity grant
funding, we also try to use open source tools as part of our workflows for
testing, building and deploying our products wherever possible. We do support
commercial operating systems and platforms — we build and test OMERO.insight
on Windows, actively work to support accessing the OMERO and Bio-Formats APIs
in Matlab, and actively support browsing OMERO.web using IE.
Since the beginning of the OMERO project, we have actively tested and
supported builds and tests of OMERO.server on Windows. Several users—
students, faculty and institutions—have highlighted the importance of this
support over many years. Therefore we are frustrated and saddened to
announce that we have to withdraw support for OMERO.server on Windows starting
with the 5.3.0 release. This means OMERO.web hosting will not be possible on
Windows either. We will of course still support running OMERO.insight on
Windows, OMERO.web browsing on IE and continue to provide full support for
Bio-Formats on Windows (including the C++ components of this project). The
reasons for this decision are outlined below.
Ever-increasing technical challenges
Our Continuous Integration (CI) system uses Travis
and allows the OME Consortium’s work to be built and tested on a per-commit
basis. One of the challenges of running OME’s CI system is including tests for
the numerous products we release, across several operating systems with
different configurations e.g. Python 2.7, Openjdk 1.8, Ice 3.5. The testing
matrix is constantly growing; already we are adding Ice 3.6 and soon Java 1.9
will need to be on our radar too. The resources we have for building and
maintaining our CI system aren’t infinite. We have to balance these resources
with core development work. There’s always a tension between rapid development
of new functionality and robust, reliable testing.
The focus of our OMERO 5.2.1 release was on deployment, following feedback
from system administrators (e.g. this forum thread).
We improved our server installation guides and OMERO.web deployment
documentation, and provided stepwise deployment scripts, e.g. for CentOS 6
with
Python 2.7. We extensively used Docker to test our
Linux installation scripts and also to test our installation documentation.
All the installation scripts are available (see
omero-install) and run on the CI
system each time a commit is pushed.
During the development phase of OMERO 5.2.1, we dedicated a large amount of
extra resources—from the devops team and in terms of computing hardware—to
test the Windows installation
scripts and improve our installation documentation. This level of effort will
not scale with the introduction of new elements to our testing matrix e.g. Ice
3.6 support. Moreover, we have other critical priorities—public repositories
and improved support for ontologies to name but two—so we are forced to make
difficult decisions.
Our usage statistics indicate that a large majority of production servers are
installed on Ubuntu 14.04 and CentOS 6, and less than 10 % are run on Windows,
so this decision should have relatively limited impact. We are, nonetheless,
an open source project and aim to support our incredibly diverse community and
the needs they have to deploy our software. So, even if only a few
OMERO.server installations run on Windows, we very much want to support them.
We simply can’t afford to do so.
For all these reasons, from version 5.3, we will not be able to support
OMERO.server on Windows. Again, this is a build and testing issue, so if
anyone out there would like to contribute their time, energy, expertise and
compute resources to provide Windows support, we’d welcome them doing so.
Instead, we will
focus on ensuring we provide the best support we can for a range of UNIX-like
systems, continuing the effort to make OMERO.server easier to install,
maintain and manage.
This decision is for OMERO.server support and OMERO.web hosting only; we
will continue to support and test OMERO.insight, OMERO.web browsing and all
aspects of the Bio-Formats project on Windows.
