This blog post is part of a series introducing 3rd-party open-source technologies
used within the free software solutions created by Nexedi. One of these solutions is NEO, our
distributed, redundant and transactional ZODB storage designed as alternative
and FileStorage. With flexibility and reliability in mind, Nexedi chose MariaDB
as a core component of the NEO system architecture.
Photo: fotolia.fr - pixel
The volume of data handled by Nexedi is growing across the board with
IoT, Big Data and Machine Learning becoming more relevant and projects such as
gaining industry traction. As Nexedi services often range from network
infrastructure to business applications great emphasis is put on
efficient data handling making accessability, availability, distribution and
performance high-priority and research topics.
NEO (short for Nexedi Enterprise Object database
- not to be confused with the neo graph database)
was the result of such a research project intended to replace all implementations of
ZEO (Zope Enterprise Object database) being used at Nexedi and to
switch to an architecture that could be scaled automatically while also providing
persistence and resiliency. During development, SQL and MariaDB proved
to be a good choice because they met our immediate needs while also being flexible
enough to reshape tables as the design of NEO grew. This was key, because frequent
schema changes in early project phases were expected. MariaDB allowed us to focus
on specific features of NEO while not having to call the low-level InnoDB BTree
implementation directly which reduced development time considerably.
The initial challenge for the NEO project was to find the minimal set of things
which could not be distributed while retaining and being able to closely monitor
all the strong transactional consistency guarantees a ZODB implementation had to
provide. In doing so we found that the we could reduce the critical section of
the generic ZODB commit protocol to a shorter version than what ZODB provided
at that time. We determined elements that had to be constant and areas where
persistence was needed with MariaDB and SQL letting us redesign on-the-fly
and eventually settle on a data structure to implement a working prototype.
The resulting NEO cluster features different "personalities": configuration-only
master, client and admin nodes acting besides pure storage nodes with configuration
NEO uses MariaDB to store low level object, transaction and location data and makes
these accessible via the "medium" ZOPE layer which our main software solutions
such as ERP5
are running on. Using MariaDB we could overcome the shortcomings of ZODBs
traditional way of maintaining indices and are now able to easily scale our
application architectures automatically and depending on need. MariaDB also
provides well-test durability, which is especially important for the more
advanced features of NEO, such as storing data in non-chronological order
when lagging nodes are catch up or a node's workload is reduced by moving data
to another node. Finally, since most of our implementations are
running long-term, MariaDB also provides us the needed stability in terms of
source code availability - it's a proven open-source embeddable solution.
As part of NEO, MariaDB was included in all latest ERP5 implementations as well
as Nexedi's internal system and test infrastructure (30 users, 70 test nodes) currently
handling indices of about 97GB (TokuDB) or 133GB (InnoDB backup) which without
NEO compression would equate to a FileStorage equivalent between 250GB and 350GB.
Initially no customizations were made in MariaDB but we have since modified the
underlying storage engines using TokuDB
for its performance and space-efficiency characteristics instead of InnoDB.
For the past couple of years we have increased our efforts to contribute back to
MariaDB, because we believe there should be open-source industry-grade software
solutions available and in terms of database technologies MariaDB is an essential
solution which we want to support into the future.
For us this translates to working with MariaDB in joint research projects, sponsoring
some of the MariaDB developer salaries and helping to improve MariaDB code itself
by reporting and collaborating on issues related to data availability, consistency and
Nexedi is one of the largest open-source Software Publishers in Europe providing
services around their solutions for business (ERP5),
cloud management (SlapOS)
and Big Data analytics (Wendelin).
As creator and maintainer of ERP5, Nexedi has deployed its no-license-cost and
fully customizable ERP and related software solutions in a variety of industries
across the globe ranging from aerospace, apparel and banking to healthcare,
finance and government.