Announcing ODE 0.3
Today, Opallios is pleased to announce the general availability of ODE 0.3, Opallios Distribution of ELSA. The first version, ODE 0.1, was released two months back on July 14, 2015. Since then our development team has been hard at work to make further improvements to ELSA as we listed out in our first blog. The focus of ODE 0.3 is to add some new functionalities that our customers have asked for in past and improve on the charting experience of ELSA. Besides some bug fixes, the changes in 0.3 can be divided in three main areas,
- Improved Charting and Dashboard User Experience
- New Aggregate Functions
- Integration with Fluentd
ODE 0.3 is available for download as both debian and rpm packages as well as pre-built AWS images. Currently, deb is supported on Ubuntu 12.x and 14.x, and rpm has been successfully tested on RedHat 6.5 and Centos 6.6. For more details on download and installation please refer to ODE’s github site, https://github.com/opallios/ode.
We found NVD3’s svg rendering very attractive, but it’s not very flexible and poorly documented. Due to some of its chart reusability requirements we found it little hard to integrate with the current ELSA DOM based code, which would have required dynamic loading of SVG charts. Chart.js is a canvas based library, which is quite small and highly customizable. It’s very popular among open source projects and have a big user community. With all the pros and cons we finally opted for Chart.js to replace Google Charts in ODE 0.3.
Chart.js produces responsive, animated and great looking charts. Charting plays a big role in ELSA. It’s primarily used for aggregated queries (groupby clause) and dashboard. Using Chart.js we were able to make charts load 2-3 times faster and look cool. We also fixed some of the long table and charting alignment issues for aggregated queries. Here are some snapshots,
On Dashboard, we cleaned up the links for each chart and improved charting layout. The result is cleaner and attractive dashboards. Here is one example,
Chart.js has a strong active community around it, which allows for regular enhancements and releases. We will continue to update ODE with any major changes in Chart.js to further improve user experience.
ODE 0.3 adds five new aggregate functions, which are also known as transforms in ELSA. Transforms allow you to pass the results of a query to a backend plugin. These functions serve as backbone of ELSA analytics. The plugins that currently ship with ELSA include whois, dnsdb, and CIF (Collective Intelligence Framework). There are also utility transforms filter, grep, and sum. There have requests from users to add new plugins for some of the basic aggregate functions. ODE 0.3 adds four new such transforms,
- min – finds the field’s minimum value for the given result of the subquery. For ex., “class:xxx | min(eventid)”
- max – finds the field’s maximum value for the given result of the subquery. For ex., “class:xxx | max(eventid)”
- avg – finds the field’s average value for the given result of the subquery. For ex., “class:xxx | avg(bytes)”
- median – finds the field’s median value for the given result of the subquery. For ex., “class:xxx | median(bytes)”
Few key things to note,
- As ELSA queries return only 100 records by default, you would have to set limit directive to apply the aggregation over more than 100 records. For ex., “class:xxx nobatch:1 timeout:0 limit:0 | max(eventid)”
- You can pipe multiple transforms within the same query
- The field passed to the above aggregated functions have to be a number type
ELSA was created as a centralized syslog server and such has syslog-ng at its heart. With syslog-ng you can collect logs from any source (both structured and unstructured) and process them in near real-time. Syslog-ng’s powerful db-parser() allows to extract information and tag messages for later classification. Even though, db-parser() is a generic and supports various log formats, the parsing language for it can become quite complex depending on the log format. In one of our project we were required to process large number of nested json messages. After quite a bit of trial with syslog-ng json parser we were still not able to effectively create parsing for our needs. We ended up writing json parser in java and pass its output through syslog-ng for ELSA’s consumption.
Though, syslog-ng is pretty reliable most of the times, but there are cases when it’s parser may come up little short. In comes Fluentd, a simple, but extensible log collector that has over 100 plugins to parse different log formats. It’s widely used in industry and has a strong active community behind it. We think Fluentd can work well with syslog-ng in ODE.
By integrating fluentd we open up ODE to consume a wide range of data sources. With ODE 0.3 we have created fluentd configuration for the following log types,
- Apache Log
User can easily add more fluentd plugins as needed. The current configuration allows for both stream and file based inputs. Fluentd is currently disabled by default in ODE 0.3 and requires some manual configuration to enable it. The messages that are chosen to be processed by fluentd would first enter fluentd and then processed by syslog-ng to be consumed by ELSA. Fluentd would transform the messages in the form that’s easily parsable by syslog-ng.
The fluentd installation and configuration for ODE is documented in more details on the ODE site, http://opalliosode.org/documentation/fluentd/.
Opallios is committed to continue make improvements to ELSA. With our first release, ODE 0.1, we made the installation process smoother and now, with ODE 0.3 we are trying to improve on the user experience and add new functionalities. In coming months, we’ll continue to work on some of the usability features and add new functions as desired by our customers. Here are some of the items we’re currently looking at for our next release, ODE 0.5,
- Support ODE installation on newer OS versions
- Speed up the installation process
- More analytical functions
- Better documentation with use cases
- Tighter integration with Fluentd
- Bug fixes
Our users can dictate the future roadmap of ODE by getting involved in the product development and requesting new features. Drop us a line so we can prioritize our ODE work list based on what’s important to our users. We appreciate your feedback and comments, http://opalliosode.org/feature-request/.
Opallios provides software consulting for big data analytics and cloud computing. Over last five years it has helped several organizations simplify their big data projects and develop cloud based softwares. Opallios business model is that of building long term partnerships to help our clients lower their software development and maintenance cost by offering our expertise customized to client’s’ needs and requirements.