Announcing ODE 0.3 release

Announcing ODE 0.3

 

Today, Opallios is pleased to announce the general availability of ODE 0.3, Opallios Distribution of ELSA. The first version, ODE 0.1, was released two months back on July 14, 2015. Since then our development team has been hard at work to make further improvements to ELSA as we listed out in our first blog. The focus of ODE 0.3 is to add some new functionalities that our customers have asked for in past and improve on the charting experience of ELSA. Besides some bug fixes, the changes in 0.3 can be divided in three main areas,

  1. Improved Charting and Dashboard User Experience
  2. New Aggregate Functions
  3. Integration with Fluentd

ODE 0.3 is available for download as both debian and rpm packages as well as pre-built AWS images. Currently, deb is supported on Ubuntu 12.x and 14.x, and rpm has been successfully tested on RedHat 6.5 and Centos 6.6. For more details on download and installation please refer to ODE’s github site, https://github.com/opallios/ode.

Charting

ELSA, since beginning, has been leveraging Google Charts for visualization. Google Charts, although extensive, lacks basic responsiveness and coolness of some of the more modern javascript based charting libraries. After discussion with Martin, creator of ELSA, we agreed that visualization is one area which will make the most impact in terms of user experience. Out of several popular charting libraries out there we narrowed down our investigation to couple namely, NVD3 and Chart.js, for replacing flash based Google Charts.

We found NVD3’s svg rendering very attractive, but it’s not very flexible and poorly documented. Due to some of its chart reusability requirements we found it little hard to integrate with the current ELSA DOM based code, which would have required dynamic loading of SVG charts. Chart.js is a canvas based library, which is quite small and highly customizable. It’s very popular among open source projects and have a big user community. With all the pros and cons we finally opted for Chart.js to replace Google Charts in ODE 0.3.

Chart.js produces responsive, animated and great looking charts. Charting plays a big role in ELSA. It’s primarily used for aggregated queries (groupby clause) and dashboard.  Using Chart.js we were able to make charts load 2-3 times faster and look cool. We also fixed some of the long table and charting alignment issues for aggregated queries. Here are some snapshots,

ode c1

 

 

 

 

 

 

 

ode c2

On Dashboard, we cleaned up the links for each chart and improved charting layout. The result is cleaner and attractive dashboards. Here is one example,

ode c3

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ode c4

 

 

 

 

 

 

 

Chart.js has a strong active community around it, which allows for regular enhancements and releases. We will continue to update ODE with any major changes in Chart.js to further improve user experience.

Analytical Functions

ODE 0.3 adds five new aggregate functions, which are also known as transforms in ELSA. Transforms allow you to pass the results of a query to a backend plugin. These functions serve as backbone of ELSA analytics. The plugins that currently ship with ELSA include whois, dnsdb, and CIF (Collective Intelligence Framework). There are also utility transforms filter, grep, and sum. There have requests from users to add new plugins for some of the basic aggregate functions. ODE 0.3 adds four new such transforms,

  1. min – finds the field’s minimum value for the given result of the subquery. For ex.,  “class:xxx | min(eventid)”
  2. max – finds the field’s maximum value for the given result of the subquery. For ex., “class:xxx | max(eventid)”
  3. avg – finds the field’s average value for the given result of the subquery. For ex., “class:xxx | avg(bytes)”
  4. median – finds the field’s median value for the given result of the subquery. For ex., “class:xxx | median(bytes)”

 

Few key things to note,

 

  • As ELSA queries return only 100 records by default, you would have to set limit directive to apply the aggregation over more than 100 records. For ex., “class:xxx nobatch:1 timeout:0 limit:0 | max(eventid)”
  • You can pipe multiple transforms within the same query
  • The field passed to the above aggregated functions have to be a number type

FluentD

ELSA was created as a centralized syslog server and such has syslog-ng at its heart. With syslog-ng you can collect logs from any source (both structured and unstructured) and process them in near real-time. Syslog-ng’s powerful db-parser() allows to extract information and tag messages for later classification. Even though, db-parser() is a generic and supports various log formats, the parsing language for it can become quite complex depending on the log format. In one of our project we were required to process large number of nested json messages. After quite a bit of trial with syslog-ng json parser we were still not able to effectively create parsing for our needs. We ended up writing json parser in java and pass its output through syslog-ng for ELSA’s consumption.

Though, syslog-ng is pretty reliable most of the times, but there are cases when it’s parser may come up little short. In comes Fluentd, a simple, but extensible log collector that has over 100 plugins to parse different log formats. It’s widely used in industry and has a strong active community behind it. We think Fluentd can work well with syslog-ng in ODE.

By integrating fluentd we open up ODE to consume a wide range of data sources. With ODE 0.3 we have created fluentd configuration for the following log types,

  1. JSON
  2. NetFlow
  3. Apache Log

User can easily add more fluentd plugins as needed. The current configuration allows for both stream and file based inputs. Fluentd is currently disabled by default in ODE 0.3 and requires some manual configuration to enable it. The messages that are chosen to be processed by fluentd would first enter fluentd and then processed by syslog-ng to be consumed by ELSA. Fluentd would transform the messages in the form that’s easily parsable by syslog-ng.

ode-c6

The fluentd installation and configuration for ODE is documented in more details on the ODE site,  http://opalliosode.org/documentation/fluentd/.

What’s Next?

Opallios is committed to continue make improvements to ELSA. With our first release, ODE 0.1, we made the installation process smoother and now, with ODE 0.3 we are trying to improve on the user experience and add new functionalities. In coming months, we’ll continue to work on some of the usability features and add new functions as desired by our customers. Here are some of the items we’re currently looking at for our next release, ODE 0.5,

  1. Support ODE installation on newer OS versions
  2. Speed up the installation process
  3. More analytical functions
  4. Better documentation with use cases
  5. Tighter integration with Fluentd
  6. Bug fixes

Our users can dictate the future roadmap of ODE by getting involved in the product development and requesting new features. Drop us a line so we can prioritize our ODE work list based on what’s important to our users. We appreciate your feedback and comments, http://opalliosode.org/feature-request/.

About Opallios

Opallios provides software consulting for big data analytics and cloud computing. Over last five years it has helped several organizations simplify their big data projects and develop cloud based softwares. Opallios business model is that of building long term partnerships to help our clients lower their software development and maintenance cost by offering our expertise customized to client’s’ needs and requirements.

 



Introducing ODE

Today, we are announcing the general availability of ODE 0.1. This is the first release of ODE, the Opallios Distribution of ELSA. For people who are not familiar with ELSA, it stands for Enterprise Log Search and Archive. ELSA is an open-source enterprise ready log management system. It leverages syslog-ng for processing incoming syslog data and Sphinx full-text indexing for log searching. ELSA stands out from other well-known log management system in its high performance and scalability. For more details on ELSA refer to its Github site, https://github.com/mcholste/elsa.

Why ODE?

The first version of ELSA was released back in 2010 and since, has gone through various updates and releases. Over time it has gained a loyal user group with active community. We at Opallios first came across ELSA for one of our customer’s project where we needed a high performing syslog data indexer. There was a rigid requirement on scalability and hardware usage. We needed a system that could ingest syslog data at the rate of 10Mb/sec on a single machine and persist data for over 6 months. Of all the open source log management systems we evaluated ELSA came out ahead, both in terms of performance and scalability. After some rigorous testing and use ELSA impressed us a lot, but at the same time we saw some opportunity for improvement and thus, originated the idea of ODE. Till now, we have noticed from our customer requests that there are some minor changes that if included in the open-source edition, would benefit both – our customers and the community. As a result to provide a committed roadmap and delivery schedule for these changes, we decide to fork a branch and initiate ODE (Opallios Distribution of ELSA).

Martin C Holste, the creator of ELSA, has put together an impressive log management system, but, as with any open source project the success of the project depends on its adaption. We thought ELSA had all the necessary ingredients to be one of the leading log management solution, but lacked in some key areas,

  1. Installation Process – Though, the installation is pretty straight forward, but is prone to failures in some cases. There is also not much of updates to support newer version of OS.
  2. Data Sources – ELSA uses syslog-ng for to process input data. syslog-ng allows you to write parsers for varying number of data formats, but there is limited support for various log formats out-of-box.
  3. Analytical Functions – ELSA’s strong data correlation along with dynamic full-text search bodes for impressive analytics, but there is a room for adding more analytical functions that would bring it on par with other leading data analytics engines.
  4. Debugging Tools – ELSA works quite reliably there could be instances when we need to troubleshoot some unseen issues. Improving debugging ability that would reduce TTI (time to investigate) & TTR (time to resolve) would enhance usability of ELSA.
  5. Documentation – ELSA has just enough documentation that helps a technology expert to configure and get started. However the community would benefit with some documentation like a user guide and use case examples explaining different configurations.
  6. Regular Updates – As there are limited number of committers to ELSA, the updates lag sometimes. There are features the community want to see in the future ELSA, with Martin’s approval that we can help bring to the market faster either in main ELSA branch or in ODE.

After discussion with Martin we agreed to fork out ELSA into ODE to address some of these items. ODE will continue to be in sync with ELSA branch for any updates.

ODE 0.1

The goal behind ODE 0.1 release is to make the ELSA installation reliable and consistent with the primary focus on “the out of the box experience”.  As part of ODE 0.1 we provide support for both debian and rpm packages. ODE 0.1 uses the latest ELSA codebase from github. Distributing ODE via standard packages has allowed us to have a better control over the flow of installation and support the basic install, remove and update software features. There are some other slight changes to make the software more reliable, like using Starman in place of Apache http server. Following OSs have been tested and verified with the ODE 0.1 packages,

  1. Debian package – Ubuntu 12.04, Ubuntu 14.04
  2. RPM package – Red Hat 6.6, Centos 6.5

We will also provide AWS images for the above mentioned linux distributions for easier installs and quick ODE evaluation. The ODE 0.1 packages and images are available for download at ODE github site, https://github.com/opallios/ode.

Roadmap

We are a big supporter of ELSA and are committed to assist Martin in taking it to the next level. Our goal is to increase its adaptability by filling in the gaps in its implementation that exist today. In the next few months we will continue to develop on the items we listed earlier in this blog. We hope users will find ELSA much more appealing as we continue to check-off items from our to-do list. The roadmap of ODE will also very much depend on our users’ feedback and their wish-list.

 Summary

ELSA is a very well thought out and implemented log management system, but has limited popularity due to inconsistent installation process, limited support for various log formats, and minimal documentation/support. Opallios would like to see better adoption of ELSA and is ready to invest its resources to drive higher adoption, as it ELSA provides a real low total cost of ownership option in log management space.  The first step in that direction is to improve ELSA installation process, which we are doing with the release of ODE 0.1. Over next few months Opallios will continue to update ODE adding new features and contributing to ELSA’s growth.