Big data, a new historian

While openSCADA and Eclipse SCADA already have some simple value archive that can store values for years, including master/slave replication and composite servers, we know that the archive won’t scale beyond the boundaries of one server.

That is why we started thinking of having one “big data” like historian for Eclipse SCADA. We started playing around a little bit with HBase and came up with some pretty nice ideas. Hadoop and HBase as a backend allow us to scale the way we want to.

Since we started by building an extension to our current value archive system, we started the development inside the Eclipse SCADA project, and are still tracking some design thoughts there. However it became obvious very soon that this system would not only work for Eclipse SCADA, but could also be used by other systems as long as we keep the interfaces as open as possible. So hopefully this project will end up as a new Eclipse project beside Eclipse SCADA.

So what is the current state of all this? It is an early development stage! We do have a specific use case in the moment that we want to build and for this it seems to be ready. We also have a lot of ideas in our minds that could be implemented. For the moment our focus is on realizing our use case and once we achieved that, we want to make a first release of the source code.

Here is what already works:

  • Creating value stores and storing data in raw format
  • Compacting raw data to more efficient storage formats
  • Extracting data from the storage using a CSV format
  • Query before and after the query region

Things that we need to do first:

  • We do want to compress doubles depending in their value. Since this will influence the storage format, we do want to make this as soon as possible.
  • Create a build system. Building from the IDE is ok for some time. But not for long.

Next:

  • Create a HTTP based collector framework with buffering
  • Implement an Eclipse SCADA collector module
  • REST API
  • Extend the Native API
  • Lots more…

Peek:

Haystack data exported to Eclipse Birt using the HTTP CSV servlet.

Haystack data exported to Eclipse Birt using the HTTP CSV servlet.

A simple Eclipse RAP based Query UI

A simple Eclipse RAP based Query UI

The output of the CSV export servlet

The output of the CSV export servlet

Posted in News | 2 Comments

Eclipse SCADA MSI downloads

Finally we do have some MSI packages for the windows platform again. In the past we used AdvancedInstaller as our installation tool for the OSTC (openSCADA Admin Client). But this tool is not open source and therefore does not fit into the Eclipse SCADA project. As an alternative we use the Wix Toolset now.

The process of creating MSI files using Wix needs to be performed on some Windows machine, which is the reason why this is no automated process right now. But we will create MSI files for the milestone releases. On the bright side these MSI files will now be signed by the Eclipse Foundation, after bug 426371 has been resolved.

Release32bit64bit
Milestone (0.1.0-M3)esac_0.1.0-M3_x86.msiesac_0.1.0-M3_x64.msi
Posted in News | Leave a comment

DA item trace console

It was there is in the past, a long time ago. A console view that traces changes of a data item, in detail. Now it is back, but this time based on the Eclipse Console framework which solves all the issues the previous version had. It is available in the recent version of the Eclipse SCADA Admin Client.

os_da_console

Posted in News | Leave a comment

Wireshark dissector for NGP

This is something we wanted to do a long time ago. But now we needed it … a wireshark dissector for the Eclipse SCADA NGP protocol stack. You can get the source from github (ctron/eclipse-scada-ngp-dissector).

es_ws_dis1

Due to the complexity of the NGP protocol stack, we were only able to implement to message channel layer of the protocol. Also the streaming compression and SSL/TLS are not supported at the moment. But on the bright side it can handle TCP fragmentation and multiple packets in one TCP packet. Hopefully we can extend it a little bit in the future. For the moment it is a good start.

Posted in News | Leave a comment

Milestone release M2 of Eclipse SCADA

On friday we released M2 of the ongoing development towards 0.1.0. It contains new features and bug fixes. We also tested the whole new release workflow for the second time.

The most interesting new features might be the extensions to modbus and the REST API.

Posted in News | Leave a comment

Milestone Release 0.1.0.M1 of Eclipse SCADA

It was hard work. But we finally have started the release cycle for 0.1.0 with the first milestone build today.

https://hudson.eclipse.org/scada/job/aggregator-release/2/

The documentation of how to perform the release build is here in the Eclipse Wiki.

The “adminclient” (formerly “ostc”) can be downloaded here:

The yum repository is here: http://download.eclipse.org/eclipsescada/repos/milestone/0.1.0/yum/
The debian packages (currently no APT repository): http://download.eclipse.org/eclipsescada/repos/milestone/0.1.0/deb/

The P2 repository is here: http://download.eclipse.org/eclipsescada/updates/milestone/0.1.0/

Posted in News | Leave a comment

First integration builds from download area

After the code base has been committed and nearly everything is in the integration builds we started pushing the artifacts to download.eclipse.org/eclipsescada. The composite P2 repository for integration builds is now at http://download.eclipse.org/eclipsescada/updates/integration/. This is a P2 repository location!

But beware! This is not a release, these are only the most recent integration builds from the build server.

Continue reading

Posted in News | Leave a comment

Major code base is contributed

The majority of the code base of openSCADA is now contributed to the Eclipse Foundation and will by checked by the IP team in the next weeks. This process will take some time since we have a rather larger code base. Fortunately we are in the incubating phase and can take advantage of the parallel IP process, so after an initial check we are already allowed to check in the code. So this means that in a few days the majority of the code base will be in the Eclipse SCADA GIT repositories and will be built on your hudson instance.

The major tasks now will be to restore all the nice deployment options we have with openSCADA, like the MSI installer for OSTC and the configuration system that will generate “ready-to-run” debian and rpm packages. We hope that we can offer an apt and yum repository at Eclipse like we have it now for openSCADA.

For the moment the new version of OSTC (called ESAC) can already be downloaded from the continous build server. However this is not a release, but a work in progress.

So it looks like we can stick to our plan and target a first release of Eclipse SCADA 0.1.0 for the end of the year.

Posted in News | Leave a comment

First builds at Eclipse

Today we got our Hudson instance at eclipse and were able to successfully run the first builds.

The Hudson instance is at: https://hudson.eclipse.org/scada/

Please be aware of the fact that the migration of the code from openSCADA to Eclipse SCADA is still in progress and that these are continuous integration builds and not releases!

Posted in News | Leave a comment

First code reached Eclipse GIT

After the creation review was successfull we went through the project provisioning and filed for our first initial contribution. We decided to start with some basic components from org.openscada.aurora since we do want to use the step for two changes. First of all we want to restructure the repositories a little bit. Our repository structure has some history now and the fancy names and grown structure don’t make it easy for newcommers to join the project, although the current structure served as pretty well. The new plan is located in the Eclipse Wiki at http://wiki.eclipse.org/EclipseSCADA/Plan/RepositoryLayout and will be extended when necessary.

Second we do want to switch to Tycho for building Eclipse SCADA on the side of Eclipse. While we still think that the PDE build has several advances over Tycho, it truly eases building Eclipse SCADA for people that have no experience in the PDE build. We may keep the PDE build alive though.

So if you like to grab some “Eclipse SCADA” code, feel free: http://git.eclipse.org/c/eclipsescada/org.eclipse.scada.utils.git/

But be warned, this is only a small portion of openSCADA. The rest will follow in the next few weeks.

Posted in News | Leave a comment