6th
QlikView data integration in a Java world with QVX & QVD
After creating a Java lib called QVXConverter to read and write QlikView data eXchange files (QVX) which got integrated in several solutions:
- Hadoop QVX Converter (Incentro HQC)
- Pentaho Kettle QVX Writer Plugin (QlikCommunity)
both from from Incentro, Netherlands - Lavastorm Analytics
I will start to work on a Java implementation of the QVD format.
The QVX file format still don’t shows the wanted loading performance in QlikView with large files even if it’s optimized for numericals like numbers and date/datetime and a blocksize is given.
Therefor the QVD file format - which can be used in an optimized QlikView load - must do the job in Big Data environments. Hence the QVD format is a bit more complex (but very delicious) it needs a different implementation approach and intelligent data caching (maybe I use Ehcache) in a Java application.
But I think this is doable and I can use some snippets of the QVXConverter because of some similarities..
Update: The QVDReader is finished! Now I can read and convert any QVD file outside of QlikView, under all OS platforms which can process Java.
Btw. Ehcache was not the right solution. I had to use a persisten key/value store to cover unlimited QVD file size. I have chosen JDBM3 which runs very stable so far.
Update 2: This is the very first demo of the Java-based QVDReader implementation. You can convert QlikView QVD files to CSV. Try it out!
I’m waiting for your feedback..
Update 3: Now I was able to increase performance of the QVDReader by 300% using a better JDBM3 implementation. It seems now more likely to go into a productive state..
Update 4: Our Java-based QVDWriter implementation is working now! This means we’re able to create QVD files in nearly any environment independently from QlikView.
