Hi all,
I plan to port our data to Hana. Each data row contains, among others, a text document that should be searchable, its size might be anything from 200 byte to say 20kbyte. From the documentation I follow that to enable the text search on this column it has to be a columnar table. So far so good. The main point is, however, that new rows are being added all the time, with a frequency of a couple of rows per second. As we'd like to keep at least tens of millions rows in memory, I'm wondering what will be the effect of the DELTA MERGE on this data. If I understand it right, columnar tables are updated/inserted using the delta storage and during the merge the main storage and the delta storage are copied to a new main storage, so effectively the memory usage during this operation will be doubled.
Is this really necessary? I understand that columns that benefit from compression will also benefit from reordering to group identical values, so real merging is necessary. In our case, however, there will be no identical values, each document is completely different and quite large, so copying the whole main storage to another place will hardly have any effect. The downside would be that we'd actually have to keep half of available memory empty just to accomodate this operation.
Thanks for any hits,
-- Micha