Hi all,
I'm experiencing quite a few problems with importing CSV files to Hana and wondering if it's just only me, because they make my using Hana quite difficult:
- I'm loading data from another database to Hana. I extract it from there and convert to CSV. I then load a CSV file with 100-10000 rows and it takes anything between 2 and 30 minutes. Is this normal? I'm loading it into a table that has 3M rows, I don't see why it should be so slow. I've tried various combinations of threads and batch, the command looks like
import from csv file '/dropbox/P1717/fe.value8.arch2.csv' into fe_document with threads 10 batch 100 table lock record delimited by '\n' field delimited by '<#>' error log '/dropbox/P1717/errfe8.2.txt'
0 rows affected (1875.899724 sec) - this has loaded whole 96 rows - From the loaded rows a sizeable fraction is not inserted but ends up in the error log. When I edit the error log, remove the error message and import the file again, some rows are inserted and again some are not. Does it mean that when a thread encounters an error, it abandons the whole batch? And writes bogus error message to good rows, too? What is the best approach to insert as many rows as possible, knowing that a few may be bad?
- The error message is in 99% cases wrong. Either it is connected with a perfectly valid row, or the row is really bad but the error message does not say anything to help to find the problem. For instance
[ROLLBACK#1] inserted value too large for column: >> 560550752<#>ES<#>Guindos: el sistema bancario español es "muchísimo más seguro" gracias al rescate<#>2013-01-21 17:19:01<#>http://www.cincodias.com/articulo/mercados/guindos-sistema-bancario-espanol-muchisimo-seguro-gracias-rescate/20130121cdscdsmer_3/<#>El ministro de Economía, Luis de Guindos, ha destacado a su llegada al Eurogrupo que el sistema bancario español es más transparente y "muchísimo más seguro" gracias al rescate de 40.000 millones de euros concedido por la UE,...<#>ES-010-015622195<#><#>WWW_CINCODIAS_COM_RSS_FEED_HTML_1311904392722<#>NORMAL<#>Cinco Días<#>El ministro de Economía, Luis de Guindos, ha destacado a su llegada al Eurogrupo que el sistema bancario español es más transparente y "muchísimo más seguro" gracias al rescate de 40.000 millones de euros concedido por la UE, que a su juicio ha logrado acabar con las incertidumbres y las dudas que afectaban al sector.
does not say which column is the bad one (there is none anyway). Or
[ROLLBACK#0] cannot insert NULL or update to NULL: >> 560063282<#>CN<#>格鲁吉亚5000人示威要求总统下台<#>2013-01-21 02:51:23<#>http://news.xinhuanet.com/yzyd/world/20130121/142612620557278380671n.jpg<#>示威者聚集在总统官邸外面 据外国媒体报道，多达5000名格鲁吉亚人当地时间1月20日在总统官邸外示威，要求总统萨卡什维利下台。 “他的第二任期今天就结束了，1月20日，他无权再留任，”一名组织者告诉媒体。记者在现场看到，人们举着反对萨卡什维利的各种标语和讽刺漫...<#>CN-010-014787578<#>90672802<#>NEWS_BAIDU_COM_N_1281447986973<#>SHORT_SUMMARY<#><#>示威者聚集在总统官邸外面 据外国媒体报道，多达5000名格鲁吉亚人当地时间1月20日在总统官邸外示威，要求总统萨卡什维利下台。 “他的第二任期今天就结束了，1月20日，他无权再留任，”一名组织者告诉媒体。记者在现场看到，人们举着反对萨卡什维利的各种标语和讽刺漫画，有人甚至威胁在总统府周边设置路障，并安营扎寨。 萨卡什维利2003年在西方支持下通过“玫瑰革命”上台，2008年连任。格鲁吉亚宪法规定，总统任期5年，也就是说萨卡什维利任期已于20日到期。但格鲁吉亚议会2010年修改宪法，将下一届总统选举定于今年10月举行。
again it does not say which column, and the only NULL value is in a column where NULLs are allowed.
Is there some other way to load data from another database than using CSV?
Cheers,
-- Micha