Hello!
Summary: I am trying to enable changing recording (feature introduced in SPS6 and enhanced a lot in SPS7) using HALM (I'm on SPS09) but I get the following error:
"message not found;mark active objects: SQL preparation or execution failed. Details can be found in the indexserver trace file."
Inside the browser console, I can see a failing POST call to /sap/hana/xs/lm/core/ConfService.xsjs service - it receives a HTTP 500 error. The HTTP request has the folling Form Data
{"action":"set_change_tracking","value":true}
---
What I'm 100% sure
After reading several OSS Notes, debugging this service I am aware of what is **not** the root cause:
Privileges
I'm using SYSTEM user and it has:
- role sap.hana.xs.lm.roles::Administrator (among other lm roles)
- system privilege REPO.CONFIGURE
- execute privilege on procedure SYS.REPOSITORY_REST (discussed later)
Timeout
The error takes ~5 seconds to happen, so I'm discarding a timeout issue.
OSS Notes
- 2073243 - Release Notes for SAP HANA Application Lifecycle Management for SAP HANA SPS 09
This OSS Notes is the only one I found it was useful somehow as it describes frequent issues on HALM and other restrictions according to SPS and revisions. The only item I could not check in my system is "JSVM memory" mentioned in that note. According to such OSS:
The change recording function in SAP HANA Application Lifecycle Management, which was introduced with HANA SPS 07, has the following restrictions:
Change recording requires enough JSVM memory to handle changelists, for example, search for changes and export/transport changelists and enable/disable change tracking. If you have received a 500 Internal Server Error in the browser console and the xsengine_* trace file is out of memory, then enhance the memory configuration to a proper value. The following example provides information about how to do this.
xsengine.ini > jsvm > max_runtime_byte
Using the Administration view inside Eclipse, I don't even have an option called jsvm inside xsengine.ini
Browser
I'm using Chrome but I'm sure the issue is in the back end according to the log files presented afterwards.
---
What I am **not** 100% sure
I don't know if the error I'm getting is due to the fact that at least on artifact in the system has a syntax error. To explain why I guess this could be an error I have to explain what I learn watching Change Management in SAP HANA 1.0 SPS07 from Willi Petri and reading SAP HANA Lifecycle Management from Boris Zarske among other slides.
When you enable change recording, HANA takes all your artifacts and include them inside a change (the very first one in the system). After releasing this change, every activation you do on an artifact prompts for a new change associated with it (similar to how CTS works in ABAP).
Maybe, when HANA tries to assign all artifacts to a change, it checks the syntax of such objects and aborts the operation if at least one failed.
What I have done to confirm this was to manually regenerate all my packages (which contains almost 60000 artifacts) and check whether they had syntax errors or not. I found some and fixed them. I can regenerate all my packages without error. The only one I get an error is the system-local package, which although it's empty, gives me an error)
---
The error in detail
Debugging the ConfService.xsjs service, I came to the conclusion that the root cause is inside function execRepoCall. It calls the procedure SYS.REPOSITORY_REST but the call fails as it returns false.
Looking at the application and SQL trace files, I see a failing UPDATE statament on table '_SYS_REPO:ACTIVE_OBJECT'
Trace on sap.hana.xs.lm Application
[6307]{6307}[-1/-1] 2015-09-02 18:36:29.640156 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00085) : Execute WebRequest returned with: 0
[6307]{6307}[-1/-1] 2015-09-02 18:36:29.640164 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00095) : end JS request handler for /sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.511661 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00058) : begin JS request handler for /sap/hana/xs/lm/csrf.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.511891 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00085) : Execute WebRequest returned with: 0
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.511897 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00095) : end JS request handler for /sap/hana/xs/lm/csrf.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.587800 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00058) : begin JS request handler for /sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.588039 d xsa:sap.hana.xs. GlobalFunctions.cpp(00159) : check access for URI: /sap/hana/xs/lm/core/lib/Config.xsjslib
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589362 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "sap"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589378 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "hana"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589381 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "xs"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589383 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "lm"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589384 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "core"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589396 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "lib"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589397 d xsa:sap.hana.xs. GlobalFunctions.cpp(00214) : create "Config"
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.589482 d xsa:sap.hana.xs. GlobalFunctions.cpp(00159) : check access for URI: /sap/hana/xs/lm/xsts/utils.xsjslib
...
lots of other checkAccess for stuff
...
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.645294 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00873) : getTrackingStatus Starts
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.645326 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00156) : execRepoCall: {"action":"get","what":"changeTrackingStatus","session":{"sessionType":"2","owner":"SYSTEM","workspace":"HALM"}}
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.647147 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm, ApplicationSource=/sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.647145 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00167) : Starts: Wed Sep 02 2015 18:36:32 GMT-0400 (EDT)
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651268 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm, ApplicationSource=/sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651266 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00169) : Done: Wed Sep 02 2015 18:36:32 GMT-0400 (EDT)
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651285 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00170) : Statement Result: false
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651308 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00172) : Result: [object ArrayBuffer]
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651325 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:utils.xsjslib(00420) : ab2str(): byte array length: 6
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651366 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00089) : Header: repoV2
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651370 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00091) : No. of Attachments: 1
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651374 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00094) : JSON Length: 94
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651380 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:utils.xsjslib(00420) : ab2str(): byte array length: 94
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.651393 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00101) : DECODED JSON: {
"error-arg": "",
"error-code": "0",
"error-msg": "No error",
"status": "disabled"
}
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.654434 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00905) : setTrackingStatus Starts: Wed Sep 02 2015 18:36:32 GMT-0400 (EDT)
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.654464 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00156) : execRepoCall: {"action":"set","what":"changeTracking","status":"on","ignoreOpenChanges":"1","session":{"sessionType":"2","owner":"SYSTEM","workspace":"HALM"}}
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.656030 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm, ApplicationSource=/sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:32.656028 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00167) : Starts: Wed Sep 02 2015 18:36:32 GMT-0400 (EDT)
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143529 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm, ApplicationSource=/sap/hana/xs/lm/core/ConfService.xsjs
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143527 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00169) : Done: Wed Sep 02 2015 18:36:38 GMT-0400 (EDT)
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143539 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00170) : Statement Result: false
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143622 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00172) : Result: [object ArrayBuffer]
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143644 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:utils.xsjslib(00420) : ab2str(): byte array length: 6
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143656 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00089) : Header: repoV2
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143660 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00091) : No. of Attachments: 1
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143664 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00094) : JSON Length: 191
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143671 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:utils.xsjslib(00420) : ab2str(): byte array length: 191
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.143805 d xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00101) : DECODED JSON: {
"error-arg": "",
"error-code": "129",
"error-msg": "message not found;mark active objects: SQL preparation or execution failed. Details can be found in the indexserver trace file."
}
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.147330 e xsa:sap.hana.xs. sap.hana.xs.lm.xsts:repoUtils.xsjslib(00929) :
Stack Trace:
UserReportableException@/sap/hana/xs/lm/xsts/utils.xsjslib:41
setTrackingStatus@/sap/hana/xs/lm/xsts/repoUtils.xsjslib:922
setChangeTracking@/sap/hana/xs/lm/core/lib/Config.xsjslib:26
processPOSTRequest@/sap/hana/xs/lm/core/lib/Config.xsjslib:90
@/sap/hana/xs/lm/core/lib/Config.xsjslib:104
process@/sap/hana/xs/lm/xsts/utils.xsjslib:1219
processRequest@/sap/hana/xs/lm/core/lib/Config.xsjslib:98
@/sap/hana/xs/lm/core/ConfService.xsjs:2
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.147355 e xsa:sap.hana.xs. sap.hana.xs.lm.xsts:utils.xsjslib(00161) : message not found;mark active objects: SQL preparation or execution failed. Details can be found in the indexserver trace file.
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.147381 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00085) : Execute WebRequest returned with: 0
[6307]{6307}[-1/-1] 2015-09-02 18:36:38.147386 d xsa:sap.hana.xs. JsSuffixHandler.cpp(00095) : end JS request handler for /sap/hana/xs/lm/core/ConfService.xsjs
SQL Trace
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862357 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862345 e AEText core.cpp(00324) : Invalid term id in scaled document: 32625
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862366 e AEText core.cpp(00325) : Document id: 13156
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862368 e AEText core.cpp(00326) : Size of scaled document: 114460
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862369 e AEText core.cpp(00327) : Number of distinct terms: 8240
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862370 e AEText core.cpp(00328) : Min term id: 1
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862379 e AEText core.cpp(00329) : Max term id: 121040
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862388 e textattr TextAttribute.cpp(03606) : _getNormalizedDoc._SYS_REPO:ACTIVE_OBJECT (5).$_SYS_SHADOW_CDATA$(63378): mCore->scaledDocument(id, doc, mapping, fromTemp), in _getNormalizedDoc: 6900
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862394 e textattr TextAttribute.cpp(03647) : _reconstructDaf._SYS_REPO:ACTIVE_OBJECT (5).$_SYS_SHADOW_CDATA$(63378): _getNormalizedDoc(id, dict, doc, fromTemp), in _reconstructDaf: 6900
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862397 e textattr TextAttribute.cpp(03734) : _getDaf._SYS_REPO:ACTIVE_OBJECT (5).$_SYS_SHADOW_CDATA$(63378): _reconstructDaf(id, daf, fromTemp), in _getDaf: 6900
[9732]{212705}[1577/3965391] 2015-09-02 18:45:21.862398 e textattr TextAttribute.cpp(03700) : getDocObject._SYS_REPO:ACTIVE_OBJECT (5).$_SYS_SHADOW_CDATA$(63378): _getDaf(id, simpleDaf), in getDocObject: 6900
[5949]{212705}[1577/3965391] 2015-09-02 18:45:21.862922 i TraceContext TraceContext.cpp(00827) : UserName=SYSTEM, ApplicationUserName=SYSTEM, ApplicationName=sap.hana.xs.lm
[5949]{212705}[1577/3965391] 2015-09-02 18:45:21.862917 e table_update TableUpdate.cpp(12201) : TableUpdate::writeDataIntoAttributeEngine(): commitOptimizeAttributes() failed with rc=6900, Attribute engine failed; table '_SYS_REPO:ACTIVE_OBJECT$delta_1$'(6900)
[5949]{212705}[1577/3965391] 2015-09-02 18:45:21.862927 e table_update TableUpdate.cpp(15614) : lastError: commitOptimizeAttributes() failed with rc=6900, Attribute engine failed; table '_SYS_REPO:ACTIVE_OBJECT$delta_1$'(6900)
[5949]{212705}[40/3965389] 2015-09-02 18:45:21.902318 e REPOSITORY dbutils.cpp(01014) : mark active objects: SQL execution of
'UPDATE _sys_repo.active_object SET change_number = ?, released_at = ?
(65835, '2015-09-02 18:45:16.5740000')
SQL string with parameter values instead of '?':
UPDATE _sys_repo.active_object SET change_number = 65835, released_at = '2015-09-02 18:45:16.5740000''
failed:
transaction rolled back by an internal error: TrexUpdate failed on table '_SYS_REPO:ACTIVE_OBJECT' with error: commitOptimizeAttributes() failed with rc=6900, Attribute engine failed; table '_SYS_REPO:ACTIVE_OBJECT$delta_1$', rc=6900 - enforce TX rollback at ptime/query/plan_executor/trex_wrapper/trex_wrapper_body/trex_update.cc:2374
exception 1: no.71000129 (ptime/query/plan_executor/trex_wrapper/trex_wrapper_body/trex_update.cc:2374)
TrexUpdate failed on table '_SYS_REPO:ACTIVE_OBJECT' with error: commitOptimizeAttributes() failed with rc=6900, Attribute engine failed; table '_SYS_REPO:ACTIVE_OBJECT$delta_1$', rc=6900 - enforce TX rollback
NO exception throw location recorded. Stack generation suppressed.
[5782]{-1}[-1/-1] 2015-09-02 18:46:30.975267 w Logger SavepointImpl.cpp(02136) : NOTE: BACKUP DATA needed to ensure recoverability of the database
Using the SQL console to reproduce the error
If I take the UPDATE call from the SQL file and try to run it using SQL console, I get the same error:
UPDATE _sys_repo.active_object SET change_number = 65835, released_at = '2015-09-02 18:45:16.5740000'
Could not execute 'UPDATE _sys_repo.active_object SET released_at = '2015-09-02 18:45:16.5740000'' in 5.348 seconds .
[129]: transaction rolled back by an internal error: TrexUpdate failed on table '_SYS_REPO:ACTIVE_OBJECT' with error: commitOptimizeAttributes() failed with rc=6900, Attribute engine failed; table '_SYS_REPO:ACTIVE_OBJECT$delta_1$', rc=6900 - enforce TX rollback
I had no success searching for 'TrexUpdate failed on table', 'commitOptimizeAttributes() failed with rc=6900' and other key terms. I have no clue what is Trex and why this dinossaur doesn't like me.
So... after all... what do I need to do inside this system to enable change recoring?
Thanks a lot! I tried to be as detailed as possible.
Including trace files as quotes so it's possible to see it completely