I WSO2ESB start up failed and hangout due to typo of wsdl saved in governance registry

Ok first of all i will tell what happened before the start up fail.

1. We have added a wsdl file belong to one of our services into the governance registry.

2. While manually editing the wsdl file which have been added to governance registry, some miss typing has occurred which changed a letter of one of the XML tags into upper case

3. Without knowing about the typo we saved the wsdl, (and here is the funny part… ) ESB let that wsdl to be save in it’s governance registry without parsing it or neither complaining us about the typo we did .. ( really ??? an enterprise level ESB does not check the documents edited ??? )

4. After some time for some reason we have restarted the ESB. ( and here the fun began. ) ESB hangout printing an exception complaining XML parsing error when it try to load that wsdl file where the error happened. And the admin console is not loading because OSGI module that correspond to esb (Synapsys) is waiting until all the wsdl loads. (This is the biggest fault - not letting other services, and esb to start because a wsdl typo occurred where it should parse the wsdl before save it to registry )

At the end of the console we can see the following message printed time to time without starting the ESB and the services due to what happened that has been described above.

 StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.mediation.initializer.services.SynapseEnvironmentService

And this is the exception which complains about that our wsdl could not be parsed at startup.

WstxParsingException: Illegal processing instruction target (“xml”); xml (case insensitive) is reserved by the specs. at [row,col

There were 0 documentation and no entry about how to handle such a situation.

According to this exception we guessed the work around for this problem is just to remove the erroneous wsdl file from the governance registry. But wso2esb it self should not let that wsdl added to the registry at the first place. Even it did add, there should be some way to remove the erroneous wsdl related proxy and start up the esb and other proxy services. But it does not have that feature yet.

So how the hell we edit the governance registry without esb up.

Well, there is no proper way wso2 has exposed to do that. ( i could not find any thing on the web that’s why i write this blog entry apart from that to complement wso2 esb team about this s:D )

What we did is this,

1. Shut down the esb , copy and back up the databases folder in <wso2esb_home>/repository/database

2. Download any sql viewer tool that support h2 database management system. (yes wso2 default database is java h2 DBMS - in our case we used the default DBMS ). I used SQL Squirrel with h2 database jdbc driver.

3. open/connect to the h2 database in above folder using the sql tool.

4. find the “REG_RESOURCE” and “REG_CONTENT” tables in “WSO2CARBON_DB/PUBLIC/TABLE/” location.

5. search for erroneous wsdl file name inside “REG_RESOURCE” (hint: use sql select). File name for a resource in “REG_RESOURCE” saved in “REG_NAME” column. 

6. From the resulted row note down the value of “REG_CONTENT_ID” and search for that REG_CONTENT_ID in “REG_CONTENT” table. The resulting row is where the actual wsdl file saved as sql BLOB file type.

select rc.REG_CONTENT_ID from REG_RESOURCE rr, REG_CONTENT rc where rr.REG_NAME like ‘%mywsdl.wsdl’ and rc.REG_CONTENT_ID=rr.REG_CONTENT_ID

7. To recover the wso2 esb from hanging out due to this erroneous wsdl remove the two entries related to REG_CONTENT_ID of REG_NAME of the wsdl from two tables.

8. Re start the wso2esb and you may find that exceptions are no more there and server does not hanging out any more.

Yeah i know this is bit fishy and fundamental, but if we cant access the admin console due to that ESB not finishing up loading its OSGI module (because esb let it save error-nous wsdl inside the governance registry at the first place.) we cant do much rather than cracking the database and remove the wsdl manually right ??

PS : there may be other artifacts such as proxies, sequences, endpoints using the removed wsdl, that may affected by the removal. so i instruct to remove those of which use the removed wsdl.