WSO2 ESB - sample REST to REST API configuration

<api xmlns=”http://ws.apache.org/ns/synapse” name=”TestApi” context=”/helloWorld”>
   <resource methods=”GET” url-mapping=”/hello”>
      <inSequence>
         <property name=”ContentType” value=”text/xml” scope=”axis2” type=”STRING”/>
         <switch source=”$axis2:HTTP_METHOD”>
            <case regex=”GET”>
               <property name=”HTTP_METHOD” value=”GET” scope=”axis2” type=”STRING”/>
            </case>
            <case regex=”POST”>
               <property name=”HTTP_METHOD” value=”POST” scope=”axis2” type=”STRING”/>
            </case>
            <default/>
         </switch>
         <log/>
         <send>
            <endpoint>
               <address uri=”http://localhost:8080/HelloRestFul” format=”pox”/>
            </endpoint>
         </send>
      </inSequence>
      <outSequence>
         <log/>
         <send/>
      </outSequence>
   </resource>
</api>
                       

WSO2 ESB exposing Web Service Proxy to a TCP client PART 2 : Sending batch responses from ESB to TCP server

In my last post regarding how to expose a web service for a TCP client we have discussed about retrieving a request in plain text format from a TCP client, convert it to SOAP message and send to the back end webservice.

In that post I have mentioned that according to our implementation we did not need to send the response back to the TCP client immediately due to that our responses are not synchronous.

So what we did to achieve this asynchronous status is converting the ESB it self to a TCP client and send the responses as batches to a TCP server.

So in this article I am going to explain how to use the WSO2 ESB as a TCP client where we needed to convert the XML responses to delimited string sequences before sending to TCP server.

To convert the XML file to delimited string sequence we have used the smooks mediator. (Refer the smooks configuration below)

In above example we are converting a xml file that has an entity called “Login”. Each login element has three child elements.

Freemarker tool has been used to convert the XML elements to text fields and delimit two values.

<smooks-resource-list xmlns=”http://www.milyn.org/xsd/smooks-1.1.xsd” xmlns:ftl=”http://www.milyn.org/xsd/smooks/freemarker-1.1.xsd” xmlns:core=”http://www.milyn.org/xsd/smooks/smooks-core-1.3.xsd” xmlns:file=”http://www.milyn.org/xsd/smooks/file-routing-1.1.xsd”>
      <core:filterSettings type=”SAX” defaultSerialization=”false”/>
      <resource-config selector=”//batchTransfer/returnVal/login”>
         <resource>org.milyn.delivery.DomModelCreator</resource>
      </resource-config>
      <ftl:freemarker applyOnElement=”login”>
         <ftl:template>${.vars[“login”].id},${.vars[“login”].password},${.vars[“login”].userName}
           </ftl:template>
      </ftl:freemarker>
   </smooks-resource-list>

Save above smooks configuration as a Local XML entry.

Then to send a batch of responses to ESB from our back end, we have created a dummy web service proxy inside ESB. This dummy web service connect to the TCP server, thus the TCP server now acts as the back end. Also we have hand written a WSDL and expose that hand written WSDL through the ESB. This way we can configure what kind or SOAP structure accept by this dummy proxy web service and then convert the XML resides within that proxy and then send those converted text to the back end TCP server.

Below is the WSDL which we hand written for the above “Login batch response” example.

<wsdl:definitions xmlns:wsdl=”http://schemas.xmlsoap.org/wsdl/” xmlns:xsd=”http://www.w3.org/2001/XMLSchema” xmlns:soap=”http://schemas.xmlsoap.org/wsdl/soap/” xmlns:tns=”com.mobitel.test” name=”test” targetNamespace=”com.mobitel.test”>
      <wsdl:types>
         <xsd:schema targetNamespace=”com.mobitel.test”>
            <xsd:element name=”batchTransfer” type=”tns:batchTransfer”/>
            <xsd:element name=”batchTransferResponse” type=”tns:batchTransferResponse”/>
            <xsd:complexType name=”batchTransfer”>
               <xsd:sequence>
                  <xsd:element name=”returnVal” type=”tns:logins”/>
               </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name=”logins”>
               <xsd:sequence>
                  <xsd:element name=”login” type=”tns:login” nillable=”true” minOccurs=”0” maxOccurs=”unbounded”/>
               </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name=”login”>
               <xsd:sequence>
                  <xsd:element name=”id” type=”xsd:int”/>
                  <xsd:element name=”password” type=”xsd:string”/>
                  <xsd:element name=”userName” type=”xsd:string”/>
               </xsd:sequence>
            </xsd:complexType>
            <xsd:complexType name=”batchTransferResponse”/>
         </xsd:schema>
      </wsdl:types>
      <wsdl:message name=”batchTransferRequest”>
         <wsdl:part element=”tns:batchTransfer” name=”parameters”/>
      </wsdl:message>
      <wsdl:message name=”batchTransferResponse”>
         <wsdl:part element=”tns:batchTransferResponse” name=”parameters”/>
      </wsdl:message>
      <wsdl:portType name=”test”>
         <wsdl:operation name=”batchTransfer”>
            <wsdl:input message=”tns:batchTransferRequest”/>
            <wsdl:output message=”tns:batchTransferResponse”/>
         </wsdl:operation>
      </wsdl:portType>
      <wsdl:binding name=”testSOAP” type=”tns:test”>
         <soap:binding style=”document” transport=”http://schemas.xmlsoap.org/soap/http”/>
         <wsdl:operation name=”batchTransfer”>
            <soap:operation soapAction=”com.mobitel.test/NewOperation”/>
            <wsdl:input>
               <soap:body use=”literal”/>
            </wsdl:input>
            <wsdl:output>
               <soap:body use=”literal”/>
            </wsdl:output>
         </wsdl:operation>
      </wsdl:binding>
      <wsdl:service name=”test”>
         <wsdl:port binding=”tns:testSOAP” name=”testSOAP”>
            <soap:address location=”http://localhost:8282/services/TcpOutput”/>
         </wsdl:port>
      </wsdl:service>
   </wsdl:definitions>

This way the user of this dummy web service can create a web service client by using the above WSDl expose from WSO2 ESB’s dummy proxy service. Also the details such as back end is TCP can be hidden from the user.

Below is the dummy proxy for the “Login batch response” example.

<?xml version=”1.0” encoding=”UTF-8”?>
<proxy xmlns=”http://ws.apache.org/ns/synapse”
       name=”TcpOutput”
       transports=”https,http,tcp”
       statistics=”disable”
       trace=”disable”
       startOnLoad=”true”>
   <target>
      <inSequence>
         <log level=”full”/>
         <smooks config-key=”smooksXMLToText”>
            <input type=”xml”/>
            <output type=”text”/>
         </smooks>
         <log level=”full”/>
         <property name=”CONTENT_TYPE”
                   value=”text/plain”
                   scope=”axis2”
                   type=”STRING”/>
         <property name=”OUT_ONLY” value=”true” scope=”default” type=”STRING”/>
         <send>
            <endpoint>
               <address uri=”tcp://localhost:9448”/>
            </endpoint>
         </send>
         <property name=”FORCE_SC_ACCEPTED”
                   value=”true”
                   scope=”axis2”
                   type=”STRING”/>
      </inSequence>
      <outSequence>
         <send>
            <endpoint>
               <default/>
            </endpoint>
         </send>
      </outSequence>
   </target>
   <publishWSDL key=”testOutPutWsdl”/>
   <description/>
</proxy>
                               

Here we have used 3 properties ( please refer the proxy xml above)

1. <property name=”CONTENT_TYPE” value=”text/plain”               scope=”axis2” type=”STRING”/>   

This will tell ESB that the output to back end will be “text/plain”. We needed this because our TCP server accept only text formate.

2. <property name=”OUT_ONLY” value=”true” scope=”default” type=”STRING”/>

This will tell ESB that this proxy does not send a response or simply do not need a out sequence. Though we defined empty outsequance tag we does not expect web service client to wait for a response from back end TCP server.

3. <property name=”FORCE_SC_ACCEPTED” value=”true”                 scope=”axis2” type=”STRING”/>

Add this property just after the SEND mediator. We use this property to send a HTTP 202 ( no content ) response to the web service client to avoid it wait for response.

Also please note that in our dummy proxy web service we have enabled all the http,https,tcp transport types because in our proxy we retrieve the “batch response” request via HTTP or HTTPS and we use TCP to transport that “batch response” request to TCP server.

(Special thanks goes to Lalaji Sureshika from WSO2)

WSO2 ESB exposing Web Service Proxy to a TCP client. ( A.K.A TCP proxy)

In this article I am going to explain how to create a proxy to a back end web service which accept TCP messages.

In our case our TCP client was sending a delimited string to the ESB and we need ESB to convert the delimited string into a SOAP/XML file and call the web service.

In our case we does not intend to send the response immediately back to the client rather than sending response as batches of several requests. Since I will explain how the batch response in another article.

To accept the TCP requests you need to change your axis2.xml file inside <ESB_LOCATION>/repository/conf/axis2.xml file and add below two transport handler.

1. Configure TCPTransportListener to listen to in coming TCP requests.

<transportReceiver name=”tcp” class=”org.apache.axis2.transport.tcp.TCPTransportListener”>
      <parameter name=”transport.tcp.port”>6060</parameter>
    </transportReceiver>

2. Configure TCPTransportSender to send responses over TCP (this handler will not want for this article but this will be used in next article for batch response sending )

<transportSender name=”tcp” class=”org.apache.axis2.transport.tcp.TCPTransportSender”/>

Then we use inbuilt smooks handler to transform the delimited values into a XML entries.

To do this we first need to add smooks configuration which act as a template to smooks handler. We need to add the smooks configuration as a local XML entry to ESB. To do this go to Main - > Service Bus -> Local Entries. And add a in-lined XML entry.

(This is an example of smooks configuration which use to trans form delimited text to XML)

<smooks-resource-list xmlns=”http://www.milyn.org/xsd/smooks-1.1.xsd” xmlns:csv=”http://www.milyn.org/xsd/smooks/csv-1.2.xsd”>
      <resource-config selector=”org.xml.sax.driver”>
         <resource>org.milyn.csv.CSVReader</resource>
         <param name=”fields”>id,password,userName</param>
         <param name=”rootElementName”>addLogin</param>
         <param name=”recordElementName”>login</param>
      </resource-config>
   </smooks-resource-list>
                                   

Then add a new proxy which use the smooks mediator in “Advance -> smooks”, with TCP transport type allowed, and finally with port and content type properties as below.

Below is a sample

<?xml version=”1.0” encoding=”UTF-8”?>
<proxy xmlns=”http://ws.apache.org/ns/synapse”
       name=”LoginProxy”
       transports=”tcp”
       statistics=”disable”
       trace=”disable”
       startOnLoad=”true”>
   <target>
      <inSequence>
         <smooks config-key=”LoginDetails”>
            <input type=”text”/>
            <output type=”xml”/>
         </smooks>
         <send>
            <endpoint>
               <wsdl service=”HelloServiceImplService”
                     port=”HelloServiceImpl”
                     uri=”http://localhost:8080//HelloWebService/services/HelloServiceImpl?wsdl”/>
            </endpoint>
         </send>
         <log level=”full”/>
      </inSequence>
      <outSequence>
         <log/>
      </outSequence>
   </target>
   <parameter name=”transport.tcp.port”>9446</parameter>
   <parameter name=”transport.tcp.contentType”>text/plain</parameter>
   <description/>
</proxy>

How to move your blogs around.

In this post I am going to describe how to move your blog from one platform to another. Since there are many famous platforms that offer various kind of user experience, it is more tend to keep several blogs in different platforms that has the same content. To synchronize these blogs we have to have some kind of automation.

Blogspot powered by google has functionality that support Wordpress but not Tumblr. In the other hand Tumblr support both Wordpress and Blogspot. Also Wordpress support backs both Tumblr and Wordpress.

1. Moving from Blogspot to Tumblr.

This tool  gives an api to move your Blog in Google’s Blogspot platfrom to Tumblr.

2. Moving from Tumblr to Wordpress

This tool is giving you the ability to move your Tumblr blog posts to Wordpress platform. Its source code also available in Git repository to download and host in your own server.

3. Moving from Wordpress to Blogspot.

This tool gives the ability to move your blogs from Wordpress platform to Blogspot platform.

4. Moving from Tumblr to Blogspot.

This is a bit tricky task since google’s Blogspot does not expose an Api to directly support Tumbler platform. Hence we have to be bit tricky. First convert your Tumblr into Wordpress format by using the tool given in section 2. This will download a XML file into your computer which is in Wordpress format. Now using the tool given in section 3 convert the downloaded xml file into Blogspot fomrat and save the XML file. Go to settings>import and import the downloaded XML file.

Google App Engine DataNucles Enhancer error

When you update your GAE version to the latest, you might have been struck in the following error.

java.lang.UnsupportedClassVersionError: com/google/appengine/tools/enhancer/Enhance : Unsupported major.minor version 51.0

at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(Unknown Source)
at java.lang.ClassLoader.defineClass(Unknown Source)
at java.security.SecureClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.defineClass(Unknown Source)
at java.net.URLClassLoader.access$000(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)

Reason for this is that the JDK versions and JRE versions you use are conflict with either each other or the version compatible with the latest GAE SDK version.

First thing I did is check for Java versions installed in my machine. I found out that I have java 1.6 and 1.7 both installed. So I change the default JDK to 1.7 (since newer GAE tends to be more support for java 7).

Change your JDK_HOME, JRE_HOME and PATH variables to reflect the location of newest JDK in your machine.

Though this solve the above error, it might introduce a new error which is eclipse is not able to recognize the appengine-sdk at all.

“Failed to initialize App Engine SDK at [path]”

The reason might be that when eclipse launch it do not use the default Java runtime. To force eclipse to use the default runtime, add the “-vm” argument to <eclipse-folder>\eclipse.ini file.

-startup
plugins/org.eclipse.equinox.launcher_1.3.0.v20130327-1440.jar
—launcher.library
plugins/org.eclipse.equinox.launcher.win32.win32.x86_64_1.1.200.v20130807-1835
-vm
C:\Program Files\Java\jdk1.7.0_45\bin\javaw.exe

WSO2 ESB exposing a secure backend through a proxy.

(Special thanks goes to Sumedha Kodithuwakku @ wso2 support team)

In this post i will describe how to generate a custom certificate using keytool and use that certificate to secure a your web service and finally how to expose the secured web service using a proxy in wso2 ESB.

1. Generate a custom certificate.

Open a command prompt and use the keytool to generate a certificate. Here I use the CN as localhost since we use locally host web service(keep this in mind for later use). The rule of thumb is that CN should be matches to the hostname of the wsdl endpoint to resolve it by the ESB.

(Also using IP address as a CN would be a problematic so always stick to a host name rather than using an IP address.)

Use the following command to generate a certificate

keytool -genkey -alias localhost -keyalg RSA -keystore localhost.jks

*** When asked for the first and last name give the hostname which is localhost in our case.

Here you can use what ever name for -alias and - localhost.jks which is the alias of our certificate and name of the keystore if it newly create.

then export that certificate to a “.crt” file using command below.

keytool -export -alias localhost -file localhost.crt -keystore localhost.jks

2 Adding a keystore to web server (tomcat in our example)

Open the server.xml file in <tomcat-location>/conf/ folder and edit the <Connector port=”8443”tag as below.

<Connector port=”8443” protocol=”HTTP/1.1” SSLEnabled=”true”
               maxThreads=”150” scheme=”https” secure=”true”
               clientAuth=”false” sslProtocol=”TLS”
               keystoreFile=”C:\keyStores\localhost.jks” keystorePass=”123456” />

Here we have added the keystore and the password we newly created.

3. Adding imported certificate to wso2 keystore and truststore.

Browse to <wso2-esb>\repository\resources\security folder. There is two keystore files namely “client-truststore.jks” and “wso2carbon.jks”. Use the below command to add the newly created and stored certificate (localhost.crt as our example in part 1 ) to these two keystore. Provide alias as the host name used as CN.

keytool -importcert -v -trustcacerts -file “localhost.crt” -alias localhost -keystore “wso2carbon.jks”

keytool -importcert -v -trustcacerts -file “localhost.crt” -alias localhost -keystore “client-truststore.jks

Since we are using localhost as our host we have update “HostnameVerifier" parameter in <wso2-esb>repository/conf/axis2.xml file to allow all hosts.

<parameter name=”HostnameVerifier”>AllowAll</parameter>

(This parameter can be found in <transportSender name=”https” tags.)

4. Create a proxy as normal and add the https://localhost:8443/<your-service>?wsdl file as normally do.

Resources :

[1] http://blog.facilelogin.com/2011/02/wso2-esb-invoking-web-service-via-https.html

[2] http://stackoverflow.com/questions/8443081/how-are-ssl-certificate-server-names-resolved-can-i-add-alternative-names-using/8444863#8444863

[3] http://stackoverflow.com/questions/19540289/how-to-fix-the-java-security-cert-certificateexception-no-subject-alternative?rq=1

I WSO2ESB start up failed and hangout due to typo of wsdl saved in governance registry

Ok first of all i will tell what happened before the start up fail.

1. We have added a wsdl file belong to one of our services into the governance registry.

2. While manually editing the wsdl file which have been added to governance registry, some miss typing has occurred which changed a letter of one of the XML tags into upper case

3. Without knowing about the typo we saved the wsdl, (and here is the funny part… ) ESB let that wsdl to be save in it’s governance registry without parsing it or neither complaining us about the typo we did .. ( really ??? an enterprise level ESB does not check the documents edited ??? )

4. After some time for some reason we have restarted the ESB. ( and here the fun began. ) ESB hangout printing an exception complaining XML parsing error when it try to load that wsdl file where the error happened. And the admin console is not loading because OSGI module that correspond to esb (Synapsys) is waiting until all the wsdl loads. (This is the biggest fault - not letting other services, and esb to start because a wsdl typo occurred where it should parse the wsdl before save it to registry )

At the end of the console we can see the following message printed time to time without starting the ESB and the services due to what happened that has been described above.

 StartupFinalizerServiceComponent Waiting for required OSGi services: org.wso2.carbon.mediation.initializer.services.SynapseEnvironmentService

And this is the exception which complains about that our wsdl could not be parsed at startup.

WstxParsingException: Illegal processing instruction target (“xml”); xml (case insensitive) is reserved by the specs. at [row,col

There were 0 documentation and no entry about how to handle such a situation.

According to this exception we guessed the work around for this problem is just to remove the erroneous wsdl file from the governance registry. But wso2esb it self should not let that wsdl added to the registry at the first place. Even it did add, there should be some way to remove the erroneous wsdl related proxy and start up the esb and other proxy services. But it does not have that feature yet.

So how the hell we edit the governance registry without esb up.

Well, there is no proper way wso2 has exposed to do that. ( i could not find any thing on the web that’s why i write this blog entry apart from that to complement wso2 esb team about this s:D )

What we did is this,

1. Shut down the esb , copy and back up the databases folder in <wso2esb_home>/repository/database

2. Download any sql viewer tool that support h2 database management system. (yes wso2 default database is java h2 DBMS - in our case we used the default DBMS ). I used SQL Squirrel with h2 database jdbc driver.

3. open/connect to the h2 database in above folder using the sql tool.

4. find the “REG_RESOURCE” and “REG_CONTENT” tables in “WSO2CARBON_DB/PUBLIC/TABLE/” location.

5. search for erroneous wsdl file name inside “REG_RESOURCE” (hint: use sql select). File name for a resource in “REG_RESOURCE” saved in “REG_NAME” column. 

6. From the resulted row note down the value of “REG_CONTENT_ID” and search for that REG_CONTENT_ID in “REG_CONTENT” table. The resulting row is where the actual wsdl file saved as sql BLOB file type.

select rc.REG_CONTENT_ID from REG_RESOURCE rr, REG_CONTENT rc where rr.REG_NAME like ‘%mywsdl.wsdl’ and rc.REG_CONTENT_ID=rr.REG_CONTENT_ID

7. To recover the wso2 esb from hanging out due to this erroneous wsdl remove the two entries related to REG_CONTENT_ID of REG_NAME of the wsdl from two tables.

8. Re start the wso2esb and you may find that exceptions are no more there and server does not hanging out any more.

Yeah i know this is bit fishy and fundamental, but if we cant access the admin console due to that ESB not finishing up loading its OSGI module (because esb let it save error-nous wsdl inside the governance registry at the first place.) we cant do much rather than cracking the database and remove the wsdl manually right ??

PS : there may be other artifacts such as proxies, sequences, endpoints using the removed wsdl, that may affected by the removal. so i instruct to remove those of which use the removed wsdl.

When WSO2 ESB does not read groups from external LDAP

Once I configured external LDAP (In my case using Apache DS) to use by both WSO2 Identity Server and WSO2 ESB, I came across a situation where WSO2 ESB does not show any of the user roles that has been created through Identity Server. Neither Identity server show any roles created in ESB.

After going through lot of configurations I found it was due to this small line. So thought to blog about this for future use of other.

Within “<UserStoreManager>” tags in your external LDAP configuration (in {wso2-esb-root}/repository/conf/user-mgt.xml folder), just add following line.

<Property name=”ReadLDAPGroups”>true</Property>

Configure WSO2 Identity server to use Apache DS LDAP

1. Configure a LDAP from Apache DS.

To configure Apache DS first download the latest version of apache directory studio from here.

Then install the setup (for windows users) and start the Apache Directory Studio.

Then go to “Window - > show view -> other ” and select “LDAP servers”.

In the server view click on “new server” and create a new server instance. Start the server by right clicking on the server instance and click on “start”.

Then go to connections tab next to the LDAP servers tab and create a new connection.

This will open “New LDAP Connection" wizard.

Enter the connection details and click next. (Here in my example i run on localhost port 10389.)

In the next window you will be asked for parameters for authentication details.

Enter the parameters for simple authentication. In the Bind DN or user field, enter the DN of the administrator’s account on the directory server (for the default instance of the Apache directory server, this is uid=admin,ou=system). In the Bind password field, enter the administrator’s password (for the default instance of the Apache directory server, the administrator’s password is secret). Click Finish

If the connection is successfully established, you should see an outline of the Directory Information Tree (DIT) in the LDAP Browser view. In the LDAP Browser view, drill down to the ou=users node, as shown.

Right-click on the ou=users node and select New Entry. The New Entry wizard appears. Leave the Entry Creation Method by clicking next. In the Object Classes pane, select inetOrgPerson from the list of Available object classes on the left and then click Add to populate the list of Selected object classes. Click Next.

In the Distinguished Name pane, complete the RDN field, putting uid in front and jdoe after the equals sign. Click Next.

Fill in the remaining mandatory attributes in the Attributes pane. Set the cn (common name) attribute to John Doe and the sn (surname) attribute to Doe. Click Finish.

To add a userPassword attribute to the user entry. In the LDAP Browser view, you should now be able to see a new node, uid=jdoe. Select the uid=jdoe node. Now, right-click in the Entry Editor view and select New Attribute. The New Attribute wizard appears. From the Attribute type drop-down list, select userPassword. Click Finish. The Password Editor dialog appears. In the Enter New Password field, enter the password, secret. Click Ok.

Point wso2 Identity server to the newly created external LDAP

Goto Identity server root folder and browse in to “wso2esb-{version}\repository\conf”. Open “user-mgt.xml” in the editor.

Un-comment the commented section about external LDAP.

Change the password of connection as following line. (since our previously entered default password is “secret” we entered that below)

<Property name=”ConnectionPassword”>secret</Property>

Run the Identity server and create a new user by signup.

If all goes correctly you may be able to add new users by Identity server and, in Apache DS you may see the newly added  users.