Ruby + SOAP4R + WSDL Hell

I’ve been spending a bit of time lately playing around with Ruby on Rails the last couple days, giving it a bit of a test to see what everyone’s raving about and understand how it could be used. Overall it’s a pretty impressive framework, and worth a little time investment to give it a whirl. For those without the patience, I highly recommend viewing the impressive presentation that demonstrates development of a blogging tool in 15 minutes.

That said, all is not sunshine and chocolate in the world of Ruby. I spent the better part of today trying to get some basic SOAP functionality operating to allow me to interact with Amazon Web Services’ E-Commerce Service. I was using Hiroshi Nakamura’s soap4r library to auto-generate Ruby class definitions from a WSDL file, but I was running into a bit of pain. There seems to be next to no information out there about how to use soap4r and the associated wsdl2ruby class generation utility, and even less about the current shortcomings of the current stable release of the library. In the interest of saving someone a day of time, I thought I’d put together some details about the wsdl2ruby tool, the files it generates, and what does and doesn’t work.

To complete this exercise, I assume you have already:

  1. Downloaded and installed Ruby (I used 1.8.2)
  2. Downloaded and installed soap4r (I used 1.5.5)
  3. Downloaded and installed http-access2 (I used 2.0.6)
  4. Signed up for an Amazon Web Services developer token (it’s free)

Just a disclaimer: I’m no whiz in the whole SOAP/WSDL arena, but I think the information I’m about to provide you with will be enough to help you figure out what’s going on when using soap4r. Your mileage may vary.

Generating Classes with wsdl2ruby

To create applications capable of accessing web services via SOAP, you could compose raw SOAP requests yourself (see the “Behind the Screens” article for more detail), but that would be a bit painful and require a fair amount of manual labor. I’m a lazy, lazy programmer, and I’m betting you’re the same.

A better approach is to use a framework that can automatically generate class definitions for a framework that can be used to create objects, map those objects to SOAP, and vice-versa. This is exactly what soap4r and the wsdl2ruby provides. Using soap4r, a developer can easily generate both client and server classes to handle consuming and providing SOAP-accessible services. For my purposes, I’m only interested in generating client code to allow me to develop an application that can consume services.

The wsdl2ruby application does exactly what it name implies: it takes a Web Services Description Language definition of a web service, and transforms it into Ruby code. For this exercise, I’m going to use the 2006-03-08 WSDL definition of the Amazon Web Services’ E-Commerce Service web service available here.

To generate client code for the Amazon.com web service from the WSDL description, run wsdl2ruby like this (your platform may require the path to be set appropriately):
wsdl2ruby.rb --wsdl http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl --type client --force

This command will generate three files:

  • AWSECommerceServiceClient.rb: An example client that provides skeleton code for exercising the web service. This code can’t really be run “out-of-the-box”, something I’ll talk about in a moment.
  • default.rb: The set of class definitions for all elements defined by the WSDL file. Using these class definitions, a developer will be able to produce and consume the various building blocks required to interact with the web service without needing to search an XML tree, or perform any other similar ugliness.
  • defaultDriver.rb: This file contains a single class, AWSECommerceServicePortType, which is used to conduct all requests of the web service.

Although I won’t be doing it for this exercise, you could easily rename default.rb and defaultDriver.rb as you see fit; however, you’ll have to make sure to update any require statements to reflect your new naming.

Using the Generated Sample Client

AWSECommerceServiceClient.rb provides a skeleton application that initializes a AWSECommerceServicePortType object:

#!/usr/bin/env ruby
require 'defaultDriver.rb'

endpoint_url = ARGV.shift
obj = AWSECommerceServicePortType.new(endpoint_url)

# run ruby with -d to see SOAP wiredumps.
obj.wiredump_dev = STDERR if $DEBUG

and then uses it to call each of the operations made available by the web service. The wsdl2ruby application auto-generates a skeleton for each operation that looks something like this:

# SYNOPSIS
#   ItemLookup(body)
#
# ARGS
#   body            ItemLookup - {http://webservices.amazon.com/AWSECommerceService/2006-03-08}ItemLookup
#
# RETURNS
#   body            ItemLookupResponse - {http://webservices.amazon.com/AWSECommerceService/2006-03-08}ItemLookupResponse
#
body = nil
puts obj.itemLookup(body)

Notice that body is set to nil, whereas the web service requires an ItemLookup object to work. To make this code work, you’d need to create an ItemLookup object – as it turns out, ItemLookup relies on ItemLookupRequest, so you’ll have to create one of those as well. The parameters required to create these objects are determined by the WSDL definition, and the order of parameters to pass to new are documented in part in default.rb; the meaning of those parameters are given in the Amazon Web Services’ E-Commerce Service API documentation.

As an example, let’s say I want to perform a simple lookup for an item with an ASIN of B00005JLXH – which just happens to be the unique Amazon identifier for Star Wars, Episode III (it was the first thing I saw on the Amazon home page, I swear). First I create the specific ItemLookupRequest object for that item:

itemLookupRequest = ItemLookupRequest.new("", "", "", "", "", "", "", ["B00005JLXH"], [], "", "", "", "")

Note that the class constructor generated by wsdl2ruby requires all parameters to be specified (their default value is nil), so you need to provide all the parameters. Use empty strings for the ones you don’t need or want to provide.

Next, I create an ItemLookup object, adding both my developer token and the ItemLookupRequest object I created above.

body = ItemLookup.new("", "Your Amazon Web Services developer token goes here", "", "", "", "", "", [itemLookupRequest])

Finally, I call the itemLookup method on my AWSECommerceServicePortType instance to execute the call to the web service, and print some output using the resulting ItemLookupResponse object as the source of the response data:

itemLookupResponse = obj.itemLookup(body)
 
itemLookupResponse.Items.each do |item|
  item.Item.each do |innerItem|
    puts "ASIN: #{innerItem.ASIN}"
    puts "Detail Page URL: #{innerItem.DetailPageURL}"
    puts "Title: #{innerItem.ItemAttributes.Title}"
  end
end

If you run the AWSECommerceServiceClient application with the –d option and you’ll see should see something like this:
Wire dump:

= Request

! CONNECT TO soap.amazon.com:80
! CONNECTION ESTABLISHED
POST /onca/soap?Service=AWSECommerceService HTTP/1.1

SOAPAction: "http://soap.amazon.com"

Content-Type: text/xml; charset=us-ascii

User-Agent: SOAP4R/1.5.5 (/114, ruby 1.8.2 (2004-12-25) [i386-mswin32])

Date: Sat Apr 01 19:59:04 Pacific Standard Time 2006

Content-Length: 1136

Host: soap.amazon.com

< ?xml version="1.0" encoding="us-ascii" ?>
<env :Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
</env><env :Body>
<itemlookup xmlns="http://webservices.amazon.com/AWSECommerceService/2006-03-08">
<marketplacedomain></marketplacedomain>
<awsaccesskeyid> Your Amazon Web Services developer token goes here</awsaccesskeyid>
<subscriptionid></subscriptionid>
<associatetag> </associatetag>
<validate></validate>
<xmlescaping></xmlescaping>
<shared></shared>
<request>
<condition></condition>
<deliverymethod></deliverymethod>
<futurelaunchdate></futurelaunchdate>
<idtype></idtype>
<ispupostalcode></ispupostalcode>
<merchantid></merchantid>
<offerpage></offerpage>
<itemid>B00005JLXH</itemid>
<reviewpage></reviewpage>
<searchindex></searchindex>
<searchinsidekeywords></searchinsidekeywords>
<variationpage></variationpage>
</request>
</itemlookup>
</env>

= Response

HTTP/1.1 200 OK

Date: Sun, 02 Apr 2006 04:00:37 GMT

Server: Server

x-amz-id-1: 1KSRGANDWP7TE6SSCSPP

x-amz-id-2: aO3Y8m2+yt5uR7NPGKwLAeY6RG9w0BfZ

nnCoection: close

Transfer-Encoding: chunked

Content-Type: text/xml; charset=UTF-8

607

< ?xml version="1.0" encoding="UTF-8"?><soap -ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"></soap><soap -ENV:Body><itemlookupresponse xmlns="http://webservices.amazon.com/AWSECommerceService/2006-03-08"><operationrequest><httpheaders><header Name="UserAgent" Value="SOAP4R/1.5.5 (/114, ruby 1.8.2 (2004-12-25) [i386-mswin32])"></header></httpheaders><requestid>1KSRGANDWP7TE6SSCSPP</requestid><arguments><argument Name="Service" Value="AWSECommerceService"></argument></arguments><requestprocessingtime>0.0273821353912354</requestprocessingtime></operationrequest><items><request><isvalid>True</isvalid><itemlookuprequest><itemid>B00005JLXH</itemid></itemlookuprequest></request><item><asin>B00005JLXH</asin><detailpageurl>http://www.amazon.com/exec/obidos/redirect?tag=brendonwilson-20%26link_code=sp1%26camp=2025%26creative=165953%26path=http://www.amazon.com/gp/redirect.html%253fASIN=B00005JLXH%2526tag=brendonwilson-20%2526lcode=sp1%2526cID=2025%2526ccmID=165953%2526location=/o/ASIN/B00005JLXH%25253FSubscriptionId=0VS96BNQBVY904T3XZ02</detailpageurl><itemattributes><actor>Hayden Christensen</actor><actor>Ewan McGregor</actor><actor>Natalie Portman</actor><productgroup>DVD</productgroup><title>Star Wars, Episode III - Revenge of the Sith (Widescreen Edition)</title></itemattributes></item></items></itemlookupresponse></soap>

0

Once the request completes, the itemLookup method will return an ItemLookupResponse object, which will allow you to programmatically access all of the returned data in a simple programmatic fashion via the accessor methds generated by wsdl2ruby. In theory.

In theory, Communism works. In theory.

Unfortunately, wsdl2ruby is still seems to be a work in progress, and therefore doesn’t work as cleanly “out of the box” as one might like. For one thing, it seems to have some difficult with complex types. The example above will undoubtedly choke with something like:


C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:202:in `const_from_name': private method `sub' called for nil:NilClass (NoMethodError)
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:221:in `class_from_name'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:302:in `add_elements2stubobj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:298:in `each'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:298:in `add_elements2stubobj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:283:in `soapele2stubobj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:265:in `any2obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:59:in `soap2obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:146:in `_soap2obj'
... 11 levels...
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:178:in `call'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:232:in `help'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:227:in `help'
from D:/Permanent Backup/Development/soap4r-1_5_5/sample/wsdl/test/AWSECommerceServiceClient.rb:20

This problem arise from the lack of a class name being generated for the OperationRequest associated with an ItemLookupResponse. As defined by the WSDL, an ItemLookupResponse is defined as:

<xs:element name="ItemLookupResponse">
<xs:complexType>
<xs:sequence>
<xs:element ref="tns:OperationRequest" minOccurs="0"/>
<xs:element ref="tns:Items" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>

Which results in the following code:

class ItemLookupResponse
  @@schema_type = "ItemLookupResponse"
  @@schema_ns = "http://webservices.amazon.com/AWSECommerceService/2006-03-08"
  @@schema_qualified = "true"
  @@schema_element = [["operationRequest", [<strong>nil</strong>, XSD::QName.new("http://webservices.amazon.com/AWSECommerceService/2006-03-08", "OperationRequest")]], ["items", [<strong>nil</strong>, XSD::QName.new("http://webservices.amazon.com/AWSECommerceService/2006-03-08", "Items")]]]

The problem here is the two highlighted nil class names generated by wsdl2ruby. The ClassDefCreator class in soap4r is responsible for generating this class definition – I took a lookup at the definition and found the following issue in dump_classdef (follow along in your own install of Ruby, in {ruby install path}lib/ruby/1.8/wsdl/soap/classDefCreator.rb):

if element.type == XSD::AnyTypeName
  type = nil
elsif klass = element_basetype(element)
  type = klass.name
elsif element.type
  type = create_class_name(element.type)
else
  type = nil      # means anyType.
  # do we define a class for local complexType from it's name?
  #type = create_class_name(element.name)
  # &lt;element&gt;
  #   &lt;complextype&gt;
  #     &lt;seq ...&gt;
  #   &lt;/seq&gt;&lt;/complextype&gt;
  # &lt;/element&gt;
end

The problem is that last type = nil statement. Basically, if ClassDefCreator encounters a complex type in the WSDL, it assigns a nil type. What’s odd is that there appears to be a perfectly good solution currently commented out of the code. If we change:

type = nil

to

type = create_class_name(element.name)

and regenerate the only the class definitions using wsdl2ruby:


wsdl2ruby.rb --wsdl http://webservices.amazon.com/AWSECommerceService/AWSECommerceService.wsdl --classdef –force

then the definition of schema_element in the ItemLookupResponse class changes to:

  @@schema_element = [["operationRequest", ["OperationRequest", XSD::QName.new("http://webservices.amazon.com/AWSECommerceService/2006-03-08", "OperationRequest")]], ["items", ["Items[]", XSD::QName.new("http://webservices.amazon.com/AWSECommerceService/2006-03-08", "Items")]]]

Now, the soap4r framework will be able to find the appropriate class to use to represent the OperationRequest when transforming the response SOAP XML into a object. I’m a little puzzled why this fix is currently commented out in ClassDefCreator – I assume there’s probably a good reason. In all likelihood, this solution is probably commented out because the class name alone isn’t enough to avoid namespace clashes. I’m sure for a more complicated application consuming several web services this would undoubtedly be an issue, but for my purposes, this is not an issue.

Everything works flawlessly. Kinda.

Running the AWSECommerceServiceClient with these changes in place, the application throws another error:

C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:71: warning: Object#type is deprecated; use Object#class
C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/wsdlliteralregistry.rb:71:in `soap2obj': cannot map SOAP::SOAPElement to Ruby object (SOAP::Mapping::MappingError)
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:146:in `_soap2obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:59:in `soap2obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:55:in `protect_threadvars'
from C:/Program Files/ruby/lib/ruby/1.8/soap/mapping/mapping.rb:55:in `soap2obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:479:in `response_doc_lit'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:478:in `collect'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:478:in `each'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:478:in `collect'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:478:in `response_doc_lit'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:444:in `response_doc'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:348:in `response_obj'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/proxy.rb:149:in `call'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:178:in `call'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:232:in `help'
from C:/Program Files/ruby/lib/ruby/1.8/soap/rpc/driver.rb:227:in `help'
from D:/Permanent Backup/Development/soap4r-1_5_5/sample/wsdl/test/AWSECommerceServiceClient.rb:20

Further investigation reveals that wsdl2ruby did not generate a Header class, required as part of the HTTPHeaders returned as part of the ItemLookupResponse:

<xs:element name="HTTPHeaders">
<xs:complexType>
<xs:sequence>
<xs:element name="Header" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:attribute name="Name" type="xs:string" use="required"/>
<xs:attribute name="Value" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>

Hence, soap4r is unable to map the returned Headers XML to a Headers class. For some reason, wsdl2ruby seems to choke on nested complex type definitions. Saving the WSDL as a local file and changing the code above to:

<xs:element name="HTTPHeaders">
<xs:complexType>
<xs:sequence>
<xs:element ref="tns:Header" minOccurs="0" maxOccurs="unbounded"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element name="Header" minOccurs="0" maxOccurs="unbounded">
<xs:complexType>
<xs:attribute name="Name" type="xs:string" use="required"/>
<xs:attribute name="Value" type="xs:string" use="required"/>
</xs:complexType>
</xs:element>

And regenerating the class definitions using

wsdl2ruby.rb --wsdl AWSECommerceService.wsdl --classdef –force

tricks wsdl2ruby into generating the correct Header class definition. As this similar construct exists throughout the WSDL, I performed similar changes throughout – primarily I changed the Arguments WSDL definition to make sure an Argument class is properly generated.

With those changes in place, regenerate the class definitions as before, and run the AWSECommerceServiceClient. This time it should provide the desired output:

ASIN: B00005JLXH
Detail Page URL: http://www.amazon.com/exec/obidos/redirect?tag=ws%26link_code=sp1%26camp=2025%26creative=165953%26path=http://www.amazon.com/gp/redirect.html%253fASIN=B00005JLXH%2526tag=ws%2526lcode=sp1%2526cID=2025%2526ccmID=165953%2526location=/o/ASIN/B00005JLXH%25253FSubscriptionId=0VS96BNQBVY904T3XZ02
Title: Star Wars, Episode III - Revenge of the Sith (Widescreen Edition)

Ah, Closure

Lesson of the day: my pain is your gain. While the wsdl2ruby utility is not fully baked to handle the full flexibility provided by WSDL, it can be cajoled into doing the right thing to get the results you desire. Although I was able to get my simple example working, I’m sure there’s any number of esoteric cases that the soap4r libraries don’t currently handle. Until they do, you’ll have to tinker a bit to get them working. Good luck!

Less Software, Please

A couple weeks back, I watched in horror as Microsoft Word insisted on butchering a contract that I was working on with my company’s general counsel. No matter what he did, Word unhelpfully shifted formatting and mangled clauses. An arsenal of editing tools, over a dozen years in the making, and yet it still fails making a basic word-processing task simple.

Now, in the past I’ve blamed the user for not learning how to use their tools. Sure, I thought, office productivity software requires some education, but even a pencil requires effort to learn how to use (remember grades one through seven) – why should general office productivity be any different? Shouldn’t we expect some general level of proficiency from a user?

Therein lies the problem – the user needs to invest in learning a set of generalized functionality that far exceeds the actual functionality they need to achieve their given task. The company lawyer doesn’t need a generalized word-processor, he needs an application that understands the standard form of legal contracts (clauses, sections, subsections), and provides limited spell-checking and redlining capabilities. Instead, Word provides a plethora of infinitely reconfigurable editing and formatting options that simply make no sense in the context of the production of a contract. The application is simply too powerful.

Users are paying for it – not only in terms of the price of a piece of software that contains more functionality than they could even hope to use, but also in terms of lost productivity.


The Innovator's Dilemma
Prior to this realization, I didn’t really think online word-processing tools like Writely, ZohoWriter, and Writeboard could hope to compete against Word. Now I’m not so sure. A little refresher of the tenets of disruptive innovation highlights the characteristics that indicate online word-processors may have what it takes to undermine Word over the long term. According to “The Innovator’s Dilemma“, disruptive technologies generally:

  • Underperform established products in established markets
  • Are cheaper, simpler, smaller, and, frequently, more easy to use
  • Provide features that a few fringe (and generally new) customers value

The last point is the critical one – providing new features that a few fringe customers value. Perhaps profession-specific toolsets might be one such way to achieve such differentiation from the behemoth that is Microsoft Word. Provided they don’t give Microsoft too many free ideas along the way.