Services:Pluggable Services Infrastructure

From CyberGIS

Jump to: navigation, search

Contents

Objective

Develop pluggable services infrastructure to enable scientists to contribute state-of-the-art LiDAR processing tools and algorithms operating on OT data, for use by the larger community.

Technology Survey

Service Registry & Repository
It is a key piece for enhancing and achieving reuse of services in SOA.
It allows service providers/consumers to organize information about services and provide facilities to publish and discover services.
Application metadata, WSDLs and Schemas will be stored in the registry and repository
  1. WSO2 Governance Registry(http://wso2.com/products/governance-registry/)
    It is a SOA integrated registry/repository for storing and managing metadata related to service artifacts and other artifacts, for example, WSDL, XML, Word/Excel documents, JPEG Images, etc.
    It can be used to discover and manage service contracts, versioning and policies.
    It can be combined with other products (ex. ESB) to achieve service virtualization.
    Registry Web Service API is supported.
    It support the UDDI interface. Apache JUDDI has been embedded as the UDDI server.
    It seems the use of this registry will give us more potential development environments because of handling broader artifacts and supporting multiple APIs and Open standards.
  2. Apache jUDDI (http://juddi.apache.org/)
    UDDI(Universal Description, Discovery and Integration) protocol is one of the major building blocks required for successful Web services.
    It provides an infrastructure that supports the description, publication, and discovery of service providers.
    This registry is designed to store information about Businesses and Services and it holds references to detailed documentation.
  3. Membrane SOA Registry (http://www.membrane-soa.org/soa-registry/)
    It is open source Web Services Registry.
    It provides monitoring of WSDL changes and versioning.
    It is built a generic Web Services(SOAP) client for testing.
    There is no concept of repository.

Service Registry/Repository

  1. Since OT infrastructure has been built around SOAP based Web services, I have explored UDDI as a service registry in more details. UDDI registry holds WSDL service description and is used for Web services in a systematic way.
  2. In a nutshell, UDDI registry consists of four primary data types, "BusineeEntity", "BusinessService", "BindingTemplate", and "tModel". The "BusinessEntity" type provides information about a business, and contain one or more "BusinessService" types. The technical and business descriptions for a Web service are defined in a "BusinessService" type and its "BindingTemplate" types. Each "BindingTemplate" type contains a reference to one or more "tModel" types. A "tModel" type is used to define the technical specification for a service.
  3. I have installed and tested Apache jUUDI server for publishing the WSDL service metadata.
    I implemented the publish interface for the process for mapping WSDL service descriptions into a UDDI registry. The example (http://cybergis.sdsc.edu:8080/ot_cybergis/sdem_uddi.txt) contains the UDDI BusinessEntity, BusinessService, BindingTemplate, and tModel types.
  4. A typical client inquiry can be one of two objectives, one is for finding an implementation of a known interface (WSDL location) of a tModel ID, the other is for finding the updated value of the service endpoint of a known binding template ID.
  5. WSO2 Governance registry (http://wso2.com/products/governance-registry/) stores any type of data or metadata as resources including contracts, models, workflows, WSDLs, Word documents, server configurations and more. It seems to me it is flexible service registry for any type of services including REST services, JSON services, SOAP services, etc. I have installed this registry product for testing. The management console (https://cybergis.sdsc.edu:9443/carbon/admin/login.jsp) provided by this product is a web browser based user interface for configuring, managing and monitoring WSO2 Governance Registry. It can be used to create, retrieve, update and delete any type of resource(s). Through this user interface, administrators can also make configuration changes to the server. If you are sign in on it, please use the credentials (username: admin Password: admin).
  6. As other service registry system, OGC Catalogue service (http://www.opengeospatial.org/standards/cat) can be considered. OGC CSW(Catalogue Service for the Web) is one part of OGC Catalogue Service, which defines standardized catalog interfaces to discover, browse, and query metadata about geospatial data, processing services, and other potential resources. OT has been published the metadata of OT datasets as the resource type on CSW server. Please reference this wiki page (http://cgwiki.cigi.uiuc.edu:8080/mediawiki/index.php/Open_Metadata_Service_Framework). As of example, I looked into the Global Earth Observation System of Systems (GEOSS) (http://www.earthobservations.org/index.shtml). It provides the flexible network of content providers allowing decision makers to access an extraordinary range of information about data from the thousands of different instruments and services at their desk. The GEOSS registry system (http://geossregistries.info/index.htm) includes mechanisms to register services and associate them with GEOSS-recognized standards. A taxonomy of standards types is also proposed to assist in the discovery and classification of GEOSS service implementations. This registry system leverages the Geonetwork opensource (http://geonetwork-opensource.org/), which we are using for the publication of OT dataset resources, as the back-end geo-spatial cataloge.

Test Site for Service Registry

  1. I have implemented ISO-based application profile for ISO 19115 metadata for geodata/geospatial applications and ISO 19119 metadata for tightly and loosely coupled geospatial services. This profile is used for OpenGIS Catalogue service (CSW). Catalogue services for the Web (CSW) are for locating, managing, and maintaining distributed geo resources such as geospatial data, application and services. With OGC CSW, client applications are capable of discovering and binding geo resources in a standardized way and, theoretically, they are based on a well-known information model, for example, ISO metadata standards, which includes spatial references, service instance metadata, and further descriptive information that enables client applications to search and bind geo resources in more efficient way. ISO metadata model that I have chosen as the specific information model, includes the information which can be inserted in the catalogue available search terms, response/result sets, etc. The overall objective of this profile is to improve interoperability between systems/science gateways conforming to this specific profile that fits CyberGIS environments properly, even though there is no single solution for catalogue services which fits every user/community's needs. Temporarily I set up the base profile that provides a basic set of information objects when I think to be needed to publish service metadata. Here is the link for the geoportal's user guide. http://cybergis.sdsc.edu:8080/ot_cybergis/guguide/.
  2. I have prepared WSO2 governance registry for integrating service metadata with Service Oriented Architecture (SOA). Unlike the esri's geoportal that is based on ISO metadata standard and CSW protocol, WSO2 governance registry stores any metadata and information on defined metadata model. Currently, WSO2 registry provides many different service artifact types as the examples. Choosing one of types completely depends on how the service metadata is consumed in the cyberinfrastructure, or system. So, I opened some service artifacts such as API, Service, URI, WADL and WSDL. Please review them about whether or not they are useful. And I have created and added a new service model, called "CyberGIS service" for the testing, based on registered service metadata in geoportal. This is a initial draft for designing the metadata model to collect the service metadata from the service providers in CyberGIS. It is possible to edit this model based on your review comments easily. I have registered WSDL service and OGC WPS service using "WSDL" and "CyberGIS service" service artifact types. For registering your services in more details, please look into the below link: http://cybergis.sdsc.edu:8080/ot_cybergis/wso2uguide/

Service Orchestration/Composition

  1. The service orchestration plays important role in pluggable service infrastructure. With OT, it would make most sense to run LiDAR processing algorithms/apps/tools as the part of OT data ingestion workflow and create the derivative data products as part of OT data package. We can build and deploy composite SOA apps. Mainly, the Apache ODE (Orchestration Director Engine) (http://ode.apache.org/index.html) and Apache Synapse (http://synapse.apache.org/) are suggested.
  2. Apache ODE runs business processes written OASIS Standard, WS-BPEL in order to orchestrate all the web services that are part of apps. WS-BPEL(Business Process Exceution Language) (http://ode.apache.org/ws-bpel-20.html) is an XML-based language which defines a set of basic control structures and elements to invoke web services and receive messages from services. All external resources and partners are represented as WSDL services for the interaction with each partner.
  3. OGC Standard WPS(Web Process Service), http://www.opengeospatial.org/standards/wps,defines how to implement geographic calculations or models as a geospatial web service. And it defines a standardized interface that facilitates the publishing of geospatial processes, and the discovery of and binding to those processes by clients.
  4. WPS acts as a stand-alone service as well as middleware service for data, by obtaining data from an external resource in order to run a process on the local implementation. Also, WPS allows existing software interfaces and resources to be wrapped up, and presented to the network as web services.
  5. The WPS interface specifies three operations: "GetCapabilities" for service metadata documents, "DescribeProcess" for detailed information about the processes that can be run on the service instance, and "Execute" for running a specified process using input(s)/output(s) parameter values.
  6. Service chaining with WPS: A WPS process is normally an atomic function as a web service. WPS processes can be incorporated into service chains in a number of ways. A BPEL engine can be used to orchestrate a service chain that includes one or more WPS processes. A WPS process can be designed to call a sequence of web services including other WPS processes, thus acting as the service chaining engine.
  7. WPS may not be appropriate as the service orchestration/composition engine. For the data injection workflow using SOAP based Web services, WS-BEPL (or BPEL) may be appropriate.
  8. OPAL WSDL as the service interface defines 12 operations. Mainly three operations are used for accessing the OPAL service in an application. First, the "launchJob" operation launches the job for running the application. Second the "queryStatus" operation queries the status of a running job. Finally, the "getOutputs" operation returns the outputs from a job. A sequence of three operations is performed on an application. Using Apache ODE that is a system for executing business process using the WS-BPEL 2.0 OASIS standard and the BPEL4WS 1.1 vendor specification, this sample example is developed for the data injection workflow with LiDAR data processing algorithms.
  9. Development and testing examples using Apache ODE(Orchestration Director Engine) software (http://ode.apache.org/index.html).
    • Get the ODE WAR file(ode.war) and copy it into Tomcat's webapp directory. Currently, I am using tomcat 6 with apache ode 1.3.5 version. Start tomcat and ODE is up and running. The development server is http://cybergis.sdsc.edu:7070/ode/index.html.
    • Example 1: The example shown in Figure Schematic Diagram accesses the external Web service, that is, one of OPAL Web services using WS-BPEL. The raster service for accessing standard DEM is used for the development. It invokes this Web service to get the application metadata information. Mainly WS-BPEL consists of "receive" activity, "invoke" activity, and "reply" activity.The "receive" and "reply" activities does a blocking wait for a matching message to arrive / send a message in reply. And the "invoke" activity invokes a one-way or request-response operation of the Web service.
      • Download Source file.
      • Deployment: copy uncompressed source directory to tomcat/webapps/ode/WEB-INF/processes/ directory, this deploys the process automatically.
      • Test: use the "apache-ode-war-1.3.5/bin/sendsoap" command line to send test messages. For example, "apache-ode-war-1.3.5/bin/sendsoap http://localhost:7070/ode/processes/WebserviceProcessServicePort testRequest.soap".
    • Example 2: The example shown in Figure Schematic Diagram has a set of control structures to invoke OPAL Web Service and receive and reply messages from services. The raster service for accessing standard DEM is used for the development. After getting incoming messages on "receive" activity, the "launchJob" operation on "invoke" activity is invoked for submitting the job. After getting the job result such as the job id and job status, sequentially, the "queryStatus" operation is invoked for checking the job status every 5 second in this example. Depending on the job status, the different messages are replied to the client.
      • Download Source file.
      • Deployment: copy uncompressed source directory to tomcat/webapps/ode/WEB-INF/processes/ directory, this deploys the process automatically.
      • Test: use the "apache-ode-war-1.3.5/bin/sendsoap" command line to send test messages. For example, "apache-ode-war-1.3.5/bin/sendsoap http://localhost:7070/ode/processes/launchjobService testRequest.soap".

Template-based Service Orchestration/Composition

  1. Main Approach: It is designed to create WS-BPEL processes in a modular and dynamic way based on precompiled templates. The generated process can be packaged and deployed on a BPEL engine on the fly before it will be executed. The template framework for designing the WS-BPEL processes allows the interpretation and combination of templates based on parameters. For this framework, a generic master process can be defined as the main template. It represents a generic workflow where requirements can be modeled without being bound to specific tasks. The concrete modules of WS-BPEL activities for invoking Web services are modeled as templates and stored in a template library. A template library contains combinations of WS-BPEL activities that perform Web service functionality. The templates with the generic master process together are pieces for the controller that generates a tailored WS-BPEL process. The controller composes the generic master WS-BPEL process by interpreting various parameters associated with WS-BPEL activities and by selecting appropriate templates from the template library.
  2. Implementing blocks
    • Design and development of WS-BPEL sub-process for Opal Web service.
      • Opal toolkit provides a mechanism to wrap up any command line application as a Web Service. Since its service provides job and data management, and state management, there is a predictable workflow behavior from job submission to job completion. An Opal-specific WS-BPEL process to ease Web service composition is developed. This Schematic Workflow Diagram shows the start and end of a opal job process, invoking three operations.
    • Template Definition: The syntax used for defining the templates that describing WS-BPEL activities.
    • Template Library: the storage available for the design of the workflow. It will contains templates for developed Opal Web services, for example.
    • Generic Master WS-BPEL process: As the main service template, this process is designed as a regular process and will contain the replaceable parameters.
    • Template Manager: It controls the logic that decides which templates to use, generates executable WS-BPEL process, deploys this process, manages the BPEL processes on runtime environments, and etc.