Government Information Sharing and Interoperability via Data Services and SOAJune 21, 2006 Rob Cardwell CTO, MetaMatrix rcardwell @ metamatrix.com
Government Information Sharing and Interoperability via Data Services and SOA
June 21, 2006 Rob Cardwell CTO, MetaMatrix rcardwell @ metamatrix.com
1
Agenda
MetaMatrix Architectural Fit MetaMatrix Products & Capabilities Achieving Information Sharing Service Enabling Data Assets (Data Services) Resolving Semantics Enabling Enterprise (Deep) Search Customers Conclusions
2
MetaMatrix Enterprise Data Services
3 Project-level or Enterprise-wide data services layer Integrated views of data from multiple sources Metadata-driven Optimized performance Interoperable security Complements BI, ETL, ESB/EAI, DQ, CDI, Search
3 Problem: data sources throughout the enterprise, vs. the need to use these data sources for purposes other than for what they were intended Solution: data services – abstraction/decoupling of the data sources from the consumers of the data Features: each of the bullet points on the slide, some more relevant than others, depending on the audience Scope: solution can scale to enterprise deployment where appropriate; many customers start by deploying data services that are more project-specific or COI-specific
MetaMatrix Fit in FEA DRM Data Sharing
4 DRM Version 2 Data Access Services Context Awareness Services Structural Awareness Services Transactional Services Data Query Services Content Search and Discovery Services Retrieval Services Subscription Services* Notification Services*
4 What are Data Services? What kinds of Data Services are there? Version 2 of the FEA DRM does a good job of defining the types of data services that might exist, both within and across COI’s Metadata vs. Data – 1st 2 bullet points = metadata Read vs. Write – bullet points 3 & 4 Structured vs. Unstructured – bullets 3&4 vs. 5&6 Pull vs. Push – bullets 4,5,6 vs. 7,8
Data Service Layer in SOA
5 ESB Client Process & Applications Data Sources Data Services Layer Message Services (ESB) Business Services Business Process Services App App App App App App Data Service Data Service Data Service Data Service Data Service Data Service
5 Some analysts and organizations think of the ESB not as a single product or vendor, but as the sum of all products/vendors that implement the stack Most agree that no single vendor will provide the whole stack For inter-agency SOA, separate data services layer allows agencies to retain control of their data and access to it MetaMatrix is generally regarded as a best-of-breed data services vendor – for creating, deploying, and managing data services
Agenda
MetaMatrix Architectural Fit MetaMatrix Products & Capabilities Achieving Information Sharing Service Enabling Data Assets (Data Services) Resolving Semantics Enabling Enterprise (Deep) Search Customers Conclusions
6
MetaMatrix Products
MetaMatrix Integration Server Information Consumers Web Svc XML RDBMS Packaged Connectors Siebel, SAP Oracle Apps IMS IDMS MetaMatrix Catalog MetaMatrix Designer - Design and deploy data services Query Processor Processor Optimizer Integration Server Virtual Data Bases VDB VDB Integrated Security Users Roles Entitle ments Access Models Views XML Docs … Services in out proc MetaMatrix Connector Framework
7
MetaMatrix Approach to Data Services
8 Designing data services xml databases warehouses spreadsheets services geo-spatial rich media … Enterprise Information Sources (EIS) Information Consumers Reusable, Integrated Data Objects Exposed Data Services (contract) (contract) (contract) Custom Apps Web Services, Business Processes Packaged Apps Reporting, Analytics EAI, Data warehouses ODBC JDBC SOAP Logistics Intelligence
8 Re-useable data services Enterprise-wide data abstraction layer Integrated views of data from multiple sources Metadata-driven Optimized performance Interoperable security Complements other tools (ETL, EAI, ESB, DQ, BI)
MetaMatrix Product Lines
9 MetaMatrix Enterprise MetaMatrix Dimension MetaMatrix Enterprise Web services & SQL Modeling enterprise data Scalable deployment server Metadata management Application/legacy connectors MetaMatrix Dimension Web service-enablement of data sources Expose business views as XML Lightweight modeling – rapid integration Standard WAR-based deployment Enterprise Project, Node
9 Introducing Dimension Historically Enterprise or Query, where Query = the embeddable engine (minus the design tool) for MM software partners looking to bundle MM for std relational DB’s Now Dimension doesn’t have all the bells/whistles of Enterprise - instead is solely focused on providing data services solution for information sharing purposes Single-purpose, therefore more intuitive Designer tool Smaller footprint, lower cost Rapid creation and deployment of web service interfaces to databases Also good for distributed nodes in a larger info sharing network (federal vs. state/local)
Information Exchange Architecture
Search Engine Index / Metadata Catalog Master Data Person / Facility / Vehicle MetaMatrix Enterprise Stage SOA App’s Federal Agencies Data Access Services SQL, Web Service/XML Staged Data (optional) Ontology Mgmt / Reasoning Enterprise Service Bus MetaMatrix Dimension State/Local State/Local Security/Authentication Operations Management Error / Exception Management Orchestration Encryption High Availability Mediation XSLT, Multi-source
10
MetaMatrix Dimension Differentiators
Dimension adds the following capabilities to an ESB… Rich, advisor-based, model-driven design tool Ability to leverage data models and manage metadata Clear way to visualize and define mappings between non-XML sources and XML views (even for complex industry schemas – NIEM, GJXDM, HL7, XBRL) Ability to do SQL-based transformations, not just XSLT (including multi-source, complex joins and unions) Query planner/optimizer that makes intelligent decisions about whether to execute transformations "at the source" vs. "on the bus" Automated semantic matching & generation of transformations Data Services to connect ESB’s to Enterprise Data
11
Agenda
MetaMatrix Architectural Fit MetaMatrix Products & Capabilities Achieving Information Sharing Service Enabling Data Assets (Data Services) Resolving Semantics Enabling Enterprise (Deep) Search Customers Conclusions
12
Mediation: XML From Non-XML Sources
13 «XML» … … «Text File» «Relational» «Application» MetaMatrix: Mapping from Data to XML Source: Data Sources containing Information to integrate Target: Fixed (complex) XML Schema Need: Data complying to schema’s vocabulary T
13 One of the biggest challenges within an SOA environment is to take data from one or more relational or legacy sources and transform it to XML. Often the XML output needs to conform to a specific data model or schema such as FXml and NIEM. MetaMatrix can easily map data from any relational, file or legacy source to the required XML format – all without coding. This mediation capability is one of the most important roles of the data services layer.
Map Data Sources to XML & Deploy
Model XML Docs, Schemas Build XML Doc. models from XML Schemas Map XML Doc. models to other data models Enable data access via XML MetaMatrix Designer – for XML-centric Data Services
14
Rapid Web Service-Enablement
15 MODEL Web service 1 PACKAGE Web service 2 DEPLOY Web service 3 Dimension Designer Model-based Expose multiple sources, integrated Expose business views of data Output = integration container (vdb) Standard WAR file bundling: Integration container Query engine Web service fully defined Deploy WAR file to Web Server Executable Web service Access via SOAP Data exposed using standard vocabulary Web Svc App (WAR) MetaMatrix Query Engine Web Server
15 Model & test, package, then deploy – rapid web services development - Different approaches discussed in the next section Avoid writing brittle code Benefit from the same query optimization and performance as Enterprise engine Maintain visibility and traceability down to the element level, because 2 things are certain Data sources will change Information sharing standards will evolve and expand
MetaMatrix Dimension Modeling
16 Rapid design & deployment of Web Services Expose integrated data as XML-based business views Deployment of Web Services as standard Web apps Runtime execution optimized through use of MetaMatrix Query Engine Dimension Models Web Server Data Sources Business Views Web Service Operations WSDL XSD Source Models Deploy Import Map Model WAR as to to
16 Quick design/deploy Use std vocabulary Use web services standards – XML, XSD, SOAP, WSDL Optimized performance
Dimension – Choose your approach
17 Rapid design & deployment of Web Services Expose integrated data as XML-based business views Deployment of Web Services as standard Web apps Runtime execution optimized through use of MetaMatrix Query Engine Dimension Models Web Server Data Sources Business Views Web Service Operations WSDL XSD Source Models Deploy Import Map Model WAR as to to Start Here? Start Here?
17 Two ‘typical’ use scenarios: Bottom-up: expose relational-style sources in ‘table-like’ form Top-down: starting from XSD/XML-based business views, typically based on COI schemas (GJXDM, NIEM, HL7), and sometimes starting from existing WSDL definition of Web service operations
Secure Access – Accredited
MetaMatrix Client App username password Membership Provider username password authenticates Connector Connector Data Source Optionally accesses source-specific information source- specific trusted payload MetaMatrix Client App Membership Provider Authentication Service logon info authenticates, generates payload trusted payload payload trusted payload authenticates, optionally modifies payload payload Username/Password Logon Connector connects with same ID for all queries Optional: Integrated with existing authentication system Trusted Payload Logon: Connector uses different credentials per connection, per query Optional: Integrated with existing authentication system Data Source
18
Agenda
MetaMatrix Architectural Fit MetaMatrix Products & Capabilities Achieving Information Sharing Service Enabling Data Assets (Data Services) Resolving Semantics Enabling Enterprise (Deep) Search Customers Conclusions
19
COI Data Dictionary
T Authoritative Sources: Mapped to logical Multiple Internal/External Information Sources Application views of information: Relational, XML T T XML Document … T T T ODBC/JDBC JDBC SOAP Web Services Search Applications Business Intelligence Applications Logical Data Model: Agency or COI-specific Rationalize, harmonize, mediate C2, Logistics, Intelligence, … bldg_id SITENUM Facility_ID Location_ID bldg_type Depot_Number Location_Type
20
Comments