Hi All,

In my project I have an requirement to encrypt a zip file. Any suggestion how to encrypt a Zip file through BODS job Or to encrypt the flat file before Zipping through BODS job/Script ? Your help is awaited. Thanks.

Hi,

I am new to BODS. I have below requirement to loop through the input records in one run.

Input Column and Data:

L_DETAIL

600000

500000

400000

600000

T_KEY_SEQ=IF(SUBSTR(L_DETAIL,1,1)='6', $G_T_KEY_SEQ+1,$G_T_KEY_SEQ)

I need output like below

L_DETAL   T_KEY_SEQ

600000        1

500000       0

400000       0

600000       2

There are scenarios where you are required to delete files older than certain days from archive folder using Data Services. Below script will make that happen which will work on UNC paths.

Replace 10 with number of days files to be kept, you can use variable. Any files older than this will be deleted.

$GV_FILES_DELETED =FN_DOS('PushD "[$GV_ARCHIVE_DIRECTORY]" && (forfiles -s -m *.* -d -10 -c "cmd /c echo @file") & PopD');
Print( 'Files that will be deleted from Archive folder are ' || $GV_FILES_DELETED );
$GV_FILES_DELETED =FN_DOS('PushD "[$GV_ARCHIVE_DIRECTORY]" && (forfiles -s -m *.* -d -10 -c "cmd /c del /q @path") & PopD');
Print('Files Older than ' || $GV_FILE_RETENTION_DAYS || ' days are being deleted now');

FN_DOS will have following code

DS 4.2 SP05 INSTALLATION:

Architecture:

The following figure outlines the architecture of standard components of SAP BO Data Services.

• Run setup.exe

Prerequisite check

• The installation program checks for the required components review the results and decide whether to continue with the installation
• Click Next to continue

Deployment platform message

• Click Next to continue

Welcome Screen

• Review the recommendations made by the installation program Welcome Screen.
• Click Next to continue

License Agreement

• Click next to continue

Product Registration

• Enter the key & Click next to continue

Specify the Installation Path

• Click next to continue

Choose Language Packages

• If need more languages, select here
• Click next to continue

CMS Configure Information: CMS for use with the ES Repository

CMS - Central Management Service, a process running as part of your BO Enterprise servers, including the CMS database, authenticating users, storing access rights, etc. The CMS is the heart of a BO Enterprise system. Provide the CMS User details

and Click next to continue

PW: *********

Choose - Yes

Installation Type Selection

• Choose Install with default configuration
• Click next to continue

Select Features

• Choose the Features that need to Install
• Click next to continue

Specify Repository Database Type

• Choose Repository Database Type
• Click next to continue

Specify JBDC Driver

JDBC driver is a software component enabling a Java application to interact with a database. JDBC drivers are analogous to ODBC drivers.

• Browse the JDBC Driver.
• Click next to continue

Repository Database Connection

Specify the connection information for the database to create repository

• Provide Database Server Version, Server name and Port
• Username and Password
• Click next to continue

PW : *********

Login Information

• Provide the User details
• Click next to continue

PW : *********

Solution Manager Integration

• Choose if you want to integrate with Solution Manager
• Click next to continue

Feature Summary

• Provide the Installed and Not Installed Features
• Click next to continue

Installation Finished

• Choose Finish

IS 4.2 SP05 INSTALLATION

The SAP BO Information Steward application provides the tools you have to understand and analyse the trustworthiness of you are enterprise information. With integrated data profiling and metadata management functionality, the solution provides continuous insight into the quality of you are data. Giving you the power to improve the effectiveness of you are operational, analytical, and governance initiatives.

Data profiling - Improve information trustworthiness and reduce the risk of propagating bad data.

Metadata management - Consolidate, integrate, and audit your metadata from all relevant sources.

Root cause and impact analysis - Determine the origin of data quality problems and how they impact downstream processes or information assets.

Validation rule management - Define data validation rules against data sources and apply rules continuously to monitor data quality.

Creation of a metadata business glossary - Promote a common understanding and acceptance of business terms and build a central location for organizing them.

Development of cleansing packages - Create and reuse the rules, patterns, and dictionary that comprise data cleansing packages.

• Run setup.exe

• Choose Yes

Prerequisite check

• The installation program checks for the required components review the results and decide whether to continue with the installation
• Click Next to continue

Deployment platform message

• Click Next to continue

Welcome Screen

• Review the recommendations made by the installation program Welcome Screen

  Click Next to continue

License Agreement

• Click next to continue

Product Registration

• Enter the key & Click next to continue

Specify the Installation Path

• Click next to continue

Choose Language Packages

• If need more languages, select here
• Click next to continue

To be continued.....

ODBC/NETWORK CONNECTION PROTOTYPE FOR SQL ANYWHERE 16

1. Download Sybase SQL Anywhere 16 from the link Sybase Sql Anywhere 16

2. Install SQL Anywhere 16 with default settings, which installs Sybase Central

3. Open Sybase Central from Start→ Programs → SQL Anywhere 16 → Sybase Central

4. Connect to the SQL Anywhere server 16 Database from Sybase Central → Connections → Connect with SQL Anywhere 16 database

5. Enter the Connection Details for the SQL Anywhere 16 database connection and click on Tools buttons Test connection

6. The connection will open the connected database
7. The default authentication details are user id:DBA Password:sql
8. The Connection to the SQL Anywhere 16 is of different types
   1. Connect with an ODBC data source
      1. Download and install the SQL Anywhere 16 ODBC drivers and Create an ODBC connection with the database connection details 

2. Check the connection
2. Connect to a Running Database on this computer
   1. Enter the Authentication details
   2. Select the Connect to a running database on this computer from Action list
   3. Run the Personal Server or Personal Server sample from the start→ SQL Anywhere 16 → SQL Anywhere → Personal Server / Personal Server sample

4. Select the DB file path enter the server name and click on OK to start the local personal server with the selected database
5. Now select the server Name and Database Name in the connect to SQL Anywhere 16 database window

6. Check the connection

3. Connect to a running database on another computer
   1. Enter the details of connection parameters

2. The port is assigned by the Network server

3. To connect to the database running on the another computer, we need to create and start the Network server like how personal server created with the selected database
3. Connect to a running database in a cloud
   1. Enter the cloud connection details

9. Open DPL/SSIS and create new connection with the ODBC connection to the SQL Anywhere 16
10. Select the tables you want to import and export to SQL Server through export option.

FIREBIRD DATA CONNECTION PROTOTYPE

Process

 

To download the files from the remote server, create an FTP server for the folder

• FTP server creation for the firebird database file

• Creating script file in ScriptFTP to download the FTP files

Download the “DBConvert for firebird and MS SQL”  tool

3. Use DB Convert to convert from .fdb format to .sql format (FireBird to SQL Server)
4. Use the target connection to staging database

Steps

1. Enable FTP service from Start → Control Panel → Programs and Features → Turn windows features on and off

2. Expand the IIS and select FTP Service

 

3. Open IIS window from Administrative tools → IIS

4. Right click on the sites folder and select Add FTP site option

5. Enter the name for the FTP Site and click Next

6. Enter the Binding and SSL settings

 

7. Select the Authentication and Authorisation information and click finish

 

8. Now check the FTP connection by opening explorer and enter the ftp address as ftp://<ipaddress>

9. Now open ScriptFTP tool select New→ download.ftp → save the .ftp file and enter the script

# Connect to FTP serverOPENHOST("10.100.1.21","RISECORP\ppandela","T3mp2014")                        <ip address>,      <user name>,       <password># Go to C:\MyDirLOCALCHDIR("D:\BODS Videos")                             <file path to where the downloaded file has to place># Get all the files in the FTP serverGETFILE("*.FDB")                <file name to download># Transfer finished, close the connectionCLOSEHOST

10. Click on Run to execute the script and download the file

11. Click on scheduler menu, select the script file and create scheduling for download the files

.

12. Now open the DB Convert tool
13. Select the source connection to .fdb database file

14. Enter the target staging database connection details

15. click on commit to convert the .fdb file to .sql database

16. click on schedule button to schedule the conversion

Hi

   In this blog I am going to share my experience while exploring and working on one of the Customer's data cleansing and reporting project. This Project involves multiple SAP Technologies like :SAP BODS ,SAP Information Steward,SAP Business Objects(BO) , SQL Server(database,client's requirement,Non SAP),

Such kind of projects have lots of data to be cleansed , put in other words, required to be in the desired format as the client needs.The Client will be sharing their requirements beforehand. Here we used SAP Information Steward to create various rules(involving Regex functions) , which are used in SAP BODS to cleanse and transform the data. Using SAP BODS we create various types of transforms which helps in cleansing the data process very smoother and easier. The data might be cleansed in various ways:Automated or Manual.However majority of the tasks are being carried by the automated cleansing process through the tools(Information steward and BODS).

                                SQL Server is the back-end database, being used at the client side to maintain their data.Hence the created tables in  SAP BODS and SAP Information Steward are being stored on it.

The volume of data to be cleansed may vary across projects. In our case, we divided the data load and cleanse processes into 4 batch job load process (comprising 250 K data in each batch).

                                The project involved reporting as well. The customers wanted to display the amount of records cleansed,the changes which were affected to specific fields which were requested, count of records before cleanse and after cleanse and many more things, for which a universe fetching data from SQL server( which holds the cleansed data during each batch load) was created, and then using the Webi (Web-Intelligence) the reports were created.

                                Very soon ,will be adding few images describing the process flow.

Till Then Happy Learning

Thank You

This blog will provide some basic information about how to use the File location object to access files in SFTP and FTP after  Document Version: 4.2 Support Package 6 (14.2.6.0).

1. Go to file file location object, right click on it and create new.

2.

• Name:Name of the file location
• Protocol: Type of protocol(SFTP, FTP or SCP)
• Port: you can leave it default. Usually for SFTP it is 22 and for FTP it is 21
• HostKey fingerprint: It is required for SFTP not for FTP. But don't worry system it self will propose this key once user and password is valid.

      copy and paste(copy is not allowed here but you can copy it in job error log)  the above marked key in the host key finger prints.

• Authorization type: In my case, I use password
• Connection retry count and interval can leave as it is.
• Remote directory: is the folder path in side server('/' is not required at the end)
• Local directory: Required only if you are copying it to local system folder.

3. Now you are ready with file location object and go to the file in data flow  and double click on it.

• Chose the file location from dropdown
• provide the file name and the execute job. you should be able to access the file.

[First of all, sorry for my English]

It is interesting to have some control in our jobs, to be able to detect problems in our ETLs processes, as soon as possible.

Data Services Designer have the possibility to Audit the queries. To activate this, inside the Work Flow, in the DF (Data Flow), go to menu, TOOLS --> Audit

In the first tab, LABEL, select the query and right click, and select COUNT to activate audit on it.

In the second tab you can define the validation to audit:

First, create a new rule. This rule could be "count = 0" or perhaps "count < 1000", if you know that in your source input you have always more than 1000 rows.

Second, the action. You can define three actions:

* Raise an exception

* Send an E-mail to someone or some list

* Script

The script could be some kind of insert in AUDIT_TABLE yo have a log of all executions.

I hope you can easy use this functionality.

Comments are wellcome. Thanks.

The purpose of this document, is to guide implementations where real time processing is needed instead batch in background. I'll try to clarify all the common points related with real time jobs included in DS 4.X based on my recently experiencie with this kind of objects.

The main use of RTJB (real time jobs) is for online processing  where by an input message whit a PK or ID, you can get by response, all the information about your "request", in this case the KEY ID provided in the input message.

Usually RT processing is asociated to a Web service where you have an Input Message and a output as well, eventhough what happend in the middle of the job.

Anyway, you can use RTJB to write in tables, create flat files or any other action that you can perform in batch processing. Please note there are some limitations that also I going to describe on this document.

Important informatión that you must consider:

Realtime at a Glance - Enterprise Information Management - SCN Wiki

Demo on Real time job and configuration in Data service

1592957 - Data Services Real Time Frequently Asked Questions

Why Real time instead Batch?

It depends of the processing type that you are looking for. An example of this:

a) I need to load my DW hourly or once a day  ( programed batch jobs)

b) I need a reproceses a lookup table (demand batch job exec or batch job as web service)

c) third party system, need the status of a client by a XML nested schema response ( Real time job web service with custom XML input and response)

Is Web service the only way to use RTJB?

I guess so, maybe there is another use that I hope you discover in your own experience.

But as I said, Real time processing involves you send a message and wait for the response, like WS works. That's why they work together

What diferences / limitations in deployment have RTJB vs Batch Jobs?

A) RTJB starts with a message and ends with a message.  Input message must be the first step in the job. and the end message the last one. The main limitation here, is that you cannot englobe this with a try catch for example. Supose you have an XML with 3 parameter to handle your Real time job, you need to save these

on some persistent area in a DF in order to use them in the following steps. this first DF cannot be handle by try catch.

B) there are diferences in the use of flat files and Template tables. In batch job, the check to delete the table o delete the file, performs the action before the load. In real time jobs if done when jobs ends. This means that TT and Flatfiles keep empty after execution. To avoid this, use persistent tables and use scritps to delete the files instead use the check in the object

C) When the realtime job is published, DS checks the job inside (tables, sintax, and all objects) and if everything is OK, the job starts and can accept request.

D) by design, flat files used in the real time job, keeps blocked by the process. Maybe you need to update the file while the real time job is ready for accept requests. This is not possible if the quantity of requests procesed is greater than 0. In order to update source files you need to restart the job or use Recycle count property.

1779884 - What does Recycle Count mean for real-time services? - Data Services

E) When use flat files or TT, query object does not work as expected or does not have the same behavior as batch job. use XML MAPS or XML Pipeline instead

F) Some transformations that works in Batch job with "lookup_ext" does not work in real time, eventhough validation is OK. As far I know there is a limitation when use XML map where you cannot use another table not involved in the function.

Note: all this information was relevated on DS 4.2 SP6

hope this informatión can help us like helped me.

fell free to ask and I will try to response and update this doc.

regards

We have come a long way from LSMW to modern ETLs.

I attended a late late night webinar on HCI-DS (date - 24th June 2015); it was very well presented and articulated byBen Hofmans.

Currently we have the below ETL -

BODS - primarily for one off - during the data migration phase of a project (on premise; is it cloud compatible?)

SLT - real-time data synchronization; on going activity (on premise and cloud)

HCI-DS (on premise and cloud)

HANA EIM (on premise now; will be cloud by year end)

All these products require "tunneling". Except HCI-DS.

SAP says HANA EIM is the future.

What are the best use cases for each of these?

Are there any matrices (things to evaluate would be SAP's strategic direction, and other factors like price, easy to install and use, cloud vs on-premise, efficiency etc) that would help a customer / systems integrator decide one tool over another for a given scenario?

Using SAP Data Services 4.2 we will walk through an example of consuming a REST web services via a Web Service REST Datastore.

To get a basic idea of what we will do, you can read through the SAP DS Integrator Guide, specifically the REST portions of these three sections:

Consuming external web services in SAP Data Services

>> Accessing a web service using the designer

>> Adding web service calls to a job

Also the processes is essentially the same as consuming as SOAP web service for which Saurav Mitra has posted an excellent article over at dwbi.org

A summary of the steps we will take:

1. Procure the WADL file for the REST web service we want to consume

2. Create the Datastore and function from the WADL file

3. Create our Dataflow that will consume the function (REST web service)

The REST service we will use in our example is Google's Geocoding API.  I'm not advocating using this as a way to mass geocode addresses in Data Services, it simply is a nice open API to use for demoing the REST capabilities.

Here's an example Call:

http://maps.googleapis.com/maps/api/geocode/xml?address=1600%20Amphitheatre%20Parkway,%20Mountain%20View,%20CA&sensor=false

In a browser you can see the results as:

Let's get started!

1. Procuring the WADL

This can be one of the most difficult part of this effort.  Sometimes REST service publishers do make a WADL available and sometimes they do not.  The WADL is what defines the REST service call for Data Services, similar to the WSDL for a SOAP service.  You will very likely have to create your own WADL, or use the help of some tools to do so.

If you want to skip this step you can simply download the attached WADL file to your PC with DS Designer on it, and save the file without the .txt extension (GoogleGeocodeAPI.wadl).  Note: if Google changes their API after this blog is posted, you may have to tweak or recreate these. Also, passing in different address types to Google's API can yield different XML results, meaning the schema we are tying into here isn't necessarily complete.  For the sake of demo we will use some addresses that yield similar results

One free tool that can help us generate the WADL is SoapUI by Smart Bear.

If you download the free version of SoapUI you can

a. Create a new REST Project (File menu --> New REST Project)

b. Enter the service URI when prompted, and click OK

http://maps.googleapis.com/maps/api/geocode/xml?address=1600%20Amphitheatre%20Parkway,%20Mountain%20View,%20CA&sensor=false

c. In your new project window, you can test the service by clicking the submit button and viewing the result pane.

d. To generate the WADL, right click on the service in the project navigator and choose "Export WADL"

Note: wherever you save it, it will be called "_1.wadl"

e. If you open the file in your favorite XML editor, you'll notice that it contains information on the parameters to pass to the service (outlined in green), but no details about the response format (outlined in red).  Note: the geocode api has more parameters than we show here, but because we only specified these parameters in our sample call in SoapUI, they are the only ones present in the generated WADL.

f. To address the missing response info we need to add a <grammars> section just after the first <doc /> tag

g. Within the grammars section we need to add the schema of the response.  To do this, copy the XML Response from the SoapUI test we did, and paste into an XML to XSD generator.  For example here is one.

h. Copy the schema output into the grammars section

i. Finally we have to tie the response to the schema.  Add the following attributes to the response <representation /> tag:

element="GeocodeResponse" xmlns:xs="http://www.w3.org/2001/XMLSchema"

j. Save the WADL file.... preferably as something other than _1.wadl

2. Creating the Datastore from the WADL file

a. Launch SAP Data Services Designer

b. On the Datastore tab of the Object Library view, right click on a blank area and choose New

c. Give your data store a name, select type: Web Service REST, and in the Web Service URI browse to the WADL file we created in step 1, and click OK to create.

If the create was successful, GREAT JOB!  You will see your new Datastore listed.  If it was not successful, there is something wrong with your WADL file, designer will indicate the line and column where it had the issue, try to find and correct the related issue and then try to recreate the Datastore.  We worked through several xml validation issues until we got it right!

d. Next we need to import the function call.  In the Datastore browser, double-click on the f(x) Functions node.  This will open the Datastore explorer.  Expand the tree until you see the XML node.  Right-click the XML node and select Import.  (note: this was another area where we had to work through some XML validation issues originally).

e. You should now see the function listed!

3. Creating a Dataflow that will consume the function (REST web service)

Next we will walk through a simple data flow where we consume a CSV file that contains our parameters for the service call (address and sensor), calls the service, and writes the the address along with geo coordinates out to another CSV.

CSV (address, sensor) --> REST service --> (address, latitude, longitude).

a. In designer we've created a test project with a batch job and a new dataflow.

b. Next we'll create a test CSV file to use as the source of our parameters to the REST call.  The file as two columns, address and sensor.  A copy of this file is attached (googleresttest.txt).

address, sensor
"1600 Amphitheatre Parkway, Mountain View, CA",false
"3999 West Chester Pike, Newtown Square, PA",false

c. To use our file we'll create a new flat file format.  Here are the settings used to create it

d. Add the Flat file as a source to our data flow, and then add a query in which we will call the function module, and link the two together.

e. Inside the Query we add the function call by right clicking the target schema and choosing "new function call..."

f. Then we select our function

g. Next we see the parameter mapping screen and notice we are unable to map because of the nested structure that the function takes.  We'll take care of that in the next step, just hit finish for now.

You should see this in your target schema of the query.

h. Let's go back and add an additional query step in between our flat file and our REST call.  We will use this to structure, or nest, the data in the format needed by the function.  We also need to add a vanilla Row Generator feeding into the nest query, without which we would end up with only one address getting geocoded.

i. In our nest query, right click on the target schema, and choose "New Output Schema...", and give fill in a new schema name.

j. in the target schema, make sure the PARAMETERS sub-schema is select, then in the source schema, select the address and sensor fields, right click and choose map to output.

k. Next with PARAMETERS still selected ad the Schema Out, we need to hook in the row generator by checking the Row_Generation in the From tab, and unchecking our file format.

l. Now we go back to the REST function call query, where we can see our new incoming structure.  Next right click on the function call and choose "Modify Function Call...".  Click on the new PARAMETERS sub-schema and drag it down to the URI_Parameters box. Then click finish.

m. Now let's add a new Query after the REST function query to unnest the results.  Map the GeocodeResonse to the output, then right click the QRY_UNEST and choose "Unnest with Sub-shemas"

n. Now we'll add another query after the unnest, and map the columns we want to dump to a file.  However after you map the columns, you'll notice if we run a validation that it comes up with an error about each column.  I'm not sure if this is a bug in how DS maps unnested columns, but you can correct by removing the original subschema names from the new mapping.  The other thing we need to do is mark the select "distinct rows" option because of the format of the response, the unnesting can result in multiple rows per address.

o. Lastly we can take the result and dump it out to a file, by generating a file format from our final query, and making it a target.

p. Now if we execute our test, the resulting file contains:

formatted_address,lat,lng
"1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA",37.4224607000,-122.0842702000
"3999 West Chester Pike, Newtown Square, PA 19073, USA",39.9880211000,-75.4153618000

We hope this example proves useful. Again, the Google geocode API has many variations that are not all accounted for in this example, so if you try other addresses, don't be surprised if it breaks due to the different elements in the response.  Also, there are probably other better ways to mass geocode addresses, again we are just using it as an easy example.  Thanks!

Recent days I am working with a source database that contains Arabic data.   If a source is a flat file and a target is template table, we need to set source flat format code page settings to be cp1256 and target table settings enable support nvarchar(if  it is MS SQL server) template tables.

But we will face a lot of issues when we are migrating the data from one database to another database with different code page (char set) settings.      I have worked recently with MS SQL server, Oracle & Sybase ASE as the source database and Sybase IQ as the target.   I have worked the source code pages   for oracle are AL32UTF8 and ARISO8859P6, Sybase ASE is UTF8 and MSSQL Server is Arabic_CI_AI_KS etc.

While handling the code page, we need to keep mind the below points else we may face the junk characters issue.  Here I am giving example for Arabic data and we can change it other character sets accordingly. SAP Data Services job server code page by default Unicode. No need to change the settings. If you give UTF-8  as server setting it won’t be affected.

• Source database code page and need to set datastore code page settings accordingly.
  
  

For Oracle, we need to get code page (char set) using below command

SELECT * FROM NLS_DATABASE_PARAMETERS where parameter = 'NLS_CHARACTERSET'

If Code page is : ARISO8859P6

Language: we can mention the language Arabic etc depending on the code page.

Code page: Database code page iso-8859-6

Server code page:  Job server code page. We can keep as default or we can set UTF-8 if we change in the DSconfig file .

If code page is: AL32UTF8

• Target database code page and need to set datastore code page settings accordingly

For Sybase IQ:

                Select db_property('Collation')

                 1256ARA

select db_property('CharSet')

Windows-1256

• Use the environment variables related to databases if required. For example oracle database NLS_LANG environment variable required.  For MS SQL Server, Sybase ASE & IQ not required of any environment variables.

NLS_LANG should be sync with OS char set settings and source database code settings if the source is Oracle.

In the above case, AMERICAN is English language and AMERICA is regional setting.

• Set the charset options in the ODBC settings. ODBC settings are depending on the database.  For Sybase IQ setting are required but for Oracle , SQL server not required.

• Set the region & language settings in the control panel.

But for the above NLS_LANG settings current system locale should be like below.

• As per the region & language settings in the control panel need to provide the settings in the Data Services Locale selector. Data Services Locale Selector settings written to the DSConfig file. System regional & language settings and DSConfig locale settings should match.

Based on the system local, our data services local selector should be there.  Language and territory in the Arabic (Saudi Arabia) in region settings and we gave the same in the in DS settings. Once we give our DSConfig file looks should be like this.

• Restart the system as we are changing the system settings.

Vast is an Ocean,So is vast the World of Knowledge. With my diving suit packed, loaded with imaginative visions, and lots of curiosity, started diving deep into the world of BODS.Lots of work is going on. Got attracted towards the "Key_Generation" transform and was fascinated at its features.Now it was time for me to fuse and adapt myself into its world.

THE KEY_GENERATION TRANSFORM:-

This transform is categorized under the "Data Integrator Transforms". This generates new keys for source data, starting from a value based on existing keys in the table we specify.

If needed to generate Artificial keys in a table, the Key_Generation transform looks up the maximum existing key value from a table and   uses it as the starting value to generate new keys.

The transform expects the generated key column to be part of the input schema.

STEPS TO USE KEY GENERATION TRANSFORM:-

    Scenario:- Here the target data source for which the keys is needed to be added, have certain newly added rows without a Customer_ID. This could be easily understood in the following snap:-

Our aim here is to automatically generate the keys(Customer_ID) in this case , for the newly inserted records which have no Customer_Id. Accordingly we have taken the following as our input (the modified data without Customer_ID)

INPUT DATA (to be staged in the db):-

TARGET TABLE(which contains the data initially contained in the source table before the entry of new records in the database):-

THE GENERATED DATA FLOW:-

CONTENT OF SOURCE DATA:- (containing the modified entry alone)

CONTENT OF QUERY_TRANSFORM:-

CONTENT OF THE KEY_GENERATION TRANSFORM:-

THE CONTENTS OF THE TARGET TABLE PRIOR JOB EXECUTION:-

The JOB_EXECUTION:-

THE OUTPUT AFTER THE JOB EXECUTION:-

We can now see from the output how Keys have been generated automatically to those records which did not have the Customer_ID initially.

I explored this  little process of the Key_Generation transform, and it seems a savior at times when huge amount of data have the missing entries(wrt to the keys or any sequential column fields).

Now its time to go back to the surface of waters........

Hello Readers,

We can use Pre-load and post-load commands present in the Target table properties tabs

These two works in opposite way to each other, commands written in Pre-load will execute before the DS Job starts executing.

Commands written in Post-Load will execute after the finishing of the DS job.

I will show you how to work with these commands by executing Stored Procedures written in SQL.

1. Writing a stored procedure to fetch a record from one table to another empty table

1.Open SQL Server management Studio.

2.Connect to a database

3.Expand to Programmability folder-> Storedprocedures

4.Right click on the StoredProcedures and select New StoredProcedures option

5.Now write a storedprocedure to extact a record from one table and insert into another empty table(source of DF)

My Pre-Load storedProcedure is like this

USE [NORTHWND]

GO

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

ALTER PROCEDURE [dbo].[spINSERT_dbo_Preloaded]

AS

BEGIN

  insert into [NORTHWND].[dbo].Preloadedtable([CategoryID],[CategoryName],[Description],[Picture]) select * from [NORTHWND].[dbo].[Categories] where CategoryID=1;

END

through this sp, a record from categories table is inserted in Preloadedtable.

My Post-Load SP is

USE [NORTHWND]

GO

SET ANSI_NULLS ON

GO

SET QUOTED_IDENTIFIER ON

GO

ALTER PROCEDURE [dbo].[spINSERT_dbo_Preloadedtable]

AS

BEGIN

  insert into [NORTHWND].[dbo].Preloadedtable([CategoryID],[CategoryName],[Description],[Picture]) select * from [NORTHWND].[dbo].[Categories] where CategoryID=4;

END

This sp will insert another record with categroryID 4

Now will start designing the dataflow in DS Designer

The flow of the DF is like this

1.Attach the source table onto the Dataflow.

2.Map the source columns to the target table using Query transform

3.attach the target table to the query transform

check the source and target table data

before executing the DF it will be zero records in both the tables

Open the Pre-Post table and navigate to pre-load and post load commands tabs

Enter the sql commands in both pre-load and post-load to execute the storedprocedures written in sql server

1.

2.

validate the DF and execute the DF

based on the SP written, the source will contain 2 records, 1 at the time of pre-load and 1 after the execution of the DF flow completely till the target table.

as per the mapping specified and conditions mentioned in the query transform the data will be transferred to target table.

this is how we can use the Pre-load and Post-load commands in the target table.

more blogs coming soon..with spl concepts...

Thanks

PrasannaKumar.P

How to read multiple sheet’s from excel

Step 1: Take a script and define a global variable $G_SHEET_NO  and assign it as 1 and a sql command to delete the data from table before loading.

          $G_SHEET_NO = 1;

          sql('SOURCE_DB', 'delete TBL_TARGET_EXCEL');

Step 2:While loop condition will be $G_SHEET_NO <= 3 (In my case there are 3 sheet’s you can give any number here or to make it dynamic use global                    variable and pass the value to that variable at run time.

Step 3:First script in above picture will have code like below:

          $G_LIST_SHEET = 'Sheet'||$G_SHEET_NO;

          print( 'LOADING DATA FROM :'|| $G_LIST_SHEET);

          Note: we’ll take another variable here ‘$G_LIST_SHEET’ and print function here is to print the sheet no in trace log that which sheet we are loading right           now.

Step 4:Then Workflow ‘WF_SHEET_LIST’ will have a Dataflow

Step 5:Here SHEET_LIST EXCEL file properties are –

Step 6:Now the last script in while loop will have code like –

          $G_SHEET_NO = $G_SHEET_NO + 1;

          So this will just increment the sheet no and next it will load data from second sheet then third and so on.

Tip: Format of data should be the same for all sheets you want to load data from.

In market, we have numerous solutions for Business Address search and validation like QAS (Quick Address Systems by Experian), SAP DQM (Data Quality Management). These are the few listed ones which I came across.

This time the Customer wanted to implement a SAP based solution for the Address validation (for Great Britain) and we couldn’t think other than SAP DQM. This is where challenges started falling in place.

Customer system was a third party e-commerce system that is SAP Hybris where it wanted to validate and provide Address suggestions to the customers.

Challenge was – SAP DQM has its support for SAP CRM, ECC and MDG environments and achieving same for third party was a real trouble.

This document discusses about the challenges that we faced during the implementation of this functionality and the shortcomings out of it.

Implementation:

The implementation involved challenges like:

1. How DQM functionality can be used by the Third party system?
2. Address suggestion input parameters
3. Address suggestions as output
4. Issues
5. Helpful threads on SCN

1. How DQM functionality can be used by the Third party system?

DQM provides set of web services for Data de-duplication, data cleansing, and address validation. We decided to deploy a custom address validation functionality using a web service, which would be consumed by the Third party system.

However the only thing required in place is the appropriate Address directory to be installed at the desired location.

2. Address suggestion input parameters –

Address suggestions would be provided based on the parameters like Postcode, Street, City, Region and many more. Be careful while deciding the parameters for address suggestions. Selecting wrong set of parameters can result in n number of address suggestions irrelevant to our search and even in some cases the directory won’t return any results.

That’s not the end, the real work of Address suggestions in DQM is done by the GAC (Global address cleanse) and the GSL (Global suggestion list transform). Choosing one of them is entirely requirement specific. GAC works as the address assignment engine whereas GSL works like a query tool. Each one of them has different set of input parameters, option level configurations and output fields.

Note: Before proceeding with the implementation have a detail look at the functioning of DQM transforms in SAP Data services reference guide.

3. Address suggestions as output –

Crucial step. How are you delivering the output?

If you are deploying using the web service xml/json data exchange format is going to come in play. SAP provides the SUGGESTION_LIST field which is a nested XML structure where all the suggestions related to the input parameters will be staged at run-time and you need to consume these results at other end.

4. Limitations of DQM functionality

• GAC returns results on a fuzzy search basis and if you want the result on exact search basis, it’s not possible as of now. Have raised the same concern on SAP IDEA Place. Refer here.
• In some cases GSL may return results on exact search basis but, it has lesser number of output fields.
• It is extremely difficult to return all related addresses to a specific postcode in a single go.
• Again, the SUGGESTION_LIST field has its size limitation.
• You cannot modify the processing logic of GAC or GSL transforms.

5. Issues:

     5.1 XML Parsing error:

A major issue which consumed too much of our time in analysis. Posted about the same issue on SCN and got it resolved.

It was the issue with the size limitation of the SUGGESTION_LIST field.

     5.2 Testing the Web service on SOAP UI.

Also faced an issue while testing. Refer here.

6. Helpful threads on SCN

SCN has always been helpful in resolving the issues and people voluntarily help to solve the issue.

Some threads –

    6.1 https://wiki.scn.sap.com/wiki/display/EIM/Global+Address+Cleanse+Transform

         Refer main and child pages for better understanding about how the GAC transform works.

    6.2 https://wiki.scn.sap.com/wiki/pages/viewpage.action?pageId=272795473 

         Very helpful thread. Helped to understand how the directory logic works and also helped to define the approach for our requirement.

    Special thanks to Virginia Hagen for helping us out through the troublesome situations.

Closing note:

SAP DQM is meant for CRM, ECC and MDG type environments as of now and delivering the same functionality over a web-service is a challenge in itself. One more thing, SAP updates the processing logic of the address suggestions with every new version update and if you are continuing with a custom functionality like this then, you’ll have to be cautious before the system upgrade. It all began here and I would ask you to go through some thoughtful inputs from experts which would help you in taking a good business decision.

I’ll be happy to accept any suggestions and corrections on this…

Thanks – Santosh G.

Talk less. Work Smart!