Partitioning and Collecting Icons

These icons keep appearing in the jobs. Here, is a description what each of these actually mean:

1.AutoPartitioning: 

2.Repartitioning: 

3.Partitioning: 

4.SamePartitioning: 

5.Collector: 

Note : Though matter related to different datastage concepts are prevalent on the net and can be gathered from various sources, mine is an attempt to present the matter in an easy to understand manner.  

Reference:
http://publib.boulder.ibm.com/infocenter/iisinfsv/v8r5/index.jsptopic=%2Fcom.ibm.swg.im.iis.ds.parjob.dev.doc%2Ftopics%2Fpartitioningrepartitioningandcollectingdata.html

Partitioning and Collecting Data - Related Stage Properties

1.   Preserve Partitioning Flag

This flag is present in the stage properties:

Stage Properties window -> Stage Page -> Advanced tab

Values it can take:

a.     Set:

Setting this flag requests the next stage in the job to preserve whatever partitioning the current stage has implemented. Note, however, that the next stage might ignore this request.

b.    Clear: The next stage is free to partition the data.

c.     in the job has set. If the stage has multiple inputs and has a flag set to Propagate, its Preserve Partitioning flag is set if it is set on any of the inputs, or cleared if all the inputs are clear.

2.   Partition Type (where do we specify)

This flag is present at in the stage properties:

Stage Properties window -> Inputs Page -> Partitioning tab 

Partitioning and Collecting Data in Datastage

Partitioning of data helps to implement parallel processing in our jobs. This makes it one of the most important topics of study.

Simplest scenarios do not require delving into the nitty-gritty of the subject, but others will.

The different partitioning methods available in datastage are as follows:

S.No.	Partition	Method of Distribution	Nature of resulting Partitions	Important points
	Round Robin	1st record to 1st partition and so on	approximately equal-sized	This method is the one normally used when InfoSphere DataStage initially partitions data.
	Random	Random distribution	approximately equal-sized	Slightly higher overhead than round robin because of the extra processing required to calculate a random value for each record.   It doesn't primarily load balances as it's random. But, for a large enough number of rows, random distribution will be close enough to 1/N rows per node.
	Same	no repartitioning , records stay on the same processing node; that is, they are not redistributed	Not Applicable	Fastest. This is normally the method InfoSphere® DataStage® uses when passing data between stages in your job.
	Entire	Every instance of a stage on every processing node receives the complete data set as input.	Every partition consists of the complete dataset.	Used when every instance of the operator needs access to the entire input data set.
	Hash	Partitioning is based on a function of one or more columns (the hash partitioning keys) in each record.	Can be skewed depending on the data distribution.	The data type of a partitioning key might be any data type except raw, subrecord, tagged aggregate, or vector. By default, the hash partitioner does case-sensitive comparison. This means that uppercase strings appear before lowercase strings in a partitioned data set. You can override this default if you want to perform case insensitive partitioning on string fields.
	Modulus	partition_number = fieldname mod number_of_partitions , as is obvious fieldname is a numeric data field.	Can be skewed depending on the value of the key field.	Only one column can be specified as key as opposed to other keyed partitions.
	Range	divides a data set into approximately equal size partitions based on one or more partitioning keys. In order to use a range partitioner, we need a range map. This can be done using the Write Range Map stage.	approximately equal-sized.	The only keyed partitioning method that created equal-sized partitions. [The write range map stage uses a probabilistic splitting technique to range partition a data set.]
	DB2	Partition an input data set in the same way that DB2® would partition it.	Not Applicable.
	Auto	We leave it to InfoSphere DataStage to determine the best partitioning method to use depending on the type of stage, and what the previous stage in the job has done.	Usually round-robin / same so equi-sized.	Typically InfoSphere DataStage would use round robin when initially partitioning data, and same for the intermediate stages of a job.

We need to take care of some of the specific stage properties to handle partitioning effectively.

Here is a description of the same.

Datastage uses different icons to specify the kind of partitioning that is happening inside the stages. I have a detailed explanation of these icons here , this will help us understand the partitioning techniques used in our jobs.

BASIC Transformer Stage Vs Transformer Stage

BASIC Transformer stage is available in the repository under:

Stage Types ->Parallel -> Processing

It can be added to the palette from here, not available by default. This can be inferred as a suggestion to use this stage as less as possible. 

Both stages can be used in both server and parallel jobs.

The various differences between BASIC Transformer and Transformer stage as listed below:

S. No.	BASIC Transformer	Transformer
1.	Basic transformer will operate in Sequential mode, i.e., does not run on multiple nodes.	Transformer operates in Parallel mode, i.e., can run on multiple nodes giving better performance.
2.	It gives access to BASIC transforms and functions (BASIC is the language supported by the server engine and available in server jobs).	It gives access to C++ transforms and functions (C++ is the language supported by the parallel engine and available in parallel jobs).
3.	BASIC transformer stages can be used only on SMP systems, not on MPP or cluster systems.	Transformer stages can be used on SMP, MPP or cluster systems.
4.	Can access built-in server routines.	Cannot access built-in server routines, only parallel, written in C/C++.
5.	Basic transformer compiles in BASIC language.	Transformer compiles in C++.
6.	Takes less time to compile.	Takes more time to compile.

Why would we need the BASIC Transformer stage at all?

To access built-in server routines in parallel jobs.