Apache Pig Tutorial 

---

1. Apache Pig Tutorial

This Apache Pig tutorial provides the basic introduction to Apache Pig – high-level tool over MapReduce. This tutorial helps professionals who are working on Hadoop and would like to perform MapReduce operations using a high-level scripting language instead of developing complex codes in Java.

Apache Pig Tutorial

2. Apache Pig Introduction

a. History of Apache Pig

As a research project at Yahoo in the year 2006, Apache Pig was developed in order to create and execute MapReduce jobs on large data-sets. In 2007 Apache Pig was open sourced, later in 2008, Apache Pig’s first release came out.

b. Introduction to Apache Pig

Pig was created to simplify the burden of writing complex Java codes to perform MapReduce jobs. Earlier Hadoop developers have to write complex java codes in order to perform data analysis. Apache Pig provides a high-level language known as Pig Latin which helps Hadoop developers to write data analysis programs. By using various operators provided by Pig Latin language programmers can develop their own functions for reading, writing, and processing data.

In order to perform analysis using Apache Pig, programmers have to write scripts using Pig Latin language to process data stored in Hadoop Distributed File System. Internally, all these scripts are converted to Map and Reduce tasks. A component known as Pig Engine is present inside Apache Pig in which Pig Latin scripts are taken as input and these scripts gets converted into Map-Reduce jobs.

c. Need for Pig

For all those Programmers who are not so good at Java normally, have to struggle a lot for working with Hadoop, especially when they need to perform any MapReduce tasks. Apache Pig comes up as a helpful tool for all such programmers.

There is no need of developing complex Java codes to perform MapReduce tasks. By simply writing Pig Latin scripts programmers can now easily perform MapReduce tasks without having need of writing complex codes in Java.

Apache Pig reduces the length of codes by using multi-query approach. For example, to perform an operation we need to write 200 lines of code in Java that we can easily perform just by typing less than 10 lines of code in Apache Pig. Hence, ultimately our almost 16 times development time gets reduced using Apache Pig.

If developers have knowledge of SQL language, then it is very easy to learn Pig Latin language as it is similar to SQL language.

Many built-in operators are provided by Apache Pig to support data operations like filters, joins, ordering, etc. In addition, nested data types like tuples, bags, and maps which are not present in MapReduce are also provided by Pig.

d. Features of Pig

Apache Pig comes with the below unique features:

Apache Pig Features

Rich Set of Operators: Pig consists of a collection of rich set of operators in order to perform operations such as join, filer, sort and many more.

Ease of Programming: Pig Latin is similar to SQL and hence it becomes very easy for developers to write a Pig script. If you have knowledge of SQL language, then it is very easy to learn Pig Latin language as it is similar to SQL language.

Optimization opportunities: The execution of the task in Apache Pig gets automatically optimized by the task itself, hence the programmers need to only focus on the semantics of the language.

Extensibility: By using the existing operators, users can easily develop their own functions to read, process, and write data.

User Define Functions (UDF’s): With the help of facility provided by Pig of creating UDF’s, we can easily create User Defined Functions on a number of programming languages such as Java and invoke or embed them in Pig Scripts.

All types of data handling: Analysis of all types of Data (i.e. both structured as well as unstructured) is provided by Apache Pig and the results are stored inside HDFS.

3. Conclusion

So finally we have seen what is Apache Pig, Pig History, Why Pig is required and the key features of Apache Pig that make it different from other similar technologies.

Execute Pig Script | Apache Pig Running Scripts

---

1. Execute Pig Script

In this article on how to execute pig script, we will see the whole concept of PigScripts Execution. Also, we will cover the basic comments in Pig Script, that will help while writing a script in a file. Moreover, we will see how to Execute Pig Script in a Batch mode as well as how to Execute a Pig Script from HDFS with proper steps and examples.

Execute Pig Script

2. Introduction to Apache Pig Running Scripts

Basically, to place Pig Latin statements and Pig commands in a single file, we use Pig scripts. It is good practice to identify the file using the *.pig extension, even while not required.

Moreover, we can run Pig scripts from the command line and from the Grunt shell.

Also, to pass values to parameters using parameter substitution, Pig scripts allows us to do so.

3. Comments in Pig Script

We can include comments in Pig Script while writing a script in a file. like:

a. Multi-line comments

The multi-line comments will begin with ‘/*’, end them with ‘*/’.

1. /* These are the multi-line comments
2. In the pig script */

b. Single –line comments

The single-line comments will begin with ‘–‘.

1. --we can write single line comments like this.

4. Executing Pig Script in Batch mode

Further, follow these steps, while we execute Pig script in batch mode.

Step 1

At very first, write all the required Pig Latin statements and commands in a single file. Then save it as a .pig file.

Step 2

Afterwards, execute the Apache Pig script. To execute Pig script from the shell (Linux), see:

• Local mode

1. $ pig -x local Sample_script.pig

• MapReduce mode

1. $ pig -x MapReduce Sample_script.pig

It is possible to execute it from the Grunt shell as well using the exec command.

1. grunt> exec /sample_script.pig

5. Executing a Pig Script from HDFS

Also, we can execute Pig script that resides in the HDFS. Let’s assume there is a Pig script with the name Sample_script.pig in the HDFS directory named /pig_data/. To execute it, see.

1. $ pig -x mapreduce hdfs://localhost:9000/pig_data/Sample_script.pig

• Pig Script Example

Suppose we have a file Employee_details.txt in HDFS with the following content.

1. Employee_details.txt
2. 001,mehul,chourey,21,9848022337,Hyderabad
3. 002,Ankur,Dutta,22,9848022338,Kolkata
4. 003,Shubham,Sengar,22,9848022339,Delhi
5. 004,Prerna,Tripathi,21,9848022330,Pune
6. 005,Sagar,Joshi,23,9848022336,Bhuwaneshwar
7. 006,Monika,sharma,23,9848022335,Chennai
8. 007,pulkit,pawar,24,9848022334,trivendram
9. 008,Roshan,Shaikh,24,9848022333,Chennai

Now, also we have a sample script with the name sample_script.pig, in the same HDFS directory. It contains statements performing operations and transformations on the Employee relation.

1. Employee = LOAD 'hdfs://localhost:9000/pig_data/Employee_details.txt' USING PigStorage(',')
2. as (id:int, firstname:chararray, lastname:chararray, phone:chararray, city:chararray);
3. Employee_order = ORDER Employee BY age DESC;
4. Employee_limit = LIMIT Employee_order 4;
5. Dump Employee_limit;

• The script will load the data in the file named Employee_details.txt as a relation named Employee, in the first statement.
• Moreover, the script will arrange the tuples of the relation in descending order, based on age, and store it as Employee_order, in the second statement.
• The script will store the first 4 tuples of Employee_order as Employee_limit, in the third statement.
• Ultimately,  last and the 4rth statement will dump the content of the relation Employee_limit.

Further, let’s execute the sample_script.pig.

1. $./pig -x mapreduce hdfs://localhost:9000/pig_data/sample_script.pig

In this way, Pig gets executed and gives you the output like:

1. (7,Pulkit,Pawar,24,9848022334,trivendram)
2. (8,Roshan,Shaikh,24,9848022333,Chennai)
3. (5,Sagar,Joshi,23,9848022336,Bhuwaneshwar)
4. (6,Monika,Sharma,23,9848022335,Chennai)
5. 2015-10-19 10:31:27,446 [main] INFO org.apache.pig.Main - Pig script completed in 12
6. minutes, 32 seconds and 751 milliseconds (752751 ms)

This was all about how to Execute Pig Script. Hope you like our explanation.

6. Conclusion

As a result, we have seen the whole concept of Apache Pig Running Scripts, along with Executing of Pig Script in Batch mode and from HDFS. Also, we have seen its comments to understand well. Still, if any doubt occurs, feel free to ask in the comment section.