Feb 5, 2016

Ways of invoking Pig or Pig script from command line

Posted by  on 



The following examples are based on Apache Pig 0.15.0 latest at the time. Please make sure to review my later post in relation to this.

Link:
Apache-Pig-commandline-Options



a) Pig in local interactive mode: 
The following will run in the single virtual machine and the files that you are working on will be in local file systems only. This command will take you on Pig's grunt interactive shell.

On unix terminal run: 
$ cd /usr/share/cloudera/pig/ 
$ bin/pig –x local 

b) Pig in Map reduce mode: 
By default Pig runs in a map reduce mode in a Hadoop cluster. This command will take you on Pig's grunt interactive shell in a Hadoop cluster.

 On unix terminal run:
$ cd /usr/share/cloudera/pig/ 
$ bin/pig –x
or  
$ bin/pig 

c) Pig in Tez mode:
It is similar to local mode, except internally Pig will invoke tez runtime engine. Tez is new framework from Hortonworks which is now a part of Apache. It is still under finalization. This command will take you on Pig's grunt interactive shell.

On unix terminal run: 
$ cd /usr/share/cloudera/pig/ 
$ bin/pig –x tez


d) Pig separate scripts in local interactive mode or Map Reduce mode:
We can list out pig scripts in a text file and invoke Pig scripts in local or Map Reduce mode.
On unix terminal run:
$ cd /usr/share/cloudera/pig/ 
$ bin/pig –x local myscript.pig
or 
$ bin/pig myscript.pig       (This will run in Map Reduce mode)



Note: 
In this post it is assumed that the pig binaries are installed in this /usr/share/cloudera/pig/ directory on a local or hadoop cluster. Thus this location will use to execute/initiate the Pig framework.  

Thanks!


References:
http://pig.apache.org

Jan 26, 2016

Apache Pig Exercises: 21. List Employees those are having five characters in their names

Posted by  on 


In this post the sample Apache Pig script will List employees wthose are having five characters in their names.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

 @ Test file data structure:

 Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more.


@ Sample data:

Employees data table:


Department data table:



 @ Apache Pig Script:

 a) List employees wthose are having five characters in their names.

grunt>
data = LOAD '/Documents/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
all_recs = foreach data generate empno,ename, job,mgr,hiredate, sal, comm,deptno;
rec_fltr = filter all_recs by (SIZE(ename)==5) ;
rec_ordr = order rec_fltr by sal;
dump rec_ordr;

@Apache Pig Output on Grunt Shell:
(7369,SMITH,CLERK,7902,1980-12-17,800.0,,20)
(7900,JAMES,CLERK,7698,1981-12-03,950.0,,30)
(7876,ADAMS,CLERK,7788,1983-01-12,1100.0,,20)
(7499,ALLEN,SALESMAN,7698,1981-02-20,1600.0,300.0,30)
(7782,CLARK,MANAGER,7839,1981-06-09,2450.0,,10)
(7698,BLAKE,MANAGER,7839,1981-05-01,2850.0,,30)
(7566,JONES,MANAGER,7839,1981-04-02,2975.0,,20)
(7788,SCOTT,ANALYST,7566,1982-12-09,3000.0,,20)

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/

_________________
Thank you!

Jan 19, 2016

List of Apache Pig v0.15 Built-in functions

Posted by  on 

Apache Pig Exercises: 20. List Employees whose annul salary is ranging from 22000 and 45000

Posted by  on 

In this post the sample Apache Pig script will display employees whose annul salary ranging from 22000 and 45000.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, sal, (int)(sal*12) as annsal, comm,deptno;

 rec_fltr = filter all_recs by (annsal>22000) and (annsal<45000) ;

 rec_ordr = order rec_fltr by annsal;

 dump rec_ordr;

@Apache Pig Output on Grunt Shell: 




@ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_ordr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!

Apache Pig Exercises: 19. List Employees who are joined in the month of Dec 1981

Posted by  on 

In this post the sample Apache Pig script will display employees  who are joined in the month of Dec 1981.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, SUBSTRING(hiredate,0,7) as hryear, sal,comm,deptno;

 rec_fltr = filter all_recs by (hryear=='1981-12');

 dump rec_fltr;


@Apache Pig Output on Grunt Shell: 



@ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_fltr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!

Apache Pig Exercises: 18. List Employees who are joined in the year 1981

Posted by  on 

In this post the sample Apache Pig script will display employees who are joined in the year 1981.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, SUBSTRING(hiredate,0,4) as hryear, sal,comm,deptno;

 rec_fltr = filter all_recs by (hryear=='1981');

 dump rec_fltr;


@Apache Pig Output on Grunt Shell: 




@ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_fltr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!

Apache Pig Exercises: 17. List Employees who are working for the deptno 10 or 20

Posted by  on 

In this post the sample Apache Pig script will display employees who are working for the deptno 10 or 20.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, sal,comm,deptno;

 rec_fltr = filter all_recs by (deptno==10) or (deptno==20);

 rec_ordr = order rec_fltr by deptno;

 dump rec_ordr;


@Apache Pig Output on Grunt Shell: 




 @ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_ordr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!

Apache Pig Exercises: 16. List Employees who joined on 1-May-81, 31-Dec-81, 17-Dec-81, 19-Jan-80 in asc order of seniority

Posted by  on 

In this post the sample Apache Pig script will display employees who joined on 1-May-81, 31-Dec-81, 17-Dec-81, 19-Jan-80 in asc order of seniority.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, (int)GetYear(CurrentTime())-(int)SUBSTRING(hiredate,0,4) as expn ,sal,(int)(sal/30) as dailysal, comm,deptno;

 rec_fltr = filter all_recs by (hiredate=='1981-05-01') or (hiredate=='1981-12-31') or (hiredate=='1981-12-17') or (hiredate=='1980-01-19');

 rec_ordr = order rec_fltr by expn desc;

 dump rec_ordr;

@Apache Pig Output on Grunt Shell: 




 @ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_ordr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!

Apache Pig Exercises: 15. List Employees who are either ‘CLERK’ or ‘ANALYST’ in descending order

Posted by  on 

In this post the sample Apache Pig script will display employees who are either ‘CLERK’ or ‘ANALYST’ in the desc order.

The examples and exercise scripts are created using Apache Pig current version r0.14.0.

@ Test data structure:
Please refer to Apache Pig learning series intro... post for the file structures, visit the reference section shown at the bottom of the post for more. 

@ Sample data:

Employees data table:


Department data table:


@ Apache Pig Script:

List all employee records in asc order of dept nbr. and desc of job title:

grunt> 
/* load data file in pig reference variable */
data = LOAD '/ALLApplnDvlpmt/All_DataSets/tbl_EMP.txt' USING PigStorage(',') as (empno:int, ename:chararray, job:chararray, mgr:int, hiredate:chararray, sal:float, comm:float, deptno:int);
/* separated only the jobs column from source file dataand store in pig reference variable */
all_recs = foreach data generate empno,ename,job,mgr,hiredate, sal, comm,deptno;

 rec_fltr = filter all_recs by (job=='CLERK') or (job=='ANALYST');

 rec_ordr = order rec_fltr by empno desc;

 dump rec_ordr;

@Apache Pig Output on Grunt Shell: 

@ Store output in a file:
Please issue following Apache Pig command on grunt shell, to store the output in a file on the disk we use store command with appropriate folder location. 

grunt> store rec_ordr into '/pig_outout/final_output.txt'

You can review the grunt shell console if the command was successful or not. Also, if you decide to reuse the Pig Output file again and again, please make sure to delete the folder when the actual output file is stored to be able to reuse the same, else Apache Pig displays failed message on the grunt shell/console.

@ Review results from the stored file:
Please note, the above command creates a sub-folder named final_output.txt in the actual folder pig_output and the actual output is stored in a file called part-r-00000 and not in a file final_output.txt

@ Apache Pig Reference/s:
  • https://pig.apache.org
  • http://pig.apache.org/docs/r0.14.0/
_________________
Thank you!