本文整理汇总了Java中org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher类的典型用法代码示例。如果您正苦于以下问题：Java MapReduceLauncher类的具体用法？Java MapReduceLauncher怎么用？Java MapReduceLauncher使用的例子？那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。
MapReduceLauncher类属于org.apache.pig.backend.hadoop.executionengine.mapReduceLayer包，在下文中一共展示了MapReduceLauncher类的13个代码示例，这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞，您的评价将有助于我们的系统推荐出更棒的Java代码示例。
示例1: executePlan
点赞 3
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
private boolean executePlan(PhysicalPlan pp) throws IOException {
boolean failed = true;
MapReduceLauncher launcher = new MapReduceLauncher();
PigStats stats = null;
try {
stats = launcher.launchPig(pp, "execute", myPig.getPigContext());
} catch (Exception e) {
e.printStackTrace(System.out);
throw new IOException(e);
}
Iterator<JobStats> iter = stats.getJobGraph().iterator();
while (iter.hasNext()) {
JobStats js = iter.next();
failed = !js.isSuccessful();
if (failed) {
break;
}
}
return !failed;
}
开发者ID:sigmoidanalytics，
项目名称:spork-streaming，
代码行数:21，
代码来源:TestMultiQueryLocal.java
示例2: compile
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public SOperPlan compile(PhysicalPlan php, PigContext pc)
throws PlanException, IOException, VisitorException {
MapReduceLauncher mrlauncher = new MapReduceLauncher();
MROperPlan mrp = mrlauncher.compile(php, pc);
MRtoSConverter converter = new MRtoSConverter(mrp, pc);
converter.convert();
return converter.getSPlan();
}
开发者ID:JamesLampton，
项目名称:piggybank-squeal，
代码行数:11，
代码来源:StormLauncher.java
示例3: explain
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
/**
* Provide information on how a pig query will be executed.
* @param alias Name of alias to explain.
* @param format Format in which the explain should be printed. If text, then the plan will
* be printed in plain text. Otherwise, the execution plan will be printed in
* <a href="http://en.wikipedia.org/wiki/DOT_language">DOT</a> format.
* @param verbose Controls the amount of information printed
* @param markAsExecute When set will treat the explain like a
* call to execute in the respoect that all the pending stores are
* marked as complete.
* @param lps Stream to print the logical tree
* @param pps Stream to print the physical tree
* @param eps Stream to print the execution tree
* @throws IOException if the requested alias cannot be found.
*/
@SuppressWarnings("unchecked")
public void explain(String alias,
String format,
boolean verbose,
boolean markAsExecute,
PrintStream lps,
PrintStream pps,
PrintStream eps) throws IOException {
try {
pigContext.inExplain = true;
buildStorePlan( alias );
if( currDAG.lp.size() == 0 ) {
lps.println("Logical plan is empty.");
pps.println("Physical plan is empty.");
eps.println("Execution plan is empty.");
return;
}
PhysicalPlan pp = compilePp();
currDAG.lp.explain(lps, format, verbose);
pp.explain(pps, format, verbose);
MapRedUtil.checkLeafIsStore(pp, pigContext);
MapReduceLauncher launcher = new MapReduceLauncher();
launcher.explain(pp, pigContext, eps, format, verbose);
if (markAsExecute) {
currDAG.markAsExecuted();
}
} catch (Exception e) {
int errCode = 1067;
String msg = "Unable to explain alias " + alias;
throw new FrontendException(msg, errCode, PigException.INPUT, e);
} finally {
pigContext.inExplain = false;
}
}
开发者ID:sigmoidanalytics，
项目名称:spork-streaming，
代码行数:53，
代码来源:PigServer.java
示例4: getMRPlan
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan getMRPlan(PhysicalPlan pp, PigContext ctx) throws Exception {
MapReduceLauncher launcher = new MapReduceLauncher();
java.lang.reflect.Method compile = launcher.getClass()
.getDeclaredMethod("compile",
new Class[] { PhysicalPlan.class, PigContext.class });
compile.setAccessible(true);
return (MROperPlan) compile.invoke(launcher, new Object[] { pp, ctx });
}
开发者ID:sigmoidanalytics，
项目名称:spork-streaming，
代码行数:9，
代码来源:TestPigStats.java
示例5: explain
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
/**
* Provide information on how a pig query will be executed.
* @param alias Name of alias to explain.
* @param format Format in which the explain should be printed. If text, then the plan will
* be printed in plain text. Otherwise, the execution plan will be printed in
* <a href="http://en.wikipedia.org/wiki/DOT_language">DOT</a> format.
* @param verbose Controls the amount of information printed
* @param markAsExecute When set will treat the explain like a
* call to execute in the respoect that all the pending stores are
* marked as complete.
* @param lps Stream to print the logical tree
* @param pps Stream to print the physical tree
* @param eps Stream to print the execution tree
* @throws IOException if the requested alias cannot be found.
*/
@SuppressWarnings("unchecked")
public void explain(String alias,
String format,
boolean verbose,
boolean markAsExecute,
PrintStream lps,
PrintStream pps,
PrintStream eps) throws IOException {
try {
pigContext.inExplain = true;
buildStorePlan( alias );
if( currDAG.lp.size() == 0 ) {
lps.println("Logical plan is empty.");
pps.println("Physical plan is empty.");
eps.println("Execution plan is empty.");
return;
}
PhysicalPlan pp = compilePp();
currDAG.lp.explain(lps, format, verbose);
pp.explain(pps, format, verbose);
MapRedUtil.checkLeafIsStore(pp, pigContext);
MapReduceLauncher launcher = new MapReduceLauncher();
launcher.explain(pp, pigContext, eps, format, verbose);
if (markAsExecute) {
currDAG.markAsExecuted();
}
} catch (Exception e) {
int errCode = 1067;
String msg = "Unable to explain alias " + alias;
throw new FrontendException(msg, errCode, PigException.INPUT, e);
} finally {
pigContext.inExplain = false;
}
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:53，
代码来源:PigServer.java
示例6: testMapAggPropFalse
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggPropFalse() throws Exception{
//test with pig.exec.mapPartAgg set to false
String query = getGByQuery();
pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "false");
MROperPlan mrp = Util.buildMRPlan(query, pc);
assertEquals(mrp.size(), 1);
assertNull("POPartialAgg should be absent", findPOPartialAgg(mrp));
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:11，
代码来源:TestPOPartialAggPlan.java
示例7: testMapAggPropTrue
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggPropTrue() throws Exception{
//test with pig.exec.mapPartAgg to true
String query = getGByQuery();
pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
MROperPlan mrp = Util.buildMRPlan(query, pc);
assertEquals(mrp.size(), 1);
assertNotNull("POPartialAgg should be present",findPOPartialAgg(mrp));
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:12，
代码来源:TestPOPartialAggPlan.java
示例8: testMapAggNoAggFunc
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggNoAggFunc() throws Exception{
//no agg func, so there should not be a POPartial
String query = "l = load 'x' as (a,b,c);" +
"g = group l by a;" +
"f = foreach g generate group;";
pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
MROperPlan mrp = Util.buildMRPlan(query, pc);
assertEquals(mrp.size(), 1);
assertNull("POPartialAgg should be absent",findPOPartialAgg(mrp));
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:13，
代码来源:TestPOPartialAggPlan.java
示例9: testMapAggNotCombinable
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggNotCombinable() throws Exception{
//not combinable, so there should not be a POPartial
String query = "l = load 'x' as (a,b,c);" +
"g = group l by a;" +
"f = foreach g generate group, COUNT(l.b), l.b;";
pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
MROperPlan mrp = Util.buildMRPlan(query, pc);
assertEquals(mrp.size(), 1);
assertNull("POPartialAgg should be absent", findPOPartialAgg(mrp));
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:13，
代码来源:TestPOPartialAggPlan.java
示例10: checkMRPlan
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
private MROperPlan checkMRPlan(PhysicalPlan pp, int expectedRoots,
int expectedLeaves, int expectedSize) throws IOException {
System.out.println("===== check map-reduce plan =====");
MapRedUtil.checkLeafIsStore(pp, myPig.getPigContext());
MapReduceLauncher launcher = new MapReduceLauncher();
MROperPlan mrp = null;
try {
java.lang.reflect.Method compile = launcher.getClass()
.getDeclaredMethod("compile",
new Class[] { PhysicalPlan.class, PigContext.class });
compile.setAccessible(true);
mrp = (MROperPlan) compile.invoke(launcher, new Object[] { pp, myPig.getPigContext() });
Assert.assertNotNull(mrp);
} catch (Exception e) {
PigException pe = LogUtils.getPigException(e);
if (pe != null) {
throw pe;
} else {
e.printStackTrace();
Assert.fail();
}
}
showPlanOperators(mrp);
System.out.println("===== Display map-reduce Plan =====");
System.out.println(mrp.toString());
Assert.assertEquals(expectedRoots, mrp.getRoots().size());
Assert.assertEquals(expectedLeaves, mrp.getLeaves().size());
Assert.assertEquals(expectedSize, mrp.size());
return mrp;
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:44，
代码来源:TestMultiQueryCompiler.java
示例11: testSuccessFileCreation1
点赞 2
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testSuccessFileCreation1() throws Exception {
PigServer ps = null;
String[] files = new String[] { inputFileName,
outputFileName + "_1", outputFileName + "_2", outputFileName + "_3"};
try {
ExecType[] modes = new ExecType[] { ExecType.LOCAL, ExecType.MAPREDUCE};
String[] inputData = new String[]{"hello\tworld", "hi\tworld", "bye\tworld"};
String multiStoreScript = "a = load '"+ inputFileName + "';" +
"b = filter a by $0 == 'hello';" +
"c = filter a by $0 == 'hi';" +
"d = filter a by $0 == 'bye';" +
"store b into '" + outputFileName + "_1';" +
"store c into '" + outputFileName + "_2';" +
"store d into '" + outputFileName + "_3';";
String singleStoreScript = "a = load '"+ inputFileName + "';" +
"store a into '" + outputFileName + "_1';" ;
for (ExecType execType : modes) {
for(boolean isPropertySet: new boolean[] { true, false}) {
for(boolean isMultiStore: new boolean[] { true, false}) {
String script = (isMultiStore ? multiStoreScript :
singleStoreScript);
// since we will be switching between map red and local modes
// we will need to make sure filelocalizer is reset before each
// run.
FileLocalizer.setInitialized(false);
if(execType == ExecType.MAPREDUCE) {
ps = new PigServer(ExecType.MAPREDUCE,
cluster.getProperties());
} else {
Properties props = new Properties();
props.setProperty(MapRedUtil.FILE_SYSTEM_NAME, "file:///");
ps = new PigServer(ExecType.LOCAL, props);
}
ps.getPigContext().getProperties().setProperty(
MapReduceLauncher.SUCCESSFUL_JOB_OUTPUT_DIR_MARKER,
Boolean.toString(isPropertySet));
cleanupFiles(ps, files);
ps.setBatchOn();
Util.createInputFile(ps.getPigContext(),
inputFileName, inputData);
Util.registerMultiLineQuery(ps, script);
ps.executeBatch();
for(int i = 1; i <= (isMultiStore ? 3 : 1); i++) {
String sucFile = outputFileName + "_" + i + "/" +
MapReduceLauncher.SUCCEEDED_FILE_NAME;
assertEquals("Checking if _SUCCESS file exists in " +
execType + " mode", isPropertySet,
Util.exists(ps.getPigContext(), sucFile));
}
}
}
}
} finally {
cleanupFiles(ps, files);
}
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:61，
代码来源:TestStore.java
示例12: buildMRPlanWithOptimizer
点赞 1
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan buildMRPlanWithOptimizer(PhysicalPlan pp, PigContext pc) throws Exception {
MapRedUtil.checkLeafIsStore(pp, pc);
MapReduceLauncher launcher = new MapReduceLauncher();
java.lang.reflect.Method compile = launcher.getClass()
.getDeclaredMethod("compile",
new Class[] { PhysicalPlan.class, PigContext.class });
compile.setAccessible(true);
return (MROperPlan) compile.invoke(launcher, new Object[] { pp, pc });
}
开发者ID:sigmoidanalytics，
项目名称:spork-streaming，
代码行数:14，
代码来源:Util.java
示例13: buildMRPlanWithOptimizer
点赞 1
import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan buildMRPlanWithOptimizer(PhysicalPlan pp, PigContext pc) throws Exception {
MapRedUtil.checkLeafIsStore(pp, pc);
MapReduceLauncher launcher = new MapReduceLauncher();
java.lang.reflect.Method compile = launcher.getClass()
.getDeclaredMethod("compile",
new Class[] { PhysicalPlan.class, PigContext.class });
compile.setAccessible(true);
return (MROperPlan) compile.invoke(launcher, new Object[] { pp, pc });
}
开发者ID:PonIC，
项目名称:PonIC，
代码行数:14，
代码来源:Util.java
版权声明：本文转自网络文章，转载此文章仅为分享知识，如有侵权，请联系管理员进行删除。
