• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

Java MapReduceLauncher类的典型用法和代码示例

java 3次浏览

本文整理汇总了Java中org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher的典型用法代码示例。如果您正苦于以下问题:Java MapReduceLauncher类的具体用法?Java MapReduceLauncher怎么用?Java MapReduceLauncher使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

MapReduceLauncher类属于org.apache.pig.backend.hadoop.executionengine.mapReduceLayer包,在下文中一共展示了MapReduceLauncher类的13个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: executePlan

点赞 3

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
private boolean executePlan(PhysicalPlan pp) throws IOException {
    boolean failed = true;
    MapReduceLauncher launcher = new MapReduceLauncher();
    PigStats stats = null;
    try {
        stats = launcher.launchPig(pp, "execute", myPig.getPigContext());
    } catch (Exception e) {
        e.printStackTrace(System.out);
        throw new IOException(e);
    }
    Iterator<JobStats> iter = stats.getJobGraph().iterator();
    while (iter.hasNext()) {
        JobStats js = iter.next();
        failed = !js.isSuccessful();
        if (failed) {
            break;
        }
    }
    return !failed;
}
 

开发者ID:sigmoidanalytics,
项目名称:spork-streaming,
代码行数:21,
代码来源:TestMultiQueryLocal.java

示例2: compile

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public SOperPlan compile(PhysicalPlan php, PigContext pc) 
		throws PlanException, IOException, VisitorException {		
	MapReduceLauncher mrlauncher = new MapReduceLauncher();
	MROperPlan mrp = mrlauncher.compile(php, pc);
	
	MRtoSConverter converter = new MRtoSConverter(mrp, pc);
	converter.convert();
	
	return converter.getSPlan();
}
 

开发者ID:JamesLampton,
项目名称:piggybank-squeal,
代码行数:11,
代码来源:StormLauncher.java

示例3: explain

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
/**
 * Provide information on how a pig query will be executed.
 * @param alias Name of alias to explain.
 * @param format Format in which the explain should be printed.  If text, then the plan will
 * be printed in plain text.  Otherwise, the execution plan will be printed in
 * <a href="http://en.wikipedia.org/wiki/DOT_language">DOT</a> format.
 * @param verbose Controls the amount of information printed
 * @param markAsExecute When set will treat the explain like a
 * call to execute in the respoect that all the pending stores are
 * marked as complete.
 * @param lps Stream to print the logical tree
 * @param pps Stream to print the physical tree
 * @param eps Stream to print the execution tree
 * @throws IOException if the requested alias cannot be found.
 */
@SuppressWarnings("unchecked")
public void explain(String alias,
                    String format,
                    boolean verbose,
                    boolean markAsExecute,
                    PrintStream lps,
                    PrintStream pps,
                    PrintStream eps) throws IOException {
    try {
        pigContext.inExplain = true;
        buildStorePlan( alias );
        if( currDAG.lp.size() == 0 ) {
            lps.println("Logical plan is empty.");
            pps.println("Physical plan is empty.");
            eps.println("Execution plan is empty.");
            return;
        }
        PhysicalPlan pp = compilePp();
        currDAG.lp.explain(lps, format, verbose);

        pp.explain(pps, format, verbose);

        MapRedUtil.checkLeafIsStore(pp, pigContext);
        MapReduceLauncher launcher = new MapReduceLauncher();
        launcher.explain(pp, pigContext, eps, format, verbose);

        if (markAsExecute) {
            currDAG.markAsExecuted();
        }
    } catch (Exception e) {
        int errCode = 1067;
        String msg = "Unable to explain alias " + alias;
        throw new FrontendException(msg, errCode, PigException.INPUT, e);
    } finally {
        pigContext.inExplain = false;
    }
}
 

开发者ID:sigmoidanalytics,
项目名称:spork-streaming,
代码行数:53,
代码来源:PigServer.java

示例4: getMRPlan

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan getMRPlan(PhysicalPlan pp, PigContext ctx) throws Exception {
    MapReduceLauncher launcher = new MapReduceLauncher();
    java.lang.reflect.Method compile = launcher.getClass()
            .getDeclaredMethod("compile",
                    new Class[] { PhysicalPlan.class, PigContext.class });
    compile.setAccessible(true);
    return (MROperPlan) compile.invoke(launcher, new Object[] { pp, ctx });
}
 

开发者ID:sigmoidanalytics,
项目名称:spork-streaming,
代码行数:9,
代码来源:TestPigStats.java

示例5: explain

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
/**
 * Provide information on how a pig query will be executed.
 * @param alias Name of alias to explain.
 * @param format Format in which the explain should be printed.  If text, then the plan will
 * be printed in plain text.  Otherwise, the execution plan will be printed in
 * <a href="http://en.wikipedia.org/wiki/DOT_language">DOT</a> format.
 * @param verbose Controls the amount of information printed
 * @param markAsExecute When set will treat the explain like a
 * call to execute in the respoect that all the pending stores are
 * marked as complete.
 * @param lps Stream to print the logical tree
 * @param pps Stream to print the physical tree
 * @param eps Stream to print the execution tree
 * @throws IOException if the requested alias cannot be found.
 */
@SuppressWarnings("unchecked")
public void explain(String alias,
                    String format,
                    boolean verbose,
                    boolean markAsExecute,
                    PrintStream lps,
                    PrintStream pps,
                    PrintStream eps) throws IOException {
    try {
        pigContext.inExplain = true;
        buildStorePlan( alias );
        if( currDAG.lp.size() == 0 ) {
            lps.println("Logical plan is empty.");
            pps.println("Physical plan is empty.");
            eps.println("Execution plan is empty.");
            return;
        }
        PhysicalPlan pp = compilePp();
        currDAG.lp.explain(lps, format, verbose);

        pp.explain(pps, format, verbose);
        
        MapRedUtil.checkLeafIsStore(pp, pigContext);
        MapReduceLauncher launcher = new MapReduceLauncher();
        launcher.explain(pp, pigContext, eps, format, verbose);

        if (markAsExecute) {
            currDAG.markAsExecuted();
        }
    } catch (Exception e) {
        int errCode = 1067;
        String msg = "Unable to explain alias " + alias;
        throw new FrontendException(msg, errCode, PigException.INPUT, e);
    } finally {
        pigContext.inExplain = false;
    }
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:53,
代码来源:PigServer.java

示例6: testMapAggPropFalse

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggPropFalse() throws Exception{
    //test with pig.exec.mapPartAgg set to false
    String query = getGByQuery();
    pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "false");
    MROperPlan mrp = Util.buildMRPlan(query, pc);
    assertEquals(mrp.size(), 1);

    assertNull("POPartialAgg should be absent", findPOPartialAgg(mrp));
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:11,
代码来源:TestPOPartialAggPlan.java

示例7: testMapAggPropTrue

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggPropTrue() throws Exception{
    //test with pig.exec.mapPartAgg to true
    String query = getGByQuery();
    pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
    MROperPlan mrp = Util.buildMRPlan(query, pc);
    assertEquals(mrp.size(), 1);

    assertNotNull("POPartialAgg should be present",findPOPartialAgg(mrp));

}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:12,
代码来源:TestPOPartialAggPlan.java

示例8: testMapAggNoAggFunc

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggNoAggFunc() throws Exception{
    //no agg func, so there should not be a POPartial
    String query = "l = load 'x' as (a,b,c);" +
            "g = group l by a;" +
            "f = foreach g generate group;";
    pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
    MROperPlan mrp = Util.buildMRPlan(query, pc);
    assertEquals(mrp.size(), 1);

    assertNull("POPartialAgg should be absent",findPOPartialAgg(mrp));
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:13,
代码来源:TestPOPartialAggPlan.java

示例9: testMapAggNotCombinable

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testMapAggNotCombinable() throws Exception{
    //not combinable, so there should not be a POPartial
    String query = "l = load 'x' as (a,b,c);" +
            "g = group l by a;" +
            "f = foreach g generate group, COUNT(l.b), l.b;";
    pc.getProperties().setProperty(MapReduceLauncher.PROP_EXEC_MAP_PARTAGG, "true");
    MROperPlan mrp = Util.buildMRPlan(query, pc);
    assertEquals(mrp.size(), 1);

    assertNull("POPartialAgg should be absent", findPOPartialAgg(mrp));
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:13,
代码来源:TestPOPartialAggPlan.java

示例10: checkMRPlan

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
private MROperPlan checkMRPlan(PhysicalPlan pp, int expectedRoots,
        int expectedLeaves, int expectedSize) throws IOException {

    System.out.println("===== check map-reduce plan =====");

    MapRedUtil.checkLeafIsStore(pp, myPig.getPigContext());
    
    MapReduceLauncher launcher = new MapReduceLauncher();

    MROperPlan mrp = null;

    try {
        java.lang.reflect.Method compile = launcher.getClass()
                .getDeclaredMethod("compile",
                        new Class[] { PhysicalPlan.class, PigContext.class });

        compile.setAccessible(true);

        mrp = (MROperPlan) compile.invoke(launcher, new Object[] { pp, myPig.getPigContext() });

        Assert.assertNotNull(mrp);

    } catch (Exception e) {
        PigException pe = LogUtils.getPigException(e);
        if (pe != null) {
            throw pe;
        } else {
            e.printStackTrace();
            Assert.fail();
        }
    }        

    showPlanOperators(mrp);
    
    System.out.println("===== Display map-reduce Plan =====");
    System.out.println(mrp.toString());
    
    Assert.assertEquals(expectedRoots, mrp.getRoots().size());
    Assert.assertEquals(expectedLeaves, mrp.getLeaves().size());
    Assert.assertEquals(expectedSize, mrp.size());

    return mrp;
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:44,
代码来源:TestMultiQueryCompiler.java

示例11: testSuccessFileCreation1

点赞 2

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
@Test
public void testSuccessFileCreation1() throws Exception {
    PigServer ps = null;
    String[] files = new String[] { inputFileName, 
            outputFileName + "_1", outputFileName + "_2", outputFileName + "_3"};
    try {
        ExecType[] modes = new ExecType[] { ExecType.LOCAL, ExecType.MAPREDUCE};
        String[] inputData = new String[]{"hello\tworld", "hi\tworld", "bye\tworld"};
        
        String multiStoreScript = "a = load '"+ inputFileName + "';" +
                "b = filter a by $0 == 'hello';" +
                "c = filter a by $0 == 'hi';" +
                "d = filter a by $0 == 'bye';" +
                "store b into '" + outputFileName + "_1';" +
                "store c into '" + outputFileName + "_2';" + 
                "store d into '" + outputFileName + "_3';";
        
        String singleStoreScript =  "a = load '"+ inputFileName + "';" +
            "store a into '" + outputFileName + "_1';" ;
        
        for (ExecType execType : modes) {
            for(boolean isPropertySet: new boolean[] { true, false}) {
                for(boolean isMultiStore: new boolean[] { true, false}) {
                    String script = (isMultiStore ? multiStoreScript : 
                        singleStoreScript);
                    // since we will be switching between map red and local modes
                    // we will need to make sure filelocalizer is reset before each
                    // run.
                    FileLocalizer.setInitialized(false);
                    if(execType == ExecType.MAPREDUCE) {
                        ps = new PigServer(ExecType.MAPREDUCE, 
                                cluster.getProperties());
                    } else {
                        Properties props = new Properties();                                          
                        props.setProperty(MapRedUtil.FILE_SYSTEM_NAME, "file:///");
                        ps = new PigServer(ExecType.LOCAL, props);
                    }
                    ps.getPigContext().getProperties().setProperty(
                            MapReduceLauncher.SUCCESSFUL_JOB_OUTPUT_DIR_MARKER, 
                            Boolean.toString(isPropertySet));
                    cleanupFiles(ps, files);
                    ps.setBatchOn();
                    Util.createInputFile(ps.getPigContext(), 
                            inputFileName, inputData);
                    Util.registerMultiLineQuery(ps, script);
                    ps.executeBatch();
                    for(int i = 1; i <= (isMultiStore ? 3 : 1); i++) {
                        String sucFile = outputFileName + "_" + i + "/" + 
                                           MapReduceLauncher.SUCCEEDED_FILE_NAME;
                        assertEquals("Checking if _SUCCESS file exists in " + 
                                execType + " mode", isPropertySet, 
                                Util.exists(ps.getPigContext(), sucFile));
                    }
                }
            }
        }
    } finally {
        cleanupFiles(ps, files);
    }
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:61,
代码来源:TestStore.java

示例12: buildMRPlanWithOptimizer

点赞 1

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan buildMRPlanWithOptimizer(PhysicalPlan pp, PigContext pc) throws Exception {
    MapRedUtil.checkLeafIsStore(pp, pc);

    MapReduceLauncher launcher = new MapReduceLauncher();

    java.lang.reflect.Method compile = launcher.getClass()
            .getDeclaredMethod("compile",
                    new Class[] { PhysicalPlan.class, PigContext.class });

    compile.setAccessible(true);

    return (MROperPlan) compile.invoke(launcher, new Object[] { pp, pc });
}
 

开发者ID:sigmoidanalytics,
项目名称:spork-streaming,
代码行数:14,
代码来源:Util.java

示例13: buildMRPlanWithOptimizer

点赞 1

import org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher; //导入依赖的package包/类
public static MROperPlan buildMRPlanWithOptimizer(PhysicalPlan pp, PigContext pc) throws Exception {
    MapRedUtil.checkLeafIsStore(pp, pc);
    
    MapReduceLauncher launcher = new MapReduceLauncher();

    java.lang.reflect.Method compile = launcher.getClass()
            .getDeclaredMethod("compile",
                    new Class[] { PhysicalPlan.class, PigContext.class });

    compile.setAccessible(true);

    return (MROperPlan) compile.invoke(launcher, new Object[] { pp, pc });
}
 

开发者ID:PonIC,
项目名称:PonIC,
代码行数:14,
代码来源:Util.java


版权声明:本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系管理员进行删除。
喜欢 (0)