• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

Java CompactionDescriptor类的典型用法和代码示例

java 1次浏览

本文整理汇总了Java中org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor的典型用法代码示例。如果您正苦于以下问题:Java CompactionDescriptor类的具体用法?Java CompactionDescriptor怎么用?Java CompactionDescriptor使用的例子?那么恭喜您, 这里精选的类代码示例或许可以为您提供帮助。

CompactionDescriptor类属于org.apache.hadoop.hbase.protobuf.generated.WALProtos包,在下文中一共展示了CompactionDescriptor类的28个代码示例,这些例子默认根据受欢迎程度排序。您可以为喜欢或者感觉有用的代码点赞,您的评价将有助于我们的系统推荐出更棒的Java代码示例。

示例1: testReplayingCompactionWithFileAlreadyDeleted

点赞 3

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
@Test
public void testReplayingCompactionWithFileAlreadyDeleted() throws IOException {
  // tests replaying compaction marker, but the compaction output file has already been compacted
  // from primary and also deleted from the archive directory
  secondaryRegion.replayWALCompactionMarker(CompactionDescriptor.newBuilder()
    .setTableName(ByteString.copyFrom(primaryRegion.getTableDesc().getTableName().getName()))
    .setEncodedRegionName(
      ByteString.copyFrom(primaryRegion.getRegionInfo().getEncodedNameAsBytes()))
    .setFamilyName(ByteString.copyFrom(families[0]))
    .addCompactionInput("/foo")
    .addCompactionOutput("/bar")
    .setStoreHomeDir("/store_home_dir")
    .setRegionName(ByteString.copyFrom(primaryRegion.getRegionInfo().getRegionName()))
    .build()
    , true, true, Long.MAX_VALUE);
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:17,
代码来源:TestHRegionReplayEvents.java

示例2: toCompactionDescriptor

点赞 3

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
@SuppressWarnings("deprecation")
public static CompactionDescriptor toCompactionDescriptor(HRegionInfo info, byte[] regionName,
    byte[] family, List<Path> inputPaths, List<Path> outputPaths, Path storeDir) {
  // compaction descriptor contains relative paths.
  // input / output paths are relative to the store dir
  // store dir is relative to region dir
  CompactionDescriptor.Builder builder = CompactionDescriptor.newBuilder()
      .setTableName(ByteStringer.wrap(info.getTableName()))
      .setEncodedRegionName(ByteStringer.wrap(
        regionName == null ? info.getEncodedNameAsBytes() : regionName))
      .setFamilyName(ByteStringer.wrap(family))
      .setStoreHomeDir(storeDir.getName()); //make relative
  for (Path inputPath : inputPaths) {
    builder.addCompactionInput(inputPath.getName()); //relative path
  }
  for (Path outputPath : outputPaths) {
    builder.addCompactionOutput(outputPath.getName());
  }
  builder.setRegionName(ByteStringer.wrap(info.getRegionName()));
  return builder.build();
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:22,
代码来源:ProtobufUtil.java

示例3: toCompactionDescriptor

点赞 3

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
public static CompactionDescriptor toCompactionDescriptor(HRegionInfo info, byte[] family,
    List<Path> inputPaths, List<Path> outputPaths, Path storeDir) {
  // compaction descriptor contains relative paths.
  // input / output paths are relative to the store dir
  // store dir is relative to region dir
  CompactionDescriptor.Builder builder = CompactionDescriptor.newBuilder()
      .setTableName(ByteStringer.wrap(info.getTableName()))
      .setEncodedRegionName(ByteStringer.wrap(info.getEncodedNameAsBytes()))
      .setFamilyName(ByteStringer.wrap(family))
      .setStoreHomeDir(storeDir.getName()); //make relative
  for (Path inputPath : inputPaths) {
    builder.addCompactionInput(inputPath.getName()); //relative path
  }
  for (Path outputPath : outputPaths) {
    builder.addCompactionOutput(outputPath.getName());
  }
  builder.setRegionName(ByteStringer.wrap(info.getRegionName()));
  return builder.build();
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:20,
代码来源:ProtobufUtil.java

示例4: toCompactionDescriptor

点赞 3

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
public static CompactionDescriptor toCompactionDescriptor(HRegionInfo info, byte[] family,
    List<Path> inputPaths, List<Path> outputPaths, Path storeDir) {
  // compaction descriptor contains relative paths.
  // input / output paths are relative to the store dir
  // store dir is relative to region dir
  CompactionDescriptor.Builder builder = CompactionDescriptor.newBuilder()
      .setTableName(HBaseZeroCopyByteString.wrap(info.getTableName()))
      .setEncodedRegionName(HBaseZeroCopyByteString.wrap(info.getEncodedNameAsBytes()))
      .setFamilyName(HBaseZeroCopyByteString.wrap(family))
      .setStoreHomeDir(storeDir.getName()); //make relative
  for (Path inputPath : inputPaths) {
    builder.addCompactionInput(inputPath.getName()); //relative path
  }
  for (Path outputPath : outputPaths) {
    builder.addCompactionOutput(outputPath.getName());
  }
  return builder.build();
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:19,
代码来源:ProtobufUtil.java

示例5: toCompactionDescriptor

点赞 3

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
public static CompactionDescriptor toCompactionDescriptor(HRegionInfo info, byte[] family,
    List<Path> inputPaths, List<Path> outputPaths, Path storeDir) {
  // compaction descriptor contains relative paths.
  // input / output paths are relative to the store dir
  // store dir is relative to region dir
  CompactionDescriptor.Builder builder = CompactionDescriptor.newBuilder()
      .setTableName(ZeroCopyLiteralByteString.wrap(info.getTableName()))
      .setEncodedRegionName(ZeroCopyLiteralByteString.wrap(info.getEncodedNameAsBytes()))
      .setFamilyName(ZeroCopyLiteralByteString.wrap(family))
      .setStoreHomeDir(storeDir.getName()); //make relative
  for (Path inputPath : inputPaths) {
    builder.addCompactionInput(inputPath.getName()); //relative path
  }
  for (Path outputPath : outputPaths) {
    builder.addCompactionOutput(outputPath.getName());
  }
  return builder.build();
}
 

开发者ID:cloud-software-foundation,
项目名称:c5,
代码行数:19,
代码来源:ProtobufUtil.java

示例6: createCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Create a compaction WALEdit
 * @param c
 * @return A WALEdit that has <code>c</code> serialized as its value
 */
public static WALEdit createCompaction(final HRegionInfo hri, final CompactionDescriptor c) {
  byte [] pbbytes = c.toByteArray();
  KeyValue kv = new KeyValue(getRowForRegion(hri), METAFAMILY, COMPACTION,
    EnvironmentEdgeManager.currentTime(), pbbytes);
  return new WALEdit().add(kv); //replication scope null so that this won't be replicated
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:12,
代码来源:WALEdit.java

示例7: getCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Deserialized and returns a CompactionDescriptor is the KeyValue contains one.
 * @param kv the key value
 * @return deserialized CompactionDescriptor or null.
 */
public static CompactionDescriptor getCompaction(Cell kv) throws IOException {
  if (CellUtil.matchingColumn(kv, METAFAMILY, COMPACTION)) {
    return CompactionDescriptor.parseFrom(kv.getValue());
  }
  return null;
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:12,
代码来源:WALEdit.java

示例8: writeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Write the marker that a compaction has succeeded and is about to be committed.
 * This provides info to the HMaster to allow it to recover the compaction if
 * this regionserver dies in the middle (This part is not yet implemented). It also prevents
 * the compaction from finishing if this regionserver has already lost its lease on the log.
 * @param mvcc Used by WAL to get sequence Id for the waledit.
 */
public static long writeCompactionMarker(WAL wal, HTableDescriptor htd, HRegionInfo hri,
    final CompactionDescriptor c, MultiVersionConcurrencyControl mvcc)
throws IOException {
  long trx = writeMarker(wal, htd, hri, WALEdit.createCompaction(hri, c), mvcc, true);
  if (LOG.isTraceEnabled()) {
    LOG.trace("Appended compaction marker " + TextFormat.shortDebugString(c));
  }
  return trx;
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:17,
代码来源:WALUtil.java

示例9: createCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Create a compacion WALEdit
 * @param c
 * @return A WALEdit that has <code>c</code> serialized as its value
 */
public static WALEdit createCompaction(final HRegionInfo hri, final CompactionDescriptor c) {
  byte [] pbbytes = c.toByteArray();
  KeyValue kv = new KeyValue(getRowForRegion(hri), METAFAMILY, COMPACTION,
    EnvironmentEdgeManager.currentTime(), pbbytes);
  return new WALEdit().add(kv); //replication scope null so that this won't be replicated
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:12,
代码来源:WALEdit.java

示例10: writeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Write the marker that a compaction has succeeded and is about to be committed.
 * This provides info to the HMaster to allow it to recover the compaction if
 * this regionserver dies in the middle (This part is not yet implemented). It also prevents
 * the compaction from finishing if this regionserver has already lost its lease on the log.
 * @param sequenceId Used by WAL to get sequence Id for the waledit.
 */
public static void writeCompactionMarker(WAL log, HTableDescriptor htd, HRegionInfo info,
    final CompactionDescriptor c, AtomicLong sequenceId) throws IOException {
  TableName tn = TableName.valueOf(c.getTableName().toByteArray());
  // we use HLogKey here instead of WALKey directly to support legacy coprocessors.
  WALKey key = new HLogKey(info.getEncodedNameAsBytes(), tn);
  log.append(htd, info, key, WALEdit.createCompaction(info, c), sequenceId, false, null);
  log.sync();
  if (LOG.isTraceEnabled()) {
    LOG.trace("Appended compaction marker " + TextFormat.shortDebugString(c));
  }
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:19,
代码来源:WALUtil.java

示例11: completeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2331.
 */
void completeCompactionMarker(CompactionDescriptor compaction)
        throws IOException {
    Store store = this.getStore(compaction.getFamilyName().toByteArray());
    if (store == null) {
        LOG.warn("Found Compaction WAL edit for deleted family:" +
                Bytes.toString(compaction.getFamilyName().toByteArray()));
        return;
    }
    store.completeCompactionMarker(compaction);
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:16,
代码来源:HRegion.java

示例12: completeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2231.
 *
 * @param compaction
 */
@Override
public void completeCompactionMarker(CompactionDescriptor compaction)
        throws IOException {
    LOG.debug("Completing compaction from the WAL marker");
    List<String> compactionInputs = compaction.getCompactionInputList();

    // The Compaction Marker is written after the compaction is completed,
    // and the files moved into the region/family folder.
    //
    // If we crash after the entry is written, we may not have removed the
    // input files, but the output file is present.
    // (The unremoved input files will be removed by this function)
    //
    // If we scan the directory and the file is not present, it can mean that:
    //   - The file was manually removed by the user
    //   - The file was removed as consequence of subsequent compaction
    // so, we can't do anything with the "compaction output list" because those
    // files have already been loaded when opening the region (by virtue of
    // being in the store's folder) or they may be missing due to a compaction.

    String familyName = this.getColumnFamilyName();
    List<Path> inputPaths = new ArrayList<Path>(compactionInputs.size());
    for (String compactionInput : compactionInputs) {
        Path inputPath = fs.getStoreFilePath(familyName, compactionInput);
        inputPaths.add(inputPath);
    }

    //some of the input files might already be deleted
    List<StoreFile> inputStoreFiles = new ArrayList<StoreFile>(compactionInputs.size());
    for (StoreFile sf : this.getStorefiles()) {
        if (inputPaths.contains(sf.getQualifiedPath())) {
            inputStoreFiles.add(sf);
        }
    }

    this.replaceStoreFiles(inputStoreFiles, Collections.EMPTY_LIST);
    this.completeCompaction(inputStoreFiles);
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:46,
代码来源:HStore.java

示例13: getCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Deserialized and returns a CompactionDescriptor is the KeyValue contains one.
 * @param kv the key value
 * @return deserialized CompactionDescriptor or null.
 */
public static CompactionDescriptor getCompaction(KeyValue kv) throws IOException {
  if (kv.matchingRow(METAROW) && kv.matchingColumn(METAFAMILY, COMPACTION)) {
    return CompactionDescriptor.parseFrom(kv.getValue());
  }
  return null;
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:12,
代码来源:WALEdit.java

示例14: writeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Write the marker that a compaction has succeeded and is about to be committed.
 * This provides info to the HMaster to allow it to recover the compaction if
 * this regionserver dies in the middle (This part is not yet implemented). It also prevents
 * the compaction from finishing if this regionserver has already lost its lease on the log.
 * @param sequenceId Used by HLog to get sequence Id for the waledit.
 */
public static void writeCompactionMarker(HLog log, HTableDescriptor htd, HRegionInfo info,
    final CompactionDescriptor c, AtomicLong sequenceId) throws IOException {
  WALEdit e = WALEdit.createCompaction(c);
  long now = EnvironmentEdgeManager.currentTimeMillis();
  TableName tn = TableName.valueOf(c.getTableName().toByteArray());
  long txid = log.appendNoSync(info, tn, e, new ArrayList<UUID>(), now, htd, sequenceId,
      false, HConstants.NO_NONCE, HConstants.NO_NONCE);
  log.sync(txid);

  if (LOG.isTraceEnabled()) {
    LOG.trace("Appended compaction marker " + TextFormat.shortDebugString(c));
  }
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:21,
代码来源:HLogUtil.java

示例15: completeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2331.
 * @param compaction
 */
void completeCompactionMarker(CompactionDescriptor compaction)
    throws IOException {
  Store store = this.getStore(compaction.getFamilyName().toByteArray());
  if (store == null) {
    LOG.warn("Found Compaction WAL edit for deleted family:" +
        Bytes.toString(compaction.getFamilyName().toByteArray()));
    return;
  }
  store.completeCompactionMarker(compaction);
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:17,
代码来源:HRegion.java

示例16: completeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2231.
 * @param compaction
 */
@Override
public void completeCompactionMarker(CompactionDescriptor compaction)
    throws IOException {
  LOG.debug("Completing compaction from the WAL marker");
  List<String> compactionInputs = compaction.getCompactionInputList();

  // The Compaction Marker is written after the compaction is completed,
  // and the files moved into the region/family folder.
  //
  // If we crash after the entry is written, we may not have removed the
  // input files, but the output file is present.
  // (The unremoved input files will be removed by this function)
  //
  // If we scan the directory and the file is not present, it can mean that:
  //   - The file was manually removed by the user
  //   - The file was removed as consequence of subsequent compaction
  // so, we can't do anything with the "compaction output list" because those
  // files have already been loaded when opening the region (by virtue of
  // being in the store's folder) or they may be missing due to a compaction.

  String familyName = this.getColumnFamilyName();
  List<Path> inputPaths = new ArrayList<Path>(compactionInputs.size());
  for (String compactionInput : compactionInputs) {
    Path inputPath = fs.getStoreFilePath(familyName, compactionInput);
    inputPaths.add(inputPath);
  }

  //some of the input files might already be deleted
  List<StoreFile> inputStoreFiles = new ArrayList<StoreFile>(compactionInputs.size());
  for (StoreFile sf : this.getStorefiles()) {
    if (inputPaths.contains(sf.getQualifiedPath())) {
      inputStoreFiles.add(sf);
    }
  }

  this.replaceStoreFiles(inputStoreFiles, Collections.EMPTY_LIST);
  this.completeCompaction(inputStoreFiles);
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:45,
代码来源:HStore.java

示例17: getCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Deserialized and returns a CompactionDescriptor is the KeyValue contains one.
 * @param kv the key value
 * @return deserialized CompactionDescriptor or null.
 */
public static CompactionDescriptor getCompaction(KeyValue kv) throws IOException {
  if (CellUtil.matchingRow(kv, METAROW) && CellUtil.matchingColumn(kv, METAFAMILY, COMPACTION)) {
    return CompactionDescriptor.parseFrom(kv.getValue());
  }
  return null;
}
 

开发者ID:shenli-uiuc,
项目名称:PyroDB,
代码行数:12,
代码来源:WALEdit.java

示例18: writeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Write the marker that a compaction has succeeded and is about to be committed.
 * This provides info to the HMaster to allow it to recover the compaction if
 * this regionserver dies in the middle (This part is not yet implemented). It also prevents
 * the compaction from finishing if this regionserver has already lost its lease on the log.
 * @param sequenceId Used by HLog to get sequence Id for the waledit.
 */
public static void writeCompactionMarker(HLog log, HTableDescriptor htd, HRegionInfo info,
    final CompactionDescriptor c, AtomicLong sequenceId) throws IOException {
  TableName tn = TableName.valueOf(c.getTableName().toByteArray());
  HLogKey key = new HLogKey(info.getEncodedNameAsBytes(), tn);
  log.appendNoSync(htd, info, key, WALEdit.createCompaction(c), sequenceId, false);
  log.sync();
  if (LOG.isTraceEnabled()) {
    LOG.trace("Appended compaction marker " + TextFormat.shortDebugString(c));
  }
}
 

开发者ID:shenli-uiuc,
项目名称:PyroDB,
代码行数:18,
代码来源:HLogUtil.java

示例19: writeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Write the marker that a compaction has succeeded and is about to be committed.
 * This provides info to the HMaster to allow it to recover the compaction if
 * this regionserver dies in the middle (This part is not yet implemented). It also prevents
 * the compaction from finishing if this regionserver has already lost its lease on the log.
 */
public static void writeCompactionMarker(HLog log, HTableDescriptor htd, HRegionInfo info,
    final CompactionDescriptor c) throws IOException {
  WALEdit e = WALEdit.createCompaction(c);
  log.append(info, TableName.valueOf(c.getTableName().toByteArray()), e,
      EnvironmentEdgeManager.currentTimeMillis(), htd, false);
  if (LOG.isTraceEnabled()) {
    LOG.trace("Appended compaction marker " + TextFormat.shortDebugString(c));
  }
}
 

开发者ID:cloud-software-foundation,
项目名称:c5,
代码行数:16,
代码来源:HLogUtil.java

示例20: completeCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2331.
 * @param fs
 * @param compaction
 */
void completeCompactionMarker(CompactionDescriptor compaction)
    throws IOException {
  Store store = this.getStore(compaction.getFamilyName().toByteArray());
  if (store == null) {
    LOG.warn("Found Compaction WAL edit for deleted family:" +
        Bytes.toString(compaction.getFamilyName().toByteArray()));
    return;
  }
  store.completeCompactionMarker(compaction);
}
 

开发者ID:cloud-software-foundation,
项目名称:c5,
代码行数:18,
代码来源:HRegion.java

示例21: replayWALCompactionMarker

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction that was
 * not finished. We could find one recovering a WAL after a regionserver crash. See HBASE-2331.
 */
void replayWALCompactionMarker(CompactionDescriptor compaction, boolean pickCompactionFiles,
    boolean removeFiles, long replaySeqId) throws IOException {
  try {
    checkTargetRegion(compaction.getEncodedRegionName().toByteArray(),
        "Compaction marker from WAL ", compaction);
  } catch (WrongRegionException wre) {
    if (RegionReplicaUtil.isDefaultReplica(this.getRegionInfo())) {
      // skip the compaction marker since it is not for this region
      return;
    }
    throw wre;
  }

  synchronized (writestate) {
    if (replaySeqId < lastReplayedOpenRegionSeqId) {
      LOG.warn(getRegionInfo().getEncodedName() + " : " + "Skipping replaying compaction event :"
          + TextFormat.shortDebugString(compaction) + " because its sequence id " + replaySeqId
          + " is smaller than this regions " + "lastReplayedOpenRegionSeqId of "
          + lastReplayedOpenRegionSeqId);
      return;
    }
    if (replaySeqId < lastReplayedCompactionSeqId) {
      LOG.warn(getRegionInfo().getEncodedName() + " : " + "Skipping replaying compaction event :"
          + TextFormat.shortDebugString(compaction) + " because its sequence id " + replaySeqId
          + " is smaller than this regions " + "lastReplayedCompactionSeqId of "
          + lastReplayedCompactionSeqId);
      return;
    } else {
      lastReplayedCompactionSeqId = replaySeqId;
    }

    if (LOG.isDebugEnabled()) {
      LOG.debug(
          getRegionInfo().getEncodedName() + " : " + "Replaying compaction marker " + TextFormat
              .shortDebugString(compaction) + " with seqId=" + replaySeqId
              + " and lastReplayedOpenRegionSeqId=" + lastReplayedOpenRegionSeqId);
    }

    startRegionOperation(Operation.REPLAY_EVENT);
    try {
      Store store = this.getStore(compaction.getFamilyName().toByteArray());
      if (store == null) {
        LOG.warn(getRegionInfo().getEncodedName() + " : "
            + "Found Compaction WAL edit for deleted family:" + Bytes
            .toString(compaction.getFamilyName().toByteArray()));
        return;
      }
      store.replayCompactionMarker(compaction, pickCompactionFiles, removeFiles);
      logRegionFiles();
    } catch (FileNotFoundException ex) {
      LOG.warn(getRegionInfo().getEncodedName() + " : "
              + "At least one of the store files in compaction: " + TextFormat
              .shortDebugString(compaction) + " doesn't exist any more. Skip loading the file(s)",
          ex);
    } finally {
      closeRegionOperation(Operation.REPLAY_EVENT);
    }
  }
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:64,
代码来源:HRegion.java

示例22: doReplayBatchOp

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Execute a list of Put/Delete mutations. The function returns OperationStatus instead of
 * constructing MultiResponse to save a possible loop if caller doesn't need MultiResponse.
 * @param region
 * @param mutations
 * @param replaySeqId
 * @return an array of OperationStatus which internally contains the OperationStatusCode and the
 *         exceptionMessage if any
 * @throws IOException
 */
private OperationStatus [] doReplayBatchOp(final Region region,
    final List<WALSplitter.MutationReplay> mutations, long replaySeqId) throws IOException {
  long before = EnvironmentEdgeManager.currentTime();
  boolean batchContainsPuts = false, batchContainsDelete = false;
  try {
    for (Iterator<WALSplitter.MutationReplay> it = mutations.iterator(); it.hasNext();) {
      WALSplitter.MutationReplay m = it.next();

      if (m.type == MutationType.PUT) {
        batchContainsPuts = true;
      } else {
        batchContainsDelete = true;
      }

      NavigableMap<byte[], List<Cell>> map = m.mutation.getFamilyCellMap();
      List<Cell> metaCells = map.get(WALEdit.METAFAMILY);
      if (metaCells != null && !metaCells.isEmpty()) {
        for (Cell metaCell : metaCells) {
          CompactionDescriptor compactionDesc = WALEdit.getCompaction(metaCell);
          boolean isDefaultReplica = RegionReplicaUtil.isDefaultReplica(region.getRegionInfo());
          HRegion hRegion = (HRegion)region;
          if (compactionDesc != null) {
            // replay the compaction. Remove the files from stores only if we are the primary
            // region replica (thus own the files)
            hRegion.replayWALCompactionMarker(compactionDesc, !isDefaultReplica, isDefaultReplica,
              replaySeqId);
            continue;
          }
          FlushDescriptor flushDesc = WALEdit.getFlushDescriptor(metaCell);
          if (flushDesc != null && !isDefaultReplica) {
            hRegion.replayWALFlushMarker(flushDesc, replaySeqId);
            continue;
          }
          RegionEventDescriptor regionEvent = WALEdit.getRegionEventDescriptor(metaCell);
          if (regionEvent != null && !isDefaultReplica) {
            hRegion.replayWALRegionEventMarker(regionEvent);
            continue;
          }
          BulkLoadDescriptor bulkLoadEvent = WALEdit.getBulkLoadDescriptor(metaCell);
          if (bulkLoadEvent != null) {
            hRegion.replayWALBulkLoadEventMarker(bulkLoadEvent);
            continue;
          }
        }
        it.remove();
      }
    }
    requestCount.add(mutations.size());
    if (!region.getRegionInfo().isMetaTable()) {
      regionServer.cacheFlusher.reclaimMemStoreMemory();
    }
    return region.batchReplay(mutations.toArray(
      new WALSplitter.MutationReplay[mutations.size()]), replaySeqId);
  } finally {
    if (regionServer.metricsRegionServer != null) {
      long after = EnvironmentEdgeManager.currentTime();
        if (batchContainsPuts) {
        regionServer.metricsRegionServer.updatePut(after - before);
      }
      if (batchContainsDelete) {
        regionServer.metricsRegionServer.updateDelete(after - before);
      }
    }
  }
}
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:76,
代码来源:RSRpcServices.java

示例23: doReplayBatchOp

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Execute a list of Put/Delete mutations. The function returns OperationStatus instead of
 * constructing MultiResponse to save a possible loop if caller doesn't need MultiResponse.
 *
 * @param region
 * @param mutations
 * @param replaySeqId
 * @return an array of OperationStatus which internally contains the OperationStatusCode and the
 * exceptionMessage if any
 * @throws IOException
 */
private OperationStatus[] doReplayBatchOp(final HRegion region,
                                          final List<WALSplitter.MutationReplay> mutations, long replaySeqId) throws IOException {

    long before = EnvironmentEdgeManager.currentTime();
    boolean batchContainsPuts = false, batchContainsDelete = false;
    try {
        for (Iterator<WALSplitter.MutationReplay> it = mutations.iterator(); it.hasNext(); ) {
            WALSplitter.MutationReplay m = it.next();

            if (m.type == MutationType.PUT) {
                batchContainsPuts = true;
            } else {
                batchContainsDelete = true;
            }

            NavigableMap<byte[], List<Cell>> map = m.mutation.getFamilyCellMap();
            List<Cell> metaCells = map.get(WALEdit.METAFAMILY);
            if (metaCells != null && !metaCells.isEmpty()) {
                for (Cell metaCell : metaCells) {
                    CompactionDescriptor compactionDesc = WALEdit.getCompaction(metaCell);
                    if (compactionDesc != null) {
                        region.completeCompactionMarker(compactionDesc);
                    }
                }
                it.remove();
            }
        }
        requestCount.add(mutations.size());
        if (!region.getRegionInfo().isMetaTable()) {
            regionServer.cacheFlusher.reclaimMemStoreMemory();
        }
        return region.batchReplay(mutations.toArray(
                new WALSplitter.MutationReplay[mutations.size()]), replaySeqId);
    } finally {
        if (regionServer.metricsRegionServer != null) {
            long after = EnvironmentEdgeManager.currentTime();
            if (batchContainsPuts) {
                regionServer.metricsRegionServer.updatePut(after - before);
            }
            if (batchContainsDelete) {
                regionServer.metricsRegionServer.updateDelete(after - before);
            }
        }
    }
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:57,
代码来源:RSRpcServices.java

示例24: testSkipRecoveredEditsReplayTheLastFileIgnored

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
@Test
public void testSkipRecoveredEditsReplayTheLastFileIgnored() throws Exception {
  String method = "testSkipRecoveredEditsReplayTheLastFileIgnored";
  TableName tableName = TableName.valueOf(method);
  byte[] family = Bytes.toBytes("family");
  this.region = initHRegion(tableName, method, CONF, family);
  final WALFactory wals = new WALFactory(CONF, null, method);
  try {
    Path regiondir = region.getRegionFileSystem().getRegionDir();
    FileSystem fs = region.getRegionFileSystem().getFileSystem();
    byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes();

    assertEquals(0, region.getStoreFileList(
      region.getStores().keySet().toArray(new byte[0][])).size());

    Path recoveredEditsDir = WALSplitter.getRegionDirRecoveredEditsDir(regiondir);

    long maxSeqId = 1050;
    long minSeqId = 1000;

    for (long i = minSeqId; i <= maxSeqId; i += 10) {
      Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i));
      fs.create(recoveredEdits);
      WALProvider.Writer writer = wals.createRecoveredEditsWriter(fs, recoveredEdits);

      long time = System.nanoTime();
      WALEdit edit = null;
      if (i == maxSeqId) {
        edit = WALEdit.createCompaction(region.getRegionInfo(),
        CompactionDescriptor.newBuilder()
        .setTableName(ByteString.copyFrom(tableName.getName()))
        .setFamilyName(ByteString.copyFrom(regionName))
        .setEncodedRegionName(ByteString.copyFrom(regionName))
        .setStoreHomeDirBytes(ByteString.copyFrom(Bytes.toBytes(regiondir.toString())))
        .setRegionName(ByteString.copyFrom(region.getRegionInfo().getRegionName()))
        .build());
      } else {
        edit = new WALEdit();
        edit.add(new KeyValue(row, family, Bytes.toBytes(i), time, KeyValue.Type.Put, Bytes
          .toBytes(i)));
      }
      writer.append(new WAL.Entry(new HLogKey(regionName, tableName, i, time,
          HConstants.DEFAULT_CLUSTER_ID), edit));
      writer.close();
    }

    long recoverSeqId = 1030;
    Map<byte[], Long> maxSeqIdInStores = new TreeMap<byte[], Long>(Bytes.BYTES_COMPARATOR);
    MonitoredTask status = TaskMonitor.get().createStatus(method);
    for (Store store : region.getStores().values()) {
      maxSeqIdInStores.put(store.getColumnFamilyName().getBytes(), recoverSeqId - 1);
    }
    long seqId = region.replayRecoveredEditsIfAny(regiondir, maxSeqIdInStores, null, status);
    assertEquals(maxSeqId, seqId);

    // assert that the files are flushed
    assertEquals(1, region.getStoreFileList(
      region.getStores().keySet().toArray(new byte[0][])).size());

  } finally {
    HRegion.closeHRegion(this.region);
    this.region = null;
    wals.close();
  }
}
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:66,
代码来源:TestHRegion.java

示例25: createCompaction

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Create a compacion WALEdit
 * @param c
 * @return A WALEdit that has <code>c</code> serialized as its value
 */
public static WALEdit createCompaction(final CompactionDescriptor c) {
  byte [] pbbytes = c.toByteArray();
  KeyValue kv = new KeyValue(METAROW, METAFAMILY, COMPACTION, System.currentTimeMillis(), pbbytes);
  return new WALEdit().add(kv); //replication scope null so that this won't be replicated
}
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:11,
代码来源:WALEdit.java

示例26: testSkipRecoveredEditsReplayTheLastFileIgnored

点赞 2

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
@Test
public void testSkipRecoveredEditsReplayTheLastFileIgnored() throws Exception {
  String method = "testSkipRecoveredEditsReplayTheLastFileIgnored";
  TableName tableName = TableName.valueOf(method);
  byte[] family = Bytes.toBytes("family");
  this.region = initHRegion(tableName, method, CONF, family);
  try {
    Path regiondir = region.getRegionFileSystem().getRegionDir();
    FileSystem fs = region.getRegionFileSystem().getFileSystem();
    byte[] regionName = region.getRegionInfo().getEncodedNameAsBytes();

    assertEquals(0, region.getStoreFileList(
      region.getStores().keySet().toArray(new byte[0][])).size());

    Path recoveredEditsDir = HLogUtil.getRegionDirRecoveredEditsDir(regiondir);

    long maxSeqId = 1050;
    long minSeqId = 1000;

    for (long i = minSeqId; i <= maxSeqId; i += 10) {
      Path recoveredEdits = new Path(recoveredEditsDir, String.format("%019d", i));
      fs.create(recoveredEdits);
      HLog.Writer writer = HLogFactory.createRecoveredEditsWriter(fs, recoveredEdits, CONF);

      long time = System.nanoTime();
      WALEdit edit = null;
      if (i == maxSeqId) {
        edit = WALEdit.createCompaction(CompactionDescriptor.newBuilder()
        .setTableName(ByteString.copyFrom(tableName.getName()))
        .setFamilyName(ByteString.copyFrom(regionName))
        .setEncodedRegionName(ByteString.copyFrom(regionName))
        .setStoreHomeDirBytes(ByteString.copyFrom(Bytes.toBytes(regiondir.toString())))
        .build());
      } else {
        edit = new WALEdit();
        edit.add(new KeyValue(row, family, Bytes.toBytes(i), time, KeyValue.Type.Put, Bytes
          .toBytes(i)));
      }
      writer.append(new HLog.Entry(new HLogKey(regionName, tableName, i, time,
          HConstants.DEFAULT_CLUSTER_ID), edit));
      writer.close();
    }

    long recoverSeqId = 1030;
    Map<byte[], Long> maxSeqIdInStores = new TreeMap<byte[], Long>(Bytes.BYTES_COMPARATOR);
    MonitoredTask status = TaskMonitor.get().createStatus(method);
    for (Store store : region.getStores().values()) {
      maxSeqIdInStores.put(store.getColumnFamilyName().getBytes(), recoverSeqId - 1);
    }
    long seqId = region.replayRecoveredEditsIfAny(regiondir, maxSeqIdInStores, null, status);
    assertEquals(maxSeqId, seqId);

    // assert that the files are flushed
    assertEquals(1, region.getStoreFileList(
      region.getStores().keySet().toArray(new byte[0][])).size());

  } finally {
    HRegion.closeHRegion(this.region);
    this.region = null;
  }  }
 

开发者ID:tenggyut,
项目名称:HIndex,
代码行数:61,
代码来源:TestHRegion.java

示例27: replayCompactionMarker

点赞 1

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a
 * compaction that was not finished. We could find one recovering a WAL after
 * a regionserver crash. See HBASE-2331.
 * 
 * @param compaction the descriptor for compaction
 * @param pickCompactionFiles whether or not pick up the new compaction output
 *          files and add it to the store
 * @param removeFiles whether to remove/archive files from filesystem
 */
void replayCompactionMarker(CompactionDescriptor compaction,
    boolean pickCompactionFiles, boolean removeFiles) throws IOException;
 

开发者ID:fengchen8086,
项目名称:ditb,
代码行数:13,
代码来源:Store.java

示例28: completeCompactionMarker

点赞 1

import org.apache.hadoop.hbase.protobuf.generated.WALProtos.CompactionDescriptor; //导入依赖的package包/类
/**
 * Call to complete a compaction. Its for the case where we find in the WAL a compaction
 * that was not finished.  We could find one recovering a WAL after a regionserver crash.
 * See HBASE-2331.
 * @param compaction
 */
void completeCompactionMarker(CompactionDescriptor compaction)
    throws IOException;
 

开发者ID:grokcoder,
项目名称:pbase,
代码行数:9,
代码来源:Store.java


版权声明:本文转自网络文章,转载此文章仅为分享知识,如有侵权,请联系管理员进行删除。
喜欢 (0)