Flume1.6.0之Error-protobuf-This is supposed to be overridden by subclasses-白红宇

1.场景:

Apache Flume: Agent 【http-->memory-->hdfs(CDH4)】 (http发送请求,通过内存,然后写到cdh4的hdfs上)

当前flume agent机器有【CDH4】环境的文件(而没有Apache hadoop环境),

故 JAVA_HOME=/opt/cloudera/parcels/CDH/lib/hadoop

2.错误:

2016-05-21 19:31:27,756 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.hdfs.BucketWriter.open(BucketWriter.java:234)] Creating hdfs://alish1-dataservice-01.mypna.cn:8022/testwjp/2016-05-21/19/FlumeData.1463830281582.tmp

2016-05-21 19:31:27,791 (SinkRunner-PollingRunner-DefaultSinkProcessor) [ERROR - org.apache.flume.sink.hdfs.HDFSEventSink.process(HDFSEventSink.java:459)] process failed

java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)

at org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$FsPermissionProto.getSerializedSize(HdfsProtos.java:5407)

at com.google.protobuf.CodedOutputStream.computeMessageSizeNoTag(CodedOutputStream.java:749)

at com.google.protobuf.CodedOutputStream.computeMessageSize(CodedOutputStream.java:530)

3.分析:

java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

at com.google.protobuf.GeneratedMessage.getUnknownFields(GeneratedMessage.java:180)

可能出现不同版本的protobuf包

4.验证:

[root@xxx-01 ~]# find / -name protobuf*.jar

/usr/lib/hadoop-hdfs/protobuf-java-2.4.0a.jar

/usr/share/cmf/lib/cdh4/protobuf-java-2.4.0a.jar

/data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar

/data/01/local/apache-tomcat-7.0.42/webapps/logshedcollector/WEB-INF/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/flume-ng/lib/protobuf-java-2.4.1.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/oozie/libtools/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hbase/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-httpfs/webapps/webhdfs/WEB-INF/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/mahout/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-hdfs/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client-0.20/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client-0.20/protobuf-java.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/client/protobuf-java.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-yarn/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hcatalog/share/webhcat/svr/lib/protobuf-java-2.4.0a.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop-mapreduce/lib/protobuf-java-2.4.0a.jar

果然有两个版本:

/data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar

/opt/cloudera/parcels/CDH-4.7.1-1.cdh4.7.1.p0.47/lib/hadoop/lib/protobuf-java-2.4.0a.jar

5.解决方案

根据

返回"This is supposed to be overridden by subclasses"这句话是高版本protobuf-java-2.5.0.jar才会有的,

那么将高版本jar包移除,重新启动flume,模拟数据http请求,就能写进hdfs了.

mv /data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar /data/01/local/apache-flume-1.6.0-bin/lib/protobuf-java-2.5.0.jar.bak

6.反思,有个担忧

flume1.6.0 -->protobuf-java-2.5.0.jar

cdh4.8.6 -->protobuf-java-2.4.0a.jar

cdh5.4.8 -->protobuf-java-2.5.0.jar

把flume的protobuf-java-2.5.0.jar给移除了,相当于 flume使用了2.4.0包,

担忧1.6.0版本flume的会使用2.5.0版本的函数方法,那么到时就会抛错,故就只能等着看了.

7.建议

假如使用CDH的话,个人建议最好使用配套的版本flume,

进cloudera查看对应版本

CDH4.8.6 --》 flume 1.4.0

CDH5.4.8 --》 flume 1.5.0

假如使用Apache Hadoop的话,还是要配套的版本flume.