將數據從mongoDB複製到hdfs時出現hadoop jar錯誤

我試圖使用下面的代碼將收集從mongodb複製到hadoop使用HadoopMongodb連接器包hdfs;將數據從mongoDB複製到hdfs時出現hadoop jar錯誤

import java.io.*; 
import org.apache.commons.logging.*; 
import org.apache.hadoop.conf.*; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.io.*; 
import org.apache.hadoop.mapreduce.lib.output.*; 
import org.apache.hadoop.mapreduce.*; 
import org.bson.*; 
import com.mongodb.hadoop.*; 
import com.mongodb.hadoop.util.*; 

public class ImportWeblogsFromMongo { 
    private static final Log log = LogFactory.getLog(ImportWeblogsFromMongo.class); 

    public static class ReadWeblogsFromMongo extends Mapper<Object, BSONObject, Text, Text> { 
     public void map(Object key, BSONObject value, Context context) throws IOException, InterruptedException { 
      System.out.println("Key: " + key); 
      System.out.println("Value: " + value); 
      String md5 = value.get("md5").toString(); 
      String url = value.get("url").toString(); 
      String date = value.get("date").toString(); 
      String time = value.get("time").toString(); 
      String ip = value.get("ip").toString(); 
      String output = "\t" + url + "\t" + date + "\t" + time + "\t" + ip; 
      context.write(new Text(md5), new Text(output)); 
     } 
    } 

    public static void main(String[] args) throws Exception { 
     final Configuration conf = new Configuration(); 
     MongoConfigUtil.setInputURI(conf,"mongodb://localhost:27017/clusterdb.fish"); 
     MongoConfigUtil.setCreateInputSplits(conf, false); 
     System.out.println("Configuration: " + conf); 
     @SuppressWarnings("deprecation") 
     final Job job = new Job(conf, "Mongo Import"); 
     Path out = new Path("/home/mongo_import"); 
     FileOutputFormat.setOutputPath(job, out); 
     job.setJarByClass(ImportWeblogsFromMongo.class); 
     job.setMapperClass(ReadWeblogsFromMongo.class); 
     job.setOutputKeyClass(Text.class); 
     job.setOutputValueClass(Text.class); 
     job.setInputFormatClass(MongoInputFormat.class); 
     job.setOutputFormatClass(TextOutputFormat.class); 
     job.setNumReduceTasks(0); 
     System.exit(job.waitForCompletion(true) ? 0 : 1); 
    } 
    }

1.出口命名的Jar文件後importmongo.jar 我試圖去執行這個命令hadoop jar /home/yass/importmongo.jar hdfs.ImportWeblogsFromMongo，但我得到了以下錯誤：

Exception in thread "main" java.lang.NoClassDefFoundError: com/mongodb/hadoop/util/MongoConfigUtil 
    at hdfs.ImportWeblogsFromMongo.main(ImportWeblogsFromMongo.java:33) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: java.lang.ClassNotFoundException: com.mongodb.hadoop.util.MongoConfigUtil 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    ... 7 more

NB：clustedb是數據庫名稱和它的集合和hdfs.ImportWeblogsFromMongo是package.class

任何建議請

來源

2017-06-01 user3623460

你嘗試'Hadoop的JAR/home/yass/importmongo.jar -libjars HADOOP_MONGODB_JAR hdfs.ImportWeblogsFromMongo'？ – tk421

是的，我得到了一個異常在線程「主」java.lang.ClassNotFoundException異常：-libjars – user3623460

對不起，我的意思是'hadoop jar /home/yass/importmongo.jar hdfs.ImportWeblogsFromMongo -libjars HADOOP_MONGODB_JAR'。你可以看到[https://stackoverflow.com/questions/13095402/hadoop-libjars-and-classnotfoundexception]（https://stackoverflow.com/questions/13095402/hadoop-libjars-and-classnotfoundexception）或[https：/ /stackoverflow.com/questions/6890087/problem-with-libjars-in-hadoop](https://stackoverflow.com/questions/6890087/problem-with-libjars-in-hadoop）正確的語法使用-libjars 。 – tk421

我沒有解決這個樣子，但我通過將文件複製到Hdfs發現使用Mongodump解決方案，它下面的線可以幫助別人完成工作

mongodump --db clusterdb --collection CollectionName 

    bsondump file.bson > file.json 

    hadoop dfs -copyFromLocal /path/to/file/fish.json mongo

來源

2017-06-06 14:15:40 user3623460

將數據從mongoDB複製到hdfs時出現hadoop jar錯誤

回答

相關問題