我正在開發一個使用大型數據集的spring引導命令行應用程序。數據集很麻煩,所以我想將它與我的代碼分開,但我不想讓用戶不得不管理數據。掃描並從依賴關係jar中讀取文件
我認爲最理想的解決方案是創建一個數據jar並從中讀取主應用程序。不幸的是,當它在一個依賴關係jar中時,我無法成功讀取數據。
我做了一個示例應用程序來證明什麼,我至今試圖
https://github.com/LewisWatson/java-data-jar
. ├── data-jar │ ├── pom.xml │ └── src │ └── main │ └── resources │ └── data │ ├── hellos │ │ └── hello.txt │ └── loremIpsums │ └── loremIpsum.txt ├── data-jar-reader │ ├── pom.xml │ └── src │ └── main │ ├── java │ │ └── com │ │ └── example │ │ └── datajarreader │ │ └── DataJarReaderApplication.java │ └── resources │ └── application.properties ├── LICENSE ├── pom.xml └── README.md
它有兩個模塊:
data-jar
包含兩個文本文件在maven的標準資源目錄中的一個data
目錄中有兩個目錄。data-jar-reader
是彈簧引導命令行應用程序,它依賴於data-jar
並嘗試訪問其數據文件。
,做閱讀類是DataJarReaderApplication
@SpringBootApplication
public class DataJarReaderApplication implements CommandLineRunner {
private static final Logger log = LoggerFactory.getLogger(DataJarReaderApplication.class);
FileSystem fileSystem;
public static void main(String[] args) {
SpringApplication.run(DataJarReaderApplication.class, args);
}
@Override
public void run(String... args) throws Exception {
Path path = getTestDataPath();
List<String> data = loadTestData(path);
log.info("data {}", data);
}
private Path getTestDataPath() throws URISyntaxException, IOException {
Path path;
URI uri = this.getClass().getResource("/data").toURI();
log.info("uri: {}", uri);
if (uri.getScheme().equals("jar")) {
fileSystem = FileSystems.newFileSystem(uri, Collections.<String, Object>emptyMap());
log.info("fileSystem: {}", fileSystem);
path = fileSystem.getPath("/data");
} else { // not in a jar, probably running in iDE
path = Paths.get(uri);
}
return path;
}
private List<String> loadTestData(Path testDataDirectory) throws IOException {
try (Stream<Path> path = Files.walk(testDataDirectory).sorted()) {
return loadTrackData(path);
}
}
private List<String> loadTrackData(Stream<Path> walk) throws IOException {
List<String> data = new ArrayList<>();
for (Iterator<Path> it = walk.iterator(); it.hasNext();) {
Path path = it.next();
if (path.getFileName().toString().endsWith(".txt")) {
log.info("text file path: {}", path);
List<String> dataFromFile = getData(path);
data.addAll(dataFromFile);
} else {
log.info("non text file path: {}", path);
}
}
return data;
}
private List<String> getData(Path path) throws IOException {
List<String> data = new ArrayList<>();
try (BufferedReader reader = Files.newBufferedReader(path, StandardCharsets.UTF_8)) {
String line;
while ((line = reader.readLine()) != null) {
data.add(line);
}
}
return data;
}
}
輸出我得到的,當我運行的應用表明,它的查找數據在數據罐子
jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data
但是當它試圖訪問數據我得到一個沒有這樣的文件例外...
$ java -jar data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar
. ____ _ __ _ _
/\\/___'_ __ _ _(_)_ __ __ _ \ \ \ \
(()\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ))))
' |____| .__|_| |_|_| |_\__, |////
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v1.5.7.RELEASE)
2017-10-16 09:35:46.238 INFO 18756 --- [ main] c.e.d.DataJarReaderApplication : Starting DataJarReaderApplication v0.0.1-SNAPSHOT on mir with PID 18756 (/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar started by lewis in /home/lewis/workspace/java-data-jar)
2017-10-16 09:35:46.249 INFO 18756 --- [ main] c.e.d.DataJarReaderApplication : No active profile set, falling back to default profiles: default
2017-10-16 09:35:46.376 INFO 18756 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.spring[email protected]443b7951: startup date [Mon Oct 16 09:35:46 BST 2017]; root of context hierarchy
2017-10-16 09:35:47.049 INFO 18756 --- [ main] o.s.j.e.a.AnnotationMBeanExporter : Registering beans for JMX exposure on startup
2017-10-16 09:35:47.056 INFO 18756 --- [ main] c.e.d.DataJarReaderApplication : uri: jar:file:/home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar!/BOOT-INF/lib/data-jar-1.0-SNAPSHOT.jar!/data
2017-10-16 09:35:47.094 INFO 18756 --- [ main] c.e.d.DataJarReaderApplication : fileSystem: /home/lewis/workspace/java-data-jar/data-jar-reader/target/data-jar-reader-0.0.1-SNAPSHOT.jar
2017-10-16 09:35:47.098 INFO 18756 --- [ main] utoConfigurationReportLoggingInitializer :
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
2017-10-16 09:35:47.111 ERROR 18756 --- [ main] o.s.boot.SpringApplication : Application startup failed
java.lang.IllegalStateException: Failed to execute CommandLineRunner
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:735) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:716) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at org.springframework.boot.SpringApplication.afterRefresh(SpringApplication.java:703) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:304) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1118) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1107) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
at com.example.datajarreader.DataJarReaderApplication.main(DataJarReaderApplication.java:34) [classes!/:0.0.1-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_131]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_131]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [data-jar-reader-0.0.1-SNAPSHOT.jar:0.0.1-SNAPSHOT]
Caused by: java.nio.file.NoSuchFileException: /data
at com.sun.nio.zipfs.ZipPath.getAttributes(ZipPath.java:666) ~[zipfs.jar:1.8.0_131]
at com.sun.nio.zipfs.ZipFileSystemProvider.readAttributes(ZipFileSystemProvider.java:294) ~[zipfs.jar:1.8.0_131]
at java.nio.file.Files.readAttributes(Files.java:1737) ~[na:1.8.0_131]
at java.nio.file.FileTreeWalker.getAttributes(FileTreeWalker.java:219) ~[na:1.8.0_131]
at java.nio.file.FileTreeWalker.visit(FileTreeWalker.java:276) ~[na:1.8.0_131]
at java.nio.file.FileTreeWalker.walk(FileTreeWalker.java:322) ~[na:1.8.0_131]
at java.nio.file.FileTreeIterator.<init>(FileTreeIterator.java:72) ~[na:1.8.0_131]
at java.nio.file.Files.walk(Files.java:3574) ~[na:1.8.0_131]
at java.nio.file.Files.walk(Files.java:3625) ~[na:1.8.0_131]
at com.example.datajarreader.DataJarReaderApplication.loadTestData(DataJarReaderApplication.java:63) [classes!/:0.0.1-SNAPSHOT]
at com.example.datajarreader.DataJarReaderApplication.run(DataJarReaderApplication.java:40) [classes!/:0.0.1-SNAPSHOT]
at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:732) [spring-boot-1.5.7.RELEASE.jar!/:1.5.7.RELEASE]
... 14 common frames omitted
你確定把數據在一個罐子裏是最好的方式去?有很多不同的選項來分發大型只讀文件。如果你打算把這個大的jar放到Maven服務器上,你很可能會遇到麻煩。 – jurez
你可能是正確的,但我的具體使用情況下,我只是希望能夠孤立地運行應用程序。這是一個測試工具,所以主要要求是它需要易於使用和可重複使用。其他解決方案涉及單獨傳輸文件(大約100MB),併爲文件引入更改機會,從而影響結果。 –