2016-01-22 240 views
0

我想加載一個CSV文件到我的Neo4j數據庫託管在GrapheneDB上。它在5000行的第一個文件上工作得很好。大約需要16秒來完成這個文件。neo4j加載CSV很慢

我現在使用相同的模式和相同數量的行導入第二個文件。數據是不同的。 Cypher查詢已運行超過30分鐘,但仍未完成。我不確定它在做什麼,爲什麼它如此緩慢。這裏是我的暗號:

USING PERIODIC COMMIT 500 
LOAD CSV WITH HEADERS FROM 'http://example.com/some.csv' AS line 
Match (c:Customer {customerID: line.CustomerID}) 
MERGE (c)<-[r:DEPT_OF]-(dept:Dept { name: line.Category}) 
ON CREATE 
SET dept.name = line.Category, dept.deptID=line.DeptID, dept.createdDTS=1453478149463 
MERGE (dept)<-[r1:IN_DEPT]-(pt:ProductType {name: dept.name}) 
ON CREATE 
SET pt.name = dept.name, pt.packQty = line.PackQty, pt.createdDTS = 1453478149463, 
pt.productTypeID = line.ProductTypeID 
MERGE (pt)<-[r2:OF_TYPE]-(st:Style {name: line.Style}) 
ON CREATE 
SET st.name = line.Style, st.styleID = line.StyleID, st.styleNum = line.StyleNo, st.price = line.Price 
MERGE (st)<-[r3:OF_STYLE]-(p:Product {productNum: line.UPC}) 
ON CREATE 
SET p.floorMin = line.MinFloor, p.floorMax = line.FloorMax, p.color = line.Color, p.createdDTS = 1453478149463, 
p.size = line.Size, p.productID = line.ProductID; 

對於從我的CSV行:

UPC,Category,Style,StyleNo,Color,Size,MinFloor,MaxFloor,Price,ProductID,CustomerID,ProductType,PackQty,DeptID,StyleID,ProductTypeID,ProductID 
33383605005,FRESH VEGETABLES,GREEN ONIONS 24/10 OZ,NA,NA,NA,0,0,1.79,,5f795a69-47cb-49c8-a334-0cf5d67be423,FRESH VEGETABLES,1,538a02c6-b6b7-4d0d-8dca-5ff3a513d59e,3e08dabb-415a-4826-86e8-44efb9813892,cc6a0f3c-1c05-44a0-b603-37cbbb60954e,3324b2b1-954a-4547-a82d-553be66d7b54 
52867010005,FRESH VEGETABLES,GREEN ONIONS 24/10 OZ,NA,NA,NA,0,0,1.79,,5f795a69-47cb-49c8-a334-0cf5d67be423,FRESH VEGETABLES,1,edfa998f-3749-4d1f-bd96-3f4a5db0de67,fb11a8e5-de49-44da-924a-9ebc5f7f01d2,d47fd5d8-dbf0-4110-b701-543e6ed0ae40,28a9d206-96c6-4fe4-b528-84e446ba3c16 

更新1:

我加入了基於關閉妮可響應以下指標。

  1. CREATE INDEX ON:客戶(的customerID)
  2. CREATE INDEX ON:部門(名稱)
  3. CREATE INDEX ON:ProductType(名稱)
  4. CREATE INDEX ON:樣式(名稱)
  5. CREATE INDEX ON:產品(productNum)

這樣做有很多幫助,但對於5K行仍需要大約20秒。這是正常的嗎?

任何幫助表示讚賞。

更新2:基於

斷從@邁克爾反應,我研究遠一點,發現下面的文章是非常有用的:

http://graphaware.com/neo4j/2014/07/31/cypher-merge-explained.html

更新3:

我已將我的密碼更新爲以下內容以避免重複。我希望這看起來正確嗎?

USING PERIODIC COMMIT 500 
LOAD CSV WITH HEADERS FROM '...' AS line 
Match (c:Customer {customerID: line.CustomerID}) 
MERGE (c)<-[r:DEPT_OF]-(dept:Dept { name: line.Category }) 
ON CREATE 
SET dept.name = dept.name, dept.deptID=line.categoryID, dept.createdDTS=1453742532269, dept.modifiedDTS = 1453742532269 
MERGE (c)<-[r22:DEPT_OF]-(dept) 
MERGE (dept)<-[r1:IN_DEPT]-(pt:ProductType {name: dept.name}) 
ON CREATE 
SET pt.name = dept.name, pt.packQty = line.PackQty, pt.createdDTS = 1453742532269, pt.productTypeID = line.ProductTypeID, pt.modifiedDTS = 1453742532269 
MERGE (c)<-[r2:DEPT_OF]-(dept) 
MERGE (dept)<-[r3:IN_DEPT]-(pt) 
MERGE (pt)<-[r4:OF_TYPE]-(st:Style {name: line.Style}) 
ON CREATE 
SET st.name = line.Style, st.styleID = line.StyleID, st.styleNum = line.StyleNo, st.price = line.Price, st.modifiedDTS = 1453742532269, st.createdDTS = 1453742532269 
MERGE (c)<-[r5:DEPT_OF]-(dept) 
MERGE (dept)<-[r6:IN_DEPT]-(pt) 
MERGE (pt)<-[r7:OF_TYPE]-(st) 
MERGE (st)<-[r8:OF_STYLE]-(p:Product {productNum: line.UPC}) 
ON CREATE 
SET p.floorMin = line.MinFloor, p.floorMax = line.FloorMax, p.color = line.Color, p.createdDTS = 1453742532269,p.modifiedDTS = 1453742532269, p.size = line.Size, p.productID = line.ProductID; 
+0

你有任何索引? –

+0

不可以。我如何添加索引? –

+0

我不確定我應該在哪些節點上添加索引。你能建議一些基於密碼的? –

回答

0

您也可以將其設置爲本地,以便您可以使用其他方法。然後將其上傳到石墨烯數據庫?