Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: too many vertices for one time post in spark-loader #312

Merged
merged 2 commits into from
Jul 25, 2022
Merged

fix: too many vertices for one time post in spark-loader #312

merged 2 commits into from
Jul 25, 2022

Conversation

simon824
Copy link
Member

The option batch.max_vertices_per_batch default value is 500, if spark/flink default value of batch_size is bigger than 500 ,will cause the below exception.

class java.lang.IllegalArgumentException: Too many vertices for one time post, the maximum number is '500'
  at com.baidu.hugegraph.exception.ServerException.fromResponse(ServerException.java:47)
  at com.baidu.hugegraph.client.RestClient.checkStatus(RestClient.java:93)
  at com.baidu.hugegraph.rest.AbstractRestClient.post(AbstractRestClient.java:231)
  at com.baidu.hugegraph.rest.AbstractRestClient.post(AbstractRestClient.java:211)
  at com.baidu.hugegraph.api.graph.VertexAPI.create(VertexAPI.java:58)
  at com.baidu.hugegraph.driver.GraphManager.addVertices(GraphManager.java:96)
  at com.baidu.hugegraph.loader.spark.HugeGraphSparkLoader.flush(HugeGraphSparkLoader.java:253)
  at com.baidu.hugegraph.loader.spark.HugeGraphSparkLoader.loadRow(HugeGraphSparkLoader.java:150)
  at com.baidu.hugegraph.loader.spark.HugeGraphSparkLoader.lambda$null$0(HugeGraphSparkLoader.java:111)

closed #311

@codecov
Copy link

codecov bot commented Jul 22, 2022

Codecov Report

Merging #312 (a2905c7) into master (a594254) will decrease coverage by 0.33%.
The diff coverage is 2.85%.

@@             Coverage Diff              @@
##             master     #312      +/-   ##
============================================
- Coverage     67.83%   67.49%   -0.34%     
  Complexity      877      877              
============================================
  Files            86       86              
  Lines          4004     4024      +20     
  Branches        474      475       +1     
============================================
  Hits           2716     2716              
- Misses         1084     1104      +20     
  Partials        204      204              
Impacted Files Coverage Δ
.../hugegraph/loader/flink/HugeGraphOutputFormat.java 0.00% <0.00%> (ø)
...u/hugegraph/loader/spark/HugeGraphSparkLoader.java 0.00% <0.00%> (ø)
...baidu/hugegraph/loader/mapping/ElementMapping.java 87.83% <100.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us.

Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@javeme javeme added the loader hugegraph-loader label Jul 22, 2022
@imbajin imbajin changed the title fix: too many vertices for one time post in SparkLoader fix: too many vertices for one time post in spark-loader Jul 25, 2022
@imbajin imbajin merged commit 278fbea into apache:master Jul 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
loader hugegraph-loader
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug] Too many vertices for one time post in SparkLoader
3 participants