add DataDirManager by houzhizhen · Pull Request #43 · apache/hugegraph-computer

houzhizhen · 2021-04-28T11:03:18Z

No description provided.

codecov · 2021-04-28T11:07:07Z

Codecov Report

Merging #43 (6c336a5) into master (c9e5ef4) will increase coverage by 0.08%.
The diff coverage is 92.68%.

@@             Coverage Diff              @@
##             master      #43      +/-   ##
============================================
+ Coverage     83.14%   83.23%   +0.08%     
- Complexity     1362     1373      +11     
============================================
  Files           149      150       +1     
  Lines          4522     4563      +41     
  Branches        379      381       +2     
============================================
+ Hits           3760     3798      +38     
  Misses          534      534              
- Partials        228      231       +3

Impacted Files	Coverage Δ	Complexity Δ
...hugegraph/computer/core/store/DataFileManager.java	`91.42% <91.42%> (ø)`	`11.00 <11.00> (?)`
...ugegraph/computer/core/config/ComputerOptions.java	`99.37% <100.00%> (+0.02%)`	`2.00 <0.00> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9e5ef4...6c336a5. Read the comment docs.

javeme · 2021-04-29T09:19:40Z

+    @Override
+    public File nextDir() {
+        int index = this.sequence.incrementAndGet();
+        // Index can't be negative.


just add assert index >= 0

javeme · 2021-04-29T09:40:13Z

+
+import java.io.File;
+
+public interface DataDirHandler {


prefer DataFilesHandler
and seems this is not handler

DataFilesHandler -> DataFileGenerator

javeme · 2021-04-29T09:41:39Z

+    /**
+     * @return the next file to persist data like vertices, edges and messages.
+     */
+    File nextFile(String label, int superstep);


what does the label mean

label means message type, different message type in different directory.

javeme · 2021-05-07T05:19:05Z

+    public static final ConfigOption<Long> WORKER_RECEIVE_BUFFERS_SIZE_LIMIT =
+            new ConfigOption<>(
+                    "worker.receive_buffers_size_limit",
+                    "The limit of received buffers that total size can't " +


The limit bytes of buffers of received data, the total size of all buffers can't excess this limit.

javeme · 2021-05-07T05:19:31Z

+                    "worker.receive_buffers_size_limit",
+                    "The limit of received buffers that total size can't " +
+                    "excess this limit. If received buffers reach this limit," +
+                    " they will be merged into a file. ",


put space to the previous line as possible

javeme · 2021-05-07T05:20:06Z


+    public static final ConfigOption<Long> WORKER_RECEIVE_BUFFERS_SIZE_LIMIT =
+            new ConfigOption<>(
+                    "worker.receive_buffers_size_limit",


received_buffers_bytes_limit

javeme · 2021-05-07T05:20:58Z

+            new ConfigListOption<>(
+                    "worker.data_dirs",
+                    true,
+                    "The dirs separated by ',' that received vertices and " +


dirs -- use full name

javeme · 2021-05-07T05:22:20Z

+                    "messages can persist into. ",
+                    disallowEmpty(),
+                    String.class,
+                    ImmutableList.of("_temporary")


set "jobs" instead of "_temporary"

javeme · 2021-05-07T05:24:21Z

+    public void init(Config config) {
+        String jobId = config.get(ComputerOptions.JOB_ID);
+        List<String> paths = config.get(ComputerOptions.WORKER_DATA_DIRS);
+        LOG.info("Worker data dirs: {}", paths);


improve message

javeme · 2021-05-07T05:24:39Z

+            File dir = new File(path);
+            File jobDir = new File(dir, jobId);
+            this.mkdirs(jobDir);
+            LOG.info("Added data dir: {}", jobDir);


improve message, add more details and context

javeme · 2021-05-08T02:35:30Z

            );

-    public static final ConfigOption<Long> WORKER_RECEIVE_BUFFERS_SIZE_LIMIT =
+    public static final ConfigOption<Long> WORKER_RECEIVED_BUFFERS_SIZE_LIMIT =


WORKER_RECEIVED_BUFFERS_BYTES_LIMIT

javeme · 2021-05-08T02:36:13Z

-                    " they will be merged into a file. ",
+                    "worker.received_buffers_bytes_limit",
+                    "The limit bytes of buffers of received data, the " +
+                    "total size of all buffers can't excess this limit. If " +


adjust wrap line

javeme · 2021-05-08T02:37:58Z

        String jobId = config.get(ComputerOptions.JOB_ID);
        List<String> paths = config.get(ComputerOptions.WORKER_DATA_DIRS);
-        LOG.info("Worker data dirs: {}", paths);
+        LOG.info("The directories '{}' configured to persist data", paths);


The directories '{}' configured to persist data for job {}

javeme · 2021-05-08T02:38:11Z

            File jobDir = new File(dir, jobId);
            this.mkdirs(jobDir);
-            LOG.info("Added data dir: {}", jobDir);
+            LOG.info("Initialized directory {} to directory list", jobDir);


houzhizhen force-pushed the data-server-manager branch from c89310d to 63d548b Compare April 28, 2021 11:07

javeme reviewed Apr 29, 2021

View reviewed changes

houzhizhen force-pushed the data-server-manager branch from 63d548b to ef125fa Compare April 30, 2021 02:53

houzhizhen added 3 commits May 6, 2021 20:12

add DataDirManager

1f08be6

rename DataDirHandler DataFileGenerator

36f27d2

remve to package store

489403d

houzhizhen force-pushed the data-server-manager branch from ef125fa to 489403d Compare May 6, 2021 12:22

tiny improve

417e17f

javeme reviewed May 7, 2021

View reviewed changes

tiny improve

43008b2

javeme reviewed May 8, 2021

View reviewed changes

tiny improve

6c336a5

javeme approved these changes May 8, 2021

View reviewed changes

coderzc approved these changes May 8, 2021

View reviewed changes

houzhizhen merged commit 1aaf9c0 into master May 8, 2021

houzhizhen deleted the data-server-manager branch May 8, 2021 11:25

Conversation

houzhizhen commented Apr 28, 2021

Uh oh!

codecov Bot commented Apr 28, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Apr 28, 2021 •

edited

Loading