Bug: Poor OpenGraph Performance

[Daniel-CS-Team]

Description:

Ingest of OpenGraph Edges takes very long. I tried to ingest an OpenGraph json with ca. 240.000 Edges, which took almost 11h to process.
We try now again with neo4j enterprise backend, which should support running on more than one CPU core, but when checking on the neo4j server, it seems only one core is used.
Additionally, bloodhound itself does not use all available resources

Are you intending to fix this bug?

no

Component(s) Affected:

• Neo4j

Steps to Reproduce:

1. Try to ingest larger OpenGraph Dataset (~200.00 Edges)
2. Observe utilization on neo4j and bloodhound instances

Expected Behavior:

Either memory or CPU should be utilized fully during ingest and analysis

Actual Behavior:

neo4j: Only one core is used, even with neo4j enterprise
bloodhound: Only 3 out of 8 cores are used

Screenshots/Code Snippets/Sample Files:

Environment Information:

BloodHound: 8.6.1

Collector: -

Database (if persistence related): noe4j 4.4.48 enterprise

Docker (if using Docker): AWS ECS

Additional Information:

Potential Solution (optional):

Split ingest tasks into smaller chunks and run them in parallel.

Related Issues:

Contributor Checklist:

• I have searched the issue tracker to ensure this bug hasn't been reported before or is not already being addressed.
• I have provided clear steps to reproduce the issue.
• I have included relevant environment information details.
• I have checked that any JSON files I am attempting to upload to BloodHound are valid.

[wes-mil]

Hi @Daniel-CS-Team, this is a known limitation of using the neo4j graph driver. The enterprise version may help, but the problem has to do with using unindexed types backing the OpenGraph objects. Officially, we only support OpenGraph using the postgres driver as listed here.

Would it be possible for your team to enable using the postgres driver and beefing up the postgres instance?

[Daniel-CS-Team]

Hi @wes-mil,
thank you for the info. Unfortunately, we use some more complex Cypher-Queries, which do not (yet) work with postgres, so we are stuck with neo4j for now.
But maybe we can solve our issue at least partially by adding some indices manually?
I would be very thankful if you could give us a hint on what indices might improve performance.

[wes-mil]

@Daniel-CS-Team I see, we can definitely give it a try! I think the most important question to ask is, are you inserting OpenGraph edges against existing AD/AZ types or are you inserting these edges against custom node kinds you've made?

If it is the first one, we may have some more digging to do, as I can't see an immediate problem there. However, if it is the second one I can think of something to try:

I suspect that when inserting these edges, they have to do a full table scan to find each node associated with your insert. The best way to improve the speed of ingest here is to ensure that we're indexing the source_kind for your opengraph payloads. As you can see here, we create a unique B-Tree index against AZBase and Base

This way, there aren't table scans when trying to match nodes based on these objectids. If you wanted to create a unique objectid index for your source_kind you would run this command in your neo4j console:

CREATE CONSTRAINT sourcekind_objectid_constaint IF NOT EXISTS FOR (n:SourceKind) REQUIRE n.objectid IS UNIQUE

Let me know if I'm shooting in the right direction here, or if we need to look elsewhere 😄

[Daniel-CS-Team]

@wes-mil I did create the constraints as suggested.
With these in place, the Ingest of the following example file took 26 Minutes, with no resources (CPU or RAM) maxed out on either bloodhound nor neo4j.
example.json

I have a much larger test-file, which I will ingest next, as ingest time does not seem to be linear with the number of nodes or edges ingested.

[wes-mil]

@Daniel-CS-Team I haven't had an opportunity to try this yet, but the sample file you provided does not have a source_kind. Thus, the nodes would still not be indexed. I'm not sure if this is just an artifact of the example

[Daniel-CS-Team]

No this is intentional. Apparently, there is a bug with source_kind also applied to start- and end-nodes in the edges-array.
I have to dig deeper into this an file another issue, but had not the time yet.

Anyway, I created constraints for the kinds defined on each node in the sample, so this should not be the problem.

[wes-mil]

Oh you're correct, the source_kind does append to the kinds of the start and end nodes in edges. We're working through a handful of these hiccups still 😓 Seeing that you've created constraints for them, that shouldn't be a factor.

I will continue to try and troubleshoot this as I find time, I apologize for how troublesome this has been