Logstash pipeline viewer unusable with large pipelines

[elasticmachine]

Original comment by @robelastic:

Kibana version: 6.0.0-beta1

Elasticsearch version: 6.0.0-beta1

Server OS version: MacOS

Browser version: Chrome Version 60.0.3112.101 (Official Build) (64-bit)

Browser OS version: MacOS

Original Kibana install method (e.g. download page, yum, from source, etc.): tar file from website

Original X-Pack install method (e.g. by name, offline install, from source, etc.): zip file download from link in docs

Description of the problem including expected versus actual behavior:
When pipeline viewer is used with moderately complex pipelines it cannot create a useful layout and slows to a point of being unusable.

Steps to reproduce:

1. Use the pipeline from es-6.0.0-pre branch of github.com/robcowart/elastiflow
2. Select in the pipeline viewer
3. Result...

NOTE: This isn't even close to the most complex pipeline that I use.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

[elasticmachine]

Original comment by @acchen97:

@andrewvc @ycombinator @suyograo this looks hairy, can you please investigate?

[elasticmachine]

Original comment by @robelastic:

In the time I was filing this issue I got this...

[elasticmachine]

Original comment by @suyograo:

@ycombinator FYI that @andrewvc is working on a solution for this..

[elasticmachine]

Original comment by @andrewvc:

Yes!

The good news is that there are really three issues that need to be fixed:

• Fix Infinite recursion in Graph#lineage in some topologies: This will fix the outright crash LINK REDACTED
• Set maximum time for constraint solver to run: For large graphs the solver shouldn't run more than maybe 10s after its first tick, otherwise the browser is very slow, and the solution will not get much better LINK REDACTED
• Use grouping to speed solving and improve layout. I have working PoC code here but its all hacky.

These three things I've been messing with locally, and they work really well together, including on large configs like the netflow one. I'll start putting up PRs tonight for the first two, the last one will take a couple days and probably be targetted at 6.1.

[elasticmachine]

Original comment by @ycombinator:

Sweet, thanks @andrewvc! Let me know where I can help --- happy to review PRs, etc.

[elasticmachine]

Original comment by @robelastic:

Just want to share another example of a much simpler pipeline that display issues. The pipeline is...

input {
  beats {
    id => "input_beats"
    port => "${LS_BEATS_PORT:5044}"
    client_inactivity_timeout => 180
    add_field => { "[metadata][raw_type]" => "beats" }
  }
}

filter {
  mutate {
    id => "init_control_metadata"
    add_field => {
      "[metadata][output_stdout]" => "${LS_OUTPUT_STDOUT:false}"
      "[metadata][output_rawcap]" => "${LS_RAWCAP:false}"
    }
  }
}

output {
  if [beat][name] == "packetbeat" and [type] == "flow" {
    redis {
      id => "output_redis_packetbeat_flow"
      data_type => "list"
      key => "logstash:beats:packetbeat:flow:raw"
    }
  } else if [beat][name] == "metricbeat" and [metricset][module] == "system" {
    redis {
      id => "output_redis_metricbeat_system"
      data_type => "list"
      key => "logstash:beats:metricbeat:system:raw"
    }
  }
  redis {
    id => "output_redis"
    data_type => "list"
    key => "logstash:beats:%{[beat][name]}:raw"
  }
}

This is displayed in the pipeline viewer as...

Notice how one of the redis outputs is hidden behind first conditional.

[elasticmachine]

Original comment by @ycombinator:

Quick update on this issue: We are working on a new layout algorithm which will be able to handle arbitrarily large configs. It's a pretty big change so we're working towards this solution in parts. Once all the parts are done, we will resolve this issue.