Metricbeat + AWS Module: runtime error: index out of range [0] with length 0

[dudumiquim]

I'm trying to configure the AWS module but when I try to start the metricbeat, I receive this error:

FATAL [metricbeat] instance/beat.go:164 Failed due to panic

This error are not "too clear" and I started my metricbeat with debug logging level. The log level are not clear about what are happening.

I checked all the configs that I know, but the problem persist.

I removed all the modules (was only aws and system) and test only with aws. If I disable aws module, the metricbeat works.

Can someone help me?

I stated a thread on discuss, but someone recommended post here because look like a issue.

metricbeat.yml

logging.level: debug
metricbeat.modules:
- module: aws
  metricsets: 
  - billing
  period: 12h

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false
setup.ilm:
    overwrite: true

output.elasticsearch:
  hosts: ["elasticsearch-url:9200"]
  username: "username"
  password: "password"

setup.kibana:
  host: "elasticsearch-url:5601"
  username: "username"
  password: "password"
  ssl.verification_mode: none
  protocol: "https"

Metricbeat version:

# metricbeat version
metricbeat version 7.9.1 (amd64), libbeat 7.9.1 [ad823eca4cc74439d1a44351c596c12ab51054f5 built 2020-09-01 20:03:55 +0000 UTC]

Complete output of debug:

# metricbeat -c metricbeat.yml -e -d "*"
2020-09-14T20:34:30.662-0300	INFO	instance/beat.go:640	Home path: [/usr/share/metricbeat] Config path: [/etc/metricbeat] Data path: [/var/lib/metricbeat] Logs path: [/var/log/metricbeat]
2020-09-14T20:34:30.662-0300	DEBUG	[beat]	instance/beat.go:692	Beat metadata path: /var/lib/metricbeat/meta.json
2020-09-14T20:34:30.662-0300	INFO	instance/beat.go:648	Beat ID: ad2b2e0b-e8b5-4cb3-81a2-b6199b6eec7f
2020-09-14T20:34:30.662-0300	DEBUG	[seccomp]	seccomp/seccomp.go:117	Loading syscall filter	{"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-09-14T20:34:30.663-0300	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/metricbeat", "data": "/var/lib/metricbeat", "home": "/usr/share/metricbeat", "logs": "/var/log/metricbeat"}, "type": "metricbeat", "uuid": "ad2b2e0b-e8b5-4cb3-81a2-b6199b6eec7f"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T20:03:55.000Z", "version": "7.9.1"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.14.7"}}}
2020-09-14T20:34:30.663-0300	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-09-14T20:03:54-03:00","containerized":false,"name":"ip-172-29-39-170.us-east-1.compute.internal","ip":["127.0.0.1/8","172.29.39.170/22"],"kernel_version":"4.14.193-149.317.amzn2.x86_64","mac":["16:83:7c:6a:e5:5b"],"os":{"family":"redhat","platform":"amzn","name":"Amazon Linux","version":"2","major":2,"minor":0,"patch":0,"codename":"Karoo"},"timezone":"-03","timezone_offset_sec":-10800,"id":"ec2eb2a972ed3b680c44e709588d1e20"}}}
2020-09-14T20:34:30.664-0300	INFO	[beat]	instance/beat.go:1021	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/etc/metricbeat", "exe": "/usr/share/metricbeat/bin/metricbeat", "name": "metricbeat", "pid": 5467, "ppid": 4955, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-09-14T20:34:30.200-0300"}}}
2020-09-14T20:34:30.664-0300	INFO	instance/beat.go:299	Setup Beat: metricbeat; Version: 7.9.1
2020-09-14T20:34:30.664-0300	DEBUG	[beat]	instance/beat.go:325	Initializing output plugins
2020-09-14T20:34:30.664-0300	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'metricbeat-7.9.1' as ILM is enabled.
2020-09-14T20:34:30.664-0300	INFO	eslegclient/connection.go:99	elasticsearch url: http://xxxxxxx:9200
2020-09-14T20:34:30.664-0300	DEBUG	[publisher]	pipeline/consumer.go:148	start pipeline event consumer
2020-09-14T20:34:30.664-0300	INFO	[publisher]	pipeline/module.go:113	Beat name: ip-xxxxx.us-east-1.compute.internal
2020-09-14T20:34:30.675-0300	DEBUG	[modules]	beater/metricbeat.go:151	Available modules and metricsets: Register [ModuleFactory:[aws, azure, beat, cloudfoundry, docker, elasticsearch, kibana, linux, logstash, mongodb, mssql, mysql, oracle, postgresql, system, uwsgi], MetricSetFactory:[aerospike/namespace, apache/status, appsearch/stats, aws/cloudwatch, aws/ec2, aws/rds, aws/s3_daily_storage, aws/s3_request, aws/sqs, azure/app_insights, azure/billing, azure/compute_vm, azure/compute_vm_scaleset, azure/monitor, azure/storage, beat/state, beat/stats, ceph/cluster_disk, ceph/cluster_health, ceph/cluster_status, ceph/mgr_cluster_disk, ceph/mgr_cluster_health, ceph/mgr_osd_perf, ceph/mgr_osd_pool_stats, ceph/mgr_osd_tree, ceph/mgr_pool_disk, ceph/monitor_health, ceph/osd_df, ceph/osd_tree, ceph/pool_disk, cloudfoundry/container, cloudfoundry/counter, cloudfoundry/value, consul/agent, coredns/stats, couchbase/bucket, couchbase/cluster, couchbase/node, couchdb/server, docker/container, docker/cpu, docker/diskio, docker/event, docker/healthcheck, docker/image, docker/info, docker/memory, docker/network, dropwizard/collector, elasticsearch/ccr, elasticsearch/cluster_stats, elasticsearch/enrich, elasticsearch/index, elasticsearch/index_recovery, elasticsearch/index_summary, elasticsearch/ml_job, elasticsearch/node, elasticsearch/node_stats, elasticsearch/pending_tasks, elasticsearch/shard, envoyproxy/server, etcd/leader, etcd/metrics, etcd/self, etcd/store, golang/expvar, golang/heap, googlecloud/metrics, graphite/server, haproxy/info, haproxy/stat, http/json, http/server, istio/citadel, istio/galley, istio/mesh, istio/mixer, istio/pilot, jolokia/jmx, kafka/consumergroup, kafka/partition, kibana/stats, kibana/status, kubernetes/apiserver, kubernetes/container, kubernetes/controllermanager, kubernetes/event, kubernetes/node, kubernetes/pod, kubernetes/proxy, kubernetes/scheduler, kubernetes/state_container, kubernetes/state_cronjob, kubernetes/state_deployment, kubernetes/state_node, kubernetes/state_persistentvolume, kubernetes/state_persistentvolumeclaim, kubernetes/state_pod, kubernetes/state_replicaset, kubernetes/state_resourcequota, kubernetes/state_service, kubernetes/state_statefulset, kubernetes/state_storageclass, kubernetes/system, kubernetes/volume, kvm/dommemstat, kvm/status, linux/conntrack, linux/ksm, linux/pageinfo, logstash/node, logstash/node_stats, memcached/stats, mongodb/collstats, mongodb/dbstats, mongodb/metrics, mongodb/replstatus, mongodb/status, mssql/performance, mssql/transaction_log, munin/node, mysql/galera_status, mysql/query, mysql/status, nats/connections, nats/routes, nats/stats, nats/subscriptions, nginx/stubstatus, openmetrics/collector, oracle/performance, oracle/tablespace, php_fpm/pool, php_fpm/process, postgresql/activity, postgresql/bgwriter, postgresql/database, postgresql/statement, prometheus/collector, prometheus/query, prometheus/remote_write, rabbitmq/connection, rabbitmq/exchange, rabbitmq/node, rabbitmq/queue, redis/info, redis/key, redis/keyspace, sql/query, stan/channels, stan/stats, stan/subscriptions, statsd/server, system/core, system/cpu, system/diskio, system/entropy, system/filesystem, system/fsstat, system/load, system/memory, system/network, system/network_summary, system/process, system/process_summary, system/raid, system/service, system/socket, system/socket_summary, system/uptime, system/users, traefik/health, uwsgi/status, vsphere/datastore, vsphere/host, vsphere/virtualmachine, zookeeper/connection, zookeeper/mntr, zookeeper/server], LightModules:[LightModules:[activemq/queue, activemq/topic, activemq/broker, mysql/performance, cockroachdb/status, azure/container_registry, azure/container_instance, azure/container_service, azure/database_account, ibmmq/qmgr, aws/usage, aws/vpn, aws/transitgateway, aws/natgateway, aws/elb, aws/ebs, aws/lambda, aws/dynamodb, aws/billing, aws/sns, googlecloud/compute, googlecloud/pubsub, googlecloud/loadbalancing, googlecloud/storage, tomcat/threading, tomcat/memory, tomcat/requests, tomcat/cache, iis/webserver, iis/website, kafka/broker, kafka/producer, kafka/consumer, redisenterprise/node, redisenterprise/proxy]]]
2020-09-14T20:34:30.677-0300	DEBUG	[get_aws_credentials]	aws/credentials.go:74	Using shared credential profile for AWS credential
2020-09-14T20:34:30.678-0300	DEBUG	[aws.billing]	aws/aws.go:93	Metricset level config for period: 12h0m0s
2020-09-14T20:34:30.678-0300	DEBUG	[aws.billing]	aws/aws.go:94	Metricset level config for tags filter: []
2020-09-14T20:34:30.762-0300	INFO	runtime/panic.go:975	metricbeat stopped.
2020-09-14T20:34:30.762-0300	FATAL	[metricbeat]	instance/beat.go:164	Failed due to panic.	{"panic": "runtime error: index out of range [0] with length 0", "stack": "github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1.1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:165\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:969\nruntime.goPanicIndex\n\t/usr/local/go/src/runtime/panic.go:88\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws.NewMetricSet\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/aws.go:122\ngithub.com/elastic/beats/v7/x-pack/metricbeat/module/aws/cloudwatch.New\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go:113\ngithub.com/elastic/beats/v7/metricbeat/mb.(*LightMetricSet).Registration.func1\n\t/go/src/github.com/elastic/beats/metricbeat/mb/lightmetricset.go:96\ngithub.com/elastic/beats/v7/metricbeat/mb.initMetricSets\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:139\ngithub.com/elastic/beats/v7/metricbeat/mb.NewModule\n\t/go/src/github.com/elastic/beats/metricbeat/mb/builders.go:65\ngithub.com/elastic/beats/v7/metricbeat/mb/module.(*Factory).Create\n\t/go/src/github.com/elastic/beats/metricbeat/mb/module/factory.go:44\ngithub.com/elastic/beats/v7/metricbeat/beater.newMetricbeat\n\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:169\ngithub.com/elastic/beats/v7/metricbeat/beater.Creator.func1\n\t/go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:81\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:359\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:417\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.Run\n\t/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190\ngithub.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1\n\t/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:766\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:852\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/[email protected]/command.go:800\nmain.main\n\t/go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"}

• Steps to Reproduce:
  Install the Metricbeat following the documentation on the Amazon Linux 2. After that, I only enable the AWS module and disable System module.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[kaiyan-sheng]

@dudumiquim Thanks for reporting this issue. I see in your config file, there are no credential related parameters specified. Do you have a ~/.aws/credentials shared the credential file with a default profile on your Amazon Linux 2 VM? I just want to make sure I get everything before trying to reproduce it.

[dudumiquim]

Hi @kaiyan-sheng .

I'm using IAM role attached a t the ec2 instance. To discard the permission problem, I'm using an administrator role. To check if all this are working correctly, I installed the filebeat and set the sqs read to aws module. Works correctly.

If you want, I can try configuring the credentials file.

[kaiyan-sheng]

Hi @dudumiquim, thanks for the quick response! I think I found the problem. I just pushed a PR up and please feel free to verify it/review the PR.

[davidg251]

There is some workaround for this issue? I'm experiencing the same.

[kaiyan-sheng]

There is some workaround for this issue? I'm experiencing the same.

Hello! Sorry about this bug! Could you try adding an account alias to your aws account? That would be the temporary workaround for now.

[davidg251]

I created an alias. But, when I run the metricbeat binary the Kibana dashboard isn't receiving metrics :( However, Kibana is confirming that the agent is sending data from the AWS module and group the metrics on AWS provider. I'm testing it on the localhost before deploying it on AWS. This is my aws.yml fil, the system module is disabled.

metricbeat.modules:
- module: aws
  period: 300s
  metricsets:
    - ec2
    - rds
    - s3

[kaiyan-sheng]

@nan140114 Any new error message in Metricbeat log? Could you check in AWS CloudWatch to see if there are metrics there first also? Thank you for the feedback! If you see anything suspicious, please open a discuss forum ticket and you can @ me there too. Let's keep this github issue for this specific bug only 🙂