Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

procstats patterns without pgrep #3559

Merged
merged 34 commits into from
Feb 1, 2018
Merged
Show file tree
Hide file tree
Changes from 28 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
9d4c282
remove pgrep for pattern matches, use go-ps instead
Dec 8, 2017
a54fd8d
add dep
Dec 8, 2017
c47eaef
remove debug
Dec 8, 2017
a2f429b
working implementation using the new gopsutil
vrecan Dec 10, 2017
4b26b52
windows files and build changes for reg pgrep
vrecan Dec 10, 2017
8b66be9
update the make file to test the new windows tests
vrecan Dec 10, 2017
54ffbd2
update gopsutil with new wmi queries
vrecan Dec 10, 2017
62285d9
Merge branch 'patterns_no_pgrep' of github.com:vrecan/telegraf into p…
Dec 12, 2017
0722c4a
add docs
Dec 12, 2017
aff075b
doc cleanup
Dec 12, 2017
e402310
move things around in the doc
Dec 12, 2017
d85d89b
more cleanup on the docs
Dec 12, 2017
56a0296
make doc a little more clear
Dec 12, 2017
452f496
revert godeps to old version of gopsutil and implement it directly
vrecan Dec 15, 2017
e1e32ef
Merge branch 'patterns_no_pgrep' of https://github.com/vrecan/telegra…
vrecan Dec 15, 2017
93fc937
remove extra dep
vrecan Dec 15, 2017
e2f374c
revert change to pgrep
vrecan Dec 20, 2017
37c6416
windows version of nativeFinder is working
vrecan Dec 22, 2017
e97d339
begin unix native finder
vrecan Dec 22, 2017
ed23fe3
update comments
vrecan Dec 22, 2017
8f008d3
fully working end to end
vrecan Dec 22, 2017
aab50e7
use native Name function instead, much better performance and less
vrecan Dec 26, 2017
78d4544
address comments
vrecan Jan 12, 2018
50fd95a
add continue
vrecan Jan 12, 2018
690ef25
change cgroup test to skip on windows
vrecan Jan 12, 2018
0ecb670
move to single quotes
vrecan Jan 12, 2018
f115e18
add missing file
vrecan Jan 12, 2018
797ec89
Merge branch 'master' of https://github.com/influxdata/telegraf into …
vrecan Jan 24, 2018
fc0f21c
Merge branch 'master' of https://github.com/influxdata/telegraf into …
vrecan Jan 30, 2018
5a4c56e
address comments, rename files and general cleanup
vrecan Jan 30, 2018
80c07ae
update name for not_windows
vrecan Jan 30, 2018
20470f8
add missing file
vrecan Jan 30, 2018
d75e3db
rename again to see if that fixes the test
vrecan Jan 30, 2018
529fd53
Merge branch 'master' of https://github.com/influxdata/telegraf into …
vrecan Jan 30, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ test-windows:
go test ./plugins/inputs/ping/...
go test ./plugins/inputs/win_perf_counters/...
go test ./plugins/inputs/win_services/...
go test ./plugins/inputs/procstat/...

lint:
go vet ./...
Expand Down
21 changes: 20 additions & 1 deletion plugins/inputs/procstat/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,28 @@ Additionally the plugin will tag processes by their PID (pid_tag = true in the c
* pid
* process_name


### Windows
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We will have to remember to take a second pass at this documentation, so that it matches the current design

On windows we only support exe and pattern. Both of these are implemented using WMI queries. exe is on the Name field and pattern is on the CommandLine field.

Windows Support:
* exe (WMI Name)
* pattern (WMI CommandLine)

this allows you to do fuzzy matching but only what is supported by [WMI query patterns](https://msdn.microsoft.com/en-us/library/aa392263(v=vs.85).aspx).

Example:

Windows fuzzy matching:
```[[inputs.procstat]]
exe = "%influx%"
process_name="influxd"
prefix = "influxd"

```

### Linux

```
[[inputs.procstat]]
exe = "influxd"
Expand All @@ -48,7 +68,6 @@ The above configuration would result in output like:
# Measurements
Note: prefix can be set by the user, per process.


Threads related measurement names:
- procstat_[prefix_]num_threads value=5

Expand Down
66 changes: 66 additions & 0 deletions plugins/inputs/procstat/nativeFinder.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
package procstat

import (
"fmt"
"io/ioutil"
"strconv"
"strings"

"github.com/shirou/gopsutil/process"
)

//NativeFinder uses gopsutil to find processes
type NativeFinder struct {
}

//NewNativeFinder ...
func NewNativeFinder() (PIDFinder, error) {
return &NativeFinder{}, nil
}

//Uid will return all pids for the given user
func (pg *NativeFinder) Uid(user string) ([]PID, error) {
pids, err := getPidsByUser(user)
if err != nil {
return pids, err
}
return pids, nil
}

//PidFile returns the pid from the pid file given.
func (pg *NativeFinder) PidFile(path string) ([]PID, error) {
var pids []PID
pidString, err := ioutil.ReadFile(path)
if err != nil {
return pids, fmt.Errorf("Failed to read pidfile '%s'. Error: '%s'",
path, err)
}
pid, err := strconv.Atoi(strings.TrimSpace(string(pidString)))
if err != nil {
return pids, err
}
pids = append(pids, PID(pid))
return pids, nil

}

//getPidsByUser ...
func getPidsByUser(username string) ([]PID, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just merge this into func (pg *NativeFinder) Uid(user string) ([]PID, error) since it doesn't add any new abstraction.

var dst []PID
procs, err := process.Processes()
if err != nil {
return dst, err
}
for _, p := range procs {
user, err := p.Username()
if err != nil {
//skip, this can happen if we don't have permissions or
//the pid no longer exists
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs continue?

continue
}
if user == username {
dst = append(dst, PID(p.Pid))
}
}
return dst, nil
}
59 changes: 59 additions & 0 deletions plugins/inputs/procstat/nativeFinder_notwindows.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
// +build !windows

package procstat

import (
"regexp"

"github.com/shirou/gopsutil/process"
)

//Pattern matches on the process name
func (pg *NativeFinder) Pattern(pattern string) ([]PID, error) {
var pids []PID
regxPattern, err := regexp.Compile(pattern)
if err != nil {
return pids, err
}
procs, err := process.Processes()
if err != nil {
return pids, err
}
for _, p := range procs {
name, err := p.Exe()
if err != nil {
//skip, this can be caused by the pid no longer existing
//or you having no permissions to access it
continue
}
if regxPattern.MatchString(name) {
pids = append(pids, PID(p.Pid))
}
}
return pids, err
}

//FullPattern matches on the command line when the proccess was executed
func (pg *NativeFinder) FullPattern(pattern string) ([]PID, error) {
var pids []PID
regxPattern, err := regexp.Compile(pattern)
if err != nil {
return pids, err
}
procs, err := process.Processes()
if err != nil {
return pids, err
}
for _, p := range procs {
cmd, err := p.Cmdline()
if err != nil {
//skip, this can be caused by the pid no longer existing
//or you having no permissions to access it
continue
}
if regxPattern.MatchString(cmd) {
pids = append(pids, PID(p.Pid))
}
}
return pids, err
}
91 changes: 91 additions & 0 deletions plugins/inputs/procstat/nativeFinder_windows.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
package procstat

import (
"context"
"fmt"
"regexp"
"time"

"github.com/StackExchange/wmi"
"github.com/shirou/gopsutil/process"
)

//Timeout is the timeout used when making wmi calls
var Timeout = 5 * time.Second

type queryType string

const (
like = queryType("LIKE")
equals = queryType("=")
notEqual = queryType("!=")
)

//Pattern matches on the process name
func (pg *NativeFinder) Pattern(pattern string) ([]PID, error) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this looks the same as the notwindows version, lets put shared implementation in a native_finder.go file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed

var pids []PID
regxPattern, err := regexp.Compile(pattern)
if err != nil {
return pids, err
}
procs, err := process.Processes()
if err != nil {
return pids, err
}
for _, p := range procs {
name, err := p.Name()
if err != nil {
//skip, this can be caused by the pid no longer existing
//or you having no permissions to access it
continue
}
if regxPattern.MatchString(name) {
pids = append(pids, PID(p.Pid))
}
}
return pids, err
}

//FullPattern matches the cmdLine on windows and will find a pattern using a WMI like query
func (pg *NativeFinder) FullPattern(pattern string) ([]PID, error) {
var pids []PID
procs, err := getWin32ProcsByVariable("CommandLine", like, pattern, Timeout)
if err != nil {
return pids, err
}
for _, p := range procs {
pids = append(pids, PID(p.ProcessID))
}
return pids, nil
}

//GetWin32ProcsByVariable allows you to query any variable with a like query
func getWin32ProcsByVariable(variable string, qType queryType, value string, timeout time.Duration) ([]process.Win32_Process, error) {
var dst []process.Win32_Process
var query string
// should look like "WHERE CommandLine LIKE "procstat"
query = fmt.Sprintf("WHERE %s %s '%s'", variable, qType, value)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use %q here to get quoting with escaping of value. "WHERE %s %s %q"

q := wmi.CreateQuery(&dst, query)
ctx, cancel := context.WithTimeout(context.Background(), timeout)
defer cancel()
err := WMIQueryWithContext(ctx, q, &dst)
if err != nil {
return []process.Win32_Process{}, fmt.Errorf("could not get win32Proc: %s", err)
}
return dst, nil
}

// WMIQueryWithContext - wraps wmi.Query with a timed-out context to avoid hanging
func WMIQueryWithContext(ctx context.Context, query string, dst interface{}, connectServerArgs ...interface{}) error {
errChan := make(chan error, 1)
go func() {
errChan <- wmi.Query(query, dst, connectServerArgs...)
}()

select {
case <-ctx.Done():
return ctx.Err()
case err := <-errChan:
return err
}
}
40 changes: 40 additions & 0 deletions plugins/inputs/procstat/nativeFinder_windows_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
package procstat

import (
"fmt"
"testing"

"os/user"

"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)

func TestGather_RealPattern(t *testing.T) {
pg, err := NewNativeFinder()
require.NoError(t, err)
pids, err := pg.Pattern(`procstat`)
require.NoError(t, err)
fmt.Println(pids)
assert.Equal(t, len(pids) > 0, true)
}

func TestGather_RealFullPattern(t *testing.T) {
pg, err := NewNativeFinder()
require.NoError(t, err)
pids, err := pg.FullPattern(`%procstat%`)
require.NoError(t, err)
fmt.Println(pids)
assert.Equal(t, len(pids) > 0, true)
}

func TestGather_RealUser(t *testing.T) {
user, err := user.Current()
require.NoError(t, err)
pg, err := NewNativeFinder()
require.NoError(t, err)
pids, err := pg.Uid(user.Username)
require.NoError(t, err)
fmt.Println(pids)
assert.Equal(t, len(pids) > 0, true)
}
7 changes: 0 additions & 7 deletions plugins/inputs/procstat/pgrep.go
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,6 @@ import (
"strings"
)

type PIDFinder interface {
PidFile(path string) ([]PID, error)
Pattern(pattern string) ([]PID, error)
Uid(user string) ([]PID, error)
FullPattern(path string) ([]PID, error)
}

// Implemention of PIDGatherer that execs pgrep to find processes
type Pgrep struct {
path string
Expand Down
7 changes: 7 additions & 0 deletions plugins/inputs/procstat/process.go
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,13 @@ type Process interface {
RlimitUsage(bool) ([]process.RlimitStat, error)
}

type PIDFinder interface {
PidFile(path string) ([]PID, error)
Pattern(pattern string) ([]PID, error)
Uid(user string) ([]PID, error)
FullPattern(path string) ([]PID, error)
}

type Proc struct {
hasCPUTimes bool
tags map[string]string
Expand Down
26 changes: 21 additions & 5 deletions plugins/inputs/procstat/procstat.go
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ var (
type PID int32

type Procstat struct {
PidFinder string `toml:"pid_finder"`
PidFile string `toml:"pid_file"`
Exe string
Pattern string
Expand All @@ -32,13 +33,19 @@ type Procstat struct {
CGroup string `toml:"cgroup"`
PidTag bool

pidFinder PIDFinder
finder PIDFinder

createPIDFinder func() (PIDFinder, error)
procs map[PID]Process
createProcess func(PID) (Process, error)
}

var sampleConfig = `
## pidFinder can be pgrep or native
## pgrep tries to exec pgrep
## native will work on all platforms, unix systems will use regexp.
## Windows will use WMI calls with like queries
pid_finder = "native"
## Must specify one of: pid_file, exe, or pattern
## PID file to monitor process
pid_file = "/var/run/nginx.pid"
Expand Down Expand Up @@ -74,7 +81,15 @@ func (_ *Procstat) Description() string {

func (p *Procstat) Gather(acc telegraf.Accumulator) error {
if p.createPIDFinder == nil {
p.createPIDFinder = defaultPIDFinder
switch p.PidFinder {
case "native":
p.createPIDFinder = NewNativeFinder
case "pgrep":
p.createPIDFinder = NewPgrep
default:
p.createPIDFinder = defaultPIDFinder
}

}
if p.createProcess == nil {
p.createProcess = defaultProcess
Expand Down Expand Up @@ -252,14 +267,15 @@ func (p *Procstat) updateProcesses(prevInfo map[PID]Process) (map[PID]Process, e

// Create and return PIDGatherer lazily
func (p *Procstat) getPIDFinder() (PIDFinder, error) {
if p.pidFinder == nil {

if p.finder == nil {
f, err := p.createPIDFinder()
if err != nil {
return nil, err
}
p.pidFinder = f
p.finder = f
}
return p.pidFinder, nil
return p.finder, nil
}

// Get matching PIDs and their initial tags
Expand Down
Loading