The Wayback Machine - https://web.archive.org/web/20200613070237/https://github.com/sirensolutions/sentinl/issues/751
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deleted watcher from kibana , but still get alerts from previous actions #751

Open
amulyamalla opened this issue Sep 4, 2019 · 6 comments
Open

Comments

@amulyamalla
Copy link

@amulyamalla amulyamalla commented Sep 4, 2019

ES - v6.5.4
Kibana - v6.5.4
Search guard - v6.5.4
sentinl plugin - v6.5.4

Configure sentinl with some test watcher and action , but when i deleted the watcher from kibana GUI , but still alarm get fired at the regular interval , as i already given required permission at search guard , subsequent index get created at elastic search , manually deleted watcher index but it will auto recreated again .

No watcher :
image

Listed alarm :
image

@amulyamalla amulyamalla changed the title Deleted watcher from kibana , but still get alerts from existing actions Deleted watcher from kibana , but still get alerts from previous actions Sep 4, 2019
@amulyamalla
Copy link
Author

@amulyamalla amulyamalla commented Sep 4, 2019

After debugging found something at kibana logs , thats related to orphan watcher , here the log i found

{"type":"log","@timestamp":"2019-09-04T10:06:54Z","tags":["plugin","debug"],"pid":26052,"message":"Checking Elasticsearch version"}
{"type":"log","@timestamp":"2019-09-04T10:06:55Z","tags":["debug","Sentinl","scheduler"],"pid":26052,"message":"cluster disabled"}
{"type":"log","@timestamp":"2019-09-04T10:06:55Z","tags":["debug","Sentinl","scheduler"],"pid":26052,"message":"reloading watchers..."}
{"type":"log","@timestamp":"2019-09-04T10:06:55Z","tags":["debug","Sentinl","get_elasticsearch_client"],"pid":26052,"message":"auth via Kibana server elasticsearch plugin"}
{"type":"log","@timestamp":"2019-09-04T10:06:55Z","tags":["debug","Sentinl","get_elasticsearch_client"],"pid":26052,"message":"auth via Kibana server elasticsearch plugin"}
{"type":"log","@timestamp":"2019-09-04T10:06:55Z","tags":["debug","Sentinl","get_elasticsearch_client"],"pid":26052,"message":"auth via Kibana server elasticsearch plugin"}
{"type":"log","@timestamp":"2019-09-04T10:06:56Z","tags":["debug","Sentinl","scheduler"],"pid":26052,"message":"deleting orphan watchers: 501e4c10-cd63-11e9-aaa0-a100e45a8f06"}
{"type":"log","@timestamp":"2019-09-04T10:06:56Z","tags":["debug","Sentinl","scheduler"],"pid":26052,"message":"scheduled watcher 501e4c10-cd63-11e9-aaa0-a100e45a8f06, to run every every 2 minutes"}

i just removed sentinl and optimize again , but still issue persist 
@amulyamalla
Copy link
Author

@amulyamalla amulyamalla commented Sep 5, 2019

Hey @lmangani and @sergibondarenko , could you please respond ?

@amulyamalla
Copy link
Author

@amulyamalla amulyamalla commented Sep 13, 2019

Remove sentinl plugin from kibana and reconfigure , still get the alerts from orphan watcher
necessary Precaution has been taken by deleting watcher index from elastic search but it re create again once plugin get installed

@amulyamalla
Copy link
Author

@amulyamalla amulyamalla commented Sep 13, 2019

Trying to delete watcher by using sentinl API , but found error that watcher is not present , dont understand where is the alert came from ?
Here is the response that has been captured

curl  -Ss -XDELETE "http://admin:admin@192.168.1.10:5601/api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06?pretty" \
  -H 'kbn-version: 6.5.4'

< --- kibana.log  --->>

{"type":"error","@timestamp":"2019-09-13T05:07:45Z","tags":[],"pid":32614,"level":"error","error":{"message":"Saved object [sentinl-watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06] not found","name":"Error","stack":"Error: Saved object [sentinl-watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06] not found\n    at handleESError (/usr/share/kibana/plugins/sentinl/server/lib/handle_es_error.js:29:17)\n    at handler (/usr/share/kibana/plugins/sentinl/server/routes/watcher.js:109:22)\n    at <anonymous>"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?pretty","query":{"pretty":""},"pathname":"/api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06","path":"/api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06?pretty","href":"/api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06?pretty"},"message":"Saved object [sentinl-watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06] not found"}
{"type":"response","@timestamp":"2019-09-13T05:07:45Z","tags":[],"pid":32614,"method":"delete","statusCode":500,"req":{"url":"/api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06?pretty=","method":"delete","headers":{"user-agent":"curl/7.29.0","host":"192.168.1.10:5601","accept":"*/*","kbn-version":"6.5.4"},"remoteAddress":"192.168.1.10","userAgent":"192.168.1.10"},"res":{"statusCode":500,"responseTime":731,"contentLength":9},"message":"DELETE /api/sentinl/watcher/501e4c10-cd63-11e9-aaa0-a100e45a8f06?pretty= 500 731ms - 9.0B"}
@YanekR
Copy link

@YanekR YanekR commented Oct 14, 2019

Hi!

I got same issue. Did you have any chance to find out where this orphaned watcher is stored?

Thanks,
Jan

@amulyamalla
Copy link
Author

@amulyamalla amulyamalla commented Oct 18, 2019

@YanekR
you need to delete mapping of sentinl from .kibana index , if you have less priority saved object then just delete it and recreate

still waiting for appropriate response from @sirensolutions Team

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
2 participants
You can’t perform that action at this time.