Added the ability to pull multiple paths #3
Conversation
|
Thanks for this! I ran some tests with my setup and am getting errors. The first run was with my existing index and got the following: I then tried it with a new index and still got a similar error: Hopefully the errors above help indicate what needs fixing to you. I'll try to take a closer look at some point too. My other feedback would be to use the term "endpoints" instead of "paths" as that more accurately describes what this is for. |
|
Yeah, the host field is reserved. I just renamed it in a filter. I can adjust the plugin to rename the host hash to something else.
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: Px Mx <[email protected]>
Sent: Thursday, February 27, 2020 4:24:39 PM
To: foospidy/logstash-input-signalsciences <[email protected]>
Cc: Roy Sprague <[email protected]>; Author <[email protected]>
Subject: [EXTERNAL] Re: [foospidy/logstash-input-signalsciences] Added the ability to pull multiple paths (#3)
WARNING: This email originated outside of the Availity email system.
DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe.
________________________________
Thanks for this! I ran some tests with my setup and am getting errors. The first run was with my existing index and got the following:
2020-02-27T16:03:23,695][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{[message][id]}", :_index=>"signalsciences", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x66280d5f>], :response=>{"index"=>{"_index"=>"signalsciences", "_type"=>"doc", "_id"=>"%{[message][id]}", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Could not dynamically add mapping for field [host.cpu]. Existing mapping for [host] must be of type object but found [text]."}}}}
I then tried it with a new index and still got a similar error:
2020-02-27T16:08:26,205][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{[message][id]}", :_index=>"sigsci_pr", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x40e183a0>], :response=>{"index"=>{"_index"=>"sigsci_pr", "_type"=>"doc", "_id"=>"%{[message][id]}", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [agent.latency_time_50th] cannot be changed from type [long] to [float]"}}}}
Hopefully the errors above help indicate what needs fixing to you. I'll try to take a closer look at some point too.
My other feedback would be to use the term "endpoints" instead of "paths" as that more accurately describes what this is for.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/foospidy/logstash-input-signalsciences/pull/3?email_source=notifications&email_token=ACAJQDP75S7A3NROMXFKGJDRFAVRPA5CNFSM4K2PPJD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENGAENI*issuecomment-592183861__;Iw!!DdwENP4MQQ!wSWdYLJdsh7hWp1GLlkTRetOnUE8LBDEWCQXR7Lwvxf8nNm8QeNBk5HZPttHVOVUCUA$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ACAJQDIHNNIAUEVKMBYWD3TRFAVRPANCNFSM4K2PPJDQ__;!!DdwENP4MQQ!wSWdYLJdsh7hWp1GLlkTRetOnUE8LBDEWCQXR7Lwvxf8nNm8QeNBk5HZPttHyYIf0hg$>.
----------------------------------------------------------------------
The information contained in this e-mail may be privileged and confidential under applicable law. It is intended solely for the use of the person or firm named above. If the reader of this e-mail is not the intended recipient, please notify us immediately by returning the e-mail to the originating e-mail address. Availity, LLC is not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Availity, LLC.
|
|
After think about this, I think there should just be a note that host is a reserved field in ES and it should be renamed via a mutate filter. I have to get this info into Splunk and our SIEM outside of ES. Renaming in the plugin probably isn’t a great idea and counter to the api doc. There probably should be a template mapping that defines those as percentiles as floats. Once you rename the host hash and delete the current index it should come in as a float. At least it did for me.
Get Outlook for iOS<https://aka.ms/o0ukef>
…________________________________
From: Px Mx <[email protected]>
Sent: Thursday, February 27, 2020 4:24:39 PM
To: foospidy/logstash-input-signalsciences <[email protected]>
Cc: Roy Sprague <[email protected]>; Author <[email protected]>
Subject: [EXTERNAL] Re: [foospidy/logstash-input-signalsciences] Added the ability to pull multiple paths (#3)
WARNING: This email originated outside of the Availity email system.
DO NOT CLICK links or open attachments unless you recognize the sender and know the content is safe.
________________________________
Thanks for this! I ran some tests with my setup and am getting errors. The first run was with my existing index and got the following:
2020-02-27T16:03:23,695][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{[message][id]}", :_index=>"signalsciences", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x66280d5f>], :response=>{"index"=>{"_index"=>"signalsciences", "_type"=>"doc", "_id"=>"%{[message][id]}", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Could not dynamically add mapping for field [host.cpu]. Existing mapping for [host] must be of type object but found [text]."}}}}
I then tried it with a new index and still got a similar error:
2020-02-27T16:08:26,205][WARN ][logstash.outputs.elasticsearch][main] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>"%{[message][id]}", :_index=>"sigsci_pr", :routing=>nil, :_type=>"doc"}, #<LogStash::Event:0x40e183a0>], :response=>{"index"=>{"_index"=>"sigsci_pr", "_type"=>"doc", "_id"=>"%{[message][id]}", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [agent.latency_time_50th] cannot be changed from type [long] to [float]"}}}}
Hopefully the errors above help indicate what needs fixing to you. I'll try to take a closer look at some point too.
My other feedback would be to use the term "endpoints" instead of "paths" as that more accurately describes what this is for.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://urldefense.com/v3/__https://github.com/foospidy/logstash-input-signalsciences/pull/3?email_source=notifications&email_token=ACAJQDP75S7A3NROMXFKGJDRFAVRPA5CNFSM4K2PPJD2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENGAENI*issuecomment-592183861__;Iw!!DdwENP4MQQ!wSWdYLJdsh7hWp1GLlkTRetOnUE8LBDEWCQXR7Lwvxf8nNm8QeNBk5HZPttHVOVUCUA$>, or unsubscribe<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/ACAJQDIHNNIAUEVKMBYWD3TRFAVRPANCNFSM4K2PPJDQ__;!!DdwENP4MQQ!wSWdYLJdsh7hWp1GLlkTRetOnUE8LBDEWCQXR7Lwvxf8nNm8QeNBk5HZPttHyYIf0hg$>.
----------------------------------------------------------------------
The information contained in this e-mail may be privileged and confidential under applicable law. It is intended solely for the use of the person or firm named above. If the reader of this e-mail is not the intended recipient, please notify us immediately by returning the e-mail to the originating e-mail address. Availity, LLC is not responsible for errors or omissions in this e-mail message. Any personal comments made in this e-mail do not reflect the views of Availity, LLC.
|
|
Adjusted to path to endpoint, added config example to remove dots from host.* field for ingest into elastic, and added the ability to have alternate sites in the endpoints hash... Let me know if you have any additional issues or need additional changes. Version 1.3.0... |
|
Thanks for the updates! I hit a couple of bumps when trying it again. First, in the conf file, the config for "paths" should be renamed to "endpoints". I needed to change that to get past the error. Second, I haven't figured this one out, but the error is: |
|
I also made the message field be a json blob so it can be sent as json instead of a hash and be parsed as json in a logstash filter. Should be in the example conf file now. One thing I'm thinking about is adding a tag for the site... I have 8 sites and it might be nice to have that capability... |
|
Sorry for the delay. I've been trying to work through an issue where only one record is appearing in elasticsearch. I think I've determined why, but haven't figured out the fix yet. In the config file the output section specifies an the field [message][id] to deduplicate data. However, it's being interpreted as a string literal so only one row is imported into elasticsearch since it's the same value for every row. Here's a screen shot: I'm still working to resolve this, but let me know if you have any ideas. |
|
I changed the config to use |
|
I don't have them going into the same index so I didn't have that problem. I am just using the tags to determine which index to use as I use different timestamp field for the requests and there isn't one for the agent status. I also didn't do the document_id deal... There shouldn't be duplicates since the code now sets the @timestamp_from = @timestamp_until. The original code looked like it just calculated the interval every loop if I'm looking at it right. A sample of my output: |
|
Sorry for the delayed response. That makes sense now. My config was for the most part based off the original. I think you've filled in the gaps I had for this. I'll test again using your config setup. Thanks! |
I verified the same data is pulled between the different versions. I also added a sigsci-API attribute to key on for output filters. I think I updated everything that needed to be updated. New to ruby so hopefully it isn't too screwed up... Work for my use case. For some reason both versions are giving me a _jsonparsefailure but lint the json output doesn't indicate any issues so thinking it might be the installation on my machine.